The replica-placement rule (data-center/rack/same-node limits plus host
anti-affinity) existed three times: the shell's satisfyReplicaPlacement/isGoodMove
used by volume.balance, fix.replication, and tier.move, and a line-for-line port
in the maintenance balance worker. Move the canonical logic into
weed/topology/balancer on a shared Location type; the shell and worker keep thin
adapters that convert their own location representation and call it. Behavior is
unchanged (the shared IsGoodMove keeps the shell's reject-move-to-self guard, and
all four replica test suites pass).
EC placement scored destinations purely by free EC shard slots (derived from
maxVolumeCount) and shard counts, blind to real disk fullness — the same defect
as volume balancing. A disk that is physically full but still shows free EC slots
kept being chosen, and EC shard bytes are captured by statfs free space yet not
by any slot accounting, so the slot math is exactly the metric that can't see EC
fullness.
Treat a disk at/above 90% physical usage as having zero free EC slots at
snapshot-build time, so every existing freeSlots>0 placement predicate excludes
it. Applied in all three snapshot builders (shell countFreeShardSlots, the shared
ecbalancer FromActiveTopology, and the worker ec_balance buildBalancerTopology)
via the shared balancer.DiskTooFullAfter gate. Servers not reporting disk bytes
fall back to slot-only behavior. ec.rebuild recovery is left ungated so shard
recovery can still complete onto fuller disks.
* shell: add volume.balance -byDiskUsage to balance by actual data
The default balancer ranks servers by slot density, dividing used volumes by
MaxVolumeCount. When MaxVolumeCount is configured higher than the disk can hold,
a physically near-full server looks nearly empty and gets picked as the move
target, so balancing drains less-full servers onto an already-full one.
-byDiskUsage ranks servers by the actual data they hold (sum of volume sizes)
instead, so the fullest-by-data server is treated as full and balancing drains
it. It assumes comparable disk sizes per disk type and still respects each
server's free volume slots. Default behavior is unchanged.
* plumb physical disk usage into topology, gate volume.balance on it
Volume servers now report each disk's filesystem total/free bytes in the
heartbeat, and the master stores them in DiskInfo. volume.balance uses them to
skip any move target whose disk is already near full (-maxDiskUsagePercent,
default 90), so an over-configured maxVolumeCount can no longer make a
physically full server look empty and get drained onto. The gate judges each
server against its own disk, so heterogeneous disk sizes are fine; servers that
do not report bytes fall back to slot-only behavior.
Rust seaweed-volume mirrors the heartbeat reporting.
* admin: report real physical disk capacity when volume servers provide it
The dashboard estimated server capacity as maxVolumeCount * volumeSizeLimit,
which overstates it when maxVolumeCount is set higher than the disk holds.
Prefer the filesystem capacity now reported per disk, falling back to the
estimate for servers that do not report it.
* worker: gate automatic balance on physical disk fullness too
The maintenance balance worker selects the least slot-utilized server as the
move destination, so an over-configured maxVolumeCount makes a physically full
server look empty and get drained onto — the same defect as the shell command.
Now that DiskInfo carries real disk bytes, skip any destination whose disk is
at/above 90% used (per server, against its own disk); a full server can still be
a source. When every candidate destination is full, create no tasks. Servers
that do not report disk bytes are not gated.
* balance: share the physical-disk-fullness gate between shell and worker
The shell volume.balance command and the maintenance balance worker each grew
their own copy of the disk-fullness gate (targetDiskTooFull / destinationDiskTooFull)
and a maxDiskUsagePercent=90 constant. Pull both into weed/topology/balancer
(DiskTooFullAfter + DefaultMaxDiskUsagePercent) so the policy has one home and the
two balancers can't drift.
* balance: harden the physical-disk gate
Guard against a nil DiskInfo in the byte/slot lookups. Let a zero disk-capacity
report clear previously stored bytes (0 means "not reported" for bytes, unlike
maxVolumeCount), so a server that stops reporting falls back to slot-only instead
of trusting stale capacity. In the worker, charge each planned move's bytes to
its destination within a detection cycle so the gate sees a target fill up rather
than only its heartbeat-time free space. Note the per-location capacity summing
assumes one location per filesystem (the used ratio the gate relies on stays
correct regardless; absolute capacity can over-report).
* fix(topology): keep physical disk 0 distinct in SplitByPhysicalDisk
DiskId 0 doubles as the first physical disk (Locations[0]) and the
protobuf "unset" default. SplitByPhysicalDisk folded every DiskId-0
record onto the aggregate DiskId whenever that was non-zero, so on a
multi-disk node the first disk's volumes merged into whichever disk
held volumes[0]: the node reported one fewer disk, the sibling showed
~2x volumes, and per-disk max was smeared across the survivors. This
surfaced as cluster.status and volume.list undercounting disks.
Only treat 0 as unset when no record carries a non-zero DiskId; with a
mix, 0 is a real disk and keeps its own entry.
* fix(admin): resolve physical disk 0 in active-topology indexes
rebuildIndexes re-derived each volume/EC record's physical disk id with
the same "DiskId 0 means unset" heuristic SplitByPhysicalDisk used, so
the two agreed only by sharing the bug. Now that SplitByPhysicalDisk
keeps disk 0 distinct, the duplicated heuristic would fold disk-0 records
onto a sibling while at.disks kept them on disk 0; GetVolumeLocations and
GetECShardLocations then matched no record and silently dropped every
volume and EC shard on the first disk, starving balance and EC tasks.
Build the indexes from the same SplitByPhysicalDisk reconstruction that
builds at.disks, so the keys always resolve. One source of truth instead
of a parallel normalize.
* fix(ec): allow physical disk 0 as preferred EC shard target
pickBestDiskOnNode gated its result on bestDiskId != 0, but 0 is both a
valid physical disk and the uint32 zero value, so a best-scoring disk 0
was discarded and the non-matching fallback returned instead. Gate on
bestScore.
* test(admin): cover EC-shard index resolution for physical disk 0
rebuildIndexes builds ecShardIndex the same way as volumeIndex; pin the EC
path too so a shard on disk 0 keeps resolving via GetECShardLocations.
* proto: per-disk type/capacity in DiskTag, DiskInfo.physical_disks
DiskTag gains type + max_volume_count so the heartbeat can describe every
physical disk, including ones holding no volumes or EC shards. DiskInfo
gains physical_disks so the master can hand the full per-type disk set to
per-physical-disk consumers.
* feat(volume): report each physical disk's type and capacity
CollectHeartbeat fills DiskTag.type and the per-disk effective max for
every location, so the master can account for disks that hold no volumes
or EC shards yet. Rust heartbeat mirrors it.
* feat(master): surface empty disks in the per-physical-disk view
The master records each disk's type and max from DiskTags and lists them
on DiskInfo.physical_disks per type, including disks with no volumes or
EC shards. SplitByPhysicalDisk enumerates that full set and gives each
disk its exact max, so cluster.status, volume.list and the admin
topology count and can target empty disks. Without physical_disks the
even-split fallback is unchanged.
* fix(master): clamp per-disk free at zero for over-allocated disks
In the exact-max path FreeVolumeCount could go negative when a disk holds
more volumes than its max; a negative would reduce the node's summed free
and block placement on healthy disks. Clamp at 0.
* fix(master): rebuild disk tags fresh each heartbeat
DiskTags is the full authoritative per-disk list every heartbeat, so
rebuild dn.diskTags from scratch like dn.diskBackends; merging left stale
entries for removed disks.
* fix(master): keep zero-capacity disks in physical_disks
A disk reporting max 0 (an unavailable disk) is a valid physical disk,
not a signal to drop it. List every disk of the type, but only emit
physical_disks when the node reports real per-disk capacity, so an older
server sending all zeros still falls back to the aggregate split.
* test(volume): cover disk-space-low per-disk max in heartbeat
Assert DiskTag.max_volume_count follows the used-slots override when a
location is low on space, matching the per-type max_volume_counts.
* chore: trim comments on the empty-disk change
Drop narration; keep only the non-obvious why (disk-0 sentinel, exact-max
free clamp, EC slots not subtracted, all-zeros fallback).
* refactor(master): merge per-disk tags and capacity into one map
diskTags and diskBackends were parallel maps keyed by the same DiskId and
filled together from DiskTags. Fold them into one diskMetas map of
{tags, type, max}.
* refactor(proto): per-disk max as a map keyed by disk id
physical_disks was a repeated {disk_id, max_volume_count} whose fields
duplicated DiskInfo's own disk_id/max_volume_count. A map<uint32,int64>
keyed by disk id expresses "max per disk" directly, drops the extra
PhysicalDiskInfo message, and the consumer reads it as the disk set.
* docs(proto): note DiskInfo.disk_id's two meanings
Identity on a per-physical-disk DiskInfo (from SplitByPhysicalDisk),
representative fallback on the type-keyed aggregate.
* fix(topology): keep physical disk 0 distinct in SplitByPhysicalDisk
DiskId 0 doubles as the first physical disk (Locations[0]) and the
protobuf "unset" default. SplitByPhysicalDisk folded every DiskId-0
record onto the aggregate DiskId whenever that was non-zero, so on a
multi-disk node the first disk's volumes merged into whichever disk
held volumes[0]: the node reported one fewer disk, the sibling showed
~2x volumes, and per-disk max was smeared across the survivors. This
surfaced as cluster.status and volume.list undercounting disks.
Only treat 0 as unset when no record carries a non-zero DiskId; with a
mix, 0 is a real disk and keeps its own entry.
* fix(admin): resolve physical disk 0 in active-topology indexes
rebuildIndexes re-derived each volume/EC record's physical disk id with
the same "DiskId 0 means unset" heuristic SplitByPhysicalDisk used, so
the two agreed only by sharing the bug. Now that SplitByPhysicalDisk
keeps disk 0 distinct, the duplicated heuristic would fold disk-0 records
onto a sibling while at.disks kept them on disk 0; GetVolumeLocations and
GetECShardLocations then matched no record and silently dropped every
volume and EC shard on the first disk, starving balance and EC tasks.
Build the indexes from the same SplitByPhysicalDisk reconstruction that
builds at.disks, so the keys always resolve. One source of truth instead
of a parallel normalize.
* fix(ec): allow physical disk 0 as preferred EC shard target
pickBestDiskOnNode gated its result on bestDiskId != 0, but 0 is both a
valid physical disk and the uint32 zero value, so a best-scoring disk 0
was discarded and the non-matching fallback returned instead. Gate on
bestScore.
* test(admin): cover EC-shard index resolution for physical disk 0
rebuildIndexes builds ecShardIndex the same way as volumeIndex; pin the EC
path too so a shard on disk 0 keeps resolving via GetECShardLocations.
* fix(shell): honor explicit fs.mergeVolumes from/to direction
mergeVolumes only ever merged a smaller volume into a larger one. When the
user named both -fromVolumeId and -toVolumeId with the source larger than the
target, the planner produced an empty plan and the command printed just
"max volume size: N MB" and moved nothing.
Build the requested pair directly when both ids are given, instead of routing
through the size-descending heuristic. Read-only, empty, and wrong-collection
endpoints are rejected with a clear error rather than a silent no-op.
* fix(shell): allow fs.mergeVolumes into an empty target volume
Merging chunks into an empty volume is valid, e.g. consolidating data into a
freshly created or recently vacuumed volume. Only reject an empty source, which
has nothing to move.
* fix(shell): reject self-map in directed mergeVolumes planner
createMergePlan with from == to returned a {vid: vid} self-merge when called
directly. Guard it in the planner so it is correct independent of the Do
entrypoint.
* s3: invalidate stale reader cache locations on chunk read failure
* filer: share the chunk-read self-heal across reader cache and streaming paths
The reader cache retry added a third copy of the invalidate-relookup-compare-retry
dance already inlined in PrepareStreamContentWithThrottler and duplicated in
retryWithCacheInvalidation. Extract retryFetchWithFreshLocations and route all
three through it, parameterized by the refetch primitive.
* filer: drop redundant completedTimeNew store in reader cache success path
startCaching already stamps completedTimeNew unconditionally before the
fetchErr branch; the second store inside the success branch is dead.
* filer: make NewReaderCache cache invalidator an explicit parameter
The variadic ...CacheInvalidator only ever read the first element, so a caller
could pass two and silently get one. Take a single explicit argument and have
the non-S3 callers pass nil.
* filer: inject reader cache chunk fetch as a struct field
Replace the process-global readerCacheFetchChunkData test seam with a
per-instance fetchChunkDataFn field defaulted in NewReaderCache, matching how
lookupFileIdFn is already wired. Tests set the field on the cache instead of
swapping a shared global.
* filer: log the location count, not full URLs, on self-heal retry
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Concurrent bucket deletion across multiple filer replicas races on the
per-bucket DROP TABLE. The first replica drops the table; the rest hit
an undefined-table error (postgres 42P01, mysql 1051) which propagates
out of DeleteFolderChildren and panics the filer. On restart the same
pending DROP re-runs and the filer crash-loops.
Make the drop idempotent. Same defect class in all SQL backends, so fix
postgres, postgres2, mysql, mysql2, and sqlite together.
* fix(ec): correct EC FULL scrub for deleted needles + shard-location cache
Addresses review findings on the EC FULL distributed scrub:
- Remote EC reads now thread Go's (bytes, is_deleted) contract. A runtime EC
delete keeps the .ecx size positive (the delete lives in .ecj/memory), so the
raw-index walk verifies the needle, and its header interval is usually remote;
the peer answers is_deleted with no payload. The scrub zero-fills that interval
(so the needle reaches read_bytes -> SizeMismatch{0} -> the delete-state
suppression), the serving direct read short-circuits to not-found, and
reconstruction EXCLUDES the shard instead of feeding zeros into Reed-Solomon.
- The walk skips size.is_deleted() (not just is_tombstone), so a -originalSize
.ecx entry (pre-encode delete) can't yield empty intervals or panic parse_header.
- Restore Go's < data_shards completeness guard (per-volume, custom-ratio aware)
and per-shard merge in the location cache instead of clobber-with-partial.
- Abort the scrub with an error on mid-scan unmount instead of a false-CLEAN.
- Hoist the refreshed location map once instead of cloning it per needle.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(scrub): keep RS parity check in EC FULL until CHECKSUM lands
The per-needle FULL walk only reads live data-shard intervals, so it can't catch
bitrot in a parity shard or an unwalked cold region. Run verify_ec_shards
alongside the walk, gated on all-shards-local (single-node EC), via spawn_blocking.
A deliberate temporary divergence from Go FULL; moves to mode 4 (CHECKSUM) once
the .ecsum subsystem lands.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(ec): add scrub_ec_volume_distributed (FULL EC scrub, local+remote)
Ports Go's Store.ScrubEcVolume: walk the raw .ecx, verify every needle across
local AND remote shards without decoding (report faults, don't heal), with the
#10130 deleted-needle size-mismatch suppression gated on a force flag. Reuses
the read path's lock-drop + no-reconstruct read_remote_ec_shard_interval so no
!Send store guard is held across an .await.
Walks the unmasked index (scrub_snapshot_under_lock locates from the raw
(offset, size), not locate_needle) so logically-deleted-but-present needles are
still byte-verified, matching Go. Refreshes shard locations once up front and
hard-fails on a master-lookup error rather than retrying per needle.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(scrub): dispatch EC FULL (mode 2) to the distributed needle walk
FULL ran a local-only Reed-Solomon parity check; route it to the per-needle
local+remote walk instead, mirroring Go. The handler collects vids under a brief
lock then releases it: FULL self-locks per needle (it awaits remote reads),
INDEX/LOCAL re-acquire a brief lock. verify_ec_shards is retained but no longer
wired to a mode.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
The EC volumes, EC shards, and collection details pages each rendered a
repair (wrench) button for incomplete EC volumes. Its handler POSTed to a
/repair endpoint that the admin server never registers, so every click
returned "404 page not found" (the collection details page only had a
placeholder handler).
Remove the buttons and their JavaScript handlers, and regenerate the
templ output. Manual EC shard recovery remains available from weed shell
via ec.rebuild.
EC volumes do not propagate deletions to all shard indexes, so it is possible
to run scrubbing on a volume where a deleted needle is still present in the
index, or a needle deleted from the index is still present on the volume.
On either scenario, scrubbing will fail due to size mismatch errors.
This PR reworks the scrubbing logic so needle size mismatches are
ignored in such scenarios.
Scrubbing can still be forced to check deleted needles (f.ex. to discover
index inconsistencies); this option will be exposed in RPCs and `weed shell`
on a follow-up PR.
* fix(scrub): don't flag offset-0 logical tombstones in volume scrub
A remote-tier delete records a tombstone at .idx offset 0 with no physical .dat
bytes. Full scrub double-flagged a healthy remote-tiered volume with deletes:
scrubVolumeData counted the tombstone's GetActualSize(-1)=32 toward totalRead
(want > physical .dat), and CheckIndexFile treated it as occupying [0,31] and
flagged the first live needle as overlapping. Skip offset-0 logical tombstones
from both the size reconcile and the overlap check; they are still counted for
the index-size check. Local deletes (offset != 0) are unaffected.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* fix(scrub): mirror offset-0 logical tombstone handling into Rust
Same fix as the Go volume_checking.go + idx/check.go change: Volume::scrub skips
offset-0 logical tombstones from total_read, and check_index_file excludes them
from the overlap check (still counted for the index-size check).
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* fix(ec): suppress deleted-needle size mismatch in EC LOCAL scrub
EcVolume.ScrubLocal reassembles each fully-local needle and ReadBytes-checks
it, but appended every error unconditionally. A needle the .ecx still reports
live while its reassembled on-disk header carries size 0 (delete state
disagrees between index and header) is not corruption — the LOCAL twin of the
#10130 fix for the FULL path. Suppress the ErrorSizeMismatch in that case;
genuine (non-zero) size mismatches and CRC/tail errors are still reported.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* fix(ec): mirror EC LOCAL scrub deleted-needle suppression into Rust
Same suppression as the Go EcVolume.ScrubLocal change: a NeedleError::SizeMismatch
whose on-disk header size is 0 against a live index entry is a delete-state
disagreement, not corruption.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(ec): extract locate_ec_shard_needle_interval
Mirrors Go's EcVolume.LocateEcShardNeedleInterval; reused by locate_needle
and the upcoming local scrub walk.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(ec): add EcVolumeShard::to_ec_shard_info
Mirrors Go's ToEcShardInfo.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(ec): add EcVolume::scrub_local
Walk the .ecx and verify each needle against the locally-held shards,
reading interval-by-interval (reusing one chunk buffer); CRC-check only
fully-local needles, report short/unreadable local shards, and abort the
scan on a structural size mismatch. Mirrors Go's EcVolume.ScrubLocal.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(scrub): dispatch EC LOCAL (mode 3) to scrub_local
Splits the mode 2|3 arm: FULL (2) keeps the Reed-Solomon parity check;
LOCAL (3) now runs the per-needle local-shard walk.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* refactor(scrub): extract open_index_for_scrub shared by scrub_index
Mirrors Go's openIndex, shared by ScrubIndex and the upcoming Scrub rewrite.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* fix(scrub): walk the on-disk .idx in Volume::scrub
scrub walked the deduped in-memory map, so total_read undercounted the
physical .dat on any volume with overwrites or deletes and the size
reconcile falsely flagged healthy volumes broken. Walk every .idx row
instead (matching Go's scrubVolumeData): count all rows, CRC-verify live
needles, skip deleted, and reconcile against the .dat. Holds one data-file
read lock and reads via the unlocked path, like Go's Scrub.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(idx): add check_index_file mirroring Go idx.CheckIndexFile
Index-only structural check: walk the on-disk index, sort by (offset, size),
flag overlapping needles, and verify the file is a whole number of entries.
No data-file reads.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* refactor(ec): use idx::check_index_file in EcVolume::scrub_index
Drops the inline walk/sort/overlap copy. Walks a private fd so the structural
scan never moves the shared ecx_file cursor (read positionally elsewhere).
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* fix(scrub): make Volume::scrub_index an index-only check on the on-disk .idx
INDEX mode walked the deduped in-memory map and read .dat headers — more
than the cheap-INDEX contract allows, yet missing Go's overlap and
size-multiple structural checks. Route it through idx::check_index_file so
it matches Go's Volume.ScrubIndex and the INDEX<LOCAL<FULL cost tiering holds.
Ports openIndex's zero-size-index guard (a populated .dat with an empty .idx
is corruption) and takes the data-file read lock for a consistent snapshot.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* fix(ec): cap EcShardConfig at MAX_SHARD_COUNT, not TOTAL_SHARDS_COUNT
read_ec_shard_config rejected any .vif ratio summing past 14 shards and
silently fell back to 10/4, so wider EC volumes ran against the wrong
shard set. Match Go's MaxShardCount(32) bound.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* docs(ec): correct stale 0..14 shard-count comments
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
Master /dir/lookup JSON omits publicUrl when empty (Go json omitempty).
The Rust volume server required the field, so serde failed with "lookup
parse failed: error decoding response body" and cross-DC replicated writes
failed.
Default publicUrl to empty, fall back to url for peer filtering, and
normalize addresses with to_http_address before excluding the local peer
(so host:port.grpcPort forms do not match self incorrectly).
* test(seaweed-volume): cover type=replicate fan-out writes
A holder must accept a replicated copy and store it locally without
re-replicating. Covers raw and multipart bodies, and a multi-copy
volume where re-replication would otherwise reach the master.
* test(seaweed-volume): use port 0 for the dead-master address
Connecting to port 0 is refused at the socket layer immediately, so the
plain-write fan-out path fails fast instead of risking a connect-timeout
hang where port 1 is filtered.
* feat: add collection pattern to delete empty volumes
Co-authored-by: Codex <noreply@openai.com>
* shell: match collection pattern with wildcard matcher
Use wildcard.MatchesWildcard in the shared collection-pattern helper,
matching command_volume_fix_replication's matchCollectionPattern. The
flag only advertises '*' and '?', which is exactly what the matcher
supports.
---------
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3: support AWS object form for bucket policy Principal, add NotPrincipal
Bucket policy statements only accepted a bare string or array of strings for
the Principal element, so the AWS-documented object form was rejected:
"Principal": { "AWS": "arn:aws:iam::123456789012:root" }
"Principal": { "AWS": ["arn:...", "999999999999"] }
Add a PolicyPrincipal type that parses the bare string, the bare array
(retained for backward compatibility), and the object form keyed by AWS,
Service, Federated or CanonicalUser (each value a string or array). All keyed
values are flattened for principal matching, and the original JSON is preserved
so PutBucketPolicy/GetBucketPolicy returns the exact shape submitted - keeping
infrastructure-as-code tools (Terraform, Ansible) idempotent.
Also add NotPrincipal support (a statement applies to every principal except the
ones named), compiled and evaluated in both policy evaluators, and reject
statements that specify both Principal and NotPrincipal.
* s3: address review - validate principal object form, honor dynamic NotPrincipal
- Reject unsupported Principal object keys (only AWS/Service/Federated/
CanonicalUser) and empty values, so a form like {"AWS":[]} no longer compiles
to zero matchers and silently relies on the match-all fallback.
- Detect both Principal and NotPrincipal by field presence, not by flattened
length, so a present-but-empty field is still rejected.
- Honor dynamic (policy-variable) NotPrincipal/Principal patterns in the
compiled evaluator; previously a NotPrincipal made only of variables was
treated as absent and its exclusion bypassed.
- Add regression tests for the object-form validation and dynamic NotPrincipal.
* Review comment removed unnecessary success and failure count
* fix: use Gather.Gather() with seeded counter for EC rebuild registration test
- Restore Gather.Gather() to verify MustRegister calls as requested in review
- Seed VolumeServerECRebuildCounter before gathering because CounterVec
only appears after at least one label value is observed
- Use correct fully-qualified metric names (SeaweedFS_volumeServer_*)
* fix: remove preflight checkEcVolumeStatus failure from ec_rebuild_total counter
ec_rebuild_total should only reflect actual rebuild execution failures
(from RebuildEcFiles / RebuildEcxFile), not scan/precheck failures in
the volume status loop. The error is still returned to the caller;
only the misleading counter increment was removed.
* Review comment removed unnecessary observe
* label EC rebuild duration histogram by result
Without a result label, fast failures pull down the success-latency
quantiles shown on the EC Rebuild Duration panel. Make the histogram a
HistogramVec keyed by result, record success/failure through one
recordEcRebuild helper, and split the Grafana quantiles by (le, result).
* reset EC rebuild metric vecs in registration test
The HistogramVec needs a child before Gather emits it, so the test must
observe once; reset both vecs in cleanup so that sample doesn't leak into
other tests.
---------
Co-authored-by: Ubuntu User <ubuntu@example.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
An empty or truncated tasks/*.pb file unmarshals into a TaskStateFile
with a nil Task, and protobufToMaintenanceTask dereferenced it
immediately, panicking the whole admin process on startup. Guard the
nil case so the loader logs a warning and skips the bad file.
Under a herd of concurrent assigns with no writable volume, Assign spun
PickForWrite for the full 10s timeout, pinning a goroutine per request and
starving the master of the cycles it needs to process growth and answer
heartbeats. When growth is the relevant remedy and already in flight, stop
spinning: if free space exists, shed with a fast retryable error so clients
back off and retry once growth lands; if the cluster is out of space, fail fast
with the real out-of-space error instead of masking it as retryable.
The gRPC shed uses ResourceExhausted, not Unavailable: operation.Assign retries
it, but the client connection layer doesn't treat it as a dead channel, so a
per-request shed across a herd doesn't tear down the shared master connection
and cancel every other in-flight assign. The HTTP dirAssignHandler sheds with
503 + Retry-After.
* volume server: route VolumeMarkReadonly to raft leader
After a master raft election, volume servers may still heartbeat a follower
while admin paths such as weed shell volume.mark call notifyMasterVolumeReadonly
via vs.GetMaster(). Followers reject VolumeMarkReadonly with NotLeader, which
breaks tiering and other mark-readonly workflows until the heartbeat loop
reconnects.
Resolve the leader through GetMasterConfiguration on configured -master peers
(same Leader field filer/master clients already use) before calling
VolumeMarkReadonly. When the leader differs from the heartbeat peer, update
currentMaster so the heartbeat loop converges faster.
Adds operation.LookupRaftLeaderMaster with unit tests.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: address review feedback on volume.mark raft leader routing
Do not update currentMaster during leader lookup — heartbeat owns that
field and uses stream GetLeader() to reconnect. Try the heartbeat peer
first and only resolve the raft leader after a NotLeader rejection.
Add ctx.Err() early exit and quieter logging for context cancellation.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(operation): thread the lookup timeout ctx into connection invalidation
The 5s timeout drove only the RPC; WithMasterServerClient saw the
unbounded outer ctx, so a self-inflicted timeout (slow GetMasterConfiguration
during an election) was treated as a stale channel and tore down the shared
master connection. Pass the timeout ctx into the helper so its own expiry
leaves ctx.Err() set and spares the connection.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix(filer.sync.verify): sort listings client-side before merge
The merge walks both filers' directory listings in lockstep and needs
them in the same byte order. A filer before 4.32 with a locale SQL
collation lists case-insensitively while a 4.32+ peer lists byte-ordered,
so comparing two such clusters returns the same names in a different
order and the merge desyncs into spurious MISSING / ONLY_IN_B.
Buffer and sort each directory client-side so both sides agree on order
regardless of filer version or store backend. Trades the streaming
source's O(buffer) memory for O(directory) per side, fine for a one-shot
verify CLI; both sides still load concurrently.
Claude-Session: https://claude.ai/code/session_01BKsBdKYFNCEjeHLjJfumPF
* fix(filer.sync.verify): surface listing errors before merging
A listing that fails mid-stream leaves a partial, unsorted buffer. Now
that both sides are fully buffered anyway, check each side's error right
after the loads finish and before the merge, so partial entries can't
emit spurious MISSING / ONLY_IN_B before the error aborts the run.
Claude-Session: https://claude.ai/code/session_01BKsBdKYFNCEjeHLjJfumPF
* fix(shell): correct volume.list -writable filter unit and comparison
* fix(shell): correct volume.list -writable filter unit and comparison
* chore(shell): fix typo in EC shard helper param names
* fix(shell): use exact match for volume.balance -racks/-nodes filter
The old strings.Contains-based filter quietly included any id that was a
substring of the user-supplied flag value (e.g. -racks=rack10 also matched
rack1). Replace it with an exact-match set parsed from the comma-separated
flag value, and add regression tests for both -racks and -nodes paths.
Also fix a small typo in the "remote storage" error returned by
maybeMoveOneVolume.
* fix(shell): use exact match for volume.balance -racks/-nodes filter
The old strings.Contains-based filter quietly included any id that was a
substring of the user-supplied flag value (e.g. -racks=rack10 also matched
rack1). Replace it with an exact-match set parsed from the comma-separated
flag value, and add regression tests for both -racks and -nodes paths.
Also fix a small typo in the "remote storage" error returned by
maybeMoveOneVolume.
* refactor(shell): drop nil sentinel in splitCSVSet, use len() in callers
* fix: apply collectionPattern during detection in volume.fix.replication
* use existing wildcard.MatchesWildcard for collection matching
It returns a plain bool, so drop the up-front filepath.Match validation
and the path/filepath import that only existed to handle its error.
* trim verbose comments to terse one-liners
* drop redundant per-path collection guards
Detection already filters by replicas[0].info.Collection. The repair guard
re-checked pickOneReplicaToCopyFrom's collection (a different replica), so a
mixed-collection volume could pass detection yet be skipped in repair without
decrementing the counter, spinning the -apply loop. deleteOneVolume keeps its
collectionIsMismatch safety.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
mount: move directory cache state to a side map to shrink InodeEntry
The mount keeps an InodeEntry alive for every inode the kernel references.
On a mount that is almost entirely regular files, each entry carried the full
directory readdir-cache bookkeeping (four time.Time fields plus counters),
bloating it to 152 bytes whether or not the inode was a directory.
Move that state into a dirState held in a side map keyed by inode, and drop the
isDirectory bool: an inode is a directory iff it has a dirState. InodeEntry is
now just paths + nlookup at 32 bytes, landing in a smaller Go allocator size
class; on a mount with tens of millions of cached file inodes that is several GB
less resident heap. As a side effect the readdir-cache scan helpers iterate only
directories instead of every inode.
* fix(volume): fsync .vif and downloaded tier .dat (Rust)
save_volume_info wrote the .vif with a plain write and no fsync, and the
tier download never synced the .dat it wrote. Either could be lost on a
crash before the tier-down path acts on them. fsync both, matching the Go
volume server's util.WriteFile and DownloadFile.
* fix(volume): swap to local before deleting remote on tier-down (Rust)
The tier-down path deleted the shared remote object before trimming the
.vif, so a crash in between left the volume's .vif pointing at a deleted
object. It also dropped the remote backend only on the delete path and
never opened the downloaded local .dat, so reads broke until reload and a
keep-remote download kept serving from the slow remote object.
Trim the .vif and swap to the local .dat on both paths, bracketed by
directory fsyncs, before removing the remote object; gate only the object
removal on keep_remote_dat_file. Matches the Go volume server's crash-safe
ordering.
After VolumeTierMoveDatToRemote uploaded the .dat, the volume closed its
local backend but never opened the remote one, leaving both dat_file and
remote_dat_file empty. The needle read path has no lazy reopen, so reads
returned "dat file not open" until the volume reloaded.
Switch to the remote backend right after saving the .vif, the same as the
Go volume server's LoadRemoteFile, so the volume keeps serving from remote
storage immediately after tiering.
* ci: add per-process memory sampler for perf jobs
Samples VmRSS once a second into a CSV and records peak VmHWM per process
on stop. Linux only; reads /proc/<pid>/status.
* ci: run perf benchmarks on the Rust volume server and report memory
Matrix the throughput and S3 jobs over go/rust volume servers, using a
standalone master (plus filer for S3) and swapping only the volume binary
so the two are directly comparable. Sample peak RSS in every job and surface
it per impl in the run summary.
* ci: harden mem sampler arg handling and peak fallback
Guard against missing args under set -u, and fall back to the max RSS
sampled when a process exits before VmHWM can be read.
* ec: recover EC shards whose .ecx index lives only on a peer server
A volume server that boots with EC shard files on disk but no .ecx index
on any local disk cannot mount the shards, so the master never learns
about them. ec.rebuild works off master-registered shards, so it sees the
volume as short and gives up even though the shard data is intact.
Add an operator-triggered recovery: VolumeEcShardsMount gains a
recover_missing_index flag that makes the volume server fetch the missing
.ecx (plus .ecj/.vif) from a peer holding it and mount the on-disk shards.
ec.rebuild runs this across the cluster before planning, so orphaned
shards register and the rebuild sees the true shard set.
.ecx is an immutable encode-time index, identical on every holder. .ecj
is a per-holder deletion journal that differs across holders, so the
recovered node adopts the source peer's deletion view, like a balanced or
rebuilt shard does.
* ec: mirror missing-index recovery into the Rust volume server
Port the #10104 recovery to seaweed-volume so the Rust volume server
self-heals the same layout: EC shards on disk with the .ecx index only on
a peer. Adds collect_ec_volumes_missing_index / mount_recovered_ec_shards
to the store, recover_missing_ec_indexes (master LookupEcVolume + peer
CopyFile fetch + mount) to the server, and the recover_missing_index flag
on VolumeEcShardsMount.
.ecx is the immutable encode-time index, identical on every holder. .ecj
is a per-holder deletion journal, so the recovered node adopts the source
peer's deletion view, matching the Go path.
* fix(volume): stream copy_file from disk instead of buffering whole file
copy_file pushed every 2MB chunk into a Vec and only then returned tokio_stream::iter(results), so serving a near-limit volume as a copy source (e.g. during volume.fix.replication) held the entire .dat resident and could OOM the process. Stream chunks through a bounded mpsc channel from a spawn_blocking reader instead; caps memory at ~16MB per transfer with backpressure.
* fix(volume): stream volume_incremental_copy from disk instead of buffering
Same buffering pattern as copy_file: every 2MB chunk was pushed into a Vec and only then returned via tokio_stream::iter, holding the entire delta resident. Stream the byte range from an owned file handle through a bounded mpsc channel, mirroring the copy_file fix.
* test(volume): cover streaming copy_file and volume_incremental_copy
Adds a multi-chunk .dat fixture and tests asserting both handlers stream in 2MB chunks (multiple messages), reassemble byte-for-byte, carry modified_ts_ns only on the first copy_file message, and honor stop_offset.
* address review: use u64 byte counters; stream local incremental copy without holding the store lock
- copy_file/volume_incremental_copy: track remaining bytes and offsets as u64 instead of casting uint64 stop_offset/dat_size through i64 (CodeRabbit).
- volume_incremental_copy: for local volumes open the .dat and stream directly with no lock held; only remote/tiered volumes take the per-chunk read_dat_slice path, so a remote S3 read is never performed while holding the store read lock (Gemini).
* volume (Rust): stream tiered incremental copy off the store lock, open .dat under it
Capture the reader for volume_incremental_copy while the volume lookup is still
under the store read lock: an open File for local volumes, a cloned remote
backend handle for tiered ones. Then drop the lock and stream with none held.
Opening under the lock pins the reader to the volume that exists now, so a
concurrent delete/recreate can't stream from the wrong file, and a slow S3
fetch for a tiered .dat no longer blocks store writers (the remote path
previously re-took the store lock per chunk).
Use a non-uniform copy-test payload so chunk reassembly catches duplicated or
reordered chunks a repeated byte would hide.
* volume (Rust): return empty when incremental-copy start offset is past the .dat
A corrupt needle index could locate an offset beyond the captured .dat size,
underflowing the dat_size - start_offset subtraction (panic in debug, wrap in
release). Guard it up front like the other empty-delta early returns.
---------
Co-authored-by: adri <adri@digitalunited.net>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Drop max-parallel so the 13 per-platform builds run together instead of two
waves of 8 (rocksdb was queuing behind the cap and starting ~8 min late).
Keep cache-to mode=max for rocksdb: its RocksDB static_lib compile is
sha-independent, so it caches across releases and stops being the ~16-min
long-pole that gates the merge fan-in. go-build variants stay mode=min.
docker release: build per-platform on native runners, drop mode=max cache
The build job built every platform of a variant on one runner, so 2-4 Go
cross-compiles fought over a single 2-vCPU box and arm64 ran in an emulated
context. Split the matrix to one platform per job on a native runner
(amd64/386 on ubuntu-latest, arm64/arm-v7 on ubuntu-24.04-arm); only arm/v7
still needs QEMU, and only for its final apk stage. Each job pushes by
digest, and a new merge job assembles the multi-arch tag with imagetools
and mirrors it to Docker Hub.
cache-to mode=max -> mode=min: BRANCH=sha cache-busts the heavy go-build
layer every release, so writing all intermediate layers to the gha backend
spent 3-11 min per variant on a cache the next release's sha can never hit.
* test: add self-contained S3 read/write load tool
Concurrent PUT/GET against the S3 gateway, reporting requests/sec,
transfer rate, and latency percentiles. Built on the aws-sdk-go-v2
client the S3 tests already use, so no extra benchmark binary is needed.
* ci: add performance workflow
Three parallel jobs: cpu/heap pprof of the server under write load,
native throughput via weed benchmark plus the Go micro-benchmarks, and
an S3 read/write benchmark against the gateway. Runs on push to master
and manual dispatch with tunable duration, object count, size, and
concurrency.
Chris Lu
Aleksey
Kwak Byoung Min
Peter Dodd
Lisandro Pin
adri
jk2lx
7y-9
dependabot[bot]
github-actions[bot]
Rushikesh Deshpande
Chris Lu
qzhello
jay