* s3: add IncompleteBody error code
* operation: tag a truncated source read as ErrTruncatedBody
The chunked uploader wraps both source-read failures and volume-upload failures,
and a mid-write volume server drop also carries io.ErrUnexpectedEOF. Tag only the
source read so callers can tell a truncated input apart from a server-side fault.
* s3: return IncompleteBody for a truncated PUT body
A client abort or reverse-proxy timeout truncates the request body mid-upload.
putToFiler mapped every streaming-upload failure to InternalError (500), which a
reverse proxy relays as a 502. Classify a source-read truncation as IncompleteBody
(400) so the response matches AWS and passes through. All S3 write paths share
putToFiler, so they all benefit.
* volume: drop stale volume-location cache on under-replication
A replicated write looks up the volume's locations and caches them for 10
minutes. When the master briefly reports fewer replicas than the copy count
(e.g. a stale heartbeat drops a just-added volume), that under-replicated
result got cached, so every write failed with "replicating operations is less
than replication copy count" until the entry expired -- long after the master
re-registered the replica.
Invalidate the cached entry when the location count is below the copy count, so
the next write re-queries the master and recovers as soon as it heals.
* volume: mirror the replication copy-count guard in seaweed-volume
do_replicated_request accepted a write even when the master reported fewer
locations than the volume's copy count, silently under-replicating. Reject it,
matching Go's GetWritableRemoteReplications. lookup_volume is uncached, so the
next write recovers as soon as the missing replica re-registers.
* fix(ec): read chunk-manifest chunks stored on EC volumes
Chunk-manifest expansion read every chunk through store.read_volume_needle,
which only resolves a local regular volume. Once a chunk's volume is
EC-encoded, that lookup returns NotFound and the GET fails 500 with
"read chunk ...: not found", so a chunked object over an EC tier is
unreadable even though its parity is intact and reconstructable.
Resolve each chunk to wherever it lives — a local regular volume, a
local EC volume (reconstruct-on-read from the surviving shards), or a
peer via master lookup — matching Go's ChunkedFileReader, which never
assumes chunks are local regular needles.
* fix(ec): validate the chunk cookie on local manifest chunk reads
A chunk fetched from a peer is cookie-checked by that peer's GET handler,
but the local regular and EC reads returned data without comparing the
needle's cookie to the one in the chunk fid. Check it, matching the main
GET paths, so a stale or guessed id can't serve another needle's bytes.
* fix(ec): clamp manifest chunk copy to its declared size
Expansion writes each chunk into result[offset..] by offset, so a chunk
whose bytes exceed its declared size could overwrite the next chunk's
window. Clamp the copy to chunk.size (and reject a negative size) so an
over-long or malformed chunk stays within its own range.
* remote.meta.sync: materialize directory entries, including empty ones
Pull metadata by walking the remote tree one directory level at a time
with a delimiter, so subdirectories, including empty ones, are listed as
their own entries and created locally. The previous flat listing only
returned files, so empty remote directories never appeared locally and
non-empty ones only existed as filer-synthesized parents.
* remote.meta.sync: remove local metadata for entries deleted from remote
After reconciling each directory, drop local entries whose remote source
is gone: files are deleted outright, and a directory removed from the
remote is descended into so its remote-backed children are cleaned while
local-only entries are kept. remote.meta.sync exposes -delete (default
on) and remote.mount.buckets reconciles the same way; a plain
remote.mount stays additive.
* remote.meta.sync: reconcile type swaps and prune emptied directories
- when the remote swaps an entry's type (file <-> directory), drop the
stale local entry and recreate it with the right type; local-only
entries are left alone
- mark synced directories remote-backed and clean a directory removed
from the remote locally, deleting it once it holds no local-only
entries, instead of re-listing the missing remote path
- treat a differing remote size or mtime, not only a newer mtime, as a
change worth pulling
* Respect filerGroup in admin discovery
Admin discovery previously queried master cluster nodes with an empty filer group, so filers registered under a non-default group could not appear in the admin UI. Add an admin filerGroup flag and carry it through cluster-node discovery requests while preserving the empty default behavior.
Constraint: SeaweedFS master ListClusterNodes filters by exact filer_group.
Rejected: Discover all groups implicitly | no existing admin or shell behavior exposes cross-group discovery.
Confidence: high
Scope-risk: narrow
Directive: Keep admin cluster discovery scoped to the configured filerGroup unless an explicit all-groups API is added.
Tested: docker run --rm -v "$PWD:/src" -w /src golang:1.25 go test ./weed/admin/dash -run TestListClusterNodesRequest -count=1
Tested: docker run --rm -v "$PWD:/src" -w /src golang:1.25 go test ./weed/command -run '^$' -count=1
Not-tested: full repository test suite
* mini: pass filer group to admin cluster discovery
miniAdminOptions.filerGroup was never initialized, so startAdminServer
dereferenced a nil *string. Share the filer.filerGroup flag pointer so the
co-located admin queries the same group the filer registers under.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* feat(filer.sync.verify): reclassify chunk-slice-order ETag diffs as CHUNK_REORDER
filer.ETagChunks concatenates per-chunk MD5s in stored slice order without
normalising by offset, so byte-identical content written by two different paths
(S3 multipart part-completion order on the source vs filer.backup replication
arrival order on the destination) yields different file ETags. filer.sync.verify
reported these as ETAG_MISMATCH even though the files are equal.
Add a second-pass check on every ETAG_MISMATCH: when both sides derive their ETag
from chunks (no attr.Md5) and hold a manifest-free, non-overlapping chunk set
that, once sorted by offset, matches element-wise on (offset, size, ETag),
classify the file as CHUNK_REORDER. Such files are content-equal, so they are
not counted as errors and do not affect the exit code; they are listed only at
higher verbosity (weed -v=1), while the summary always shows their count.
The check stays conservative: a stored attr.Md5 (order-independent content
hash), a differing chunk count, an overlapping/duplicate offset (whose visible
bytes are resolved by timestamp), or a manifest chunk all remain ETAG_MISMATCH.
* filer.sync.verify: decline chunk-reorder fast path on empty per-chunk ETag
An empty or undecodable per-chunk ETag is not a content fingerprint, so
element-wise (offset, size, ETag) equality can't prove the bytes match.
Treating "" == "" as content-equal could reclassify a genuine divergence as
CHUNK_REORDER and drop it from the error count. Decline such chunk sets so they
stay ETAG_MISMATCH.
* filer.sync.verify: emit CHUNK_REORDER in JSON output regardless of -v
The -v=1 gate belongs to the human text report only. Applying it before the
jsonOutput branch dropped the per-file CHUNK_REORDER records from NDJSON while
the summary still counted them, so a machine consumer saw a non-zero count with
no records to reconcile it. Gate the text path only; JSON always emits.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
deps: bump x/image to v0.43.0, repin thrift to CVE-fixed 32-bit-safe commit
x/image v0.41.0 -> v0.43.0 clears CVE-2026-33813/-46601/-46602/-46604.
thrift's only Go-affecting advisory, CVE-2026-41602 (HIGH), is fixed in
v0.23.0, but that release compares an int against untyped math.MaxUint32 in
framed_transport.go and fails to compile on 32-bit (linux/386, arm/v7).
Upstream fixed the range check post-release without tagging it, so replace
now points at that commit: it carries the CVE fix and builds on 32-bit.
Revert to a plain require once thrift tags a release past v0.23.0.
* ec: expose force_deleted_needles_check in ScrubEcVolume RPC and shell
FULL EC scrubs can opt into strict deleted-needle verification via the
-forceDeletedNeedlesCheck shell flag, off by default since it can report
false positives when EC indexes disagree. Rejected for non-FULL modes.
The Rust volume server parses the new field and ignores it: its FULL
scrub verifies shards via RS parity, not per-needle reads.
* volume: require admin auth for ScrubEcVolume
ScrubEcVolume ran unauthenticated while its sibling ScrubVolume, and the
rest of the mutating volume handlers, gate on checkGrpcAdminAuth. Close
the gap so an EC scrub can't be triggered anonymously.
* shell: reject ec.scrub -forceDeletedNeedlesCheck outside full mode
Fail in the client before fanning out to every volume server, instead of
erroring halfway through once the servers reject the request.
* balance: extract the bytes-aware density metric to weed/topology/balancer
The shell's volume.balance ranks servers by a bytes-aware density (used volume
equivalents over free capacity). Move that math into the shared balancer package
(VolumeDensity / DensityRatio / DensityNextRatio) so the maintenance worker can
adopt the same metric next. Shell behavior is unchanged.
* balance: rank a server with no free capacity as the fullest
DensityRatio/DensityNextRatio divided by capacity, so a server past its slot
limit (negative capacity) returned a negative ratio and sorted as the emptiest
under ascending consumers — the opposite of reality. Treat any non-positive
capacity (full, or overfull mid-run after receiving volumes) as the fullest
(+Inf) so it is a move source, never a target. Covered by negative-capacity and
ordering tests.
* kms: cache decrypted data keys so cache_enabled/cache_ttl take effect
cache_enabled/cache_ttl were parsed into KMSConfig but never consulted,
so every SSE-KMS read repeated a KMS Decrypt round-trip. Add a
CachedKMSProvider decorator keyed on (ciphertext, encryption context)
and wrap providers with it wherever one is created. Decrypt is
deterministic for that pair; GenerateDataKey/DescribeKey/GetKeyID pass
through untouched.
Cache hits and stores return private copies so the read path's
ClearSensitiveData can't wipe the cached key. TTL and max_cache_size
bound the cache.
* kms: zero superseded data keys and pool cache-key hash states
Wipe the old plaintext when a cache entry is overwritten so a key
material buffer left behind by two readers racing on the same miss does
not linger in memory. Reuse sha256 states via a sync.Pool so the read
hot path stops allocating a fresh hash on every Decrypt.
* kms: drop cache writes after Close
An in-flight Decrypt miss can call set after Close scrubbed the map,
repopulating it with a data key that would then never be cleared. Guard
set with a closed flag so post-Close stores wipe their copy and no-op.
* admin: never close the worker outgoing channel while senders are live
conn.outgoing has multiple concurrent senders (heartbeat, task assignment,
log request, registration handlers). Closing it on connection teardown raced
a sender and paniced with "send on closed channel" — reliably reproduced when
a laptop goes idle: heartbeats stall past the 2-minute stale cutoff, the
cleanup routine closes the channel, and the resumed worker's heartbeat is
received and handled at the same moment.
The connection context is already the sole teardown signal, so stop closing
the channel entirely. handleOutgoingMessages exits on conn.ctx.Done(), and the
buffered channel is GC'd once the connection drops. Route sends through a
sendToWorker helper that also selects on conn.ctx.Done() so they bail on
teardown instead of blocking for the full timeout.
* admin: bail on ctx.Done() while waiting for a worker log response
* refactor: centralize genUploadUrl in UploadOption
Replace inline genFileUrlFn closures with operation.GenUploadUrl field:
- Add GenUploadUrl func(host, fileId) string to UploadOption struct
- Add GenUploadUrlProxy(filerAddress string) utility function
- Remove genFileUrlFn parameter from UploadWithRetry signature
- Update all callers: mount, gateway, mq, filer_copy, filer_sync
This matches the weed mount -filerProxy pattern exactly,
factorizing the URL generation logic across all consumers.
Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf)
* docker release: run all platform jobs in one wave, cache rocksdb compile
Drop max-parallel so the 13 per-platform builds run together instead of two
waves of 8 (rocksdb was queuing behind the cap and starting ~8 min late).
Keep cache-to mode=max for rocksdb: its RocksDB static_lib compile is
sha-independent, so it caches across releases and stops being the ~16-min
long-pole that gates the merge fan-in. go-build variants stay mode=min.
Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf)
* refactor: centralize genUploadUrl in UploadOption
Replace inline genFileUrlFn closures with operation.GenUploadUrl field:
- Add GenUploadUrl func(host, fileId) string to UploadOption struct
- Add GenUploadUrlProxy(filerAddress string) utility function
- Remove genFileUrlFn parameter from UploadWithRetry signature
- Update all callers: mount, gateway, mq, filer_copy, filer_sync
This matches the weed mount -filerProxy pattern exactly,
factorizing the URL generation logic across all consumers.
Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf)
* Remove accidental ROCmFPX submodule reference
* gofmt chunk upload option block
* Preserve broker cipher and re-read proxy filer per upload attempt
Chunk uploads must keep the configured Cipher, and both the mount and broker current filer can change on failover, so build the proxy upload URL inside the closure instead of capturing the address once.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
save_bitrot_sidecar writes payload.len() into the header as a u32; guard against
a payload > 1 GiB (which would silently truncate the length field), mirroring
Go's SaveBitrotSidecar maxBitrotPayloadSize check. The check uses encoded_len()
before serializing, so an oversized manifest never allocates a large buffer.
Never triggers for a real sidecar (a few KB).
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* proto: add EC bitrot checksum messages + CHECKSUM scrub mode
Mirror weed/pb/volume_server.proto byte-for-byte (field numbers + types) so the
.ecsum sidecar payload is wire-identical across the Go and Rust binaries:
EcBitrotProtection / EcShardChecksums / ChecksumAlgorithm, VolumeScrubMode.CHECKSUM=4,
and VolumeEcShardsCopyRequest.copy_ecsum_file. No code uses them yet — the .ecsum
format, producer, mount-load, copy, and scrub land in following commits.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(ec): port the .ecsum bitrot checksum module
ec_bitrot.rs mirrors weed/storage/erasure_coding/ec_bitrot.go: the .ecsum sidecar
format (14-byte big-endian ECSU header + CRC32C over a prost-serialized
EcBitrotProtection payload), the per-shard per-block CRC32C producer
(ShardChecksumBuilder), save/load with payload self-integrity, manifest
validation, status resolution, and verify_shard_file_blocks for the CHECKSUM
scrub. A byte-exact test pins the serialized bytes against the Go reference's
identical constant so a format drift in either binary fails loudly.
Producer wiring (encode/vacuum), mount-load, copy, and the mode-4 dispatch land
in following commits.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* test(ec): pin .ecsum sidecar bytes for cross-binary interop
Deterministic EcBitrotProtection -> exact on-disk bytes, asserted against a
canonical constant on BOTH sides (this test and ec_bitrot.rs), so a format drift
in either binary fails its own suite rather than silently desyncing a Go-written
.ecsum from a Rust-written one.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(ec): write the .ecsum bitrot sidecar during EC encode
write_ec_files now feeds each shard's bytes through a per-shard
ShardChecksumBuilder as it writes them, then persists the generation-0 sidecar
(<base>.ecsum) alongside the shards — mirroring weed's WriteEcFiles +
SaveBitrotSidecar. Best-effort: a failed sidecar write leaves the generation
unprotected rather than failing the encode. A test confirms the produced sidecar
validates and its per-block CRCs match every on-disk shard.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(ec): load the .ecsum at mount + EcVolume::checksum_scrub
EcVolume now loads and validates its generation-0 .ecsum sidecar at mount,
caching the parsed protection + BitrotStatus (Off/On/Invalid), and exposes
bitrot_protection() mirroring Go's EcVolume.BitrotProtection(). checksum_scrub()
verifies every locally-held shard's raw bytes against the sidecar block CRCs —
the only path that exercises cold parity shards — reporting mismatched shards
without mutating anything; a wholesale mismatch beyond parity is flagged as a
suspect sidecar rather than mass shard corruption. Mirrors Go's ChecksumScrub.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(scrub): dispatch EC CHECKSUM (mode 4) to checksum_scrub
Accept VolumeScrubMode.CHECKSUM=4 and route it to EcVolume::checksum_scrub,
accumulating blocks scanned + mismatched shards into the scrub response, plus the
CHECKSUM scrub-mode metric label. Read-only bitrot verification over local shards.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(ec): copy the .ecsum sidecar during VolumeEcShardsCopy
Honor copy_ecsum_file: when set, copy the generation-0 .ecsum alongside the
shards so protection travels with them, mirroring Go's non-2PC copy path.
Tolerant of a missing source (empty stream) — the 0-byte file is dropped so
mount sees no sidecar (protection off) rather than a truncated/invalid one.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(ec): remove the .ecsum sidecars when destroying an EC volume
remove_ec_volume_files now clears <base>.ecsum (and any versioned .ecsum.v<N>)
from the data and idx dirs, so a vid reuse can't load a stale sidecar. Mirrors
Go's removeBitrotSidecars.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* style(ec): align bitrot comments and test setup for merge-cleanliness
Match the shared bitrot code (write_ec_files, encode_one_batch, checksum_scrub,
the encode sidecar test) to the canonical wording/layout so the volume-server
Rust port stays line-aligned across trees, keeping periodic merges conflict-free.
No behavior change.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
The disk-fullness gate only rejected destinations already at/above the mark, so a
server just under it could take a large volume and overshoot. Project the selected
volume's bytes onto the candidate: if the move would cross the mark, drop that
destination for the rest of the cycle and re-pick instead of overshooting. Also
note the per-location capacity-summing assumption on the Rust heartbeat side, to
match the Go store.go comment.
The replica-placement rule (data-center/rack/same-node limits plus host
anti-affinity) existed three times: the shell's satisfyReplicaPlacement/isGoodMove
used by volume.balance, fix.replication, and tier.move, and a line-for-line port
in the maintenance balance worker. Move the canonical logic into
weed/topology/balancer on a shared Location type; the shell and worker keep thin
adapters that convert their own location representation and call it. Behavior is
unchanged (the shared IsGoodMove keeps the shell's reject-move-to-self guard, and
all four replica test suites pass).
EC placement scored destinations purely by free EC shard slots (derived from
maxVolumeCount) and shard counts, blind to real disk fullness — the same defect
as volume balancing. A disk that is physically full but still shows free EC slots
kept being chosen, and EC shard bytes are captured by statfs free space yet not
by any slot accounting, so the slot math is exactly the metric that can't see EC
fullness.
Treat a disk at/above 90% physical usage as having zero free EC slots at
snapshot-build time, so every existing freeSlots>0 placement predicate excludes
it. Applied in all three snapshot builders (shell countFreeShardSlots, the shared
ecbalancer FromActiveTopology, and the worker ec_balance buildBalancerTopology)
via the shared balancer.DiskTooFullAfter gate. Servers not reporting disk bytes
fall back to slot-only behavior. ec.rebuild recovery is left ungated so shard
recovery can still complete onto fuller disks.
* shell: add volume.balance -byDiskUsage to balance by actual data
The default balancer ranks servers by slot density, dividing used volumes by
MaxVolumeCount. When MaxVolumeCount is configured higher than the disk can hold,
a physically near-full server looks nearly empty and gets picked as the move
target, so balancing drains less-full servers onto an already-full one.
-byDiskUsage ranks servers by the actual data they hold (sum of volume sizes)
instead, so the fullest-by-data server is treated as full and balancing drains
it. It assumes comparable disk sizes per disk type and still respects each
server's free volume slots. Default behavior is unchanged.
* plumb physical disk usage into topology, gate volume.balance on it
Volume servers now report each disk's filesystem total/free bytes in the
heartbeat, and the master stores them in DiskInfo. volume.balance uses them to
skip any move target whose disk is already near full (-maxDiskUsagePercent,
default 90), so an over-configured maxVolumeCount can no longer make a
physically full server look empty and get drained onto. The gate judges each
server against its own disk, so heterogeneous disk sizes are fine; servers that
do not report bytes fall back to slot-only behavior.
Rust seaweed-volume mirrors the heartbeat reporting.
* admin: report real physical disk capacity when volume servers provide it
The dashboard estimated server capacity as maxVolumeCount * volumeSizeLimit,
which overstates it when maxVolumeCount is set higher than the disk holds.
Prefer the filesystem capacity now reported per disk, falling back to the
estimate for servers that do not report it.
* worker: gate automatic balance on physical disk fullness too
The maintenance balance worker selects the least slot-utilized server as the
move destination, so an over-configured maxVolumeCount makes a physically full
server look empty and get drained onto — the same defect as the shell command.
Now that DiskInfo carries real disk bytes, skip any destination whose disk is
at/above 90% used (per server, against its own disk); a full server can still be
a source. When every candidate destination is full, create no tasks. Servers
that do not report disk bytes are not gated.
* balance: share the physical-disk-fullness gate between shell and worker
The shell volume.balance command and the maintenance balance worker each grew
their own copy of the disk-fullness gate (targetDiskTooFull / destinationDiskTooFull)
and a maxDiskUsagePercent=90 constant. Pull both into weed/topology/balancer
(DiskTooFullAfter + DefaultMaxDiskUsagePercent) so the policy has one home and the
two balancers can't drift.
* balance: harden the physical-disk gate
Guard against a nil DiskInfo in the byte/slot lookups. Let a zero disk-capacity
report clear previously stored bytes (0 means "not reported" for bytes, unlike
maxVolumeCount), so a server that stops reporting falls back to slot-only instead
of trusting stale capacity. In the worker, charge each planned move's bytes to
its destination within a detection cycle so the gate sees a target fill up rather
than only its heartbeat-time free space. Note the per-location capacity summing
assumes one location per filesystem (the used ratio the gate relies on stays
correct regardless; absolute capacity can over-report).
* fix(topology): keep physical disk 0 distinct in SplitByPhysicalDisk
DiskId 0 doubles as the first physical disk (Locations[0]) and the
protobuf "unset" default. SplitByPhysicalDisk folded every DiskId-0
record onto the aggregate DiskId whenever that was non-zero, so on a
multi-disk node the first disk's volumes merged into whichever disk
held volumes[0]: the node reported one fewer disk, the sibling showed
~2x volumes, and per-disk max was smeared across the survivors. This
surfaced as cluster.status and volume.list undercounting disks.
Only treat 0 as unset when no record carries a non-zero DiskId; with a
mix, 0 is a real disk and keeps its own entry.
* fix(admin): resolve physical disk 0 in active-topology indexes
rebuildIndexes re-derived each volume/EC record's physical disk id with
the same "DiskId 0 means unset" heuristic SplitByPhysicalDisk used, so
the two agreed only by sharing the bug. Now that SplitByPhysicalDisk
keeps disk 0 distinct, the duplicated heuristic would fold disk-0 records
onto a sibling while at.disks kept them on disk 0; GetVolumeLocations and
GetECShardLocations then matched no record and silently dropped every
volume and EC shard on the first disk, starving balance and EC tasks.
Build the indexes from the same SplitByPhysicalDisk reconstruction that
builds at.disks, so the keys always resolve. One source of truth instead
of a parallel normalize.
* fix(ec): allow physical disk 0 as preferred EC shard target
pickBestDiskOnNode gated its result on bestDiskId != 0, but 0 is both a
valid physical disk and the uint32 zero value, so a best-scoring disk 0
was discarded and the non-matching fallback returned instead. Gate on
bestScore.
* test(admin): cover EC-shard index resolution for physical disk 0
rebuildIndexes builds ecShardIndex the same way as volumeIndex; pin the EC
path too so a shard on disk 0 keeps resolving via GetECShardLocations.
* proto: per-disk type/capacity in DiskTag, DiskInfo.physical_disks
DiskTag gains type + max_volume_count so the heartbeat can describe every
physical disk, including ones holding no volumes or EC shards. DiskInfo
gains physical_disks so the master can hand the full per-type disk set to
per-physical-disk consumers.
* feat(volume): report each physical disk's type and capacity
CollectHeartbeat fills DiskTag.type and the per-disk effective max for
every location, so the master can account for disks that hold no volumes
or EC shards yet. Rust heartbeat mirrors it.
* feat(master): surface empty disks in the per-physical-disk view
The master records each disk's type and max from DiskTags and lists them
on DiskInfo.physical_disks per type, including disks with no volumes or
EC shards. SplitByPhysicalDisk enumerates that full set and gives each
disk its exact max, so cluster.status, volume.list and the admin
topology count and can target empty disks. Without physical_disks the
even-split fallback is unchanged.
* fix(master): clamp per-disk free at zero for over-allocated disks
In the exact-max path FreeVolumeCount could go negative when a disk holds
more volumes than its max; a negative would reduce the node's summed free
and block placement on healthy disks. Clamp at 0.
* fix(master): rebuild disk tags fresh each heartbeat
DiskTags is the full authoritative per-disk list every heartbeat, so
rebuild dn.diskTags from scratch like dn.diskBackends; merging left stale
entries for removed disks.
* fix(master): keep zero-capacity disks in physical_disks
A disk reporting max 0 (an unavailable disk) is a valid physical disk,
not a signal to drop it. List every disk of the type, but only emit
physical_disks when the node reports real per-disk capacity, so an older
server sending all zeros still falls back to the aggregate split.
* test(volume): cover disk-space-low per-disk max in heartbeat
Assert DiskTag.max_volume_count follows the used-slots override when a
location is low on space, matching the per-type max_volume_counts.
* chore: trim comments on the empty-disk change
Drop narration; keep only the non-obvious why (disk-0 sentinel, exact-max
free clamp, EC slots not subtracted, all-zeros fallback).
* refactor(master): merge per-disk tags and capacity into one map
diskTags and diskBackends were parallel maps keyed by the same DiskId and
filled together from DiskTags. Fold them into one diskMetas map of
{tags, type, max}.
* refactor(proto): per-disk max as a map keyed by disk id
physical_disks was a repeated {disk_id, max_volume_count} whose fields
duplicated DiskInfo's own disk_id/max_volume_count. A map<uint32,int64>
keyed by disk id expresses "max per disk" directly, drops the extra
PhysicalDiskInfo message, and the consumer reads it as the disk set.
* docs(proto): note DiskInfo.disk_id's two meanings
Identity on a per-physical-disk DiskInfo (from SplitByPhysicalDisk),
representative fallback on the type-keyed aggregate.
* fix(topology): keep physical disk 0 distinct in SplitByPhysicalDisk
DiskId 0 doubles as the first physical disk (Locations[0]) and the
protobuf "unset" default. SplitByPhysicalDisk folded every DiskId-0
record onto the aggregate DiskId whenever that was non-zero, so on a
multi-disk node the first disk's volumes merged into whichever disk
held volumes[0]: the node reported one fewer disk, the sibling showed
~2x volumes, and per-disk max was smeared across the survivors. This
surfaced as cluster.status and volume.list undercounting disks.
Only treat 0 as unset when no record carries a non-zero DiskId; with a
mix, 0 is a real disk and keeps its own entry.
* fix(admin): resolve physical disk 0 in active-topology indexes
rebuildIndexes re-derived each volume/EC record's physical disk id with
the same "DiskId 0 means unset" heuristic SplitByPhysicalDisk used, so
the two agreed only by sharing the bug. Now that SplitByPhysicalDisk
keeps disk 0 distinct, the duplicated heuristic would fold disk-0 records
onto a sibling while at.disks kept them on disk 0; GetVolumeLocations and
GetECShardLocations then matched no record and silently dropped every
volume and EC shard on the first disk, starving balance and EC tasks.
Build the indexes from the same SplitByPhysicalDisk reconstruction that
builds at.disks, so the keys always resolve. One source of truth instead
of a parallel normalize.
* fix(ec): allow physical disk 0 as preferred EC shard target
pickBestDiskOnNode gated its result on bestDiskId != 0, but 0 is both a
valid physical disk and the uint32 zero value, so a best-scoring disk 0
was discarded and the non-matching fallback returned instead. Gate on
bestScore.
* test(admin): cover EC-shard index resolution for physical disk 0
rebuildIndexes builds ecShardIndex the same way as volumeIndex; pin the EC
path too so a shard on disk 0 keeps resolving via GetECShardLocations.
* fix(shell): honor explicit fs.mergeVolumes from/to direction
mergeVolumes only ever merged a smaller volume into a larger one. When the
user named both -fromVolumeId and -toVolumeId with the source larger than the
target, the planner produced an empty plan and the command printed just
"max volume size: N MB" and moved nothing.
Build the requested pair directly when both ids are given, instead of routing
through the size-descending heuristic. Read-only, empty, and wrong-collection
endpoints are rejected with a clear error rather than a silent no-op.
* fix(shell): allow fs.mergeVolumes into an empty target volume
Merging chunks into an empty volume is valid, e.g. consolidating data into a
freshly created or recently vacuumed volume. Only reject an empty source, which
has nothing to move.
* fix(shell): reject self-map in directed mergeVolumes planner
createMergePlan with from == to returned a {vid: vid} self-merge when called
directly. Guard it in the planner so it is correct independent of the Do
entrypoint.
* s3: invalidate stale reader cache locations on chunk read failure
* filer: share the chunk-read self-heal across reader cache and streaming paths
The reader cache retry added a third copy of the invalidate-relookup-compare-retry
dance already inlined in PrepareStreamContentWithThrottler and duplicated in
retryWithCacheInvalidation. Extract retryFetchWithFreshLocations and route all
three through it, parameterized by the refetch primitive.
* filer: drop redundant completedTimeNew store in reader cache success path
startCaching already stamps completedTimeNew unconditionally before the
fetchErr branch; the second store inside the success branch is dead.
* filer: make NewReaderCache cache invalidator an explicit parameter
The variadic ...CacheInvalidator only ever read the first element, so a caller
could pass two and silently get one. Take a single explicit argument and have
the non-S3 callers pass nil.
* filer: inject reader cache chunk fetch as a struct field
Replace the process-global readerCacheFetchChunkData test seam with a
per-instance fetchChunkDataFn field defaulted in NewReaderCache, matching how
lookupFileIdFn is already wired. Tests set the field on the cache instead of
swapping a shared global.
* filer: log the location count, not full URLs, on self-heal retry
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Concurrent bucket deletion across multiple filer replicas races on the
per-bucket DROP TABLE. The first replica drops the table; the rest hit
an undefined-table error (postgres 42P01, mysql 1051) which propagates
out of DeleteFolderChildren and panics the filer. On restart the same
pending DROP re-runs and the filer crash-loops.
Make the drop idempotent. Same defect class in all SQL backends, so fix
postgres, postgres2, mysql, mysql2, and sqlite together.
* fix(ec): correct EC FULL scrub for deleted needles + shard-location cache
Addresses review findings on the EC FULL distributed scrub:
- Remote EC reads now thread Go's (bytes, is_deleted) contract. A runtime EC
delete keeps the .ecx size positive (the delete lives in .ecj/memory), so the
raw-index walk verifies the needle, and its header interval is usually remote;
the peer answers is_deleted with no payload. The scrub zero-fills that interval
(so the needle reaches read_bytes -> SizeMismatch{0} -> the delete-state
suppression), the serving direct read short-circuits to not-found, and
reconstruction EXCLUDES the shard instead of feeding zeros into Reed-Solomon.
- The walk skips size.is_deleted() (not just is_tombstone), so a -originalSize
.ecx entry (pre-encode delete) can't yield empty intervals or panic parse_header.
- Restore Go's < data_shards completeness guard (per-volume, custom-ratio aware)
and per-shard merge in the location cache instead of clobber-with-partial.
- Abort the scrub with an error on mid-scan unmount instead of a false-CLEAN.
- Hoist the refreshed location map once instead of cloning it per needle.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(scrub): keep RS parity check in EC FULL until CHECKSUM lands
The per-needle FULL walk only reads live data-shard intervals, so it can't catch
bitrot in a parity shard or an unwalked cold region. Run verify_ec_shards
alongside the walk, gated on all-shards-local (single-node EC), via spawn_blocking.
A deliberate temporary divergence from Go FULL; moves to mode 4 (CHECKSUM) once
the .ecsum subsystem lands.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(ec): add scrub_ec_volume_distributed (FULL EC scrub, local+remote)
Ports Go's Store.ScrubEcVolume: walk the raw .ecx, verify every needle across
local AND remote shards without decoding (report faults, don't heal), with the
#10130 deleted-needle size-mismatch suppression gated on a force flag. Reuses
the read path's lock-drop + no-reconstruct read_remote_ec_shard_interval so no
!Send store guard is held across an .await.
Walks the unmasked index (scrub_snapshot_under_lock locates from the raw
(offset, size), not locate_needle) so logically-deleted-but-present needles are
still byte-verified, matching Go. Refreshes shard locations once up front and
hard-fails on a master-lookup error rather than retrying per needle.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(scrub): dispatch EC FULL (mode 2) to the distributed needle walk
FULL ran a local-only Reed-Solomon parity check; route it to the per-needle
local+remote walk instead, mirroring Go. The handler collects vids under a brief
lock then releases it: FULL self-locks per needle (it awaits remote reads),
INDEX/LOCAL re-acquire a brief lock. verify_ec_shards is retained but no longer
wired to a mode.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
The EC volumes, EC shards, and collection details pages each rendered a
repair (wrench) button for incomplete EC volumes. Its handler POSTed to a
/repair endpoint that the admin server never registers, so every click
returned "404 page not found" (the collection details page only had a
placeholder handler).
Remove the buttons and their JavaScript handlers, and regenerate the
templ output. Manual EC shard recovery remains available from weed shell
via ec.rebuild.
EC volumes do not propagate deletions to all shard indexes, so it is possible
to run scrubbing on a volume where a deleted needle is still present in the
index, or a needle deleted from the index is still present on the volume.
On either scenario, scrubbing will fail due to size mismatch errors.
This PR reworks the scrubbing logic so needle size mismatches are
ignored in such scenarios.
Scrubbing can still be forced to check deleted needles (f.ex. to discover
index inconsistencies); this option will be exposed in RPCs and `weed shell`
on a follow-up PR.
* fix(scrub): don't flag offset-0 logical tombstones in volume scrub
A remote-tier delete records a tombstone at .idx offset 0 with no physical .dat
bytes. Full scrub double-flagged a healthy remote-tiered volume with deletes:
scrubVolumeData counted the tombstone's GetActualSize(-1)=32 toward totalRead
(want > physical .dat), and CheckIndexFile treated it as occupying [0,31] and
flagged the first live needle as overlapping. Skip offset-0 logical tombstones
from both the size reconcile and the overlap check; they are still counted for
the index-size check. Local deletes (offset != 0) are unaffected.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* fix(scrub): mirror offset-0 logical tombstone handling into Rust
Same fix as the Go volume_checking.go + idx/check.go change: Volume::scrub skips
offset-0 logical tombstones from total_read, and check_index_file excludes them
from the overlap check (still counted for the index-size check).
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* fix(ec): suppress deleted-needle size mismatch in EC LOCAL scrub
EcVolume.ScrubLocal reassembles each fully-local needle and ReadBytes-checks
it, but appended every error unconditionally. A needle the .ecx still reports
live while its reassembled on-disk header carries size 0 (delete state
disagrees between index and header) is not corruption — the LOCAL twin of the
#10130 fix for the FULL path. Suppress the ErrorSizeMismatch in that case;
genuine (non-zero) size mismatches and CRC/tail errors are still reported.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* fix(ec): mirror EC LOCAL scrub deleted-needle suppression into Rust
Same suppression as the Go EcVolume.ScrubLocal change: a NeedleError::SizeMismatch
whose on-disk header size is 0 against a live index entry is a delete-state
disagreement, not corruption.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(ec): extract locate_ec_shard_needle_interval
Mirrors Go's EcVolume.LocateEcShardNeedleInterval; reused by locate_needle
and the upcoming local scrub walk.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(ec): add EcVolumeShard::to_ec_shard_info
Mirrors Go's ToEcShardInfo.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(ec): add EcVolume::scrub_local
Walk the .ecx and verify each needle against the locally-held shards,
reading interval-by-interval (reusing one chunk buffer); CRC-check only
fully-local needles, report short/unreadable local shards, and abort the
scan on a structural size mismatch. Mirrors Go's EcVolume.ScrubLocal.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(scrub): dispatch EC LOCAL (mode 3) to scrub_local
Splits the mode 2|3 arm: FULL (2) keeps the Reed-Solomon parity check;
LOCAL (3) now runs the per-needle local-shard walk.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* refactor(scrub): extract open_index_for_scrub shared by scrub_index
Mirrors Go's openIndex, shared by ScrubIndex and the upcoming Scrub rewrite.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* fix(scrub): walk the on-disk .idx in Volume::scrub
scrub walked the deduped in-memory map, so total_read undercounted the
physical .dat on any volume with overwrites or deletes and the size
reconcile falsely flagged healthy volumes broken. Walk every .idx row
instead (matching Go's scrubVolumeData): count all rows, CRC-verify live
needles, skip deleted, and reconcile against the .dat. Holds one data-file
read lock and reads via the unlocked path, like Go's Scrub.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(idx): add check_index_file mirroring Go idx.CheckIndexFile
Index-only structural check: walk the on-disk index, sort by (offset, size),
flag overlapping needles, and verify the file is a whole number of entries.
No data-file reads.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* refactor(ec): use idx::check_index_file in EcVolume::scrub_index
Drops the inline walk/sort/overlap copy. Walks a private fd so the structural
scan never moves the shared ecx_file cursor (read positionally elsewhere).
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* fix(scrub): make Volume::scrub_index an index-only check on the on-disk .idx
INDEX mode walked the deduped in-memory map and read .dat headers — more
than the cheap-INDEX contract allows, yet missing Go's overlap and
size-multiple structural checks. Route it through idx::check_index_file so
it matches Go's Volume.ScrubIndex and the INDEX<LOCAL<FULL cost tiering holds.
Ports openIndex's zero-size-index guard (a populated .dat with an empty .idx
is corruption) and takes the data-file read lock for a consistent snapshot.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* fix(ec): cap EcShardConfig at MAX_SHARD_COUNT, not TOTAL_SHARDS_COUNT
read_ec_shard_config rejected any .vif ratio summing past 14 shards and
silently fell back to 10/4, so wider EC volumes ran against the wrong
shard set. Match Go's MaxShardCount(32) bound.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* docs(ec): correct stale 0..14 shard-count comments
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
Master /dir/lookup JSON omits publicUrl when empty (Go json omitempty).
The Rust volume server required the field, so serde failed with "lookup
parse failed: error decoding response body" and cross-DC replicated writes
failed.
Default publicUrl to empty, fall back to url for peer filtering, and
normalize addresses with to_http_address before excluding the local peer
(so host:port.grpcPort forms do not match self incorrectly).
* test(seaweed-volume): cover type=replicate fan-out writes
A holder must accept a replicated copy and store it locally without
re-replicating. Covers raw and multipart bodies, and a multi-copy
volume where re-replication would otherwise reach the master.
* test(seaweed-volume): use port 0 for the dead-master address
Connecting to port 0 is refused at the socket layer immediately, so the
plain-write fan-out path fails fast instead of risking a connect-timeout
hang where port 1 is filtered.
* feat: add collection pattern to delete empty volumes
Co-authored-by: Codex <noreply@openai.com>
* shell: match collection pattern with wildcard matcher
Use wildcard.MatchesWildcard in the shared collection-pattern helper,
matching command_volume_fix_replication's matchCollectionPattern. The
flag only advertises '*' and '?', which is exactly what the matcher
supports.
---------
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3: support AWS object form for bucket policy Principal, add NotPrincipal
Bucket policy statements only accepted a bare string or array of strings for
the Principal element, so the AWS-documented object form was rejected:
"Principal": { "AWS": "arn:aws:iam::123456789012:root" }
"Principal": { "AWS": ["arn:...", "999999999999"] }
Add a PolicyPrincipal type that parses the bare string, the bare array
(retained for backward compatibility), and the object form keyed by AWS,
Service, Federated or CanonicalUser (each value a string or array). All keyed
values are flattened for principal matching, and the original JSON is preserved
so PutBucketPolicy/GetBucketPolicy returns the exact shape submitted - keeping
infrastructure-as-code tools (Terraform, Ansible) idempotent.
Also add NotPrincipal support (a statement applies to every principal except the
ones named), compiled and evaluated in both policy evaluators, and reject
statements that specify both Principal and NotPrincipal.
* s3: address review - validate principal object form, honor dynamic NotPrincipal
- Reject unsupported Principal object keys (only AWS/Service/Federated/
CanonicalUser) and empty values, so a form like {"AWS":[]} no longer compiles
to zero matchers and silently relies on the match-all fallback.
- Detect both Principal and NotPrincipal by field presence, not by flattened
length, so a present-but-empty field is still rejected.
- Honor dynamic (policy-variable) NotPrincipal/Principal patterns in the
compiled evaluator; previously a NotPrincipal made only of variables was
treated as absent and its exclusion bypassed.
- Add regression tests for the object-form validation and dynamic NotPrincipal.
* Review comment removed unnecessary success and failure count
* fix: use Gather.Gather() with seeded counter for EC rebuild registration test
- Restore Gather.Gather() to verify MustRegister calls as requested in review
- Seed VolumeServerECRebuildCounter before gathering because CounterVec
only appears after at least one label value is observed
- Use correct fully-qualified metric names (SeaweedFS_volumeServer_*)
* fix: remove preflight checkEcVolumeStatus failure from ec_rebuild_total counter
ec_rebuild_total should only reflect actual rebuild execution failures
(from RebuildEcFiles / RebuildEcxFile), not scan/precheck failures in
the volume status loop. The error is still returned to the caller;
only the misleading counter increment was removed.
* Review comment removed unnecessary observe
* label EC rebuild duration histogram by result
Without a result label, fast failures pull down the success-latency
quantiles shown on the EC Rebuild Duration panel. Make the histogram a
HistogramVec keyed by result, record success/failure through one
recordEcRebuild helper, and split the Grafana quantiles by (le, result).
* reset EC rebuild metric vecs in registration test
The HistogramVec needs a child before Gather emits it, so the test must
observe once; reset both vecs in cleanup so that sample doesn't leak into
other tests.
---------
Co-authored-by: Ubuntu User <ubuntu@example.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Chris Lu
Kwak Byoung Min
Jaehoon Kim
MorezMartin
Aleksey
Peter Dodd
Lisandro Pin
adri
jk2lx
7y-9
dependabot[bot]
github-actions[bot]
Rushikesh Deshpande