* volume: validate remote S3 endpoints in FetchAndWriteNeedle (Rust)
Port the Go volume server's SSRF guard to the Rust volume server. The
gRPC FetchAndWriteNeedle reads from a caller-supplied S3 endpoint, so an
unguarded server can be pointed at loopback / link-local / RFC1918 /
CGNAT / cloud-metadata hosts to read internal services. Resolve and
reject those endpoints unless -volume.allowUntrustedRemoteEndpoints is
set (default off), mirroring weed/server/volume_grpc_remote.go.
Connect-time re-validation against DNS rebinding (Go's guardedDialer) is
not yet ported: the aws-sdk-s3 client builds its own connector, so the
up-front resolve-and-check leaves a narrow TOCTOU window. Left as a
follow-up.
* volume: harden remote endpoint guard (IPv4-mapped IPv6, all S3 types)
Address SSRF review feedback:
- Normalize IPv4-mapped IPv6 (::ffff:a.b.c.d) to IPv4 before the deny
checks, so ::ffff:127.0.0.1 / ::ffff:169.254.169.254 no longer slip
past the IPv4 rules.
- Validate the endpoint for every S3-compatible backend, not just type
"s3"; wasabi/backblaze/aliyun/... all dial a caller-supplied endpoint
through the same client. Skip validation when the endpoint is empty
(the provider default, e.g. real AWS S3, which cannot reach internal
hosts).
* volume: set allow_untrusted_remote_endpoints in integration-test state
The tests/http_integration.rs VolumeServerState literal was missed, which
broke cargo test compilation (it builds the integration tests, unlike the
cargo test --lib used locally).
EC bitrot protection is now a fixed default: always on at 16 MiB block
granularity. These volume-server flags exposed needless configurability;
the package defaults in erasure_coding (BitrotProtectionEnabled,
BitrotBlockSize) are retained and still drive sidecar generation. Drop the
now-unused erasure_coding import.
TopicExists can return a transient false-negative for a topic that is in
fact present (a broker/filer blip under load, or a just-created topic whose
existence cache is still stale). The metadata and produce handlers then tried
to auto-create, hit "topic already exists", treated that as a failure, and
dropped the topic from the Metadata response - so the client saw a spurious
UNKNOWN_TOPIC_OR_PARTITION right after the topic was created.
Return a sentinel ErrTopicAlreadyExists from the create paths and add
ensureTopicExists, which treats it as confirmation the topic exists. It also
folds in the repeated TopicExists -> invalidate -> recheck -> create logic
shared by every metadata handler (v0-v8) and both produce paths. v2+ produce
now auto-creates too, matching the auto.create.topics.enable=true behavior the
rest of the gateway already simulates.
Add path filters to workflows that fired on every PR/push regardless
of the diff: CodeQL, go build, the e2e/EC/vacuum/TLS/plugin-worker
integration suites, the Kafka and Postgres gateways, the S3 suites
(Ceph s3tests, s3-go, s3-tables, proxy-signature, https, example,
filer-group), TUS, and the dev binary/container builds. Each scopes
to its subsystem under weed/, its test dir, go.mod/go.sum, and the
workflow file, so docs-, helm-, terraform-, rust- or java-only
changes no longer trigger a full compile-and-test fleet.
* mount: apply cached remote entry without blocking the read
Reading an uncached remote-mounted file hung forever. The read holds the
file-handle shared lock across the on-demand download, then synchronously
waits on the metadata apply loop to apply the cached entry. The filer's
update event for that same object reaches the apply loop first and runs
invalidateFunc, which wants the file-handle exclusive lock — held in shared
mode by the still-running read. The loop blocks on the read; the read blocks
on the loop.
Enqueue the apply without waiting so the read never blocks on the apply loop
while holding the lock. The handle is already updated via SetEntry, and the
filer subscription delivers the same event regardless.
* mount: regression test for uncached remote read deadlock
Drives the real Read path through downloadRemoteEntry with a stub filer that
broadcasts the matching invalidate event before returning, reproducing the
lock-ordering deadlock. Fails (times out) without the fix.
* s3api: add optional request interceptor to circuit breaker
Add an optional Interceptor func(next http.HandlerFunc, action string) http.HandlerFunc
field on CircuitBreaker, applied at the very top of Limit() -- before upload
concurrency limiting and before the 'if !cb.Enabled' early return -- so it runs
for every route regardless of breaker state.
It is nil by default (no behavior change). An interceptor may reject a request
(write its own response and skip next) or wrap next to observe/shape it. This
provides a single per-request extension point at the existing per-route
chokepoint (cb.Limit already wraps nearly every S3 route), useful for tracing,
auditing, or request rate limiting without touching the route table.
* s3api: evaluate circuit breaker interceptor per request
Move the Interceptor nil-check into the returned handler so it is consulted
per request instead of captured at registration time. This:
- lets the interceptor be installed after registerRouter runs (handlers are
built during construction, before request-time dependencies exist), and
- avoids dereferencing a nil CircuitBreaker at registration time, which
panicked tests that register routes on a server with a nil cb
(e.g. TestRouting_STSWithQueryParams).
Adds a regression test for installing the interceptor after Limit() returns.
* fix(shell): return fs.verify topology errors
Problem: fs.verify can silently ignore master topology lookup failures before verifying files.
Root cause: commandFsVerify.Do returned parseErr after collectVolumeIds failed, but parseErr is nil once path parsing succeeds.
Fix: Return the actual collectVolumeIds error so VolumeList and master client failures stop the command.
Co-authored-by: Codex <noreply@openai.com>
* remove the tests
---------
Co-authored-by: Codex <noreply@openai.com>
isLocalOnlyEntry resolved the pinned-child check through inodeToPath. A kernel
Forget drops the path→inode mapping once the lookup count reaches zero, but an
async writeback flush — and the file handle, still in fhMap during the drain —
is keyed by inode and outlives that mapping. Between Release dispatching the
async flush and the flush reaching the filer, a Forget could unpin an in-flight
create, so a concurrent directory rebuild would wipe it and the file would
ENOENT until the flush lands and the cache refreshes.
Key the check off the inode the store entry already carries (createFile stamps
it into the placeholder), so the pin no longer depends on a mapping Forget can
remove.
* s3: bound streaming remote-cache wait so large cold GetObject returns 503, not a hang
A remote-only object whose download outlasts the 30s dedup attempt fell through
to cacheRemoteObjectForStreaming, which waited on the raw request context with no
bound. For multi-GB blobs the client read timeout fires first, so the request was
canceled and surfaced as InternalError rather than the 503 + Retry-After the
handler already emits. The filer keeps caching on a detached context, so the
retry would have streamed from the cached chunks.
Bound the wait under common client read timeouts: the 503 fires while the client
is still connected, and a retry picks up the finished cache.
* s3: make the streaming remote-cache timeout atomic
The bound is a package-level var so tests can shorten it. Hold it as an
atomic int64 so a test that stores a short value can never race the request
path that loads it.
* s3: assert the streaming-cache timeout test waits on the bound
The upper-bound-only check would also pass if the RPC returned immediately on a
setup error. Add a lower bound so the test fails unless the bounded wait fired.
wfs.Create defers the filer create to Flush and inserts a local-only
placeholder into the metaCache (dirtyMetadata=true), so a just-created file
exists locally before the filer has it. When the parent directory falls out of
cache (hot-dir read-through, idle evict) and is rebuilt, EnsureVisited wipes the
store and refills from a filer listing that does not yet include the un-flushed
create, then markCachedFn publishes the directory authoritatively cached without
it. lookupEntry then returns an authoritative ENOENT and ReadDir returns nothing
— the file disappears from the mount although the client created it. Under
concurrent read+write churn on one directory this is the ConcurrentReadWrite
flake.
Preserve children the mount flags as local-only (open dirty handle or pending
async flush — the signal lookupEntry already trusts) across the rebuild's wipe
instead of blind-deleting them. Unpinned stale children are still dropped so a
rebuild cannot resurrect a deleted entry.
The s3 and rclone tiered-storage backends were registered via blank imports
in weed/storage (volume_tier.go and volume_info/volume_info.go). That forced
every library consumer of weed/storage -- weed/shell, and through it external
tools -- to link aws-sdk-go and, under the rclone build tag, the full rclone
backend set and its cloud-storage SDKs, even though those consumers never tier
volumes.
Move the registrations into a new weed/storage/backend/all aggregator and
blank-import it once from weed/command, the binary's composition root. The weed
binary still registers both backends; weed/storage and its library consumers no
longer pull the backend SDKs into their dependency graph.
* SECURITY.md: require working repro and trust-boundary impact
Funnel reports through GitHub private vulnerability reporting, require a
reproduction against a supported version, and spell out the trust model so
internal-cluster exposure and example-config issues are out of scope.
Unvalidated scanner/AI output is handled as hardening, not as an advisory.
CVEs minted by third parties outside this process may be disputed.
* SECURITY.md: drop email channel, GitHub private reporting only
* mount: tolerance-window write pattern detection for concurrent writeback
WriterPattern decides whether each dirty chunk is buffered in RAM
(NewMemChunk) or spilled to an on-disk swap file (NewSwapFileChunk), so a
sequential stream misclassified as random is pushed through disk-backed
swap.
The old detector compared each write's start to the previous write's
exact stop offset (atomic.Swap of lastWriteStopOffset, then
lastOffset == offset). That is brittle under concurrent FUSE writeback:
MonitorWriteAt runs before the per-handle write lock, so parallel Write
upcalls interleave the swap, and writeback flushes dirty pages slightly
out of offset order. A genuinely sequential stream then repeatedly reads
as random and spills to swap.
Replace the exact match with the same frontier + tolerance approach used
on the read side: track a never-regressing max frontier (offset+size) via
a CAS loop and treat a write whose start is within SeqTolerance (8 MiB) of
that frontier as sequential. The existing +/-ModeChangeLimit hysteresis is
kept, so a single outlier write can't flip the mode.
Adds page_writer_pattern_test.go covering sequentiality, hysteresis,
reorder tolerance, the inclusive tolerance boundary, frontier
non-regression (the CAS invariant), and recovery back to sequential mode.
* mount: read write frontier inside the CAS loop
Capture the frontier from the CAS loop's own load rather than a separate
up-front snapshot, so the sequentiality diff is judged against the freshest
pre-image even if a concurrent writeback upcall advances the frontier while
we loop. Also drops the now-redundant load. Behavior is unchanged for the
single-threaded tests; addresses review feedback on #9984.
* filer: tolerance-window read pattern detection for concurrent readahead
ReaderPattern detected sequential access via strict contiguity (lastReadStopOffset
== offset). Under concurrent/reordered ReadAt (parallel readahead, which ReadAt
already supports via RLock + atomics), that exact match frequently misses even for
a genuinely sequential stream, drifting toward random mode and disabling prefetch.
Track a read frontier (max offset+size, advanced with a lock-free CAS loop) and
treat a read as sequential when its start is within SeqTolerance (8 MiB) of the
frontier. The window absorbs reordered/concurrent readahead while still rejecting
far random jumps; the existing ModeChangeLimit hysteresis is unchanged. API
(NewReaderPattern/MonitorReadAt/IsRandomMode) is unchanged, so reader_at is
unaffected. Adds reader_pattern_test.go.
* filer: read the read frontier inside the CAS loop
Capture the frontier from the CAS loop's own load rather than a separate
up-front snapshot, so the sequentiality diff is judged against the freshest
pre-image even if a concurrent readahead advances the frontier while we loop.
Also drops the now-redundant load. Mirrors the same fix on the write-side
ReaderPattern twin (review feedback on the companion PR).
* filer: add frontier/boundary/recovery guard tests for ReaderPattern
Mirror the regression guards added to the write-side WriterPattern: assert
the max-frontier never regresses on a backward read (the CAS invariant), pin
both sides of the inclusive SeqTolerance boundary, and verify sustained near
reads recover sequential mode out of the negative floor. Each guards an
invariant the original four tests left uncovered.
* test: add FUSE database load/durability/perf benchmark
Runs MySQL (InnoDB) and SQLite with ~1GB datadirs on a SeaweedFS FUSE
mount. Two parts:
- durability: normal shutdown, kill -9, and crash-during-write all keep
every fsync-committed row (verified by integrity check + row count +
contiguous prefix + per-row CRC).
- performance: FUSE vs the same local disk. fsync/commit latency is the
dominant cost (~0.13ms -> ~1.18ms), so small transactions run ~9-12x
slower while bulk loads and warm reads stay close.
Harness is path-independent (runtime under $SEAWEED_BENCH_WORK) and only
touches its own processes on non-default ports.
* test/benchmark/fuse_db: portable to Linux + crash-safe progress
- export MYSQL_BIN; mysql_bench.py falls back to PATH when unset
- unmount via fusermount/fusermount3 (non-root Linux), then umount/diskutil
- atomic progress write (tmp+fsync+rename); treat empty progress file as 0
- reuse a single PRNG in the perf probe so RNG init doesn't skew timings
* test/benchmark/fuse_db: validate inputs, add subprocess timeout
- mysql_bench.py: 1800s timeout on mysql CLI calls; reject db names that
aren't plain identifiers (interpolated into SQL)
- sqlite_gen.py / sqlite_verify.py: allowlist journal/synchronous modes and
the verify mode so a typo can't silently weaken durability or relax checks
- run_mysql.sh: durable atomic progress write (tmp+fsync+rename), matching
sqlite_gen.py; quote $LB in both crash-test verify calls
security.toml: document WEED_ env override for the jwt signing keys
These keys are HMAC secrets; spell out the env-var mapping so they can be
injected from a secret store instead of living in the config file.
* fix(command): preserve fuse option after writers
Problem: FUSE mount option parsing skipped the option immediately following concurrentWriters, so values such as concurrentReaders could be silently ignored.
Root cause: runFuse incremented the options loop index inside the concurrentWriters case in addition to the loop increment.
Co-authored-by: Codex <noreply@openai.com>
* test: save and restore mountOptions pointers, not their values
The fields are reassigned to fresh heap variables during runFuse, so
dereferencing to back up/restore mutated throwaways instead of the
flag-bound originals and could nil-panic on unset fields.
---------
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* admin: add connected mount clients page and dashboard section
The filer is the authority on who is subscribed to its metadata stream
(FUSE/VFS mounts, S3, peer filers, ...), but its in-memory listener
registry only tracked clientId->epoch and was not exposed.
- Enrich the filer subscriber registry with name/type/address/path/
connected-time, populated in addClient and cleared in deleteClient so
it reflects currently-connected clients only.
- Add a ListMetadataSubscribers filer gRPC (optional client-type filter).
- Admin server fans out to every filer, filters to mount types
("mount" Go weed mount, "sw-vfs" Rust VFS), and renders a new
Cluster > Mount Clients page plus a Mount Clients dashboard section.
Read-only; no behavior change to the subscribe hot path.
* admin: address review — parallelize filer fan-out, guard nil map, robust CSV
- GetMountClients now queries filers concurrently, each under a 5s
timeout, so a slow/unreachable filer can't stall the admin dashboard.
- Defensively initialize fs.subscribers before first write.
- Mount Clients CSV export uses a Blob with quote-escaping instead of a
data: URI, so special characters in paths export correctly.
* filer: keep metadata-subscription send gauge fresh on idle heartbeat
last_send_timestamp_of_subscribe only advanced when a real matching
metadata event was streamed to a subscriber. On a quiet path an idle but
perfectly healthy subscriber therefore looked increasingly stale, and the
dashboard panel rendered a large, misleading 'lag'.
An idle heartbeat is a send too, so advance the gauge when one is emitted.
Subscribers that opt into idle heartbeats (filer.sync) now report true
freshness; the rest still show time since the last real event.
Rename the dashboard panel 'Metadata Subscription Lag' ->
'Time Since Last Subscription Send' and clarify its description to match.
* filer: guard nil option when advancing heartbeat gauge
maybeSendIdleHeartbeat is unit-tested with a bare &FilerServer{} (nil
option), so dereferencing fs.option.Host for the sourceFiler label
panicked. Guard it: production always has option set; the test now gets
an empty sourceFiler label instead of a nil-pointer panic.
* s3: register account-less identities' synthesized account in the lookup
#9962 gave each account-less identity a distinct account id derived from
its name (instead of collapsing into admin), but never registered that
account in the id->account map. GetAccountNameById then returned empty
for such ids, so ACL grantee validation rejected canonical grants to the
caller's own account with InvalidRequest, and bucket/object owner display
was dropped as 'owner is invalid'.
This broke a canned PutObjectAcl by an account-less identity (e.g.
TestVersionedObjectAcl with the default 'some_admin_user' identity):
ValidateAndTransferGrants -> GetAccountNameById -> 'account id is not
exists' -> 400 InvalidRequest.
Register the synthesized account at config load so its id resolves to a
display name. Add a regression test.
* s3: reuse explicitly-configured account for account-less identity
Address review: if an account with the same id as an account-less
identity's synthesized account is explicitly configured (custom display
name/email), reuse it instead of the synthesized one. Add a test.
* metrics: add per-bucket S3 panels and volume slot utilization to dashboard
The S3 Gateway/Buckets rows sliced requests only by type, never by bucket,
though SeaweedFS_s3_request_total and the request/TTFB histograms all carry a
bucket label. Add a Bucket template variable and four per-bucket panels to the
S3 Buckets row (request rate, response codes, request p95, TTFB p95), plus a
Volume Slots Utilization panel (volumes/max_volumes) to the Volume Servers row.
* metrics: scope bucket variable to cluster and guard slot utilization divide-by-zero
Source the Bucket variable from SeaweedFS_s3_bucket_size_bytes (a gauge emitted
for every bucket, including idle ones) scoped to the selected cluster, so the
dropdown lists all buckets in the cluster rather than only those that received
requests. Wrap the volume slot utilization denominator in clamp_min(..., 1) to
avoid +Inf/NaN when max_volumes is briefly zero or absent.
* metrics: guard time()-based panels against unset (zero) gauges
Several dashboard panels compute "time() - <gauge>" (uptime, time-since-
last-scrub, lifecycle cursor lag, last daily walk). When the underlying
gauge is unset it reports 0, so the panel rendered ~56 years (time since
the Unix epoch). This is the common case for "Volume Server Uptime": the
Rust volume server doesn't set SeaweedFS_volumeServer_start_time_seconds,
and Go components expose it registered-but-zero.
Guard each such expression with "> 0" so unset/zero series drop out (the
panel shows no data) instead of rendering a nonsensical epoch value.
Affected panels: Master Uptime, Volume Server Uptime, Time Since Last
Scrub, Lifecycle Cursor Lag, Time Since Last Daily Walk.
* metrics: guard Filer Sync Offset Lag and Metadata Subscription Lag panels too
Extend the > 0 guard to the two remaining time()-<gauge> panels that
share the same unset/zero failure mode (filerSync sync offset and the
metadata subscribe last-send timestamp), so they don't render ~56-year
lag when the gauge is absent.
admin: fold dashboard sparklines into the existing cards
The trend sparklines added in #9957 lived in a separate "Cluster Trends"
row that duplicated the existing summary cards (Volumes, Files, Disk Used,
EC Shards). Remove that row and instead render each sparkline inside the
matching summary card, so every headline number shows its recent trend
without duplication. The two maintenance metrics that have no existing
card — Active Tasks and Workers — now fill the previously-empty columns of
the EC row (also with sparklines).
DashboardTrends changes from a Cards slice to named per-card sparkline
SVGs (+ current values for the two maintenance cards). Drops the now-unused
trendBytes helper (disk size keeps using the existing formatBytes).
* s3tables: resolve account-less identities to a distinct principal
Static identities with no account block default to the shared admin
account, so getAccountID returned "admin" for every such user and the
permission checks treated them all as the admin principal. Only keep the
admin account when the identity actually carries an admin action;
otherwise fall back to the unique identity name.
* s3tables: limit the open-by-default fallback to anonymous access
The legacy permission path allowed any request that no policy explicitly
denied whenever default-allow was on, which is the zero-config default.
That let an authenticated identity without table permissions reach table
resources owned by others. Restrict the fallback to requests with no
identity or the anonymous identity; authenticated callers must pass an
explicit action or policy check. Zero-config and anonymous access are
unchanged.
* s3tables: drop the no-op ListTableBuckets account gate
The top-level check passed the principal as its own owner, so it always
allowed. Per-bucket filtering in the loop is the real authority; remove
the dead gate and the now-unused locals.
* s3tables: derive the Iceberg catalog's default-allow from auth state
The Iceberg catalog reuses the S3 Tables Manager, which hardcoded
default-allow on. Authenticated callers were enforced only because the
identity struct happens to propagate into the handler; if it were ever
dropped, a secured catalog would fall open. Mirror the S3 port and set
the Manager's default-allow from the authenticator, so an authenticated
caller is enforced regardless. Shell and admin keep their own trusted
Manager. Regression test covers the struct, name-only, and admin paths.
* s3tables: drop redundant ACTION_ADMIN string conversion
ACTION_ADMIN is an untyped string constant, so the conversion is a no-op.
* s3tables: enforce name-only authenticated callers, add trusted bypass
defaultAllowFor treated a request with no identity object as anonymous,
but the Manager path forwards only the identity name (not the struct).
A name-only authenticated caller could therefore be misclassified as
anonymous and allowed under the open default. Treat a server-set identity
name as authenticated too, and add an explicit trusted flag for the local
shell/admin tooling that legitimately bypasses authorization.
* s3tables: trim verbose comments
* admin: native at-a-glance trend sparklines on the dashboard
Add a "Cluster Trends" row to the admin Dashboard with inline-SVG
sparklines for volumes, EC shards, disk used, files, active maintenance
tasks, and workers.
The data comes entirely from what the admin already holds — the cached
cluster topology and the in-process maintenance queue — sampled into a
small bounded ring buffer on the existing maintenance-metrics ticker
(~15 min of history). No Prometheus/Grafana dependency, no JS chart
library, no extra goroutine: the sparklines are self-contained SVG
rendered server-side via templ.
This gives basic trend visibility out of the box for clusters that don't
run Prometheus, and a quick glance next to the cluster controls; Grafana
remains the place for deep/historical dashboards.
* admin: cap trendBytes unit index to avoid out-of-bounds panic
A value >= 1 ZiB would push exp past the end of the units string and
panic on units[exp]; cap exp at the last unit (EiB).
* s3: give STS sessions a distinct owner account, not admin
STS sessions were built with Account: &AccountAdmin, so every assumed-role
session shared the admin account for ownership and ACL checks. Use the
assumed-role user as the account id instead, matching the JWT auth path.
Session permissions are unchanged: they come from the session policies,
and admin is granted only through Actions.
* s3: resolve STS session identity to the OIDC subject
Use sessionInfo.Subject (falling back to the assumed-role user when
absent) for the session identity name and account id, so the SigV4 and
JWT auth paths resolve the same session to the same identity instead of
diverging on AssumedRoleUser vs Subject.
* s3: trim verbose comments
* s3: stop collapsing account-less identities into the admin account
Identities configured without an account block all defaulted to the
shared admin account, so distinct users got the same owner id and
ownership checks could not tell them apart. checkAccessByOwnership also
treated that id as an admin bypass, so any account-less caller passed
ownership for any bucket. Give such identities a distinct account id from
their name, and decide the ownership admin bypass by Admin capability
rather than by the account id. isUserAdmin is now nil-safe.
* s3: use the context identity in isUserAdmin before re-authenticating
The Auth middleware already verifies and stores the identity in the
request context. Read it there first so the ownership/admin checks don't
re-run signature verification, which is redundant and fails once the
request body has been consumed.
* s3: nil-guard the context identity in isUserAdmin
A non-nil interface wrapping a typed-nil *Identity passes the type
assertion; guard against it before calling isAdmin().
* s3: trim verbose comments
The bundled dashboard (other/metrics/grafana_seaweedfs.json) covered only
18 of the 84 metrics weed/stats exposes and was a legacy Grafana 8 export
(graph panels, schemaVersion 30). Rebuild it as a modern dashboard
(timeseries panels, schemaVersion 39) with 100% metric coverage, targeting
the direct-scrape model used by Prometheus / seaweed-up / Kubernetes.
- Full coverage of every weed/stats metric: master, volume server, filer,
filer store/sync, s3, s3 buckets, s3 lifecycle, admin/maintenance, build,
wdclient, upload errors, plus Go runtime/process per component.
- Organized into collapsible rows with an always-on Overview.
- Scrape label model: group by `instance`; generic go_*/process_* panels
use `job=~"seaweedfs-.*"` to separate components; an optional `cluster`
template variable (from SeaweedFS_build_info, defaults to All) supports
multi-cluster setups and is transparent when no cluster label is present.
- Same uid (nh02dOVnz) and title so it upgrades in place; drops the dead
"AWS monthly cost" panel.
This is also the single source of truth bundled by seaweed-up's
`cluster dashboard install`.
* refactor(ec_balance): make the balance planner per-volume ratio-capable
Thread a per-volume EC ratio through the balance planner: Plan resolves each
volume's data/parity from a new Options.VolumeRatio (falling back to the
collection Ratio, then the build default, when it reports 0), and keys the
global phase's ratio maps by volume instead of collection. The shell and
worker balance paths build the per-volume lookup from each shard's heartbeat
via the new ecbalancer.VolumeShardRatio.
In OSS this is behavior-preserving: VolumeShardRatio returns 0 because the
per-volume data_shards/parity_shards heartbeat fields are an enterprise
feature, so every volume falls back to the collection ratio -- the existing
standard-scheme behavior. The refactor keeps the shared planner in sync with
the enterprise fork, which overrides VolumeShardRatio to classify and spread
a mixed-ratio collection by each volume's own data/parity split.
* perf(ec_balance): hoist the collection ratio out of the per-volume loop
The collection ratio is constant for every volume in a collection, so
resolve it once per collection instead of per volume; a custom Ratio func
may do map lookups or locking. Addresses a review comment.
* fix(ec): recover EC shards with the volume's own ratio, not the build default
recoverOneRemoteEcShardInterval rebuilt a missing shard with a hardcoded
10+4 Reed-Solomon matrix (and counted sufficiency / iterated shards
against the 10+4 constants). For a custom-ratio volume (e.g. 9+3) that
reconstructs with the wrong matrix and corrupts the recovered bytes, and
cachedLookupEcShardLocations could wrongly reject a degraded but
recoverable custom-ratio read. Use the volume's own ECContext (loaded
from its .vif) for the encoder, the shard-iteration bound, and the
data-shard sufficiency checks. In OSS the ratio is always 10+4 so this is
a no-op; it brings the Go volume server in line with the Rust one, which
already reconstructs with the volume's ratio.
* fix(ec): close data races in the EC read-recovery path
Address review: the freshness check in cachedLookupEcShardLocations read
ecVolume.ShardLocations / ShardLocationsRefreshTime without the lock while
recover goroutines mutate them via forgetShardId -- snapshot both under
ShardLocationsLock.RLock(). The recover goroutines also wrote the shared
is_deleted return concurrently -- collect it via an atomic and fold it in
after they join. Also size availableShards/missingShards by the volume's
ECContext ratio rather than the 10+4 constants.
* fix(ec): remove the shared EC index only when no shard remains node-wide
deleteEcShardIdsForEachLocation removed the shared .ecx/.ecj/.vif index
as soon as a single disk's shard count hit 0, even when a sibling disk
of the same node still held shards of the volume (split-disk reconciled
layout) -- orphaning those shards without their index. Split the
non-teardown delete into two passes: delete the requested shard files
(and now-orphaned per-disk bitrot sidecars) on every disk, then remove
the shared index only once no shard of the volume remains on ANY disk.
This brings the Go volume server in line with the Rust one, which already
gates the index removal on a node-wide check.
* refactor(ec): reuse checkEcVolumeStatus across the two delete passes
Address review: cache hasEcxFile/hasIdxFile from the node-wide count pass
and pass them to removeEcSharedIndexFiles instead of re-listing each
location's directory.
* fix(ec): clean an orphaned EC .vif even when its .ecx is already gone
Address review: removeEcSharedIndexFiles returned early on !hasEcxFile,
so a node-wide teardown left a stale EC .vif behind when its .ecx was
already removed. Decouple the .vif removal (gated on !hasIdxFile) from
.ecx presence so the generation metadata doesn't leak once no shard
remains node-wide.
* fix(storage): delete and unmount every copy of a duplicate volume id
NewStore has no cross-disk duplicate guard (unlike the Rust volume
server, which refuses to start in that state), so a stale twin of a
volume id can mount on a second disk after a disk repair. DeleteVolume
and UnmountVolume returned after the first matching disk, leaving the
twin to survive and re-register as the volume's content. Walk every disk
and act on all copies, emitting one heartbeat delta per copy.
* fix(storage): surface partial delete/unmount failures across duplicate copies
Address review: if removing one copy of a duplicate volume id fails with a
real error (disk IO, permissions), the loop logged it and could still
return success once another copy was removed -- leaving the stale copy to
re-register, the exact divergence this guards against. DeleteVolume and
UnmountVolume now accumulate such errors and return them (still attempting
every disk), so a copy left behind is never reported as success. Add a
DeleteVolume duplicate-copies regression test.
* feat(ec): add encode_ts_ns to the EC task params, shard-unmount, and shard-delete RPCs
The generation fence for stale EC-worker cleanup needs the encode
generation on three messages: ErasureCodingTaskParams (admin issues it),
VolumeEcShardsUnmountRequest, and VolumeEcShardsDeleteRequest (the worker
carries it to the volume server). Additive fields only; 0 preserves the
existing unfenced behavior. Mirror the two volume-server fields in the
Rust volume server's proto copy.
* feat(ec): issue the EC encode generation from the admin and carry it on the worker
Stamp each EC proposal's encode_ts_ns from the admin's per-cycle
DetectionSequence (a single-clock value) so generations are globally
ordered even though detection runs on a rotating worker. The worker
writes that generation into the distributed .vif and passes it on its
shard unmount/delete RPCs; it falls back to a local timestamp for the
.vif only on the unfenced legacy/shell path (keeping the read guard on).
* fix(ec): fence the stale-worker EC shard unmount and teardown by generation
A reaped-but-still-running EC worker's cleanupStaleEcShards issued a
generation-blind unmount + full teardown that could unmount and then
overwrite a newer run's live shards on a shared node. Both RPCs now
carry the encode generation: the volume server unmounts/deletes a disk
only when its .vif generation is strictly older than the request, and
preserves a same-or-newer generation, a generation-0 (recovered or
pre-upgrade) volume, and an unreadable .vif. Unload is per-disk, never
node-wide. Request generation 0 keeps the blanket teardown for the shell
pre-encode cleanup and pre-upgrade callers. Mirrored in the Rust volume
server.
* test(ec): cover the generation-fenced teardown and unmount
End-to-end volume-server tests: a fenced FullTeardown wipes a strictly-
older generation, preserves a newer one, preserves a generation-0 volume,
and blanket-wipes on request generation 0; the gen-aware unmount preserves
a same-or-newer mounted generation; and the .vif generation reader handles
present/absent/no-config cases.
* test(ec): pin the fenced .vif==teardown generation and the unreadable-.vif preserve
A fenced run must stamp the admin generation verbatim into the .vif so it
matches the generation sent on the teardown RPCs; add a regression test
that sets the task generation and asserts the .vif carries it exactly.
Also cover the present-but-unparseable .vif case (reads as generation 0,
preserved) and correct the readEcGenerationTsNs docstring accordingly.
* fix(ec): surface EC full-teardown filesystem errors in the Rust volume server
remove_ec_volume_files(_full_teardown) discarded every fs::remove_file
error, so a teardown that failed on permissions or a full disk still
returned full_teardown_done=true and left stale artifacts to collide with
the next encode. Return io::Result, ignore NotFound, propagate the first
real error, and have the teardown RPC surface it -- matching the Go
contract. The best-effort reconcile/load-cleanup callers keep ignoring it.
* refactor(ec): reuse the EC volume lookup on unmount and short-circuit the gen read
Address review: the Rust unmount fence reuses the ec_vol it already
fetched instead of a second find_ec_volume; the Go .vif generation reader
breaks out of the data/idx loop early when the two dirs are the same.
* s3: preserve checksums for copied multipart parts
* s3: return checksums from multipart copy
* s3: pin the upload's checksum algorithm on copy-part re-stream
* s3: note why UploadPartCopy uses the re-stream slow path
* s3: explain the TLS proxy in the multipart copy checksum test
* s3: cover nil and unknown-algorithm edge cases in copy checksum tests
* s3: cover all checksum algorithms in the multipart copy test
* s3: run all checksum integration tests, not just presigned
* feat(ec): carry the encode generation through the topology heartbeat
Add encode_ts_ns (field 14) to VolumeEcShardInformationMessage and
populate it from each EC volume's .vif identity. The volume server emits
it on the full and incremental heartbeats; the master stores it on
EcVolumeInfo and re-emits it via GetTopologyInfo, so the admin/worker
layer can see which encode run produced each shard set. Field 14 avoids
the enterprise fork's reserved 10-13. Mirror the proto field and both
heartbeat emit sites in the Rust volume server.
* fix(ec): group orphan-source shard completeness by encode generation
countExistingEcShardsForVolume ORed EcIndexBits across every disk, so two
interrupted encode runs whose shard sets overlap unioned into a
false-complete set -- triggering the orphaned-source delete while no
single generation was actually complete. Group shards by encode_ts_ns and
return the largest single generation's count, so the trigger fires only
when one run holds the full set. Shards from pre-upgrade servers
(encode_ts_ns==0) form their own bucket.
The heartbeat carries one encode_ts_ns per (volume, disk), so this
separates generations on different disks; same-disk mixing is prevented
upstream by the pre-encode artifact wipe and the cross-run read guard.
* fix(ec): guard against a nil Ec shard info entry in the generation count
Defensive: a manually-constructed or corrupted topology could carry a nil
entry in EcShardInfos. Skip it rather than dereference.
* fix(ec): carry the encode generation on the EC shard unmount delta
The mount delta sets EncodeTsNs; the unmount deletion delta left it 0.
Populate it from the Ec volume before unloading so both incremental
deltas are consistent (the Rust volume server already does this via its
snapshot diff).
When an EC volume's .vif was missing, NewEcVolume wrote a stub holding
only the version. That stub implies the default 10+4 ratio with
DatFileSize=0 and no encode identity, which the custom-ratio resolver
and the startup credibility checks then read as an authoritative config
-- masking the real ratio of a custom-ratio volume and defeating the
byte-exact .vif gate. Mount with in-memory defaults instead and leave
the real .vif to the encoder or a recovery tool. The Rust volume server
already behaves this way.
* preserve compression state during replication
* explain why ParseUpload skips compression for replica writes
* fix data race on err result in FetchAndWriteNeedle
The local-write and replica-write goroutines all wrote the named err return under an unsynchronized err==nil check. Give each goroutine its own error slot and combine after wg.Wait(): local error wins, then the first replica failure.
* skip redundant decompression of compressed needles during replication
doUploadData decompressed a compressed input only to report the clear-data length on UploadResult.Size, which both replication callers discard. Skip the decompress when IsReplication.
* fix(ec): persist the EC source replica readonly mark
markReplicasReadonly marked each regular replica readonly without
persisting it, so a source-server restart during or after encoding
silently reopened the volume to writes. Those writes are not in the EC
shards, and the later orphan-source cleanup would then delete the
replica, losing them. Send Persist:true so the mark survives a restart;
rollbackReadonly still clears it via VolumeMarkWritable on a failed
encode.
* fix(ec): don't delete a writable source replica during orphan cleanup
cleanupOrphanSourceReplicas issued VolumeDelete to every regular replica
once the EC shard set looked complete, without checking the replica's
current state. A replica that came back writable may hold writes the EC
shards do not contain, so deleting it loses data. Re-probe each replica
via VolumeStatus and skip any that is no longer readonly, logging a
warning instead of deleting.
* fix(ec): check decode .idx writes and fsync decoded .dat/.idx
WriteIdxFileFromEcIndex silently dropped io.Copy and Write errors, so a
short or failed write of the reconstructed .idx went unnoticed and the
caller proceeded to delete the source EC shards. Propagate those errors.
Also fsync the decoded .dat and .idx before returning, so the bytes are
durable before the shards that produced them are removed cluster-wide.
Mirror the .idx fsync into the Rust volume server (its .dat already
syncs and its writes already propagate errors).
* fix(ec): publish decoded .dat/.idx atomically via temp file and rename
WriteDatFile and WriteIdxFileFromEcIndex wrote in place at the final
name with O_TRUNC. A crash mid-write left a truncated .dat/.idx at the
final name beside the still-present EC shards; on restart that partial
file could be mounted as the live volume even though the shards held the
real data. Write to a .tmp file, fsync it, then rename into place and
fsync the directory, so the final name is only ever absent or complete.
A failed decode removes its own temp file rather than leaking it.
Add util.FsyncDir as the shared directory-fsync primitive and reuse the
Rust volume server's fsync_dir for the mirrored change.
* fix(ec): propagate .ecj read errors in the Rust decoder
Path::exists returned false for any error (permission denied, transient
IO), silently skipping the deletion journal and resurrecting deleted
needles as live. Read the journal directly and treat only NotFound as
absent, propagating other errors. The Go decoder already behaves this
way (FileExists returns false only for IsNotExist, then the open
surfaces other errors).
* fix(ec): remove rename destination on Windows in the Rust decoder publish
std::fs::rename does not replace an existing file on every Windows
version. Remove the destination first under a Windows guard before the
atomic publish rename, matching the compaction commit path.
* tiering: stop a shared remote object being deleted while replicas still point at it
A remote-tiered volume's .dat content lives only in one cloud object that all
N replica .vif files point at. Deleting that object while destroying any one
replica, or before a downloaded replica is durable, bricks the survivors.
- volume.tier.move cleanup now deletes old replicas with keepRemoteData=true so
surviving replicas keep the shared object. Document why the alreadyPlaced
anchor needs no replica sync (same-object replicas are byte-identical).
- VolumeTierMoveDatFromRemote now fsyncs the downloaded .dat, fsyncs the
containing directory, trims the .vif (fsynced) and swaps to the local DiskFile
BEFORE deleting the remote object, on both the keep-remote and delete paths.
Only the final DeleteFile is gated by keep_remote_dat_file, so a keep-remote
download leaves the replica served from local disk rather than the shared
object, and a crash before delete merely leaks the object.
- volume.tier.download keeps the shared object for every replica except the
last, which deletes it.
- s3 and rclone download paths fsync the .dat before close.
* storage: swap the volume data backend under the data lock
The tier-download swap closed v.DataBackend and assigned the new local DiskFile
without holding dataFileAccessLock, racing concurrent reads/writes (use of a
closed file / nil deref). Add an exported Volume.SwapDataBackend that performs
the close-and-replace under the lock, and call it from the tier download.
* server: skip directory fsync on Windows in the tier download path
os.Open(dir).Sync() is unsupported on Windows and returns an error, which would
fail VolumeTierMoveDatFromRemote entirely there. Skip the directory fsync on
Windows, matching how the storage-side helper tolerates the unsupported case.
* shell: make multi-replica tier.download resilient to already-local replicas
If a multi-replica download is interrupted and retried, a replica made local
in the prior attempt returns "already on local disk", which aborted the whole
command and left the remaining remote replicas dangling. Treat that case as a
skip-and-continue so a retry completes the rest.
* server: assert downloaded .dat content, not just length, in the tier test
A length-only check passes even if the bytes are corrupted; compare the full
content of the local .dat against the original.
Chris Lu
dependabot[bot]
Chris Lu
7y-9