During leader election, MaybeLeader() returns empty string immediately (non-blocking),
causing master to return NotLeaderError without sending HeartbeatResponse.Leader.
Volume servers depend on HeartbeatResponse.Leader to discover the new leader address,
so they keep retrying the old leader and cannot switch.
Switching to Leader() restores the 20-second exponential backoff retry behavior,
ensuring volume servers receive the new leader address as soon as election completes.
* feat: add Prometheus metric for volume creation operations
Add VolumeServerVolumeCreationCounter metric to track volume creation
attempts by result (success/failure).
Changes:
- Add VolumeServerVolumeCreationCounter in weed/stats/metrics.go
- Instrument GrowByCountAndType in weed/topology/volume_growth.go
- Add unit tests in weed/stats/metrics_volume_creation_test.go
- Add Grafana dashboard panel for volume creation rate
This metric enables monitoring volume creation success/failure rates
in SeaweedFS clusters.
* fix: address CodeRabbit review - move failure counter before early return
- Move failure counter into loop else block before return statement
- Move success counter to after loop completion
- Strengthen test assertion from count < 1 to count != 2
- Add t.Cleanup() for test isolation in TestVolumeCreationCounterIncrement
* fix: count each volume create attempt and move metric to master subsystem
- topology runs on the master, so move volume_creation_total from the
volumeServer subsystem to master (SeaweedFS_master_volume_creation_total);
rename the var to MasterVolumeCreationCounter and group it with the other
master metrics.
- increment the counter per findAndGrow iteration instead of once per
GrowByCountAndType call: each logical volume is now counted, partial
successes before a failure are credited, and a targetCount==0 call no
longer records a spurious success.
- update the Grafana panel query and the unit tests; the registration test
now asserts via the shared Gather registry under the fully-qualified name.
* test: drop volume_creation metric tests
They only exercised the Prometheus client library (CounterVec increment and
registry collection), not any SeaweedFS behavior, so they carried maintenance
cost without verifying anything in this codebase.
---------
Co-authored-by: Ubuntu User <ubuntu@example.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* mini: resolve admin credentials from security.toml and env vars
weed mini started the admin UI without resolving admin.user/admin.password
(and the read-only pair) from security.toml [admin] or WEED_ADMIN_* env vars,
so the only way to protect the UI was the -admin.password flag. The standalone
weed admin command applies these fallbacks in runAdmin via applyViperFallback;
the mini path calls startAdminServer directly and skipped it, leaving
authRequired false and the UI unauthenticated.
* mini: load admin.toml maintenance settings
The mini admin path runs ApplyMaintenanceConfigFromToml (via startAdminServer)
against the global viper, but runMini never merged admin.toml, so file-based
maintenance task settings ([maintenance.vacuum], .balance, .erasure_coding)
were ignored under mini while the standalone weed admin honored them. Load it
alongside master/volume config.
* mini: support -admin.urlPrefix for the admin UI
Expose the reverse-proxy subdirectory prefix that the standalone weed admin
already supports, so the mini admin UI can run under e.g. /seaweedfs. The
prefix is normalized the same way and passed through to startAdminServer.
* feat: add Prometheus metrics for replication operations
Adds 5 metrics to instrument volume server replication (write/delete):
- Operations counter with success/failure labels
- Duration histogram for latency tracking
- Targets gauge for replica fanout
- Failures counter with error reason labels
- Under-replicated volumes gauge on master
* fix: record replication duration histogram only when replicaCount > 0
* fix: update replication targets gauge for all operations including zero
* fix: ensure symmetric replication success/failure counting and proper metrics updates
* fix: change VolumeServerReplicationTargets from Gauge to Histogram
- Replace .Set() with .Observe() in store_replicate.go (2 occurrences)
- Update test to use CollectAndCount for histogram assertion
- Rename TestReplicationTargetsGauge -> TestReplicationTargetsHistogram
- Update documentation to reflect Histogram type and PromQL examples
* Add comments to replication metrics and improve test coverage
* metrics: add replication panels to grafana dashboard
Master row gets an under-replicated volumes timeseries; Volume Servers
row gets replication operations, failures-by-reason, p99 duration, and
average fan-out panels for the new replication metrics.
* metrics: name the replication duration histogram replication_seconds
Match the volumeServer convention (request_seconds, vacuuming_seconds)
rather than the admin/lifecycle _duration_seconds spelling.
* metrics: guard replication fan-out panel against divide-by-zero
clamp_min the _count rate so the avg-targets ratio reads 0 instead of
NaN when there are no replication events in the window.
---------
Co-authored-by: Ubuntu User <ubuntu@example.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* mount: attach Content-MD5 to chunk uploads
Mount writes never set UploadOption.Md5, so FileChunk.ETag stays empty
and filer.ETag() degenerates to md5("")-N: same-size files compare
equal regardless of content, defeating metadata-level verification
like filer.sync.verify.
Compute each chunk's MD5 and send it as Content-MD5. The volume server
verifies it on ingest (rejecting in-flight corruption) and echoes it
back, persisting FileChunk.ETag like filer/S3 writes already do.
The dirty-page flush paths already pass a *util.BytesReader whose
backing slice is the whole chunk, so the digest is taken in place with
no extra read, copy, or allocation (UploadWithRetry unwraps it the same
way downstream). Only the rarer plain-reader callers (e.g. manifest
chunks) fall back to io.ReadAll. Skipped under -cipher, where only the
ciphertext reaches the server.
The digest encoding (std-base64 of the raw md5) is the contract the
volume server verifies against, so it is factored into contentMD5Base64
and covered by a unit test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* mount: compute chunk Content-MD5 in the uploader, not the caller
Move the WantMd5 hashing into UploadWithRetry, where the chunk is already
buffered for the retry path, so saveDataAsChunk stops type-switching the
reader and re-reading plain readers. One materialization point, and the
cipher exclusion lives next to the hash instead of at every call site.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* volume: detect phantom volumes held open as deleted FDs
Add disk-file validation in heartbeat collection to prevent reporting
phantom volumes that exist in memory but are deleted from disk. This
unblocks re-replication when files are unlinked while the volume server
holds them open via file descriptors.
Cache disk checks per-volume with 30-second TTL to avoid syscall overhead.
Implement in both Go and Rust volume servers.
* volume: make last_disk_check_ns field public for heartbeat access
* volume: only check for phantom volumes when size > 0
Skip phantom volume detection for zero-size volumes (e.g., test volumes).
Phantom volumes only occur when disk files are deleted while the process
holds them open via FDs - which requires the volume to have had actual data.
Test volumes with zero size should not trigger disk file existence checks.
* volume: only check for phantom volumes when size > 0
Skip phantom volume detection for zero-size volumes (e.g., test volumes).
Phantom volumes only occur when disk files are deleted while the process
holds them open via FDs - which requires the volume to have had actual data.
Test volumes with zero size should not trigger disk file existence checks.
* volume: only check for phantom volumes if file_count > 0
Use file_count as the indicator for whether a volume held actual data,
rather than volume size. Phantom volumes only occur when a volume that
had files is deleted while the process holds open file descriptors.
Test volumes with no file count won't trigger the phantom detection check.
* volume: stat the .dat with its extension when detecting phantom volumes
DataFileName()/IndexFileName() return the extensionless base path, so os.Stat
saw every volume's files as missing and dropped it from the heartbeat, leaving
the master with no locations and breaking deletes/lookups. Stat FileName(".dat")
instead, skip remote-tiered volumes whose .dat lives in cloud storage, and
re-check a missing file every heartbeat rather than caching the negative.
* filer: skip COLLATE "C" list fallback on CockroachDB
CockroachDB string comparison is already byte-ordered, so wrapping the
list queries in COLLATE "C" can never change result order. It reports
datcollate=en_US.utf8 regardless of how the database was created, so the
collation check always misfires and forces the fallback. On 22.1 and
older COLLATE "C" is rejected as an invalid locale, turning every filer
list query into a hard failure. Detect the backend via version() and
keep the default ordering.
* filer: test CockroachDB collation detection
Cover the CockroachDB skip path and the locale-aware Postgres force path
with go-sqlmock.
* fix(mount): run entry invalidations off the meta-cache apply loop
The apply loop ran invalidateFunc inline, which acquires the open file
handle's lock in fhLockTable. Meanwhile flushMetadataToFiler holds that
same fh lock and then waits on the apply loop (applyLocalMetadataEvent).
When both target the same open file concurrently, the loop blocks on the
fh lock while the lock holder blocks on the loop: an ABBA deadlock that
backs up every later readdir/flush and hangs the mount.
Fix: dispatch entry invalidations to a dedicated FIFO worker goroutine so
the apply loop never blocks on locks held by goroutines waiting on it.
Adds a regression test reproducing the interleaving.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* perf(mount): update invalidate counter once per batch
Run the batch's invalidateFunc calls without re-taking invalidateMu per
item, then bump invalidateProcessed and broadcast once after the loop.
WaitForEntryInvalidations only needs the count to reach its target and a
batch always completes together, so the per-item lock + broadcast was
wasted work.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* mount: extract the invalidate worker into util.AsyncBatchWorker
The apply loop's off-thread entry-invalidation queue was a one-off mutex +
cond + slice + counters living inside MetaCache. Pull it out as a generic
unbounded FIFO worker so the deadlock-avoidance contract (never block the
producer, drain on shutdown, wait-for-quiesce) lives in one place.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
GetObjectStoreUserDetails only returned identity.PolicyNames (attached
managed policies) and omitted per-user inline policies. Inline policies are
stored separately from the identity record and are authoritative at S3
enforcement time (they take precedence over the legacy Actions list), so an
operator could not see what actually governed a user's access via the admin
API/UI.
Include inline policy names (via credentialManager.ListUserInlinePolicies) in
the returned PolicyNames. Adds a unit test using the memory credential store.
* mount: cache supplementary group IDs to improve non-root access performance
* mount: clear supplementary group cache between tests and add cache verification test
* mount: add docstrings and benchmarks for supplementary group cache
* mount: add performance test demonstrating cache effectiveness
* mount: add TTL-based cache expiry for supplementary group IDs (5-minute refresh)
* s3: add streaming reader interface for remote storage
Add RemoteStorageStreamReader optional interface to support efficient
streaming of large remote objects without buffering entire file in memory.
This enables future stream-through caching where data can be served to
clients while simultaneously writing to volume servers.
Implement ReadFileAsStream() for S3, GCS, and Azure backends using their
native streaming APIs. This provides the foundation for improving TTFB
on large remote file access by serving data directly from remote storage
while background cache operation populates local chunks.
The streaming interface allows remote storage backends to return io.ReadCloser,
enabling efficient memory usage for multi-GB objects compared to the
current ReadFile() approach which buffers entire ranges in memory.
* s3: adaptive timeout for remote object caching to improve TTFB
Use size-aware cache polling timeout to balance cache-hit rate against
time-to-first-byte:
- Small files (<50MB): 10s timeout - more likely to complete caching
before timeout, improving subsequent request performance
- Medium files (50-500MB): 5s timeout - default balance
- Large files (>500MB): 2s timeout - fail-fast to improve initial TTFB
for very large downloads
This reduces waiting time for large remote files while maintaining
high cache-hit rate for smaller files that cache quickly.
* s3: address code review feedback for stream-through cache
- Move startBackgroundRemoteCache call after policy recheck to avoid
cache side effects for denied requests (authorization first)
- Make startBackgroundRemoteCache version-aware by accepting versionId
parameter and using buildVersionedRemoteObjectPath
- Add timeout (5 minutes) to background cache context to prevent
goroutine pile-up if RPC stalls under load
- Update cacheRemoteObjectForStreamingWithShortTimeout to return both
entry and error, allowing callers to distinguish transient errors
(timeout/cancellation) from permanent errors (not found, denied)
- Update streamFromVolumeServers to handle permanent cache errors with
appropriate HTTP status codes (404 for not found, 503 for transient)
* helm: reject emptyDir for volume idx
An ephemeral index on a separate volume is wiped on every pod restart
while the .dat/.vif persist on the data PVCs. The volume server then
finds data with no matching .idx and exits via glog.Fatalf, putting the
pod into CrashLoopBackOff with no automatic recovery.
emptyDir is never the right choice for idx: if the data is persistent it
is a durability mismatch, and if the data is also ephemeral the default
(idx co-located with the data) already covers it. Fail the render with a
clear message pointing at the default or a persistent volume instead, and
drop emptyDir from the documented idx options.
* helm: rebuild a missing volume idx on restart
With emptyDir rejected, a separate idx volume is always persistent
(hostPath/PVC/existingClaim) -- but it can still lose its .idx out of
band (e.g. a node-local PVC reprovisioned on reschedule, or a pre-9944
compaction crash that left a .dat without its matching .idx). The
seaweedfs-vol-move-idx init container already moves idx files next to the
data into the index dir; have it first regenerate, via weed fix, any .idx
absent from both the data dir and the index dir, then move it into place.
The rebuild only runs when an idx is genuinely missing, so a healthy
index adds no startup cost.
* volume: validate remote S3 endpoints in FetchAndWriteNeedle (Rust)
Port the Go volume server's SSRF guard to the Rust volume server. The
gRPC FetchAndWriteNeedle reads from a caller-supplied S3 endpoint, so an
unguarded server can be pointed at loopback / link-local / RFC1918 /
CGNAT / cloud-metadata hosts to read internal services. Resolve and
reject those endpoints unless -volume.allowUntrustedRemoteEndpoints is
set (default off), mirroring weed/server/volume_grpc_remote.go.
Connect-time re-validation against DNS rebinding (Go's guardedDialer) is
not yet ported: the aws-sdk-s3 client builds its own connector, so the
up-front resolve-and-check leaves a narrow TOCTOU window. Left as a
follow-up.
* volume: harden remote endpoint guard (IPv4-mapped IPv6, all S3 types)
Address SSRF review feedback:
- Normalize IPv4-mapped IPv6 (::ffff:a.b.c.d) to IPv4 before the deny
checks, so ::ffff:127.0.0.1 / ::ffff:169.254.169.254 no longer slip
past the IPv4 rules.
- Validate the endpoint for every S3-compatible backend, not just type
"s3"; wasabi/backblaze/aliyun/... all dial a caller-supplied endpoint
through the same client. Skip validation when the endpoint is empty
(the provider default, e.g. real AWS S3, which cannot reach internal
hosts).
* volume: set allow_untrusted_remote_endpoints in integration-test state
The tests/http_integration.rs VolumeServerState literal was missed, which
broke cargo test compilation (it builds the integration tests, unlike the
cargo test --lib used locally).
EC bitrot protection is now a fixed default: always on at 16 MiB block
granularity. These volume-server flags exposed needless configurability;
the package defaults in erasure_coding (BitrotProtectionEnabled,
BitrotBlockSize) are retained and still drive sidecar generation. Drop the
now-unused erasure_coding import.
TopicExists can return a transient false-negative for a topic that is in
fact present (a broker/filer blip under load, or a just-created topic whose
existence cache is still stale). The metadata and produce handlers then tried
to auto-create, hit "topic already exists", treated that as a failure, and
dropped the topic from the Metadata response - so the client saw a spurious
UNKNOWN_TOPIC_OR_PARTITION right after the topic was created.
Return a sentinel ErrTopicAlreadyExists from the create paths and add
ensureTopicExists, which treats it as confirmation the topic exists. It also
folds in the repeated TopicExists -> invalidate -> recheck -> create logic
shared by every metadata handler (v0-v8) and both produce paths. v2+ produce
now auto-creates too, matching the auto.create.topics.enable=true behavior the
rest of the gateway already simulates.
Add path filters to workflows that fired on every PR/push regardless
of the diff: CodeQL, go build, the e2e/EC/vacuum/TLS/plugin-worker
integration suites, the Kafka and Postgres gateways, the S3 suites
(Ceph s3tests, s3-go, s3-tables, proxy-signature, https, example,
filer-group), TUS, and the dev binary/container builds. Each scopes
to its subsystem under weed/, its test dir, go.mod/go.sum, and the
workflow file, so docs-, helm-, terraform-, rust- or java-only
changes no longer trigger a full compile-and-test fleet.
* mount: apply cached remote entry without blocking the read
Reading an uncached remote-mounted file hung forever. The read holds the
file-handle shared lock across the on-demand download, then synchronously
waits on the metadata apply loop to apply the cached entry. The filer's
update event for that same object reaches the apply loop first and runs
invalidateFunc, which wants the file-handle exclusive lock — held in shared
mode by the still-running read. The loop blocks on the read; the read blocks
on the loop.
Enqueue the apply without waiting so the read never blocks on the apply loop
while holding the lock. The handle is already updated via SetEntry, and the
filer subscription delivers the same event regardless.
* mount: regression test for uncached remote read deadlock
Drives the real Read path through downloadRemoteEntry with a stub filer that
broadcasts the matching invalidate event before returning, reproducing the
lock-ordering deadlock. Fails (times out) without the fix.
* s3api: add optional request interceptor to circuit breaker
Add an optional Interceptor func(next http.HandlerFunc, action string) http.HandlerFunc
field on CircuitBreaker, applied at the very top of Limit() -- before upload
concurrency limiting and before the 'if !cb.Enabled' early return -- so it runs
for every route regardless of breaker state.
It is nil by default (no behavior change). An interceptor may reject a request
(write its own response and skip next) or wrap next to observe/shape it. This
provides a single per-request extension point at the existing per-route
chokepoint (cb.Limit already wraps nearly every S3 route), useful for tracing,
auditing, or request rate limiting without touching the route table.
* s3api: evaluate circuit breaker interceptor per request
Move the Interceptor nil-check into the returned handler so it is consulted
per request instead of captured at registration time. This:
- lets the interceptor be installed after registerRouter runs (handlers are
built during construction, before request-time dependencies exist), and
- avoids dereferencing a nil CircuitBreaker at registration time, which
panicked tests that register routes on a server with a nil cb
(e.g. TestRouting_STSWithQueryParams).
Adds a regression test for installing the interceptor after Limit() returns.
* fix(shell): return fs.verify topology errors
Problem: fs.verify can silently ignore master topology lookup failures before verifying files.
Root cause: commandFsVerify.Do returned parseErr after collectVolumeIds failed, but parseErr is nil once path parsing succeeds.
Fix: Return the actual collectVolumeIds error so VolumeList and master client failures stop the command.
Co-authored-by: Codex <noreply@openai.com>
* remove the tests
---------
Co-authored-by: Codex <noreply@openai.com>
isLocalOnlyEntry resolved the pinned-child check through inodeToPath. A kernel
Forget drops the path→inode mapping once the lookup count reaches zero, but an
async writeback flush — and the file handle, still in fhMap during the drain —
is keyed by inode and outlives that mapping. Between Release dispatching the
async flush and the flush reaching the filer, a Forget could unpin an in-flight
create, so a concurrent directory rebuild would wipe it and the file would
ENOENT until the flush lands and the cache refreshes.
Key the check off the inode the store entry already carries (createFile stamps
it into the placeholder), so the pin no longer depends on a mapping Forget can
remove.
* s3: bound streaming remote-cache wait so large cold GetObject returns 503, not a hang
A remote-only object whose download outlasts the 30s dedup attempt fell through
to cacheRemoteObjectForStreaming, which waited on the raw request context with no
bound. For multi-GB blobs the client read timeout fires first, so the request was
canceled and surfaced as InternalError rather than the 503 + Retry-After the
handler already emits. The filer keeps caching on a detached context, so the
retry would have streamed from the cached chunks.
Bound the wait under common client read timeouts: the 503 fires while the client
is still connected, and a retry picks up the finished cache.
* s3: make the streaming remote-cache timeout atomic
The bound is a package-level var so tests can shorten it. Hold it as an
atomic int64 so a test that stores a short value can never race the request
path that loads it.
* s3: assert the streaming-cache timeout test waits on the bound
The upper-bound-only check would also pass if the RPC returned immediately on a
setup error. Add a lower bound so the test fails unless the bounded wait fired.
wfs.Create defers the filer create to Flush and inserts a local-only
placeholder into the metaCache (dirtyMetadata=true), so a just-created file
exists locally before the filer has it. When the parent directory falls out of
cache (hot-dir read-through, idle evict) and is rebuilt, EnsureVisited wipes the
store and refills from a filer listing that does not yet include the un-flushed
create, then markCachedFn publishes the directory authoritatively cached without
it. lookupEntry then returns an authoritative ENOENT and ReadDir returns nothing
— the file disappears from the mount although the client created it. Under
concurrent read+write churn on one directory this is the ConcurrentReadWrite
flake.
Preserve children the mount flags as local-only (open dirty handle or pending
async flush — the signal lookupEntry already trusts) across the rebuild's wipe
instead of blind-deleting them. Unpinned stale children are still dropped so a
rebuild cannot resurrect a deleted entry.
The s3 and rclone tiered-storage backends were registered via blank imports
in weed/storage (volume_tier.go and volume_info/volume_info.go). That forced
every library consumer of weed/storage -- weed/shell, and through it external
tools -- to link aws-sdk-go and, under the rclone build tag, the full rclone
backend set and its cloud-storage SDKs, even though those consumers never tier
volumes.
Move the registrations into a new weed/storage/backend/all aggregator and
blank-import it once from weed/command, the binary's composition root. The weed
binary still registers both backends; weed/storage and its library consumers no
longer pull the backend SDKs into their dependency graph.
* SECURITY.md: require working repro and trust-boundary impact
Funnel reports through GitHub private vulnerability reporting, require a
reproduction against a supported version, and spell out the trust model so
internal-cluster exposure and example-config issues are out of scope.
Unvalidated scanner/AI output is handled as hardening, not as an advisory.
CVEs minted by third parties outside this process may be disputed.
* SECURITY.md: drop email channel, GitHub private reporting only
* mount: tolerance-window write pattern detection for concurrent writeback
WriterPattern decides whether each dirty chunk is buffered in RAM
(NewMemChunk) or spilled to an on-disk swap file (NewSwapFileChunk), so a
sequential stream misclassified as random is pushed through disk-backed
swap.
The old detector compared each write's start to the previous write's
exact stop offset (atomic.Swap of lastWriteStopOffset, then
lastOffset == offset). That is brittle under concurrent FUSE writeback:
MonitorWriteAt runs before the per-handle write lock, so parallel Write
upcalls interleave the swap, and writeback flushes dirty pages slightly
out of offset order. A genuinely sequential stream then repeatedly reads
as random and spills to swap.
Replace the exact match with the same frontier + tolerance approach used
on the read side: track a never-regressing max frontier (offset+size) via
a CAS loop and treat a write whose start is within SeqTolerance (8 MiB) of
that frontier as sequential. The existing +/-ModeChangeLimit hysteresis is
kept, so a single outlier write can't flip the mode.
Adds page_writer_pattern_test.go covering sequentiality, hysteresis,
reorder tolerance, the inclusive tolerance boundary, frontier
non-regression (the CAS invariant), and recovery back to sequential mode.
* mount: read write frontier inside the CAS loop
Capture the frontier from the CAS loop's own load rather than a separate
up-front snapshot, so the sequentiality diff is judged against the freshest
pre-image even if a concurrent writeback upcall advances the frontier while
we loop. Also drops the now-redundant load. Behavior is unchanged for the
single-threaded tests; addresses review feedback on #9984.
* filer: tolerance-window read pattern detection for concurrent readahead
ReaderPattern detected sequential access via strict contiguity (lastReadStopOffset
== offset). Under concurrent/reordered ReadAt (parallel readahead, which ReadAt
already supports via RLock + atomics), that exact match frequently misses even for
a genuinely sequential stream, drifting toward random mode and disabling prefetch.
Track a read frontier (max offset+size, advanced with a lock-free CAS loop) and
treat a read as sequential when its start is within SeqTolerance (8 MiB) of the
frontier. The window absorbs reordered/concurrent readahead while still rejecting
far random jumps; the existing ModeChangeLimit hysteresis is unchanged. API
(NewReaderPattern/MonitorReadAt/IsRandomMode) is unchanged, so reader_at is
unaffected. Adds reader_pattern_test.go.
* filer: read the read frontier inside the CAS loop
Capture the frontier from the CAS loop's own load rather than a separate
up-front snapshot, so the sequentiality diff is judged against the freshest
pre-image even if a concurrent readahead advances the frontier while we loop.
Also drops the now-redundant load. Mirrors the same fix on the write-side
ReaderPattern twin (review feedback on the companion PR).
* filer: add frontier/boundary/recovery guard tests for ReaderPattern
Mirror the regression guards added to the write-side WriterPattern: assert
the max-frontier never regresses on a backward read (the CAS invariant), pin
both sides of the inclusive SeqTolerance boundary, and verify sustained near
reads recover sequential mode out of the negative floor. Each guards an
invariant the original four tests left uncovered.
* test: add FUSE database load/durability/perf benchmark
Runs MySQL (InnoDB) and SQLite with ~1GB datadirs on a SeaweedFS FUSE
mount. Two parts:
- durability: normal shutdown, kill -9, and crash-during-write all keep
every fsync-committed row (verified by integrity check + row count +
contiguous prefix + per-row CRC).
- performance: FUSE vs the same local disk. fsync/commit latency is the
dominant cost (~0.13ms -> ~1.18ms), so small transactions run ~9-12x
slower while bulk loads and warm reads stay close.
Harness is path-independent (runtime under $SEAWEED_BENCH_WORK) and only
touches its own processes on non-default ports.
* test/benchmark/fuse_db: portable to Linux + crash-safe progress
- export MYSQL_BIN; mysql_bench.py falls back to PATH when unset
- unmount via fusermount/fusermount3 (non-root Linux), then umount/diskutil
- atomic progress write (tmp+fsync+rename); treat empty progress file as 0
- reuse a single PRNG in the perf probe so RNG init doesn't skew timings
* test/benchmark/fuse_db: validate inputs, add subprocess timeout
- mysql_bench.py: 1800s timeout on mysql CLI calls; reject db names that
aren't plain identifiers (interpolated into SQL)
- sqlite_gen.py / sqlite_verify.py: allowlist journal/synchronous modes and
the verify mode so a typo can't silently weaken durability or relax checks
- run_mysql.sh: durable atomic progress write (tmp+fsync+rename), matching
sqlite_gen.py; quote $LB in both crash-test verify calls
security.toml: document WEED_ env override for the jwt signing keys
These keys are HMAC secrets; spell out the env-var mapping so they can be
injected from a secret store instead of living in the config file.
* fix(command): preserve fuse option after writers
Problem: FUSE mount option parsing skipped the option immediately following concurrentWriters, so values such as concurrentReaders could be silently ignored.
Root cause: runFuse incremented the options loop index inside the concurrentWriters case in addition to the loop increment.
Co-authored-by: Codex <noreply@openai.com>
* test: save and restore mountOptions pointers, not their values
The fields are reassigned to fresh heap variables during runFuse, so
dereferencing to back up/restore mutated throwaways instead of the
flag-bound originals and could nil-panic on unset fields.
---------
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* admin: add connected mount clients page and dashboard section
The filer is the authority on who is subscribed to its metadata stream
(FUSE/VFS mounts, S3, peer filers, ...), but its in-memory listener
registry only tracked clientId->epoch and was not exposed.
- Enrich the filer subscriber registry with name/type/address/path/
connected-time, populated in addClient and cleared in deleteClient so
it reflects currently-connected clients only.
- Add a ListMetadataSubscribers filer gRPC (optional client-type filter).
- Admin server fans out to every filer, filters to mount types
("mount" Go weed mount, "sw-vfs" Rust VFS), and renders a new
Cluster > Mount Clients page plus a Mount Clients dashboard section.
Read-only; no behavior change to the subscribe hot path.
* admin: address review — parallelize filer fan-out, guard nil map, robust CSV
- GetMountClients now queries filers concurrently, each under a 5s
timeout, so a slow/unreachable filer can't stall the admin dashboard.
- Defensively initialize fs.subscribers before first write.
- Mount Clients CSV export uses a Blob with quote-escaping instead of a
data: URI, so special characters in paths export correctly.
* filer: keep metadata-subscription send gauge fresh on idle heartbeat
last_send_timestamp_of_subscribe only advanced when a real matching
metadata event was streamed to a subscriber. On a quiet path an idle but
perfectly healthy subscriber therefore looked increasingly stale, and the
dashboard panel rendered a large, misleading 'lag'.
An idle heartbeat is a send too, so advance the gauge when one is emitted.
Subscribers that opt into idle heartbeats (filer.sync) now report true
freshness; the rest still show time since the last real event.
Rename the dashboard panel 'Metadata Subscription Lag' ->
'Time Since Last Subscription Send' and clarify its description to match.
* filer: guard nil option when advancing heartbeat gauge
maybeSendIdleHeartbeat is unit-tested with a bare &FilerServer{} (nil
option), so dereferencing fs.option.Host for the sourceFiler label
panicked. Guard it: production always has option set; the test now gets
an empty sourceFiler label instead of a nil-pointer panic.
* s3: register account-less identities' synthesized account in the lookup
#9962 gave each account-less identity a distinct account id derived from
its name (instead of collapsing into admin), but never registered that
account in the id->account map. GetAccountNameById then returned empty
for such ids, so ACL grantee validation rejected canonical grants to the
caller's own account with InvalidRequest, and bucket/object owner display
was dropped as 'owner is invalid'.
This broke a canned PutObjectAcl by an account-less identity (e.g.
TestVersionedObjectAcl with the default 'some_admin_user' identity):
ValidateAndTransferGrants -> GetAccountNameById -> 'account id is not
exists' -> 400 InvalidRequest.
Register the synthesized account at config load so its id resolves to a
display name. Add a regression test.
* s3: reuse explicitly-configured account for account-less identity
Address review: if an account with the same id as an account-less
identity's synthesized account is explicitly configured (custom display
name/email), reuse it instead of the synthesized one. Add a test.
Chris Lu
Bruce Zou
Chris Lu
Rushikesh Deshpande
Sergey Zinchenko
Jaehoon Kim
msementsov
Minsoo Kim
Ruslan Bel'kov
dependabot[bot]
7y-9