* fix(chunk_cache): close data/index files on initialization error
* chunk_cache: assign outer err on the .dat open path
The error-path defer keys off the function-level err, but the .dat
OpenFile used := and shadowed it, so that path relied on nothing being
open yet rather than the cleanup invariant. Assign the outer err so
every error return is uniform.
* chunk_cache: verify descriptor closure on POSIX, not just Windows
os.Remove succeeds on open files on Linux/macOS, so the removal check
only proved closure on Windows. Compare the open-fd count before and
after the failed load; gate the removal check to Windows.
---------
Co-authored-by: Contributor <contributor@example.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* 修复weedfs_xattr.go 中 XATTR_REPLACE 语义缺失
* mount: fix XATTR_CREATE/XATTR_REPLACE flag semantics in setxattr
XATTR_CREATE fell through into the XATTR_REPLACE branch: creating a new
attribute hit the empty-oldData guard and returned ENODATA instead of
creating it, while creating over an existing attribute silently succeeded
without the EEXIST that setxattr(2) requires. Drop the fallthrough chain
so CREATE returns EEXIST when the attribute already exists, REPLACE
returns ENODATA when it is missing, and otherwise the value is written.
Test existence via the map lookup so an attribute with an empty value is
still treated as present.
---------
Co-authored-by: 王郁文 <wangyuwen@cmict.chinamobile.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix: Resolve inconsistent usage of error variables
* mysql2: guard nil DB on open failure and wrap connect error
---------
Co-authored-by: muminglei <muminglei@cmict.chinamobile.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* iceberg: return 400 for invalid namespace/table names
The S3 Tables name charset (a-z, 0-9, _) is stricter than the Iceberg
REST spec, so clients sending hyphens or uppercase hit a validation
error. That error fell through to 500; it's client input, so map it to
400 BadRequestException across the namespace and table handlers.
* iceberg: tighten name-validation error matching
Match the validator's own phrasings (invalid/must/cannot) instead of a
bare "namespace name"/"table name" substring, so an unrelated fault that
happens to mention a name isn't misreported as a 400. Lowercase first to
stay robust to message capitalization.
* iceberg: support namespace property updates
Add POST /v1/namespaces/{namespace}/properties to the REST catalog. It
applies the request's removals and updates and returns the removed/updated/
missing summary the spec defines. A new UpdateNamespace op on the S3 Tables
manager rewrites the stored namespace properties; AWS S3 Tables namespaces
have no properties, so this is the SeaweedFS-side backing for the catalog.
* iceberg: dedup namespace property removals
A key repeated in removals was deleted on its first occurrence, then
reported as missing on the next — landing in both removed and missing.
Skip keys already processed.
* iceberg: map namespace-update backend errors to REST statuses
UpdateNamespaceProperties returned 500 for every manager failure, masking
the namespace being dropped between read and write, or a denied caller.
Inspect the typed S3TablesError and answer 404/403 accordingly, 500 only
for the rest. Also replaces the GetNamespace not-found string match.
* iceberg: test the namespace-properties conflict path
Cover the 422 returned when a key appears in both removals and updates.
The check runs before any backend call, so it needs no filer.
* fix(s3api): preserve requested AES256 copy encryption
Problem
CopyObject metadata processing ignored an explicit x-amz-server-side-encryption: AES256 request header. A destination copy could lose the requested SSE-S3 metadata even though KMS requests were handled.
Root cause
processMetadataBytes only wrote the destination SSE header when the requested algorithm was aws:kms. Any other explicit SSE algorithm fell through to the source-preservation branch.
Fix
Write the requested SSE algorithm whenever x-amz-server-side-encryption is present, and keep KMS-specific metadata handling limited to aws:kms.
Co-authored-by: Codex <noreply@openai.com>
* fix(s3api): reject unsupported copy encryption algorithms
A mistyped or unsupported x-amz-server-side-encryption value on a copy
request slipped past validation and got persisted as the destination's
algorithm header, advertising encryption that was never applied. Reject
anything other than AES256 or aws:kms up front.
* fix(s3api): write SSE key metadata for empty encrypted copies
A zero-byte source copied with an explicit SSE request took the
no-content branch and never ran the encryption path, leaving the object
with a bare algorithm header but no key. HEAD then advertised SSE while
the encryption-state machine saw the header as orphaned. Run the inline
encryption path when the destination requests encryption so the key
metadata is written too.
* s3api: use SSEAlgorithmKMS constant in copy metadata handling
* test(s3api): cover source SSE preservation on copy
* test(iam): allow the local client's real source IP in SourceIp tests
The aws:SourceIp allow policies hardcoded the loopback CIDRs, but a CI
runner reaching the server over localhost can be observed with one of the
host's RFC1918 addresses (the S3 endpoint is advertised on a 10.x
interface), so the positive-condition PutObject was denied and the allow
assertion flaked while the deny path passed trivially. Broaden the allow
list to loopback plus private ranges via a shared helper, and log the
denial on each failed attempt so any residual failure is diagnosable.
---------
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
In ec_task.go, 23 fmt.Errorf calls used %v verb to wrap errors,
breaking the error chain introduced in Go 1.13. This prevents
callers from using errors.Is() and errors.As() to inspect the
underlying error type.
Changed all fmt.Errorf calls from %v to %w to properly wrap
errors, preserving the error chain for upstream callers.
Note: glog.* logging calls and fmt.Sprintf calls intentionally
keep %v as they are not error wrapping contexts.
Co-authored-by: 吴奇臻 <wuqizhen@cmict.chinamobile.com>
* build(deps): bump github.com/apache/iceberg-go from 0.5.0 to 0.6.0
Bumps [github.com/apache/iceberg-go](https://github.com/apache/iceberg-go) from 0.5.0 to 0.6.0.
- [Release notes](https://github.com/apache/iceberg-go/releases)
- [Commits](https://github.com/apache/iceberg-go/compare/v0.5.0...v0.6.0)
---
updated-dependencies:
- dependency-name: github.com/apache/iceberg-go
dependency-version: 0.6.0
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* iceberg: adapt worker to iceberg-go 0.6.0 API
Fields() now yields iter.Seq2 (index, value); SortField.SourceID and
PartitionField.SourceID are methods backed by SourceIDs; RemoveSnapshots
takes a postCommit flag (false here, file cleanup runs through the filer).
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix(shell): correct volume.list -writable filter unit and comparison
* fix(shell): correct volume.list -writable filter unit and comparison
* chore(shell): fix typo in EC shard helper param names
* fix(shell): use exact match for volume.balance -racks/-nodes filter
The old strings.Contains-based filter quietly included any id that was a
substring of the user-supplied flag value (e.g. -racks=rack10 also matched
rack1). Replace it with an exact-match set parsed from the comma-separated
flag value, and add regression tests for both -racks and -nodes paths.
Also fix a small typo in the "remote storage" error returned by
maybeMoveOneVolume.
* fix(shell): use exact match for volume.balance -racks/-nodes filter
The old strings.Contains-based filter quietly included any id that was a
substring of the user-supplied flag value (e.g. -racks=rack10 also matched
rack1). Replace it with an exact-match set parsed from the comma-separated
flag value, and add regression tests for both -racks and -nodes paths.
Also fix a small typo in the "remote storage" error returned by
maybeMoveOneVolume.
* refactor(shell): drop nil sentinel in splitCSVSet, use len() in callers
* fix(shell): exclude failed EC shard copies from rebuild recoverability gate
prepareDataToRecover incremented the remote-shard counter before the copy
RPC, so in apply mode a failed VolumeEcShardsCopy was still counted toward
the DataShardsCount recoverability gate. The gate could then pass with
fewer real shards than required, deferring the failure to the deeper
generateMissingShards/reconstruct step and reporting an inflated shard
count in the "not enough shards" error.
Count the remote shard only after a successful copy (apply mode) or when
planning (dry-run), and rename wouldCopy to recoverableRemoteShards for
clarity. Add a regression test covering an apply-mode copy failure.
* fix(shell): clean up copied EC shards when the recoverability gate fails
A runtime copy failure can trip the gate after earlier copies already
succeeded, stranding those working shards on the rebuilder. Return the
copied shard ids on the error path and run the cleanup defer even when
recovery fails, so the temp shards get deleted.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix(filer): propagate proxyChunkId query params to volume server
When weed mount reads via filer proxy mode (-volumeServerAccess=filerProxy),
the mount adds query params like readDeleted=true to chunk read requests.
Two bugs prevented these from working:
1. filer_server_handlers.go extracted fileId from the raw RequestURI, which
includes query params, corrupting the fileId (e.g. '6,abc&readDeleted=true').
Fix: use r.URL.Query().Get("proxyChunkId") for clean extraction.
2. filer_server_handlers_proxy.go didn't forward query params to the volume
server. The urlStrings from LookupFileId already contain the fileId in the
path, so just append the original query string.
* filer: match chunk proxy by query param, not URI prefix order
Order-dependent prefix slicing missed proxyChunkId when it wasn't the
first query param. Gate on root path and read the parsed query value.
* filer: drop internal proxyChunkId from proxied volume query
Lookup URLs already carry the fileId in the path, so forwarding the raw
query duplicated proxyChunkId onto the volume server. Strip it and only
append the remaining caller params (e.g. readDeleted).
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
util/http: lazily init the global HTTP client
GetGlobalHttpClient returned a nil client until InitGlobalHttpClient ran,
which only happens in weed.go's main. Anything that starts a command
in-process bypasses that: the admin server's metrics goroutine seeds a
dashboard sample on startup, reaching fetchPublicUrlMap -> GetGlobalHttpClient().Do,
and nil-derefs the receiver in GetHttpScheme.
Init the client on first Get via sync.Once so it is never nil regardless of
the startup path. InitGlobalHttpClient keeps its eager-init role through the
same Once.
prometheus/common v0.67.5, golang.org/x/sys v0.45.0, klauspost/compress
v1.18.6 to match the root module; readonly build was rejecting the stale
versions.
chore(deps): bump cassandra-gocql-driver to v2.1.2
Picks up upstream fixes for the Cassandra filer store: HostFilter panic
when a keyspace isn't replicated to every DC, broken system.peers
fallback, and pool connection errors under a small Session.Timeout.
* fix(s3api): sort repeated SigV4 query values
Problem
SigV4 canonical query strings must sort repeated parameter values. SeaweedFS preserved the incoming value order, so a correctly signed request with repeated query parameters could fail verification when the request order differed from canonical order.
Root cause
getCanonicalQueryString delegated directly to url.Values.Encode(), which sorts parameter names but preserves each key's value slice order.
Fix
Sort every query value slice before encoding the canonical query string, while still excluding X-Amz-Signature for presigned URLs.
Reproduction
go test ./weed/s3api -run TestGetCanonicalQueryStringSortsRepeatedValues -count=1 failed before the fix with partNumber=2 before partNumber=10.
Co-authored-by: Codex <noreply@openai.com>
* test(s3api): expand canonical query coverage
Co-authored-by: Codex <noreply@openai.com>
---------
Co-authored-by: Codex <noreply@openai.com>
* fix(shell): correct volume.list -writable filter unit and comparison
* fix(shell): correct volume.list -writable filter unit and comparison
* chore(shell): fix typo in EC shard helper param names
* fix(shell): use exact match for volume.balance -racks/-nodes filter
The old strings.Contains-based filter quietly included any id that was a
substring of the user-supplied flag value (e.g. -racks=rack10 also matched
rack1). Replace it with an exact-match set parsed from the comma-separated
flag value, and add regression tests for both -racks and -nodes paths.
Also fix a small typo in the "remote storage" error returned by
maybeMoveOneVolume.
* fix(shell): use exact match for volume.balance -racks/-nodes filter
The old strings.Contains-based filter quietly included any id that was a
substring of the user-supplied flag value (e.g. -racks=rack10 also matched
rack1). Replace it with an exact-match set parsed from the comma-separated
flag value, and add regression tests for both -racks and -nodes paths.
Also fix a small typo in the "remote storage" error returned by
maybeMoveOneVolume.
* refactor(shell): drop nil sentinel in splitCSVSet, use len() in callers
* feat(shell): support batched EC encode and multi-volume selection
Add -volumeIds (comma-separated) and -batchSize flags to ec.encode.
When -batchSize > 0, volumes are processed in independent batches, each
committed separately: encode -> rebalance -> verify -> delete originals.
This bounds the working set and lets source volumes be reclaimed without
waiting for the entire set to finish, at the cost of per-batch rebalancing.
Because each batch deletes its originals, a failure in a later batch is
unrecoverable for already-completed batches.
To let the single-volume, multi-volume, and collection paths share one
per-batch routine, the re-balance scope is now always derived from the
volumes actually selected for encoding (collectCollectionsForVolumeIds),
rather than every collection matching the -collection regex. Practical
effect: with -collection, a collection that matches the pattern but
contributes no encodable volumes is no longer re-balanced as a side effect.
The -volumeId path is unchanged; -batchSize=0 (default) preserves the
original single-pass flow.
The per-batch routine reuses the existing assertEncodableRegularVolumes
guard, doEcEncode skipped-node handling, and verifyEcShardsBeforeDelete
retry loop. The capacity pre-flight check takes the already-fetched
topology instead of issuing another VolumeList to the master per batch.
Also clarify the -collection flag description to note it accepts a regex
pattern, matching the existing command help.
-volumeId and -volumeIds are mutually exclusive; ids in -volumeIds are
validated and de-duplicated.
During leader election, MaybeLeader() returns empty string immediately (non-blocking),
causing master to return NotLeaderError without sending HeartbeatResponse.Leader.
Volume servers depend on HeartbeatResponse.Leader to discover the new leader address,
so they keep retrying the old leader and cannot switch.
Switching to Leader() restores the 20-second exponential backoff retry behavior,
ensuring volume servers receive the new leader address as soon as election completes.
* feat: add Prometheus metric for volume creation operations
Add VolumeServerVolumeCreationCounter metric to track volume creation
attempts by result (success/failure).
Changes:
- Add VolumeServerVolumeCreationCounter in weed/stats/metrics.go
- Instrument GrowByCountAndType in weed/topology/volume_growth.go
- Add unit tests in weed/stats/metrics_volume_creation_test.go
- Add Grafana dashboard panel for volume creation rate
This metric enables monitoring volume creation success/failure rates
in SeaweedFS clusters.
* fix: address CodeRabbit review - move failure counter before early return
- Move failure counter into loop else block before return statement
- Move success counter to after loop completion
- Strengthen test assertion from count < 1 to count != 2
- Add t.Cleanup() for test isolation in TestVolumeCreationCounterIncrement
* fix: count each volume create attempt and move metric to master subsystem
- topology runs on the master, so move volume_creation_total from the
volumeServer subsystem to master (SeaweedFS_master_volume_creation_total);
rename the var to MasterVolumeCreationCounter and group it with the other
master metrics.
- increment the counter per findAndGrow iteration instead of once per
GrowByCountAndType call: each logical volume is now counted, partial
successes before a failure are credited, and a targetCount==0 call no
longer records a spurious success.
- update the Grafana panel query and the unit tests; the registration test
now asserts via the shared Gather registry under the fully-qualified name.
* test: drop volume_creation metric tests
They only exercised the Prometheus client library (CounterVec increment and
registry collection), not any SeaweedFS behavior, so they carried maintenance
cost without verifying anything in this codebase.
---------
Co-authored-by: Ubuntu User <ubuntu@example.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* mini: resolve admin credentials from security.toml and env vars
weed mini started the admin UI without resolving admin.user/admin.password
(and the read-only pair) from security.toml [admin] or WEED_ADMIN_* env vars,
so the only way to protect the UI was the -admin.password flag. The standalone
weed admin command applies these fallbacks in runAdmin via applyViperFallback;
the mini path calls startAdminServer directly and skipped it, leaving
authRequired false and the UI unauthenticated.
* mini: load admin.toml maintenance settings
The mini admin path runs ApplyMaintenanceConfigFromToml (via startAdminServer)
against the global viper, but runMini never merged admin.toml, so file-based
maintenance task settings ([maintenance.vacuum], .balance, .erasure_coding)
were ignored under mini while the standalone weed admin honored them. Load it
alongside master/volume config.
* mini: support -admin.urlPrefix for the admin UI
Expose the reverse-proxy subdirectory prefix that the standalone weed admin
already supports, so the mini admin UI can run under e.g. /seaweedfs. The
prefix is normalized the same way and passed through to startAdminServer.
* feat: add Prometheus metrics for replication operations
Adds 5 metrics to instrument volume server replication (write/delete):
- Operations counter with success/failure labels
- Duration histogram for latency tracking
- Targets gauge for replica fanout
- Failures counter with error reason labels
- Under-replicated volumes gauge on master
* fix: record replication duration histogram only when replicaCount > 0
* fix: update replication targets gauge for all operations including zero
* fix: ensure symmetric replication success/failure counting and proper metrics updates
* fix: change VolumeServerReplicationTargets from Gauge to Histogram
- Replace .Set() with .Observe() in store_replicate.go (2 occurrences)
- Update test to use CollectAndCount for histogram assertion
- Rename TestReplicationTargetsGauge -> TestReplicationTargetsHistogram
- Update documentation to reflect Histogram type and PromQL examples
* Add comments to replication metrics and improve test coverage
* metrics: add replication panels to grafana dashboard
Master row gets an under-replicated volumes timeseries; Volume Servers
row gets replication operations, failures-by-reason, p99 duration, and
average fan-out panels for the new replication metrics.
* metrics: name the replication duration histogram replication_seconds
Match the volumeServer convention (request_seconds, vacuuming_seconds)
rather than the admin/lifecycle _duration_seconds spelling.
* metrics: guard replication fan-out panel against divide-by-zero
clamp_min the _count rate so the avg-targets ratio reads 0 instead of
NaN when there are no replication events in the window.
---------
Co-authored-by: Ubuntu User <ubuntu@example.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* mount: attach Content-MD5 to chunk uploads
Mount writes never set UploadOption.Md5, so FileChunk.ETag stays empty
and filer.ETag() degenerates to md5("")-N: same-size files compare
equal regardless of content, defeating metadata-level verification
like filer.sync.verify.
Compute each chunk's MD5 and send it as Content-MD5. The volume server
verifies it on ingest (rejecting in-flight corruption) and echoes it
back, persisting FileChunk.ETag like filer/S3 writes already do.
The dirty-page flush paths already pass a *util.BytesReader whose
backing slice is the whole chunk, so the digest is taken in place with
no extra read, copy, or allocation (UploadWithRetry unwraps it the same
way downstream). Only the rarer plain-reader callers (e.g. manifest
chunks) fall back to io.ReadAll. Skipped under -cipher, where only the
ciphertext reaches the server.
The digest encoding (std-base64 of the raw md5) is the contract the
volume server verifies against, so it is factored into contentMD5Base64
and covered by a unit test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* mount: compute chunk Content-MD5 in the uploader, not the caller
Move the WantMd5 hashing into UploadWithRetry, where the chunk is already
buffered for the retry path, so saveDataAsChunk stops type-switching the
reader and re-reading plain readers. One materialization point, and the
cipher exclusion lives next to the hash instead of at every call site.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* volume: detect phantom volumes held open as deleted FDs
Add disk-file validation in heartbeat collection to prevent reporting
phantom volumes that exist in memory but are deleted from disk. This
unblocks re-replication when files are unlinked while the volume server
holds them open via file descriptors.
Cache disk checks per-volume with 30-second TTL to avoid syscall overhead.
Implement in both Go and Rust volume servers.
* volume: make last_disk_check_ns field public for heartbeat access
* volume: only check for phantom volumes when size > 0
Skip phantom volume detection for zero-size volumes (e.g., test volumes).
Phantom volumes only occur when disk files are deleted while the process
holds them open via FDs - which requires the volume to have had actual data.
Test volumes with zero size should not trigger disk file existence checks.
* volume: only check for phantom volumes when size > 0
Skip phantom volume detection for zero-size volumes (e.g., test volumes).
Phantom volumes only occur when disk files are deleted while the process
holds them open via FDs - which requires the volume to have had actual data.
Test volumes with zero size should not trigger disk file existence checks.
* volume: only check for phantom volumes if file_count > 0
Use file_count as the indicator for whether a volume held actual data,
rather than volume size. Phantom volumes only occur when a volume that
had files is deleted while the process holds open file descriptors.
Test volumes with no file count won't trigger the phantom detection check.
* volume: stat the .dat with its extension when detecting phantom volumes
DataFileName()/IndexFileName() return the extensionless base path, so os.Stat
saw every volume's files as missing and dropped it from the heartbeat, leaving
the master with no locations and breaking deletes/lookups. Stat FileName(".dat")
instead, skip remote-tiered volumes whose .dat lives in cloud storage, and
re-check a missing file every heartbeat rather than caching the negative.
* filer: skip COLLATE "C" list fallback on CockroachDB
CockroachDB string comparison is already byte-ordered, so wrapping the
list queries in COLLATE "C" can never change result order. It reports
datcollate=en_US.utf8 regardless of how the database was created, so the
collation check always misfires and forces the fallback. On 22.1 and
older COLLATE "C" is rejected as an invalid locale, turning every filer
list query into a hard failure. Detect the backend via version() and
keep the default ordering.
* filer: test CockroachDB collation detection
Cover the CockroachDB skip path and the locale-aware Postgres force path
with go-sqlmock.
* fix(mount): run entry invalidations off the meta-cache apply loop
The apply loop ran invalidateFunc inline, which acquires the open file
handle's lock in fhLockTable. Meanwhile flushMetadataToFiler holds that
same fh lock and then waits on the apply loop (applyLocalMetadataEvent).
When both target the same open file concurrently, the loop blocks on the
fh lock while the lock holder blocks on the loop: an ABBA deadlock that
backs up every later readdir/flush and hangs the mount.
Fix: dispatch entry invalidations to a dedicated FIFO worker goroutine so
the apply loop never blocks on locks held by goroutines waiting on it.
Adds a regression test reproducing the interleaving.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* perf(mount): update invalidate counter once per batch
Run the batch's invalidateFunc calls without re-taking invalidateMu per
item, then bump invalidateProcessed and broadcast once after the loop.
WaitForEntryInvalidations only needs the count to reach its target and a
batch always completes together, so the per-item lock + broadcast was
wasted work.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* mount: extract the invalidate worker into util.AsyncBatchWorker
The apply loop's off-thread entry-invalidation queue was a one-off mutex +
cond + slice + counters living inside MetaCache. Pull it out as a generic
unbounded FIFO worker so the deadlock-avoidance contract (never block the
producer, drain on shutdown, wait-for-quiesce) lives in one place.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
GetObjectStoreUserDetails only returned identity.PolicyNames (attached
managed policies) and omitted per-user inline policies. Inline policies are
stored separately from the identity record and are authoritative at S3
enforcement time (they take precedence over the legacy Actions list), so an
operator could not see what actually governed a user's access via the admin
API/UI.
Include inline policy names (via credentialManager.ListUserInlinePolicies) in
the returned PolicyNames. Adds a unit test using the memory credential store.
* mount: cache supplementary group IDs to improve non-root access performance
* mount: clear supplementary group cache between tests and add cache verification test
* mount: add docstrings and benchmarks for supplementary group cache
* mount: add performance test demonstrating cache effectiveness
* mount: add TTL-based cache expiry for supplementary group IDs (5-minute refresh)
* s3: add streaming reader interface for remote storage
Add RemoteStorageStreamReader optional interface to support efficient
streaming of large remote objects without buffering entire file in memory.
This enables future stream-through caching where data can be served to
clients while simultaneously writing to volume servers.
Implement ReadFileAsStream() for S3, GCS, and Azure backends using their
native streaming APIs. This provides the foundation for improving TTFB
on large remote file access by serving data directly from remote storage
while background cache operation populates local chunks.
The streaming interface allows remote storage backends to return io.ReadCloser,
enabling efficient memory usage for multi-GB objects compared to the
current ReadFile() approach which buffers entire ranges in memory.
* s3: adaptive timeout for remote object caching to improve TTFB
Use size-aware cache polling timeout to balance cache-hit rate against
time-to-first-byte:
- Small files (<50MB): 10s timeout - more likely to complete caching
before timeout, improving subsequent request performance
- Medium files (50-500MB): 5s timeout - default balance
- Large files (>500MB): 2s timeout - fail-fast to improve initial TTFB
for very large downloads
This reduces waiting time for large remote files while maintaining
high cache-hit rate for smaller files that cache quickly.
* s3: address code review feedback for stream-through cache
- Move startBackgroundRemoteCache call after policy recheck to avoid
cache side effects for denied requests (authorization first)
- Make startBackgroundRemoteCache version-aware by accepting versionId
parameter and using buildVersionedRemoteObjectPath
- Add timeout (5 minutes) to background cache context to prevent
goroutine pile-up if RPC stalls under load
- Update cacheRemoteObjectForStreamingWithShortTimeout to return both
entry and error, allowing callers to distinguish transient errors
(timeout/cancellation) from permanent errors (not found, denied)
- Update streamFromVolumeServers to handle permanent cache errors with
appropriate HTTP status codes (404 for not found, 503 for transient)
* helm: reject emptyDir for volume idx
An ephemeral index on a separate volume is wiped on every pod restart
while the .dat/.vif persist on the data PVCs. The volume server then
finds data with no matching .idx and exits via glog.Fatalf, putting the
pod into CrashLoopBackOff with no automatic recovery.
emptyDir is never the right choice for idx: if the data is persistent it
is a durability mismatch, and if the data is also ephemeral the default
(idx co-located with the data) already covers it. Fail the render with a
clear message pointing at the default or a persistent volume instead, and
drop emptyDir from the documented idx options.
* helm: rebuild a missing volume idx on restart
With emptyDir rejected, a separate idx volume is always persistent
(hostPath/PVC/existingClaim) -- but it can still lose its .idx out of
band (e.g. a node-local PVC reprovisioned on reschedule, or a pre-9944
compaction crash that left a .dat without its matching .idx). The
seaweedfs-vol-move-idx init container already moves idx files next to the
data into the index dir; have it first regenerate, via weed fix, any .idx
absent from both the data dir and the index dir, then move it into place.
The rebuild only runs when an idx is genuinely missing, so a healthy
index adds no startup cost.
* volume: validate remote S3 endpoints in FetchAndWriteNeedle (Rust)
Port the Go volume server's SSRF guard to the Rust volume server. The
gRPC FetchAndWriteNeedle reads from a caller-supplied S3 endpoint, so an
unguarded server can be pointed at loopback / link-local / RFC1918 /
CGNAT / cloud-metadata hosts to read internal services. Resolve and
reject those endpoints unless -volume.allowUntrustedRemoteEndpoints is
set (default off), mirroring weed/server/volume_grpc_remote.go.
Connect-time re-validation against DNS rebinding (Go's guardedDialer) is
not yet ported: the aws-sdk-s3 client builds its own connector, so the
up-front resolve-and-check leaves a narrow TOCTOU window. Left as a
follow-up.
* volume: harden remote endpoint guard (IPv4-mapped IPv6, all S3 types)
Address SSRF review feedback:
- Normalize IPv4-mapped IPv6 (::ffff:a.b.c.d) to IPv4 before the deny
checks, so ::ffff:127.0.0.1 / ::ffff:169.254.169.254 no longer slip
past the IPv4 rules.
- Validate the endpoint for every S3-compatible backend, not just type
"s3"; wasabi/backblaze/aliyun/... all dial a caller-supplied endpoint
through the same client. Skip validation when the endpoint is empty
(the provider default, e.g. real AWS S3, which cannot reach internal
hosts).
* volume: set allow_untrusted_remote_endpoints in integration-test state
The tests/http_integration.rs VolumeServerState literal was missed, which
broke cargo test compilation (it builds the integration tests, unlike the
cargo test --lib used locally).
EC bitrot protection is now a fixed default: always on at 16 MiB block
granularity. These volume-server flags exposed needless configurability;
the package defaults in erasure_coding (BitrotProtectionEnabled,
BitrotBlockSize) are retained and still drive sidecar generation. Drop the
now-unused erasure_coding import.
TopicExists can return a transient false-negative for a topic that is in
fact present (a broker/filer blip under load, or a just-created topic whose
existence cache is still stale). The metadata and produce handlers then tried
to auto-create, hit "topic already exists", treated that as a failure, and
dropped the topic from the Metadata response - so the client saw a spurious
UNKNOWN_TOPIC_OR_PARTITION right after the topic was created.
Return a sentinel ErrTopicAlreadyExists from the create paths and add
ensureTopicExists, which treats it as confirmation the topic exists. It also
folds in the repeated TopicExists -> invalidate -> recheck -> create logic
shared by every metadata handler (v0-v8) and both produce paths. v2+ produce
now auto-creates too, matching the auto.create.topics.enable=true behavior the
rest of the gateway already simulates.
AlexALei
shiftraodd
patrick
mumingl
Chris Lu
7y-9
aCuteBegCinner
DanielWu-star
dependabot[bot]
qzhello
MorezMartin
Chris Lu
Bruce Zou
Rushikesh Deshpande
Sergey Zinchenko
Jaehoon Kim
msementsov
Minsoo Kim
Ruslan Bel'kov