VolumeTierMoveDatFromRemote downloads the .dat, trims the .vif, and swaps
the data backend to the local file, but left hasRemoteFile set. The
volume.tier.download command masks this by unmounting and remounting
right after, which reloads the flag from the trimmed .vif — but in the
window before the remount the in-memory flag is wrong: doDeleteRequest
would skip appending the tombstone to the freshly local .dat, and the
phantom-.dat guard stays disabled.
Give SwapDataBackend a hasRemoteFile argument so the backend swap and the
flag move together under one lock, and route both tier directions through
it: the tier-down download passes false, LoadRemoteFile passes true. The
flag can no longer disagree with the live backend.
volume: keep tier-uploaded volume reporting to master
A live volume.tier.upload removes the local .dat and swaps the data
backend to remote, but v.hasRemoteFile was only ever set when a volume
is loaded from disk, so the running volume kept it false. The phantom
.dat guard then saw fileCount>0, !HasRemoteFile, and a missing .dat, and
stopped reporting the volume to the master. The volume vanished from the
topology even though the upload succeeded and the data was in cloud
storage.
Set hasRemoteFile in LoadRemoteFile, the single point where a volume's
backend becomes remote, so it is true both on disk-scan load and after
an in-process tier upload. Route the backend reassignment through
SwapDataBackend so it happens under dataFileAccessLock, closing the old
backend and never racing the heartbeat's concurrent DataBackend read.
Make the field atomic since the heartbeat now reads it concurrently with
the tier-upload handler that writes it.
* java: HTTP Basic Auth for filer behind an Nginx reverse proxy
Attach an "Authorization: Basic" header to the filer gRPC channel through a
ClientInterceptor and to the volume read/write HTTP requests, so the Java
client can reach SeaweedFS fronted by Nginx with auth_basic enabled.
Credentials come from a [basic_auth] section in security.toml or from
FilerSecurityContext.setBasicAuth(). On writes the volume JWT still owns the
Authorization header when present, since the two cannot share it.
* java: precompute Basic Auth header and warn on plaintext channel
Cache the Base64 Authorization value once instead of re-encoding on every
chunk read/write, and log a one-time warning when Basic Auth rides a
plaintext gRPC channel where the credentials would travel in cleartext.
* s3: fail over routed object writes when the owner filer is unreachable
A routed object write (multipart completion, PUT, delete, versioned
finalize, metadata replace) dialed the ring-selected owner filer
directly with no failover. After a filer restarts onto a new address the
lock ring can still name the old one, so every routed write hangs on the
dead address until the gateway is restarted; CompleteMultipartUpload in
particular exceeds client timeouts.
Route the transaction through withFilerClientFailover, skipping an owner
that recently failed, so a live filer forwards it to the real owner by
route_key. Mirrors the read path's getObjectEntryRoutedByKey.
* s3: fail over bucket-config writes when the owner filer is unreachable
patchBucketEntry dialed the bucket's ring owner directly, so a restarted
filer's stale ring address hung every bucket-config write (versioning,
lifecycle, object lock, ownership, ACL, policy, CORS) - the same failure
as routed object writes. Route it through objectTxnOnFiler so it skips an
unreachable owner and a live filer forwards by route_key.
* Optimize s3api encodePath to eliminate O(n^2) allocations
encodePath built its result via string concatenation in a loop, which is
O(n^2) in allocations. For non-ASCII (e.g. Chinese) runes it additionally
called make + hex.EncodeToString + strings.ToUpper per byte (~10 allocations
per character). Under high QPS with long non-ASCII object keys this produced
a very high allocation rate, frequent GC and long GC pauses, causing S3
request latency spikes.
Replace with a preallocated strings.Builder, a manual hex lookup table, and a
zero-allocation fast path. EncodePath now delegates to encodePath to remove
the duplicated implementation. Output is byte-for-byte identical, verified by
TestEncodePath and TestEncodePathEqual.
Benchmark (long Chinese path): 261 -> 1 allocs/op, 35810 -> 480 B/op, 7.5x faster.
Load test (50M calls): 212x fewer allocations, 76x fewer GC cycles.
* s3api: drop regexp from encodePath fast path
The unreserved-character scan already decides whether any byte needs
encoding, so reservedObjectNames.MatchString was a redundant second pass
that also ran the RE2 engine on every authenticated request. Rely on the
scan alone; output is unchanged. The ASCII fast path drops from ~188ns to
~11ns per call.
---------
Co-authored-by: LiuDoge <liudoge@LiuDogedeMacBook-Air.local>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* filer: require JWT authorization on TUS upload endpoints
filerHandler and readonlyFilerHandler run every request through
maybeCheckJwtAuthorization, but the TUS routes were registered only
behind filerGuard.WhiteList, which is a pass-through for the filer (its
guard is built with an empty whitelist), and no TUS handler called the
JWT check. So with jwt.filer_signing.key set, normal PUT/POST/DELETE
were authorized while the TUS endpoints were not.
Run the same check in tusHandler before routing: HEAD uses the read key,
POST/PATCH/DELETE use the write key, and creation scopes a
prefix-restricted token against the resolved target path. OPTIONS stays
open for capability discovery.
* filer: enforce read-only and WORM rules on TUS completion
completeTusUpload wrote the final entry with CreateEntry directly,
skipping the read-only and WORM checks the normal write path applies.
Reject completion when the target prefix is read-only or the existing
entry at the target is WORM-enforced.
* filer: align TUS write path with the normal write path
Resolve the create target with a guaranteed leading slash so a
prefix-restricted token and the stored path stay absolute even if
TusBasePath were set with a trailing slash. Reject read-only prefixes at
session creation before any chunk is written, and map read-only and WORM
rejections at completion to 507 and 403 instead of a generic 500.
* filer: cover method-restricted TUS tokens in the auth test
With replication 001 and one node down, ~2/3 of volumes have their
replica on the dead node. The primary write lands locally but the
replica forward fails, so the volume returns 500 "failed to write to
replicas". #9744 made that upload fail fast so the client re-assigns,
but the client retry never fired: the reassign gate matched a fixed
list of error substrings that didn't include this one. The mount
surfaced I/O error and dropped the chunk, leaving missing lines and
null-byte gaps in an append workload while a node rebooted.
Decide reassignment by HTTP status instead of matching message text.
On the write path a volume server only 5xxs on a ReplicatedWrite
failure (local disk, replica peer down, under-replication) — all of
which a different volume dodges — so any 5xx reassigns; a no-response
transport failure (the assigned target itself is down) reassigns too;
a 4xx is a genuine client error and is surfaced. doUploadData tags its
errors with the response status via uploadStatusError, and the gate
moves from util.MultiRetry(errList) to util.RetryOnError(predicate).
This drops the fragile substring list (and the message-prefix
constants and guard test it needed); store_replicate.go is untouched.
* filer: keep .system internal folder in the default SQL table
Bucket-table SQL stores read the first path segment under /buckets as a
bucket name. The ListBuckets owner index lives at /buckets/.system/..., so
every write there hit isValidBucket(".system") == false and returned
"invalid bucket name .system", flooding the filer log on the postgres/mysql
backends. Route dot-prefixed internal folders to the default table by their
full path, like any other non-bucket entry.
* filer: keep .system internal folder in the default leveldb3 DB
Same guard as the SQL stores: leveldb3 would otherwise open a separate DB
for the .system owner-index folder instead of keeping it in the default DB.
* filer: keep .system internal folder under the default ydb prefix
Skips a DescribeTable round trip per operation on the .system owner-index
path, which never resolves to a real bucket table.
* filer: keep .system internal folder in the default arangodb collection
Avoids creating a stray collection for the .system owner-index folder.
* seaweed-volume: async, buffered writes in VolumeEcShardsCopy
The EC-shards-copy RPC handler wrote each streamed chunk to disk with a
synchronous std::fs::File::write_all inside the async handler, blocking a
Tokio worker thread for the duration of every write — noticeable for a
large .ecx on a slow or busy disk.
Factor the five near-identical receive-and-write loops (.ec shards, .ecx,
.ecj, .vif, .ecsum) into drain_copy_stream_to_file, which uses tokio::fs +
BufWriter for async, buffered I/O. Behavior is otherwise unchanged: the
.ecj append mode, the .ecsum byte count and 0-byte-file cleanup, and all
error messages are preserved.
Claude-Session: https://claude.ai/code/session_01Ny5Rt1ph9VWeKmfY936GtF
* seaweed-volume: remove partial copy target on error in EC-shards-copy
Follow-up: drain_copy_stream_to_file now deletes the destination file on
any recv/write/flush error, so a failed VolumeEcShardsCopy no longer leaves
a truncated .ecNN/.ecx/.ecj/.vif/.ecsum on disk for a later reader to trip
on. Matches receive_file / the Go volume server. Best-effort cleanup; the
original stream error is still returned.
Claude-Session: https://claude.ai/code/session_01Ny5Rt1ph9VWeKmfY936GtF
CRC64NVME multipart objects now emit a full-object checksum instead of a composite base64-N value, matching AWS (CRC64NVME supports full-object only). Adds x-amz-checksum-type handling (COMPOSITE/FULL_OBJECT) for CRC32/CRC32C/CRC64NVME via CRC combination, resolved at CreateMultipartUpload and applied at completion.
* cassandra2: default connection_timeout_millisecond so env-var configs keep the 600ms timeout
The scaffold documents a 600ms default, but the value has no SetDefault. A
config supplied purely through env vars (or a minimal toml) that omits the
key read back 0, silently overriding the client timeout with no bound.
* ydb: default the table path prefix so env-var configs keep the seaweedfs sub-path
The scaffold documents prefix = "seaweedfs", but with no SetDefault a config
that omits it lands tables at the database root instead of under seaweedfs/.
* postgres2: default the filemeta CREATE TABLE when createTable is unset
An empty createTable rendered through fmt.Sprintf produced
%!(EXTRA string=filemeta), which Postgres rejected with a syntax error at
init. Fall back to a working template so a minimal config bootstraps.
* mysql2: default the filemeta CREATE TABLE when createTable is unset
Same empty-template failure the postgres2 path had: an unset createTable
rendered to %!(EXTRA string=filemeta) and MySQL rejected it at init.
Fall back to a working template.
aws-lc-rs and ring both get linked transitively, so rustls can't
auto-select a crypto provider and tonic's client TLS panics the moment
the volume server dials a master over TLS. Install aws-lc-rs as the
process default in main(), matching the provider the server config
already uses.
* shell: add -delete option to remote.copy.local
Add a -delete flag to remote.copy.local that removes files and
directories from remote storage when they no longer exist locally,
similar to rsync --delete. This makes the command usable for
scheduled one-shot backups that also propagate local deletions.
- -include/-exclude patterns also limit which remote files are deleted
- size/age filters only apply to copying, since remote entries have no
local attributes to filter on
- orphaned remote directories are removed after their contents,
deepest first, and only when no name filter is set (a recursive
RemoveDirectory could otherwise remove intentionally kept files)
- deletion is skipped entirely if any copy failed
- -dryRun shows DELETE lines for review before committing to anything
Fixesseaweedfs/seaweedfs#8609
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* shell: fix remote.copy.local -delete deleting files outside -dir
Traverse lists remote objects by key prefix with no delimiter, so a -dir
pointing at a subdirectory also matches siblings that merely share its
name prefix (foo -> foobar). Those are not under the local traversal
root, so -delete treated them as extraneous and removed them. Scope
deletion candidates to paths under dirToCopy.
Also drop the directory-removal path: RemoveDirectory is a no-op on every
backend and Traverse never emits directory entries, so it only ever
printed success for work it never did.
---------
Co-authored-by: Jason Lin <jason@jtx.com.tw>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
A read or a replicated write to a volume server that is TCP-reachable but not
answering -- one still loading its volumes after a restart, or reached over a
stale keep-alive to a container that came back on a new IP -- blocked forever:
the shared HTTP transport had a dial timeout but no response timeout.
Add ResponseHeaderTimeout so a chunk read fails over to another replica and a
replicated write fails fast for the client to retry, and IdleConnTimeout so
pooled connections to a departed server are evicted instead of reused.
* s3: answer application/x-directory for a directory without a stored mime
A real directory reached via a trailing-slash GET/HEAD answered the
octet-stream default when it had no stored mime, so Hadoop-style S3
filesystems (flink-s3-fs-presto and friends) classified the path as a
0-byte file instead of a directory and then failed reading it as one.
Answer application/x-directory, the marker type those clients probe
for. Stored mimes still echo verbatim, and a file promoted to a
directory keeps octet-stream for its data.
* s3: 404 GET and HEAD on a bare directory path consistently
A directory with no object data of its own answered differently per
path: plain GET gave an empty 200, ranged GET and non-versioned HEAD
gave 404, and on versioned buckets the null-version fallback adopted
the filer directory as a 0-byte object and answered 200. Clients that
probe HEAD-then-GET took the 200s at face value, treated the path as
an empty file, and never fell back to LIST-based directory discovery.
Answer 404 for a bare directory path everywhere, which is what AWS
returns for a prefix. A file promoted to a directory keeps its data
and stays retrievable.
* master: re-create a vacuum-committed volume the index has lost
SetVolumeAvailable dereferenced vid2location[vid] with no nil check. A
disconnect during a long vacuum can drop a single-replica volume from the
lookup index while it stays on the node; the commit then panics the master
instead of re-registering it. Re-create the entry, and seed its size tracking
so assigns are counted right away rather than after the next heartbeat.
* master: re-register a vacuumed volume the index lost on mark-writable
The maintenance worker re-enables a volume by marking it writable after the
vacuum commit, but VolumeMarkReadonly only updated nodes the lookup index
already knew. If a disconnect race dropped the volume from the index during the
vacuum, that path was a no-op and the volume stayed "not found" until the next
full heartbeat healed it. Re-register it from the node that still holds it.
* admin: move volume-server read JWT helper into dash
The Iceberg data preview page needs the same per-fileId read token the
file browser uses when streaming chunks from volume servers.
Claude-Session: https://claude.ai/code/session_015n3oKLTjnPjcnZtfigNKur
* admin: add Iceberg table data preview page
The admin UI browses the Iceberg catalog down to table details but not
the data itself. Add a Browse Data page per table that walks the
selected snapshot's manifests and shows sample rows from its Parquet
data files, plus the data file list with per-file preview, a snapshot
switcher, and a row limit selector.
Rows are read through a ranged ReaderAt over stream-content so only
the Parquet footer and needed pages are fetched, with the volume read
JWT applied when configured. Iceberg locations resolve into /buckets
with traversal guards, and the file parameter must match a
manifest-listed data file. Snapshots with delete files get a warning
that raw rows are shown.
Claude-Session: https://claude.ai/code/session_015n3oKLTjnPjcnZtfigNKur
* admin: integration test for Iceberg catalog and data preview pages
Starts a weed mini cluster with the admin UI, creates a table bucket,
namespace, and tables via the S3 Tables manager, uploads real Parquet
files via S3, writes manifests and snapshots with iceberg-go, and
asserts on the rendered pages: catalog browsing, table details,
current and historical snapshot previews, per-file preview, row
limits, unknown snapshot and file errors, and a metadata-less table.
Claude-Session: https://claude.ai/code/session_015n3oKLTjnPjcnZtfigNKur
* admin: write Iceberg preview chunk reads straight into the caller slice
ReadAt wrapped the caller's buffer in a bytes.Buffer, which would
silently allocate a fresh backing array and drop bytes if it ever grew.
Copy directly into the destination slice and reject negative offsets so
the ReaderAt contract holds.
Claude-Session: https://claude.ai/code/session_015n3oKLTjnPjcnZtfigNKur
* admin: link to snapshot history when the preview switcher truncates
The snapshot switcher caps at 25 entries; add a trailing item pointing
at the table details page so older snapshots stay reachable.
Claude-Session: https://claude.ai/code/session_015n3oKLTjnPjcnZtfigNKur
* test: hoist mini cluster context assignment out of the goroutine
Set MiniClusterCtx before launching the cluster goroutine and clear it
in stop(), so the assignment is not buried in the command loop.
Claude-Session: https://claude.ai/code/session_015n3oKLTjnPjcnZtfigNKur
* s3: enforce bucket quota on logical size, not un-vacuumed physical size
A bucket full of deleted/overwritten objects awaiting vacuum went
read-only while its live data stayed under quota, because enforcement
used the raw single-copy volume size with garbage included. Subtract
DeletedByteCount via a LogicalSize() helper in the auto-enforce loop,
the s3.bucket.quota.enforce command, and the bucket_size_bytes metric
(labeled logical but counting garbage too). Deleting objects now
relieves quota immediately and enforcement matches the UI usage figure.
* admin: surface bucket read-only state in the S3 buckets UI
Read the read-only flag quota enforcement writes to filer.conf and show
it as a badge in the bucket list and a Status row in the details modal,
so an operator can see why writes are being rejected.
* feat(k8s): add Traefik IngressRouteTCP for gRPC with TLS passthrough
Re-introduce Traefik support for the gRPC filer ingress that was
lost when the original ingress PR was merged. Previous attempts to
make the chart controller-agnostic using Ingress + ServersTransport
+ TLSOption CRDs were fragile — they required 2 separate services
(HTTP and gRPC), still failed with connection resets, and forced
Traefik to terminate and re-encrypt TLS traffic.
This approach uses a single IngressRouteTCP CRD with TLS passthrough
when enableSecurity is true, keeping the TLS stream intact. No
ServersTransport, no TLSOption, no service annotations, no values.yaml
structure changes. Fully backward compatible.
Refs: seaweedfs/seaweedfs#10205
Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf)
* refactor(k8s): only render standard gRPC Ingress when className is not Traefik
When className contains 'traefik', the IngressRouteTCP is the only
source of truth. The standard Kubernetes Ingress becomes superfluous
and potentially confusing for debugging.
Now:
- className: traefik → only IngressRouteTCP
- className: nginx/contour/... → only standard Ingress
- className: "" (default) → neither
No values.yaml changes. Fully backward compatible.
Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf)
* k8s: fix Traefik gRPC IngressRouteTCP for non-TLS and all-in-one modes
A non-TLS TCP router can only match HostSNI(`*`), so the default
enableSecurity=false path never matched. Use HostSNI(`*`) when security
is off and keep host-based SNI for TLS passthrough.
Route to the all-in-one service in all-in-one mode via the same ternary
the standard ingress uses; the hardcoded filer-client service is absent
when filer.enabled is false.
Also require grpc.enabled to render, align labels with the sibling
ingress, and put the comments in English.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
The owner index read the raw filer entry off BucketConfig, which no
longer carries one. Cache Crtime alongside IdentityId, and pass the
owner id rather than the entry through the ListBuckets visibility
checks so the scan and granted-bucket paths share one implementation.
* s3: paginate ListBuckets with max-buckets, continuation-token, and prefix
ListBuckets buffered every bucket entry into one slice and one XML body,
which falls over with very large bucket counts. Page through the filer
listing instead, cap each response at 10000 buckets like AWS, and honor
max-buckets, prefix, and an opaque keyset continuation-token.
* s3: maintain a bucket owner index under /buckets/.system/owners
Map each bucket owner to its buckets as zero-length entries at
/buckets/.system/owners/<owner>/<bucket>, with Crtime mirroring the
bucket's creation time. The bucket handlers write the index
synchronously, the /buckets metadata subscription reconciles changes
made elsewhere (weed shell, other gateways, direct filer operations),
and a startup backfill indexes pre-existing buckets before writing a
ready marker. Owner names are path-escaped so no identity name can
escape the index directory.
* s3: serve ListBuckets from the bucket owner index
Once the owner index is ready, non-admin identities list their owned
buckets straight from it, merged with any buckets their legacy actions
name explicitly, so ListBuckets costs O(own buckets) instead of a scan
of the global /buckets directory. Admins, identities with a bare List
grant or wildcard action patterns, and policy-authorized identities
whose grants cannot be enumerated keep the paged scan; policy-routed
identities get their owned buckets, matching AWS ListBuckets returning
only the caller's buckets.
* s3: keep dot-prefixed names under /buckets out of bucket surfaces
Dot-prefixed entries (.system) can never be valid bucket names, so
refuse to resolve them as buckets and skip them in the shell bucket
listing, matching what ListBuckets and the admin UI already do.
* test: cover ListBuckets pagination and the owner index end to end
* s3: fail closed on a nil identity when routing ListBuckets
* s3: decide the IAM authorization mechanism in one place
VerifyActionPermission and the ListBuckets owner-index routing each
re-derived the session-token / attached-policy / legacy-actions split;
extract the decision so the two cannot drift.
* s3: heal the owner index on concurrent bucket recreation too
The mkdir-lost-the-race path answers BucketAlreadyOwnedByYou just like
the up-front existence check, so give it the same index repair.
* s3: drop owner-index records for buckets deleted during backfill
A bucket removed between the backfill reading its page and writing the
index record became a permanent phantom in its owner's listing: the
delete's own cleanup ran before the record existed. After indexing each
page, re-list the same name range and remove records whose bucket is
gone; deletes landing after the re-list find the record and remove it
themselves.
* s3: add ContinuationToken and Prefix to the ListBuckets schema
Keep AmazonS3.xsd aligned with the generated ListAllMyBucketsResult so
a regeneration does not drop the pagination fields.
* s3: stop warming every bucket's config at startup
Listing all buckets in BucketRegistry.init() made S3 gateway startup
O(buckets) and pinned every bucket's metadata and config resident, which
does not scale past a few hundred thousand buckets. Both caches already
have lazy miss paths, so load on first access instead and let the
metadata subscription refresh only entries already resident; cold
buckets cost one filer round-trip on their first request.
* s3: bound the per-bucket caches with LRU eviction
The bucket config cache, bucket registry, and their negative caches were
plain maps that only ever grew: the config cache TTL made Get miss but
never evicted the entry, and the not-found sets grew on every probe of a
nonexistent bucket name. Cap all four at 65536 entries with LRU eviction
so a gateway keeps its hot working set and evicted buckets reload from
the filer on next access.
* s3: cache parsed bucket config instead of the full filer entry
Each cached BucketConfig retained the whole bucket entry (extended
attribute map plus raw content bytes) alongside the fields parsed from
it, roughly doubling per-bucket cache cost and keeping data the read
path never looks at. Parse everything up front in
newBucketConfigFromEntry - now also the creator identity, tags,
encryption config, and stored lifecycle XML - and drop the entry.
updateBucketConfig now reads the entry fresh from the filer and diffs
the mapped extended attributes against it, so the patch is computed
against current state instead of a cached copy; the config clone
helpers that existed for that path go away.
* s3: dedup cold bucket-registry loads per bucket
The registry's notFound lock doubled as the load serializer, holding one
global mutex across the filer round-trip so first-touch requests for
different buckets queued behind each other; the cache fill also happened
after the lock was released, so two concurrent misses for the same
bucket could both reach the filer. Replace it with a singleflight per
bucket that fills the cache inside the flight: different buckets load
concurrently, the same bucket loads once.
glog flags (-v, -logtostderr) are registered on weed's global flagset,
so passing them after the subcommand name kills the process at flag
parsing: flag provided but not defined: -logtostderr. The old stat-based
mount readiness probe masked this — TestWriteBufferCap silently ran
against the bare local directory and passed. The device-ID readiness
check now surfaces the dead mount as a not-ready timeout.
Move glog flags into MountGlobalOptions, emitted before the subcommand,
and do the same for the EnableDebug verbosity flag on mini and mount.
* util: don't let the activity timeout clobber externally-set conn deadlines
util.Conn extended the connection deadline at the start of every
Read/Write. net/http's server also sets deadlines directly on the same
conn - abortPendingRead sets one in the past to interrupt the pending
background read after each response. The activity extension raced with
and silently overwrote that interrupt, leaving the read blocked (and the
server's conn.serve goroutine stuck in abortPendingRead) until the full
-idleTimeout (default 30s) expired.
Wedged connections count as active, so the volume server's graceful HTTP
drain waited out its whole 30s StopTimeout on shutdown - observed as
weed mini taking ~30s to exit in the FUSE integration tests after any
filer->volume traffic.
Track externally-set deadlines, suspend the activity extension while one
is in force, and serialize deadline updates with a mutex. Activity still
extends both directions at once: a long write-only response must keep
the read deadline alive too, or the server's background read would time
out and cancel the in-flight request.
* util: extend read/write deadlines independently when one side is external
A server-configured WriteTimeout keeps an external write deadline in
force for the whole request, which previously suspended the activity
extension entirely - leaving the read deadline stale from before the
request and letting net/http's background read time out mid-response.
Extend each direction independently instead.
* fix(master): let the growth initiator wait for the growth it triggered
The growth-in-flight shed also fired on the request that initiated the
growth: it sets the pending flag right before the shed check, so a
cold-start assign enqueued growth and immediately failed itself with
"volume growth in progress". With no concurrent assigns around to pick
up the freshly grown volume, a single writer against an empty cluster
never completes a write despite ample free space.
Claim the pending flag with a compare-and-swap so exactly one request
becomes the initiator, triggering growth at most once, and let it wait
for that growth to land. Everyone else still sheds retryably instead of
pinning a goroutine: followers behind an in-flight growth, an initiator
whose growth concluded without yielding a writable volume, and an
initiator whose growth outlives the 10s wait budget, which previously
surfaced a non-retryable error (gRPC Unknown, HTTP 406) even though a
retry would have succeeded moments later.
* fix(master): stop assign waits when the request is cancelled
The assign retry loops slept through client cancellation, keeping a
goroutine spinning for the rest of the 10s budget after the caller had
gone; StreamAssign also ran assigns on a background context detached
from the stream. Wait on the request context and pass the stream
context through.
* topology: drop the unconditional grow-request setter
Growth is only claimed through AddGrowRequestIfAbsent's compare-and-swap
now; keeping the raw Store(true) around invites the check-then-set race
back.
* test: cover cold-start first write with a real cluster
Boot a fresh master plus three empty volume servers and require the very
first assign - HTTP and gRPC, each on a cold volume layout, no client
retries - to complete a write. The assign that triggers volume growth
must wait for it rather than answering "volume growth in progress";
unit tests stub the topology, so only a real cluster exercises the
assign-grow-wait path end to end.
* test: keep mini-allocated ports below the ephemeral floor
Allocated ports could land in 32768-55000, so a transient outbound dial
during mini startup (volume->master gRPC, etc.) could grab an allocated
port as its source port before the filer bound it, failing with
"bind: address already in use". Cap the range so port+GrpcPortOffset
stays under 32768.
* test: derive mini port cap from the ephemeral floor constant
Name the 32768 floor once and compute miniPortMax as floor-GrpcPortOffset
so the cap tracks the offset; reuse the constant in the regression test.
waitForMount probed the mount point with stat+ReadDir, which a bare local
directory also passes, so Setup could return before the weed mount process
finished mounting. Tests then wrote to the local disk underneath the mount
point; when the mount activated it shadowed those files, producing the
intermittent TestConcurrentFileOperations/ConcurrentReadWrite ENOENT with
an empty ReadDir. Require the mount point's device ID to differ from its
parent's before reporting ready.
* s3: surface multipart-list store errors instead of masking them
listMultipartUploads masked a filer/metadata-store list error as an empty
200 response, and listObjectParts masked it as NoSuchUpload. A client
resuming a multipart upload (the Docker Registry S3 driver resolves an
in-progress upload via ListMultipartUploads, then ListParts) reads either
as "upload gone" and fails the upload permanently with a non-retryable
error. Return ErrInternalError so a transient store error stays retryable,
keeping ErrNotFound as an empty list / NoSuchUpload respectively.
* s3: return retryable error for transient CopyObject source lookups
Resolving the copy source is reported as InvalidArgument ("Copy Source must
mention the source bucket and key") whenever the lookup returns any error,
including a transient store error. Clients don't retry a 400, so a resumable
blob commit fails permanently. Keep the client error for a missing, invalid,
directory, or delete-marker source; map a transient store error to
ErrInternalError.
* s3: mark versioned lookup miss with the not-found sentinel
recoverLatestVersionWithoutPointer's terminal miss (a .versions directory
with no pointer, no versions, and no null object) returned a plain error,
so errors.Is-based callers could not tell this designed NoSuchKey state
from a store failure and reported it as an internal error.
* s3: reject a delete-marker copy source with NoSuchKey
getLatestObjectVersion returns the delete-marker entry when the latest
version is a delete marker, and the copy handlers copied its empty stub
into the destination. Every other handler detects ExtDeleteMarkerKey and
answers NoSuchKey; do the same for CopyObject and UploadPartCopy.
* s3: align copy-source not-found detection with the grpc status idiom
The lookup path canonicalizes text-form not-found into the sentinel in
filer_pb.LookupEntry and wraps with %w after that, so matching sentinel
text here was unreachable -- and risky, since a store error whose message
merely mentions the sentinel would be downgraded to a terminal 400.
Match the raw grpc NotFound code instead, like the other version-lookup
sites, and give transient filer errors the 503 that the upload-entry
lookup in this file already returns.
* s3: keep source-bucket versioning lookup errors retryable in copy
CopyObject and UploadPartCopy mapped any source-bucket versioning-state
lookup error to a terminal InvalidCopySource, including transient store
errors; getVersioningState signals a missing bucket with the not-found
sentinel, so split on that and let real store errors surface as 500, the
same mapping the destination-bucket lookup already uses.
* s3: match grpc-transported not-found in multipart list error paths
The list client path returns raw grpc status errors without the sentinel
reconstruction that lookups get in filer_pb.LookupEntry, so errors.Is on
the not-found sentinel never matched a store-reported missing directory;
match the sentinel text as well via a shared helper. The common missing-
directory case still lists as empty with no error and is unaffected.
* s3: complete and abort multipart surface store errors
prepareMultipartCompletionState mapped any upload-directory list or
lookup error to NoSuchUpload, so a transient store error at completion
time made the client discard a fully-uploaded object as gone. Split on
not-found like the sibling listing paths. abortMultipartUpload did the
same on s3a.exists, whose errors are never not-found (filer_pb.Exists
reports that as false with no error) -- a store failure there answered
NoSuchUpload and silently leaked the uploaded parts.
* s3: reject part numbers below 1 in UploadPartCopy
Only the upper bound was checked, unlike PutObjectPart; partNumber=0
passed the route regex and validation and wrote an out-of-range
0000_copy.part into the upload directory.
* filer: stream offloaded metadata-log entries instead of buffering whole files
The client metadata-chunks read path (ReadLogFileRefs, used by the meta
aggregator to consume peer filers and by mounts) decoded every entry of a
log file into a slice before handing it to the consumer, and prefetched the
next whole file the same way. Peak memory scaled with log-file size: under
heavy concurrent writes the per-event chunk lists grow and minute-files reach
hundreds of MB to GBs, so a filer aggregating a few peers held many GBs of
decoded entries at once (heap dominated by readLogFileEntries ->
consumeBytesNoZero) and OOMed.
Stream entries through a bounded channel: a producer decodes one entry at a
time and the next file's read overlaps processing via the channel buffer, so
peak memory is bounded by the channel depth rather than O(file size). In a
synthetic replay peak live heap dropped from ~1.3x the file size to a flat
few MB regardless of file size.
* filer: tighten offload replay tests
Share one ordered-replay assertion between the merge-order and single-filer
tests, assert the callback's own error is what propagates, and drop atomic
counters from callbacks that run on a single goroutine.
* filer: abort offloaded log replay promptly instead of joining wedged readers
Collapse the single-filer fast path into the merged reader: it was a second
copy of the producer/stop lifecycle with its own subtler synchronization, and
a one-stream merge does the same job.
On abort (processing error or a fatal read error from one filer), the
consumer used to drain channels and join every producer. A producer blocked
in an uncancellable chunk read cannot observe stop until that read returns,
so an abort could stall the caller's retry loop behind a dead volume-server
connection. Closing stop is now the only cleanup: producers check it at every
send and file boundary and exit on their own, and the merge loop's blocking
receives also escape on stop. Producers also check stop before opening each
file, so an aborted replay no longer keeps reading remaining files whose
entries never reach the channel.
A mid-file chunk-not-found still skips to the next file, but the log line now
reports how many entries were delivered first instead of pretending the whole
file was skipped; the redundant error log before setFatal is gone since the
error propagates to callers that already log it.
* filer: cap offloaded log entry allocation against corrupt size prefix
A garbage 4-byte size prefix (torn chunk or stream desync) drove
make([]byte, size) up to 4GiB per entry. Reject sizes above the same 1GiB
bound the filer-side log readers enforce.
* feat(k8s): add HTTP + gRPC Ingress templates for filer
Add HTTP and gRPC Ingress templates for the filer component in both
standalone and all-in-one modes. The HTTP ingress handles REST API
traffic, the gRPC ingress exposes the gRPC endpoint with proper
annotations for nginx and Traefik.
Additionally add Traefik IngressRouteTCP for mTLS filer gRPC passthrough.
When the filer has mTLS enabled, the standard HTTP Ingress terminates TLS
at the ingress level which conflicts with the filer's mutual-TLS requirement.
IngressRouteTCP forwards raw TCP with tls.passthrough: true so the TLS
negotiation happens directly between client and filer.
Refs: PR #10035 (original fix-grpc-filer)
Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf)
* feat(k8s): restructure filer ingress into ingresses.{http,grpc}
Split the single filer ingress value into http and grpc sub-structures
so the HTTP Ingress and gRPC Ingress templates each have their own
configuration.
* k8s: document nginx ssl-passthrough for end-to-end mTLS gRPC
The filer's mTLS gRPC needs the TLS stream to reach the filer intact,
which an L7 Ingress can't do when it terminates TLS. Document the
ingress-nginx ssl-passthrough annotation on the gRPC ingress so the
whole chart stays on the standard Ingress kind, no controller-specific
CRD required.
* k8s: align filer ingress with the volume/admin ingress pattern
Only render ingressClassName when a class is set (an empty value opts out
of the cluster's default IngressClass), fall back to the
kubernetes.io/ingress.class annotation on k8s <1.18, version-gate
pathType, and quote the host so wildcard hosts stay valid YAML.
* k8s: route the filer gRPC ingress at / with Prefix
gRPC methods are called at /<package>.<Service>/<Method>; the HTTP UI
regex path never matches them, so gRPC requests would 404.
---------
Co-authored-by: MorezMartin <martin.morez@morez.org>
Unmount, and optionally delete, EC volume shards from the shell, so broken
or over-replicated shards can be handled without stopping volume servers and
deleting files by hand.
Default action is unmount; --delete also removes the shard files. Targets
resolve against the live topology, with shardId:host:port to disambiguate
co-located or over-replicated shards. Dry-run by default; pass --apply.
* feat(k8s): add certificates.dnsNames to inject custom SANs in cert-manager certs
Add certificates.dnsNames configuration option that allows users to
inject custom Subject Alternative Names (SANs) into all cert-manager
Certificate resources. This enables exposing SeaweedFS components
under custom hostnames/CN that aren't covered by the default
wildcard patterns (e.g., '*.filer.default.svc').
The dnsNames list is iterated over in all 6 cert templates
(admin, client, filer, master, volume, worker) and appended to
the spec.x509.subject.names list.
Refs: PR #10035 (original fix-grpc-filer)
Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf)
* k8s: quote certificates.dnsNames entries so wildcard SANs render valid YAML
---------
Co-authored-by: MorezMartin <martin.morez@morez.org>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3: embedded IAM inline policy honors prefix-scoped resources
getActions stripped the trailing wildcard from a resource like
arn:aws:s3:::bucket/prefix/*, producing a non-wildcard action
(Write:bucket/prefix) that CanDo only ever matched at bucket level, so
PutObject under the prefix was denied. Preserve the object path with its
wildcard (Write:bucket/prefix/*) to match objects under the prefix,
matching the standalone iamapi behavior.
* s3: prune bucket-confined wildcard actions on bucket delete
actionScopedToBucket treated any wildcard as multi-bucket, so a
prefix-scoped action like Write:bucket/prefix/* survived deletion of its
own bucket and could re-grant access if the bucket was recreated. Scope
the wildcard check to the bucket segment only: a wildcard in the object
path stays scoped to its bucket, while one in the bucket segment does
not.
* fix(volume [rust]): compare live compaction_revision instead of stale last_compact_revision
* fix(volume [rust]): compare live compaction_revision instead of stale last_compact_revision - unit tests
* s3: invalidate stale reader cache locations on chunk read failure (#10156)
* s3: invalidate stale reader cache locations on chunk read failure
* filer: share the chunk-read self-heal across reader cache and streaming paths
The reader cache retry added a third copy of the invalidate-relookup-compare-retry
dance already inlined in PrepareStreamContentWithThrottler and duplicated in
retryWithCacheInvalidation. Extract retryFetchWithFreshLocations and route all
three through it, parameterized by the refetch primitive.
* filer: drop redundant completedTimeNew store in reader cache success path
startCaching already stamps completedTimeNew unconditionally before the
fetchErr branch; the second store inside the success branch is dead.
* filer: make NewReaderCache cache invalidator an explicit parameter
The variadic ...CacheInvalidator only ever read the first element, so a caller
could pass two and silently get one. Take a single explicit argument and have
the non-S3 callers pass nil.
* filer: inject reader cache chunk fetch as a struct field
Replace the process-global readerCacheFetchChunkData test seam with a
per-instance fetchChunkDataFn field defaulted in NewReaderCache, matching how
lookupFileIdFn is already wired. Tests set the field on the cache instead of
swapping a shared global.
* filer: log the location count, not full URLs, on self-heal retry
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix(shell): honor explicit fs.mergeVolumes from/to direction (#10159)
* fix(shell): honor explicit fs.mergeVolumes from/to direction
mergeVolumes only ever merged a smaller volume into a larger one. When the
user named both -fromVolumeId and -toVolumeId with the source larger than the
target, the planner produced an empty plan and the command printed just
"max volume size: N MB" and moved nothing.
Build the requested pair directly when both ids are given, instead of routing
through the size-descending heuristic. Read-only, empty, and wrong-collection
endpoints are rejected with a clear error rather than a silent no-op.
* fix(shell): allow fs.mergeVolumes into an empty target volume
Merging chunks into an empty volume is valid, e.g. consolidating data into a
freshly created or recently vacuumed volume. Only reject an empty source, which
has nothing to move.
* fix(shell): reject self-map in directed mergeVolumes planner
createMergePlan with from == to returned a {vid: vid} self-merge when called
directly. Guard it in the planner so it is correct independent of the Do
entrypoint.
* fix(volume [rust]): compare compaction_revision in u32, not truncated u16
`req.compaction_revision as u16` truncates any request value above 65535, so a
stale revision of 65537 aliases to a live revision of 1 and the "is compacted"
guard wrongly passes. Widen the volume's revision to u32 and compare there,
matching Go's uint32(v.CompactionRevision) != req.CompactionRevision.
---------
Co-authored-by: adri <adri@digitalunited.net>
Co-authored-by: Aleksey <48918167+MilanFun@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
filer: count reader on cacher waitgroup under the map lock
The read's wg.Add(1) ran after ReadChunkAt released the ReaderCache lock, so a
concurrent destroy() (error eviction, LRU, or UnCache) could start wg.Wait() on
a zero counter and then race the Add - a WaitGroup reuse that trips -race and
can panic. Move the Add under the lock, before the cacher can leave the
downloaders map, so a destroy is always ordered against a counted read.
Chris Lu
Chris LuandGitHub
github-actions[bot]
Cole HelblingandGitHub
qzhelloandGitHub
