* log_buffer: share one window snapshot across all subscriber reads
Every in-memory read handed each subscriber a private pooled copy of the
window it wanted, so N subscribers reading the same data cost N copies of
up to 8MB each -- and slow consumers (grpc send backpressure) held those
copies live for their whole iteration. With hundreds of mount subscribers
that multiplied into gigabytes of live heap on the filer.
Share the bytes instead of copying per reader:
- Sealed windows get a lazily created GC-owned snapshot, made once by the
first reader and handed out zero-copy to the rest. The snapshot travels
with its window when SealBuffer shifts slots, so recycling the sealed
array never invalidates it.
- The current window keeps a shared snapshot of its append-only prefix
buf[:pos], extended on demand; each byte is copied once per window
(writer-rate-bound) instead of once per reader. At seal a fully
extended prefix becomes the sealed window's snapshot.
ReadFromBuffer now reports whether the returned buffer is a pooled copy
(flush path) or a shared view that must not be released; the read loops
only recycle pooled buffers.
With 200 subscribers consuming at grpc pace over sealed and current
windows, peak live heap drops from 5.2GB to 178MB.
* log_buffer: clear released read buffer so a panic cannot double-free it
The read loops release the previous iteration's pooled buffer and then
call ReadFromBuffer. If that call panicked before reassigning bytesBuf,
the deferred cleanup would put the same buffer into the pool a second
time, letting two future readers share one backing array. Nil the
pointer at the release site so the defer sees nothing to free.
* java client: fail cleanly when a chunk read returns short or empty data
readChunkView copied chunkView.size bytes out of the fetched chunk without
checking the fetched length, so an empty body (seen transiently right after an
append) threw an opaque IndexOutOfBoundsException. Retry empty chunk fetches and
bound the copy to the data actually returned.
* overflow-safe bounds check and null guard on fetched chunk data
LoadRemoteFile now takes dataFileAccessLock (so a live tier upload does
not race the heartbeat's DataBackend read). But load() also runs it, and
CommitCompact calls load() while already holding that lock, so reloading
a remote-tiered volume during a compaction commit re-enters the
non-reentrant lock and deadlocks.
Split the locked bodies out: swapDataBackendLocked and loadRemoteFileLocked
assume the caller holds dataFileAccessLock. load() uses loadRemoteFileLocked;
the public LoadRemoteFile keeps taking the lock for the live tier-upload
handler that does not hold it.
* log_buffer: read only ts_ns during metadata-read binary search
readTs ran a full proto.Unmarshal of each LogEntry just to compare
timestamps during ReadFromBuffer's binary search, allocating fresh
slices for the data and key byte fields on every probe. Under
metadata-subscription fan-out this decode churn dominated allocations
and drove heavy GC, inflating filer RSS well past the live heap.
Scan the wire format for ts_ns (field 1) instead and skip the data/key
payloads without copying them. readTs is now zero-alloc; the binary
search no longer touches the event payloads at all.
* log_buffer: alias event payloads instead of copying them on delivery
The subscribe read loops decoded each LogEntry with a full proto.Unmarshal,
allocating fresh slices for the data and key byte fields on every entry
(protobuf consumeBytesNoZero). When many metadata subscribers each re-read
the in-memory log window under a reconnect storm, that per-entry copy was a
large share of allocation churn -- and for entries a subscriber filters out
by prefix, the copied payload was never even looked at.
Decode with a wire scan that points data/key at sub-slices of the source
buffer instead of copying. The aliased slices are valid only for the
duration of the eachLogEntryFn callback; every current subscriber either
re-decodes the event into its own struct or hands it to a synchronous grpc
Send, so none retain them. Per delivered entry the loop decode drops from
176 B / 2 allocs to zero.
VolumeTierMoveDatFromRemote downloads the .dat, trims the .vif, and swaps
the data backend to the local file, but left hasRemoteFile set. The
volume.tier.download command masks this by unmounting and remounting
right after, which reloads the flag from the trimmed .vif — but in the
window before the remount the in-memory flag is wrong: doDeleteRequest
would skip appending the tombstone to the freshly local .dat, and the
phantom-.dat guard stays disabled.
Give SwapDataBackend a hasRemoteFile argument so the backend swap and the
flag move together under one lock, and route both tier directions through
it: the tier-down download passes false, LoadRemoteFile passes true. The
flag can no longer disagree with the live backend.
volume: keep tier-uploaded volume reporting to master
A live volume.tier.upload removes the local .dat and swaps the data
backend to remote, but v.hasRemoteFile was only ever set when a volume
is loaded from disk, so the running volume kept it false. The phantom
.dat guard then saw fileCount>0, !HasRemoteFile, and a missing .dat, and
stopped reporting the volume to the master. The volume vanished from the
topology even though the upload succeeded and the data was in cloud
storage.
Set hasRemoteFile in LoadRemoteFile, the single point where a volume's
backend becomes remote, so it is true both on disk-scan load and after
an in-process tier upload. Route the backend reassignment through
SwapDataBackend so it happens under dataFileAccessLock, closing the old
backend and never racing the heartbeat's concurrent DataBackend read.
Make the field atomic since the heartbeat now reads it concurrently with
the tier-upload handler that writes it.
* java: HTTP Basic Auth for filer behind an Nginx reverse proxy
Attach an "Authorization: Basic" header to the filer gRPC channel through a
ClientInterceptor and to the volume read/write HTTP requests, so the Java
client can reach SeaweedFS fronted by Nginx with auth_basic enabled.
Credentials come from a [basic_auth] section in security.toml or from
FilerSecurityContext.setBasicAuth(). On writes the volume JWT still owns the
Authorization header when present, since the two cannot share it.
* java: precompute Basic Auth header and warn on plaintext channel
Cache the Base64 Authorization value once instead of re-encoding on every
chunk read/write, and log a one-time warning when Basic Auth rides a
plaintext gRPC channel where the credentials would travel in cleartext.
* s3: fail over routed object writes when the owner filer is unreachable
A routed object write (multipart completion, PUT, delete, versioned
finalize, metadata replace) dialed the ring-selected owner filer
directly with no failover. After a filer restarts onto a new address the
lock ring can still name the old one, so every routed write hangs on the
dead address until the gateway is restarted; CompleteMultipartUpload in
particular exceeds client timeouts.
Route the transaction through withFilerClientFailover, skipping an owner
that recently failed, so a live filer forwards it to the real owner by
route_key. Mirrors the read path's getObjectEntryRoutedByKey.
* s3: fail over bucket-config writes when the owner filer is unreachable
patchBucketEntry dialed the bucket's ring owner directly, so a restarted
filer's stale ring address hung every bucket-config write (versioning,
lifecycle, object lock, ownership, ACL, policy, CORS) - the same failure
as routed object writes. Route it through objectTxnOnFiler so it skips an
unreachable owner and a live filer forwards by route_key.
* Optimize s3api encodePath to eliminate O(n^2) allocations
encodePath built its result via string concatenation in a loop, which is
O(n^2) in allocations. For non-ASCII (e.g. Chinese) runes it additionally
called make + hex.EncodeToString + strings.ToUpper per byte (~10 allocations
per character). Under high QPS with long non-ASCII object keys this produced
a very high allocation rate, frequent GC and long GC pauses, causing S3
request latency spikes.
Replace with a preallocated strings.Builder, a manual hex lookup table, and a
zero-allocation fast path. EncodePath now delegates to encodePath to remove
the duplicated implementation. Output is byte-for-byte identical, verified by
TestEncodePath and TestEncodePathEqual.
Benchmark (long Chinese path): 261 -> 1 allocs/op, 35810 -> 480 B/op, 7.5x faster.
Load test (50M calls): 212x fewer allocations, 76x fewer GC cycles.
* s3api: drop regexp from encodePath fast path
The unreserved-character scan already decides whether any byte needs
encoding, so reservedObjectNames.MatchString was a redundant second pass
that also ran the RE2 engine on every authenticated request. Rely on the
scan alone; output is unchanged. The ASCII fast path drops from ~188ns to
~11ns per call.
---------
Co-authored-by: LiuDoge <liudoge@LiuDogedeMacBook-Air.local>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* filer: require JWT authorization on TUS upload endpoints
filerHandler and readonlyFilerHandler run every request through
maybeCheckJwtAuthorization, but the TUS routes were registered only
behind filerGuard.WhiteList, which is a pass-through for the filer (its
guard is built with an empty whitelist), and no TUS handler called the
JWT check. So with jwt.filer_signing.key set, normal PUT/POST/DELETE
were authorized while the TUS endpoints were not.
Run the same check in tusHandler before routing: HEAD uses the read key,
POST/PATCH/DELETE use the write key, and creation scopes a
prefix-restricted token against the resolved target path. OPTIONS stays
open for capability discovery.
* filer: enforce read-only and WORM rules on TUS completion
completeTusUpload wrote the final entry with CreateEntry directly,
skipping the read-only and WORM checks the normal write path applies.
Reject completion when the target prefix is read-only or the existing
entry at the target is WORM-enforced.
* filer: align TUS write path with the normal write path
Resolve the create target with a guaranteed leading slash so a
prefix-restricted token and the stored path stay absolute even if
TusBasePath were set with a trailing slash. Reject read-only prefixes at
session creation before any chunk is written, and map read-only and WORM
rejections at completion to 507 and 403 instead of a generic 500.
* filer: cover method-restricted TUS tokens in the auth test
With replication 001 and one node down, ~2/3 of volumes have their
replica on the dead node. The primary write lands locally but the
replica forward fails, so the volume returns 500 "failed to write to
replicas". #9744 made that upload fail fast so the client re-assigns,
but the client retry never fired: the reassign gate matched a fixed
list of error substrings that didn't include this one. The mount
surfaced I/O error and dropped the chunk, leaving missing lines and
null-byte gaps in an append workload while a node rebooted.
Decide reassignment by HTTP status instead of matching message text.
On the write path a volume server only 5xxs on a ReplicatedWrite
failure (local disk, replica peer down, under-replication) — all of
which a different volume dodges — so any 5xx reassigns; a no-response
transport failure (the assigned target itself is down) reassigns too;
a 4xx is a genuine client error and is surfaced. doUploadData tags its
errors with the response status via uploadStatusError, and the gate
moves from util.MultiRetry(errList) to util.RetryOnError(predicate).
This drops the fragile substring list (and the message-prefix
constants and guard test it needed); store_replicate.go is untouched.
* filer: keep .system internal folder in the default SQL table
Bucket-table SQL stores read the first path segment under /buckets as a
bucket name. The ListBuckets owner index lives at /buckets/.system/..., so
every write there hit isValidBucket(".system") == false and returned
"invalid bucket name .system", flooding the filer log on the postgres/mysql
backends. Route dot-prefixed internal folders to the default table by their
full path, like any other non-bucket entry.
* filer: keep .system internal folder in the default leveldb3 DB
Same guard as the SQL stores: leveldb3 would otherwise open a separate DB
for the .system owner-index folder instead of keeping it in the default DB.
* filer: keep .system internal folder under the default ydb prefix
Skips a DescribeTable round trip per operation on the .system owner-index
path, which never resolves to a real bucket table.
* filer: keep .system internal folder in the default arangodb collection
Avoids creating a stray collection for the .system owner-index folder.
* seaweed-volume: async, buffered writes in VolumeEcShardsCopy
The EC-shards-copy RPC handler wrote each streamed chunk to disk with a
synchronous std::fs::File::write_all inside the async handler, blocking a
Tokio worker thread for the duration of every write — noticeable for a
large .ecx on a slow or busy disk.
Factor the five near-identical receive-and-write loops (.ec shards, .ecx,
.ecj, .vif, .ecsum) into drain_copy_stream_to_file, which uses tokio::fs +
BufWriter for async, buffered I/O. Behavior is otherwise unchanged: the
.ecj append mode, the .ecsum byte count and 0-byte-file cleanup, and all
error messages are preserved.
Claude-Session: https://claude.ai/code/session_01Ny5Rt1ph9VWeKmfY936GtF
* seaweed-volume: remove partial copy target on error in EC-shards-copy
Follow-up: drain_copy_stream_to_file now deletes the destination file on
any recv/write/flush error, so a failed VolumeEcShardsCopy no longer leaves
a truncated .ecNN/.ecx/.ecj/.vif/.ecsum on disk for a later reader to trip
on. Matches receive_file / the Go volume server. Best-effort cleanup; the
original stream error is still returned.
Claude-Session: https://claude.ai/code/session_01Ny5Rt1ph9VWeKmfY936GtF
CRC64NVME multipart objects now emit a full-object checksum instead of a composite base64-N value, matching AWS (CRC64NVME supports full-object only). Adds x-amz-checksum-type handling (COMPOSITE/FULL_OBJECT) for CRC32/CRC32C/CRC64NVME via CRC combination, resolved at CreateMultipartUpload and applied at completion.
* cassandra2: default connection_timeout_millisecond so env-var configs keep the 600ms timeout
The scaffold documents a 600ms default, but the value has no SetDefault. A
config supplied purely through env vars (or a minimal toml) that omits the
key read back 0, silently overriding the client timeout with no bound.
* ydb: default the table path prefix so env-var configs keep the seaweedfs sub-path
The scaffold documents prefix = "seaweedfs", but with no SetDefault a config
that omits it lands tables at the database root instead of under seaweedfs/.
* postgres2: default the filemeta CREATE TABLE when createTable is unset
An empty createTable rendered through fmt.Sprintf produced
%!(EXTRA string=filemeta), which Postgres rejected with a syntax error at
init. Fall back to a working template so a minimal config bootstraps.
* mysql2: default the filemeta CREATE TABLE when createTable is unset
Same empty-template failure the postgres2 path had: an unset createTable
rendered to %!(EXTRA string=filemeta) and MySQL rejected it at init.
Fall back to a working template.
aws-lc-rs and ring both get linked transitively, so rustls can't
auto-select a crypto provider and tonic's client TLS panics the moment
the volume server dials a master over TLS. Install aws-lc-rs as the
process default in main(), matching the provider the server config
already uses.
* shell: add -delete option to remote.copy.local
Add a -delete flag to remote.copy.local that removes files and
directories from remote storage when they no longer exist locally,
similar to rsync --delete. This makes the command usable for
scheduled one-shot backups that also propagate local deletions.
- -include/-exclude patterns also limit which remote files are deleted
- size/age filters only apply to copying, since remote entries have no
local attributes to filter on
- orphaned remote directories are removed after their contents,
deepest first, and only when no name filter is set (a recursive
RemoveDirectory could otherwise remove intentionally kept files)
- deletion is skipped entirely if any copy failed
- -dryRun shows DELETE lines for review before committing to anything
Fixesseaweedfs/seaweedfs#8609
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* shell: fix remote.copy.local -delete deleting files outside -dir
Traverse lists remote objects by key prefix with no delimiter, so a -dir
pointing at a subdirectory also matches siblings that merely share its
name prefix (foo -> foobar). Those are not under the local traversal
root, so -delete treated them as extraneous and removed them. Scope
deletion candidates to paths under dirToCopy.
Also drop the directory-removal path: RemoveDirectory is a no-op on every
backend and Traverse never emits directory entries, so it only ever
printed success for work it never did.
---------
Co-authored-by: Jason Lin <jason@jtx.com.tw>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
A read or a replicated write to a volume server that is TCP-reachable but not
answering -- one still loading its volumes after a restart, or reached over a
stale keep-alive to a container that came back on a new IP -- blocked forever:
the shared HTTP transport had a dial timeout but no response timeout.
Add ResponseHeaderTimeout so a chunk read fails over to another replica and a
replicated write fails fast for the client to retry, and IdleConnTimeout so
pooled connections to a departed server are evicted instead of reused.
* s3: answer application/x-directory for a directory without a stored mime
A real directory reached via a trailing-slash GET/HEAD answered the
octet-stream default when it had no stored mime, so Hadoop-style S3
filesystems (flink-s3-fs-presto and friends) classified the path as a
0-byte file instead of a directory and then failed reading it as one.
Answer application/x-directory, the marker type those clients probe
for. Stored mimes still echo verbatim, and a file promoted to a
directory keeps octet-stream for its data.
* s3: 404 GET and HEAD on a bare directory path consistently
A directory with no object data of its own answered differently per
path: plain GET gave an empty 200, ranged GET and non-versioned HEAD
gave 404, and on versioned buckets the null-version fallback adopted
the filer directory as a 0-byte object and answered 200. Clients that
probe HEAD-then-GET took the 200s at face value, treated the path as
an empty file, and never fell back to LIST-based directory discovery.
Answer 404 for a bare directory path everywhere, which is what AWS
returns for a prefix. A file promoted to a directory keeps its data
and stays retrievable.
* master: re-create a vacuum-committed volume the index has lost
SetVolumeAvailable dereferenced vid2location[vid] with no nil check. A
disconnect during a long vacuum can drop a single-replica volume from the
lookup index while it stays on the node; the commit then panics the master
instead of re-registering it. Re-create the entry, and seed its size tracking
so assigns are counted right away rather than after the next heartbeat.
* master: re-register a vacuumed volume the index lost on mark-writable
The maintenance worker re-enables a volume by marking it writable after the
vacuum commit, but VolumeMarkReadonly only updated nodes the lookup index
already knew. If a disconnect race dropped the volume from the index during the
vacuum, that path was a no-op and the volume stayed "not found" until the next
full heartbeat healed it. Re-register it from the node that still holds it.
* admin: move volume-server read JWT helper into dash
The Iceberg data preview page needs the same per-fileId read token the
file browser uses when streaming chunks from volume servers.
Claude-Session: https://claude.ai/code/session_015n3oKLTjnPjcnZtfigNKur
* admin: add Iceberg table data preview page
The admin UI browses the Iceberg catalog down to table details but not
the data itself. Add a Browse Data page per table that walks the
selected snapshot's manifests and shows sample rows from its Parquet
data files, plus the data file list with per-file preview, a snapshot
switcher, and a row limit selector.
Rows are read through a ranged ReaderAt over stream-content so only
the Parquet footer and needed pages are fetched, with the volume read
JWT applied when configured. Iceberg locations resolve into /buckets
with traversal guards, and the file parameter must match a
manifest-listed data file. Snapshots with delete files get a warning
that raw rows are shown.
Claude-Session: https://claude.ai/code/session_015n3oKLTjnPjcnZtfigNKur
* admin: integration test for Iceberg catalog and data preview pages
Starts a weed mini cluster with the admin UI, creates a table bucket,
namespace, and tables via the S3 Tables manager, uploads real Parquet
files via S3, writes manifests and snapshots with iceberg-go, and
asserts on the rendered pages: catalog browsing, table details,
current and historical snapshot previews, per-file preview, row
limits, unknown snapshot and file errors, and a metadata-less table.
Claude-Session: https://claude.ai/code/session_015n3oKLTjnPjcnZtfigNKur
* admin: write Iceberg preview chunk reads straight into the caller slice
ReadAt wrapped the caller's buffer in a bytes.Buffer, which would
silently allocate a fresh backing array and drop bytes if it ever grew.
Copy directly into the destination slice and reject negative offsets so
the ReaderAt contract holds.
Claude-Session: https://claude.ai/code/session_015n3oKLTjnPjcnZtfigNKur
* admin: link to snapshot history when the preview switcher truncates
The snapshot switcher caps at 25 entries; add a trailing item pointing
at the table details page so older snapshots stay reachable.
Claude-Session: https://claude.ai/code/session_015n3oKLTjnPjcnZtfigNKur
* test: hoist mini cluster context assignment out of the goroutine
Set MiniClusterCtx before launching the cluster goroutine and clear it
in stop(), so the assignment is not buried in the command loop.
Claude-Session: https://claude.ai/code/session_015n3oKLTjnPjcnZtfigNKur
* s3: enforce bucket quota on logical size, not un-vacuumed physical size
A bucket full of deleted/overwritten objects awaiting vacuum went
read-only while its live data stayed under quota, because enforcement
used the raw single-copy volume size with garbage included. Subtract
DeletedByteCount via a LogicalSize() helper in the auto-enforce loop,
the s3.bucket.quota.enforce command, and the bucket_size_bytes metric
(labeled logical but counting garbage too). Deleting objects now
relieves quota immediately and enforcement matches the UI usage figure.
* admin: surface bucket read-only state in the S3 buckets UI
Read the read-only flag quota enforcement writes to filer.conf and show
it as a badge in the bucket list and a Status row in the details modal,
so an operator can see why writes are being rejected.
* feat(k8s): add Traefik IngressRouteTCP for gRPC with TLS passthrough
Re-introduce Traefik support for the gRPC filer ingress that was
lost when the original ingress PR was merged. Previous attempts to
make the chart controller-agnostic using Ingress + ServersTransport
+ TLSOption CRDs were fragile — they required 2 separate services
(HTTP and gRPC), still failed with connection resets, and forced
Traefik to terminate and re-encrypt TLS traffic.
This approach uses a single IngressRouteTCP CRD with TLS passthrough
when enableSecurity is true, keeping the TLS stream intact. No
ServersTransport, no TLSOption, no service annotations, no values.yaml
structure changes. Fully backward compatible.
Refs: seaweedfs/seaweedfs#10205
Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf)
* refactor(k8s): only render standard gRPC Ingress when className is not Traefik
When className contains 'traefik', the IngressRouteTCP is the only
source of truth. The standard Kubernetes Ingress becomes superfluous
and potentially confusing for debugging.
Now:
- className: traefik → only IngressRouteTCP
- className: nginx/contour/... → only standard Ingress
- className: "" (default) → neither
No values.yaml changes. Fully backward compatible.
Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf)
* k8s: fix Traefik gRPC IngressRouteTCP for non-TLS and all-in-one modes
A non-TLS TCP router can only match HostSNI(`*`), so the default
enableSecurity=false path never matched. Use HostSNI(`*`) when security
is off and keep host-based SNI for TLS passthrough.
Route to the all-in-one service in all-in-one mode via the same ternary
the standard ingress uses; the hardcoded filer-client service is absent
when filer.enabled is false.
Also require grpc.enabled to render, align labels with the sibling
ingress, and put the comments in English.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
The owner index read the raw filer entry off BucketConfig, which no
longer carries one. Cache Crtime alongside IdentityId, and pass the
owner id rather than the entry through the ListBuckets visibility
checks so the scan and granted-bucket paths share one implementation.
* s3: paginate ListBuckets with max-buckets, continuation-token, and prefix
ListBuckets buffered every bucket entry into one slice and one XML body,
which falls over with very large bucket counts. Page through the filer
listing instead, cap each response at 10000 buckets like AWS, and honor
max-buckets, prefix, and an opaque keyset continuation-token.
* s3: maintain a bucket owner index under /buckets/.system/owners
Map each bucket owner to its buckets as zero-length entries at
/buckets/.system/owners/<owner>/<bucket>, with Crtime mirroring the
bucket's creation time. The bucket handlers write the index
synchronously, the /buckets metadata subscription reconciles changes
made elsewhere (weed shell, other gateways, direct filer operations),
and a startup backfill indexes pre-existing buckets before writing a
ready marker. Owner names are path-escaped so no identity name can
escape the index directory.
* s3: serve ListBuckets from the bucket owner index
Once the owner index is ready, non-admin identities list their owned
buckets straight from it, merged with any buckets their legacy actions
name explicitly, so ListBuckets costs O(own buckets) instead of a scan
of the global /buckets directory. Admins, identities with a bare List
grant or wildcard action patterns, and policy-authorized identities
whose grants cannot be enumerated keep the paged scan; policy-routed
identities get their owned buckets, matching AWS ListBuckets returning
only the caller's buckets.
* s3: keep dot-prefixed names under /buckets out of bucket surfaces
Dot-prefixed entries (.system) can never be valid bucket names, so
refuse to resolve them as buckets and skip them in the shell bucket
listing, matching what ListBuckets and the admin UI already do.
* test: cover ListBuckets pagination and the owner index end to end
* s3: fail closed on a nil identity when routing ListBuckets
* s3: decide the IAM authorization mechanism in one place
VerifyActionPermission and the ListBuckets owner-index routing each
re-derived the session-token / attached-policy / legacy-actions split;
extract the decision so the two cannot drift.
* s3: heal the owner index on concurrent bucket recreation too
The mkdir-lost-the-race path answers BucketAlreadyOwnedByYou just like
the up-front existence check, so give it the same index repair.
* s3: drop owner-index records for buckets deleted during backfill
A bucket removed between the backfill reading its page and writing the
index record became a permanent phantom in its owner's listing: the
delete's own cleanup ran before the record existed. After indexing each
page, re-list the same name range and remove records whose bucket is
gone; deletes landing after the re-list find the record and remove it
themselves.
* s3: add ContinuationToken and Prefix to the ListBuckets schema
Keep AmazonS3.xsd aligned with the generated ListAllMyBucketsResult so
a regeneration does not drop the pagination fields.
* s3: stop warming every bucket's config at startup
Listing all buckets in BucketRegistry.init() made S3 gateway startup
O(buckets) and pinned every bucket's metadata and config resident, which
does not scale past a few hundred thousand buckets. Both caches already
have lazy miss paths, so load on first access instead and let the
metadata subscription refresh only entries already resident; cold
buckets cost one filer round-trip on their first request.
* s3: bound the per-bucket caches with LRU eviction
The bucket config cache, bucket registry, and their negative caches were
plain maps that only ever grew: the config cache TTL made Get miss but
never evicted the entry, and the not-found sets grew on every probe of a
nonexistent bucket name. Cap all four at 65536 entries with LRU eviction
so a gateway keeps its hot working set and evicted buckets reload from
the filer on next access.
* s3: cache parsed bucket config instead of the full filer entry
Each cached BucketConfig retained the whole bucket entry (extended
attribute map plus raw content bytes) alongside the fields parsed from
it, roughly doubling per-bucket cache cost and keeping data the read
path never looks at. Parse everything up front in
newBucketConfigFromEntry - now also the creator identity, tags,
encryption config, and stored lifecycle XML - and drop the entry.
updateBucketConfig now reads the entry fresh from the filer and diffs
the mapped extended attributes against it, so the patch is computed
against current state instead of a cached copy; the config clone
helpers that existed for that path go away.
* s3: dedup cold bucket-registry loads per bucket
The registry's notFound lock doubled as the load serializer, holding one
global mutex across the filer round-trip so first-touch requests for
different buckets queued behind each other; the cache fill also happened
after the lock was released, so two concurrent misses for the same
bucket could both reach the filer. Replace it with a singleflight per
bucket that fills the cache inside the flight: different buckets load
concurrently, the same bucket loads once.
glog flags (-v, -logtostderr) are registered on weed's global flagset,
so passing them after the subcommand name kills the process at flag
parsing: flag provided but not defined: -logtostderr. The old stat-based
mount readiness probe masked this — TestWriteBufferCap silently ran
against the bare local directory and passed. The device-ID readiness
check now surfaces the dead mount as a not-ready timeout.
Move glog flags into MountGlobalOptions, emitted before the subcommand,
and do the same for the EnableDebug verbosity flag on mini and mount.
* util: don't let the activity timeout clobber externally-set conn deadlines
util.Conn extended the connection deadline at the start of every
Read/Write. net/http's server also sets deadlines directly on the same
conn - abortPendingRead sets one in the past to interrupt the pending
background read after each response. The activity extension raced with
and silently overwrote that interrupt, leaving the read blocked (and the
server's conn.serve goroutine stuck in abortPendingRead) until the full
-idleTimeout (default 30s) expired.
Wedged connections count as active, so the volume server's graceful HTTP
drain waited out its whole 30s StopTimeout on shutdown - observed as
weed mini taking ~30s to exit in the FUSE integration tests after any
filer->volume traffic.
Track externally-set deadlines, suspend the activity extension while one
is in force, and serialize deadline updates with a mutex. Activity still
extends both directions at once: a long write-only response must keep
the read deadline alive too, or the server's background read would time
out and cancel the in-flight request.
* util: extend read/write deadlines independently when one side is external
A server-configured WriteTimeout keeps an external write deadline in
force for the whole request, which previously suspended the activity
extension entirely - leaving the read deadline stale from before the
request and letting net/http's background read time out mid-response.
Extend each direction independently instead.
Chris LuandGitHub
jayandGitHub
github-actions[bot]
Chris Lu
Cole HelblingandGitHub
qzhelloandGitHub
