* admin: surface lifecycle rule counts in bucket listing
* admin: add bucket lifecycle JSON endpoint
* admin: show lifecycle rules on the buckets page
* admin: make the lifecycle count badge keyboard-accessible
* admin: match buckets empty-state colspan to the column count
* admin: drop stale lifecycle modal responses
* make
* sftp: match permission paths on path-component boundaries
Permission checks compared paths with raw string prefixes, so a
permission entry for /tenants/alice also matched sibling paths such as
/tenants/alice-archive, letting a scoped user read and overwrite
another tenant's files. The home directory containment check had the
same flaw. Route both through pathWithin, which cleans the paths and
requires exact equality or a separator-delimited descendant.
* sftp: clean permission path into a local when ranking matches
* filer: allow clearing a bucket read-only flag stuck after quota removal
Once s3.bucket.quota.enforce marks a bucket read-only, removing or
disabling the quota orphans the flag: enforcement skips buckets with a
non-positive quota, and fs.configure merges booleans with OR so
-readOnly=false could never turn it off. The only way out was deleting
the whole path rule.
- s3.bucket.quota -op=remove/disable and the admin server quota update
now lift the read-only flag on the bucket's path rule
- fs.configure now honors explicitly passed false boolean flags
(-readOnly, -fsync, -worm) instead of OR-merging them away
* filer: ClearBucketReadOnly reports unchanged when the save fails
* fix(rust-volume): remove .ecsum sidecars on EC destroy / shard delete
Rust EcVolume::destroy removed shards and .ecx/.ecj/.vif but left bitrot
checksum sidecars (.ecsum / .ecsum.v*). On clusters that run weed-volume
(not Go weed volume), collection.delete therefore orphans every sidecar
while correctly wiping shards — observed live on 4.39 (14/14 .ecsum
survived after collection.delete on a freshly encoded EC volume).
Go Destroy already calls RemoveBitrotSidecars; this brings Rust to parity:
- hoist remove_bitrot_sidecars into ec_bitrot (shared helper)
- call it from EcVolume::destroy for dir / dir_idx / ecx_actual_dir
- call it from Store::delete_ec_shards when a disk has no remaining shards
- unit test: test_destroy_removes_bitrot_sidecar
* rust volume: gate the shard-delete sidecar sweep on a local shard removal
Only sweep a disk's .ecsum when this delete actually removed a shard file
there, matching Go's found gate: a delete that never touched a disk must not
strip a sidecar it does not own — a shared -dir.idx sibling with surviving
shards, or an ec.rebuild index-prep copy that lands .ecx/.ecsum before any
shard. The shard-presence probe now treats unexpected stat errors as
"exists" so a transient failure cannot orphan-classify live shards, and
check_all_ec_shards_deleted reuses it.
* rust volume: destroy() sidecar sweep needs only the data and idx bases
ecx_actual_dir is always one of the two, so the third branch could never
run; this is now exactly Go Destroy()'s two-base sweep.
* rust volume: call the shared sidecar removal helper directly
* rust volume: unit-test remove_bitrot_sidecars
Mirrors Go's TestRemoveBitrotSidecars: legacy and versioned sidecars are
removed, a shard file and a longer-vid sidecar survive, absent is success.
* rust volume: keep the shared idx-base sidecar while a sibling disk has shards
One -dir.idx serves every location, so emptying one disk must not sweep
<idx>/<vol>.ecsum out from under a sibling that still holds shards. Nothing
reads the idx-base sidecar today, but .ecx shows index-dir files are real;
this keeps the defensive sweep safe if a writer ever lands one there.
* ec shard delete: keep the shared idx-base sidecar while a sibling disk has shards
One -dir.idx serves every disk, so emptying one disk must not sweep
<idx>/<vol>.ecsum out from under a sibling that still holds shards of the
volume — the same gate the Rust volume server applies. A status error counts
as in-use so a transient failure never strips it early.
* rust volume: drop a shard-only disk's stale .vif with the node's last shard
Go's removeEcSharedIndexFiles also clears the data-base .vif in the
all-shards-gone pass, gated on .idx absence so a disk still hosting the
source volume keeps its live .vif; the Rust delete path left it behind.
Unexpected stat errors count as .idx-present so a transient failure never
strips a live volume's .vif.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* filer: scope TUS HEAD/PATCH/DELETE against the session target path
checkTusJwtAuthorization only populated the scoped-path list for POST, so
a prefix-restricted token was scope-checked at session creation but not on
HEAD/PATCH/DELETE, which act on an existing session addressed by id. A
low-privilege tenant holding a valid write token who learns another
upload's session id could PATCH attacker bytes into that session, or
DELETE/HEAD it, landing content at a target path its own AllowedPrefixes
forbids. The session id is unguessable to an unauthenticated attacker but
is not an authorization boundary against a legitimate tenant.
Resolve the effective target for HEAD/PATCH/DELETE from the session's
stored TargetPath and scope the prefix check against it, the same way POST
scopes the create target. The scoped paths are resolved lazily so the
session read runs only for a prefix-restricted token; unrestricted tokens
and deployments without a signing key touch no extra state. An unknown
session stays unscoped so the handler still answers 404.
* filer: fail closed when a TUS session target cannot be resolved
The lazy scope resolver swallowed every readTusSessionInfo error and left
the request unscoped, so a filer read error or a corrupt session .info
would authorize a prefix-restricted token against a target the server
never resolved. Only a genuinely missing session should stay unscoped (so
the handler answers 404); any other failure now propagates and denies the
request. checkJwtAuthorizationScoped's resolver returns an error and a
failed resolution is treated as not-authorized.
* filer: fold the scoped-path resolver into checkJwtAuthorization
checkJwtAuthorization now takes the lazy resolver directly instead of a
separate checkJwtAuthorizationScoped wrapper, and the surrounding comments
are trimmed to the load-bearing why.
* shell: ec.encode batches 10 volumes by default
Encoding a whole collection as one batch means a late failure leaves
everything half-converted. Default -batchSize to 10 so each batch is
encoded, rebalanced, verified, and its originals deleted before the
next starts. -batchSize=0 keeps the all-at-once behavior. Batch
progress messages now print only when there is more than one batch,
so small runs read as before.
* shell: ec.decode decodes in batches, refreshing topology in between
ec.decode walked every volume off the single topology snapshot taken
at startup, which goes stale as earlier decodes move shards around and
create volumes. Decode 10 volumes per batch by default, re-collecting
the topology and rebuilding the free-space accounting between batches.
-batchSize=0 keeps the single-snapshot behavior.
Mounting freshly generated shards notifies the master asynchronously, while the post-encode rebalance plans from a fresh VolumeList snapshot; a snapshot that predates the delta shows zero shards, so the balance plans no moves and silently succeeds, and the original .dat is deleted with all shards on the generation host. Poll until every encoded volume reports a full shard set before balancing — across all disk types, since fresh shards register under the source volume's disk type, and scoped to the newest encode generation so an orphaned older generation neither satisfies the wait nor defeats the clump check. As defense in depth, refuse to delete originals when a volume's shards still sit on one node while another node has free EC slots.
* s3: accept raw semicolons in query strings
Go's url.ParseQuery drops any key=value pair containing a raw ';'. A
presigned PUT that signs content-type carries
X-Amz-SignedHeaders=content-type%3Bhost; when a client or proxy decodes
the %3B, the parameter vanished and the upload failed with
MissingFields, while AWS accepts the raw ';' as query data. Re-encode
it before routing so the pair survives parsing and signature
verification.
* iam, iceberg: recover raw-semicolon query pairs on the other listeners
The standalone IAM API verifies SigV4 with a canonical query recomputed
from the parsed query, and Iceberg REST warehouse/parent values may
legally contain ';'. Move the normalization middleware to util/http and
attach it to both routers.
A raw OIDC token sent as Authorization: Bearer was validated and mapped
to a role through the provider's roleMapping, then authorized against
the role's attached policies without ever consulting the role's trust
policy. A federated user could act as a role that
AssumeRoleWithWebIdentity would refuse to issue a session for with the
same token. Run the same trust-policy validation before returning the
principal on the bearer path.
* fix(filer.backup): stop silently dropping events on transient not-found
Under a write burst, filer.backup could consume a metadata event (advancing
the persisted offset) without replicating the file, with no error logged:
1. filersink CreateEntry/UpdateEntry swallowed replicateChunks errors
(glog.Warningf + return nil), so the offset advanced past entries that
were never written.
2. The manifest-chunk branch of replicateChunks resolved via LookupFileId
with no retry, unlike the data-chunk branch — transient lookup races
dropped exactly the large manifest-backed files while small
inline-content siblings landed.
3. isIgnorable404 matched "LookupFileId" / "volume id ... not found",
misclassifying those races as genuine source 404s at the backup layer.
Fix: on a replicateChunks failure the filer sink now skips only when the
live source has moved past the replayed version (deleted or strictly-newer
mtime) — lossless, a later event carries the current content — and
propagates otherwise so the event is retried. The manifest resolve retries
transient errors like the data-chunk path. isIgnorable404 is narrowed to
genuine 404s; non-filer sinks and the initial-snapshot walk, which relied
on the broad match as their only lossless-skip valve, now make the same
live-source decision (filersink.SourceSupersedes) instead of retrying
forever on a permanently gone volume.
Tests cover propagation of unconfirmed lookup failures, the narrowed 404
classification, and the supersession guards.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(filer.backup): derive supersession path with directory fallback
eventSourceSuperseded built the source path from NewParentPath alone.
Legacy metadata events (persisted by older filers) carry an empty
NewParentPath, so the probe looked up "/<name>", read the miss as
"source gone", and skipped a live file on a transient lookup error —
the silent drop this change is meant to eliminate.
Derive the path via MetadataEventTargetFullPath (the same directory
fallback genProcessFunction uses) and cover both event shapes with
TestEventSupersessionProbe_PathDerivation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(filersink): retry manifest resolve only for transient errors, bounded when unverifiable
The manifest-resolve retry stopped only when hasSourceNewerVersion proved
the source moved past the replayed version, which wedged the sink in two
cases: incremental sinks use dated target keys that cannot map back to a
source path (supersession never provable), and permanent resolve errors
(corrupt manifest data, bad file ids) fail forever while the source entry
stays live.
Gate the retry instead: keep retrying only transient errors (volume-lookup
races, network interruptions), stop after a few attempts when supersession
cannot be checked, and propagate everything else immediately so the
configured metadata error policy applies (-disableErrorRetry included).
Propagation is lossless: filer.backup's fallback decides with the event's
real source key, and both filer.backup and filer.sync re-deliver the event
(RetryForeverOnError) without advancing the offset.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(filer.backup): make error classifiers nil-safe
isIgnorable404, isSourceLookupError, and isTransientResolveError called
err.Error() without a nil guard. All current call sites pass a non-nil
error, but the guard is free and matches isRetryableNetworkError.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* admin: let waiting shell clients win the admin lock
The admin lock manager re-acquired the cluster shell lock immediately
after releasing it, while a waiting weed shell client only polls the
master once a second, so an operator could lose the race indefinitely.
Leave the lock free for slightly more than one poll interval before
re-acquiring, and once overlapping reference-counted holds have kept
the lock continuously held for over a minute, make new admin-side
acquires wait for a full release before piggybacking.
* admin/plugin: take the shell lock per detection and per job, not per batch
The default-lane scheduler acquired the cluster shell lock once and
held it across the whole pass: every due job type's detection plus all
of its dispatched jobs, each drained to completion. A manual weed
shell operation sharing that lock could stall behind the batch for up
to the extended execution window.
Hold the lock only while it protects something: around each detection
scan and around each dispatched job. Manual shell operations now wait
for at most one in-flight job. The admin UI detect+execute path stops
wrapping dispatch in an outer hold, since a nested acquire would
deadlock against the lock manager's fairness window; detection and
per-job dispatch take the lock themselves.
* admin/plugin: stop extending the dispatch window to the largest job estimate
When any proposal's estimated runtime exceeded the remaining
JobTypeMaxRuntime, the whole dispatch context was replaced with a
fresh one capped at eight hours, so a single balance backlog could
hold the default lane (and with it erasure_coding and vacuum
detection) for that long.
Keep the dispatch window at JobTypeMaxRuntime and instead detach each
started attempt onto its own estimated-runtime deadline. Large jobs
still get their full time once started; jobs not yet started when the
window closes are canceled and re-proposed by a later detection, so
sibling job types get a turn every window.
* worker/balance: re-check each planned move against the master before executing
A balance plan is computed at detection time, but the admin lock is
released between detection and execution, so a manual shell operation
can rearrange the volume in the gap. The task's own guards catch a
vanished source, but a target that gained a replica in the meantime
would be silently overwritten by VolumeCopy and the source delete
would then reduce the volume to a single copy.
Before executing each move, ask the master for the volume's current
locations (uncached) and skip the move if the volume has left the
source or the target already holds a replica. Skipped moves fail with
a stale-move error and the next detection replans them. Without master
addresses in the cluster context the check is skipped, preserving the
old behavior with older admins.
* admin: block re-acquire while the final lock release is in flight
Release dropped the manager mutex before calling ReleaseLock, so a
concurrent Acquire could see hold count zero and call RequestLock
while the locker still considered itself locked. That request no-ops,
leaving a hold with no live master lease. Track the in-flight release
and make Acquire wait for it.
Also normalize a nil release function from lock manager
implementations, and make the yield and fairness windows per-instance
fields so tests stop mutating globals.
* rust volume: accept odd-length needle id hex in file ids
Go formats the needle id with strconv.FormatUint and parses it back with
strconv.ParseUint, neither of which pads to an even number of hex digits.
hex::decode rejected such file ids with "Odd number of digits", so
volume.fsck could not purge orphans from a rust volume server. Parse the
needle id and cookie with from_str_radix, matching Go's ParseNeedleId
and ParseCookie.
* storage: emit even-length needle id hex in NeedleId.FileId
volume.fsck and volume.check_disk build purge file ids here with unpadded
FormatUint hex, while every other fid formatter strips whole leading zero
bytes. Pad to even length so the output matches the canonical fid format
and strict hex parsers accept it.
* s3: verify SigV4 against each plausible reverse-proxy host
A portless X-Forwarded-Host leaves the client's true port ambiguous: a
proxy that kept the Host header makes the backend Host port right, one
that rewrote it makes X-Forwarded-Port right, and a client on the
scheme's default port signed no port at all. The verifier bet on the
backend Host port whenever the hostnames matched, so nginx-style
$host/$server_port forwarding got SignatureDoesNotMatch whenever the
proxy and backend share a hostname. Try each plausible host value in
likelihood order instead of guessing one.
* s3: unbracket IPv6 X-Forwarded-Host before matching the request host
net.SplitHostPort strips brackets from the request host, so a bracketed
portless X-Forwarded-Host like [::1] never matched and lost its port
candidate.
* s3: cover unbracketed IPv6 forwarded-host candidates; compare with slices.Equal
TestSharedSnapshotConcurrentIntegrity writes as fast as one core can
marshal for two seconds, so its transient heap scales with host speed.
On linux/386 a fast runner walks the heap into the 4GB address-space
ceiling and the run dies with out of memory. Set a 1GB memory limit for
the duration of the test so the GC paces the writer instead of the
address space; the seal-and-recycle race being tested is unaffected.
* s3: keep listing when empty directories fill the listing window
doListFilerEntries issued a single ListEntries request per directory with
Limit = maxKeys+2. Entries that emit nothing - empty directories, the
.uploads folder, the marker echo - consume that window without consuming
maxKeys, so a bucket whose first window held only empty directories was
reported empty and not truncated, and a subdirectory whose window filled
up silently dropped the entries behind it. Keep requesting from the last
received entry until the quota is filled or a short window shows the
directory is exhausted.
* s3: test listing across windows of empty directories
The test filer client now honors StartFromFileName so repeated windows
advance like the real filer.
* s3: propagate the request context into directory listing RPCs
A disconnected client now cancels the ListEntries streams instead of
letting the listing keep issuing requests against the filer.
* s3: group per-directory listing parameters into a request struct
doListFilerEntries took ten positional parameters; call sites read as
string and bool soup. Wrap the per-directory arguments in
listDirectoryRequest so recursions and tests name what they pass.
* s3: group list request parameters into a struct
listFilerEntries took eight positional parameters ending in two bare
booleans. Wrap them in listObjectsRequest so the V1 and V2 handlers
name what they pass.
Move pickOneReplicaToCopyFrom, pickOneReplicaToDelete,
pickOneMisplacedVolume, and isMisplaced into weed/topology/balancer,
following satisfyReplicaPlacement. Selection functions take the shared
balancer.Replica shape and return indices, so the shell keeps its own
replica type; behavior is unchanged and covered by the existing shell
tests.
* volume.fix.replication: parallelize under-replicated volume copies
Fan out fixOneUnderReplicatedVolume up to -maxParallelization at a time.
Destination selection and free-slot accounting stay atomic behind a
scheduler mutex, so concurrent fixes see each other's reservations; a
failed copy returns its reserved slot. A per-server in-flight cap
(-maxParallelizationPerServer, default 1) keeps many simultaneous copies
from swamping a single destination server; when every eligible
destination is at the cap the fix waits for a slot instead of failing.
* volume.fix.replication: isolate the scheduler's location ordering
Clone allLocations before the per-volume fan-out so the scheduler's
re-sorting cannot alias the slice shared with the concurrent delete
phases, and make the test's per-iteration location copy explicit.
* s3: routed last-version delete removes the emptied .versions directory
The routed versioned delete (routedDeleteSpecificVersion) repoints the
latest pointer and deletes the version file, but unlike the lock-path
fallback (updateLatestVersionAfterDeletion) it never tears down the
.versions/ directory it just emptied. The residue keeps the key's read
path in the self-heal rescan loop: every GET of the deleted key logs
event=surfaced plus a GetObject error until the background
EmptyFolderCleaner gets to the directory — at least two minutes away on
its delay queue, and possibly never (the queue is in-memory, bounded,
and gated on the bucket's allow-empty-folders policy). Veeam's lock
arbitration probes deleted lock keys continuously, so those windows are
always open and the log spam is chronic.
ObjectMutation DELETE gains remove_empty_parent: after the child delete,
the filer best-effort removes the parent directory in the same locked
transaction. Non-recursive on purpose — a concurrent write that lands a
new version fails the removal instead of being lost with it. The routed
last-version delete sets it on the version-file DELETE, matching the
lock-path fallback's contract.
Claude-Session: https://claude.ai/code/session_014mMYAHXZySkCCUfpRFNtSv
* s3: drain empty .versions residue on read heal and in s3.versions.audit
Directories already stranded by pre-teardown deletes (or dropped from
the EmptyFolderCleaner's bounded in-memory queue) previously re-entered
the self-heal rescan on every GET forever: the heal only cleared the
pointer and nothing ever removed the directory, and s3.versions.audit
counted the state as clean.
When the heal rescan finds no remaining version, remove the directory
outright (non-recursive, so orphan children still block and fall back to
the pointer clear) and log event=healed mode=empty_dir_removed; the next
GET takes the clean not-found path. The audit gains an empty category so
the residue is visible, and -heal removes such directories in bulk.
Claude-Session: https://claude.ai/code/session_014mMYAHXZySkCCUfpRFNtSv
Align the usage percentage with s3.bucket.quota.enforce and the admin
UI, which measure quota against the live data size (size minus
un-vacuumed deleted bytes). Also print the logical size so both the
raw and live views stay visible.
* s3: keep empty-folder cleanup out of the multipart .uploads staging tree
The async EmptyFolderCleaner deleted <bucket>/.uploads once it looked
empty. A concurrent CreateMultipartUpload inserting its marker between
the cleaner's emptiness check and the bulk child delete had its row
wiped, so the upload silently vanished: ListMultipartUploads then omits
it and the following part/complete/copy calls fail. Skip the .uploads
subtree in both the queue and the delete path (including the eager
parent cascade); the multipart upload lifecycle owns it.
* s3: slice the bucket-relative path instead of reallocating it
* volume.balance: rank by physical disk usage
* volume.balance: keep -byDiskUsage ranking on one scale across the fleet
A server that does not report disk bytes ranks at whole volume-equivalents
while reporting servers stay below 1.0, so during a rolling upgrade the
balancer drains nearly empty old-build servers onto physically fuller ones.
Decide the scale once: rank by physical used percent only when every server
reports disk bytes, otherwise fall back to the data-size ranking for all.
Normalizing the fallback by maxVolumeCount instead would reintroduce the
over-configured-maxVolumeCount distortion this flag exists to avoid.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Drives the full path: AssumeRoleWithWebIdentity embeds the claims in the
session JWT, AuthenticateJWT restores them, and AuthorizeAction evaluates
ForAnyValue:StringEquals on jwt:groups / jwt:roles per bucket. The mock
OIDC provider now carries token claims through and surfaces roles,
matching the real provider.
* iam: surface OIDC groups into STS session request context for resource-policy ABAC
The groups claim was added to ExternalIdentity.Groups but excluded from Attributes
(processedClaims), so it never reached the session RequestContext. As a result
group membership was usable only in role trust policies (assume-time), not in
resource/permission policies (request-time) - so a single role could not scope
access by the caller's groups (aggregate ABAC). Surface Groups as a []string in
the request context; the string-condition evaluator already handles multi-valued
context keys, and the S3 middleware exposes it as jwt:groups.
* iam: also surface OIDC roles into the STS request context (companion to groups)
The 'roles' claim is excluded from the OIDC attributes by the same processedClaims
set that excluded 'groups', so it never reached the session request context and was
usable only via provider-configured role mapping - not a raw token roles claim. Add
ExternalIdentity.Roles, populate it from the token's roles claim, and surface it as a
[]string in the request context so resource policies can gate on jwt:roles, exactly
as the previous commit did for jwt:groups.
* log_buffer: share one window snapshot across all subscriber reads
Every in-memory read handed each subscriber a private pooled copy of the
window it wanted, so N subscribers reading the same data cost N copies of
up to 8MB each -- and slow consumers (grpc send backpressure) held those
copies live for their whole iteration. With hundreds of mount subscribers
that multiplied into gigabytes of live heap on the filer.
Share the bytes instead of copying per reader:
- Sealed windows get a lazily created GC-owned snapshot, made once by the
first reader and handed out zero-copy to the rest. The snapshot travels
with its window when SealBuffer shifts slots, so recycling the sealed
array never invalidates it.
- The current window keeps a shared snapshot of its append-only prefix
buf[:pos], extended on demand; each byte is copied once per window
(writer-rate-bound) instead of once per reader. At seal a fully
extended prefix becomes the sealed window's snapshot.
ReadFromBuffer now reports whether the returned buffer is a pooled copy
(flush path) or a shared view that must not be released; the read loops
only recycle pooled buffers.
With 200 subscribers consuming at grpc pace over sealed and current
windows, peak live heap drops from 5.2GB to 178MB.
* log_buffer: clear released read buffer so a panic cannot double-free it
The read loops release the previous iteration's pooled buffer and then
call ReadFromBuffer. If that call panicked before reassigning bytesBuf,
the deferred cleanup would put the same buffer into the pool a second
time, letting two future readers share one backing array. Nil the
pointer at the release site so the defer sees nothing to free.
* java client: fail cleanly when a chunk read returns short or empty data
readChunkView copied chunkView.size bytes out of the fetched chunk without
checking the fetched length, so an empty body (seen transiently right after an
append) threw an opaque IndexOutOfBoundsException. Retry empty chunk fetches and
bound the copy to the data actually returned.
* overflow-safe bounds check and null guard on fetched chunk data
LoadRemoteFile now takes dataFileAccessLock (so a live tier upload does
not race the heartbeat's DataBackend read). But load() also runs it, and
CommitCompact calls load() while already holding that lock, so reloading
a remote-tiered volume during a compaction commit re-enters the
non-reentrant lock and deadlocks.
Split the locked bodies out: swapDataBackendLocked and loadRemoteFileLocked
assume the caller holds dataFileAccessLock. load() uses loadRemoteFileLocked;
the public LoadRemoteFile keeps taking the lock for the live tier-upload
handler that does not hold it.
* log_buffer: read only ts_ns during metadata-read binary search
readTs ran a full proto.Unmarshal of each LogEntry just to compare
timestamps during ReadFromBuffer's binary search, allocating fresh
slices for the data and key byte fields on every probe. Under
metadata-subscription fan-out this decode churn dominated allocations
and drove heavy GC, inflating filer RSS well past the live heap.
Scan the wire format for ts_ns (field 1) instead and skip the data/key
payloads without copying them. readTs is now zero-alloc; the binary
search no longer touches the event payloads at all.
* log_buffer: alias event payloads instead of copying them on delivery
The subscribe read loops decoded each LogEntry with a full proto.Unmarshal,
allocating fresh slices for the data and key byte fields on every entry
(protobuf consumeBytesNoZero). When many metadata subscribers each re-read
the in-memory log window under a reconnect storm, that per-entry copy was a
large share of allocation churn -- and for entries a subscriber filters out
by prefix, the copied payload was never even looked at.
Decode with a wire scan that points data/key at sub-slices of the source
buffer instead of copying. The aliased slices are valid only for the
duration of the eachLogEntryFn callback; every current subscriber either
re-decodes the event into its own struct or hands it to a synchronous grpc
Send, so none retain them. Per delivered entry the loop decode drops from
176 B / 2 allocs to zero.
VolumeTierMoveDatFromRemote downloads the .dat, trims the .vif, and swaps
the data backend to the local file, but left hasRemoteFile set. The
volume.tier.download command masks this by unmounting and remounting
right after, which reloads the flag from the trimmed .vif — but in the
window before the remount the in-memory flag is wrong: doDeleteRequest
would skip appending the tombstone to the freshly local .dat, and the
phantom-.dat guard stays disabled.
Give SwapDataBackend a hasRemoteFile argument so the backend swap and the
flag move together under one lock, and route both tier directions through
it: the tier-down download passes false, LoadRemoteFile passes true. The
flag can no longer disagree with the live backend.
volume: keep tier-uploaded volume reporting to master
A live volume.tier.upload removes the local .dat and swaps the data
backend to remote, but v.hasRemoteFile was only ever set when a volume
is loaded from disk, so the running volume kept it false. The phantom
.dat guard then saw fileCount>0, !HasRemoteFile, and a missing .dat, and
stopped reporting the volume to the master. The volume vanished from the
topology even though the upload succeeded and the data was in cloud
storage.
Set hasRemoteFile in LoadRemoteFile, the single point where a volume's
backend becomes remote, so it is true both on disk-scan load and after
an in-process tier upload. Route the backend reassignment through
SwapDataBackend so it happens under dataFileAccessLock, closing the old
backend and never racing the heartbeat's concurrent DataBackend read.
Make the field atomic since the heartbeat now reads it concurrently with
the tier-upload handler that writes it.
* java: HTTP Basic Auth for filer behind an Nginx reverse proxy
Attach an "Authorization: Basic" header to the filer gRPC channel through a
ClientInterceptor and to the volume read/write HTTP requests, so the Java
client can reach SeaweedFS fronted by Nginx with auth_basic enabled.
Credentials come from a [basic_auth] section in security.toml or from
FilerSecurityContext.setBasicAuth(). On writes the volume JWT still owns the
Authorization header when present, since the two cannot share it.
* java: precompute Basic Auth header and warn on plaintext channel
Cache the Base64 Authorization value once instead of re-encoding on every
chunk read/write, and log a one-time warning when Basic Auth rides a
plaintext gRPC channel where the credentials would travel in cleartext.
* s3: fail over routed object writes when the owner filer is unreachable
A routed object write (multipart completion, PUT, delete, versioned
finalize, metadata replace) dialed the ring-selected owner filer
directly with no failover. After a filer restarts onto a new address the
lock ring can still name the old one, so every routed write hangs on the
dead address until the gateway is restarted; CompleteMultipartUpload in
particular exceeds client timeouts.
Route the transaction through withFilerClientFailover, skipping an owner
that recently failed, so a live filer forwards it to the real owner by
route_key. Mirrors the read path's getObjectEntryRoutedByKey.
* s3: fail over bucket-config writes when the owner filer is unreachable
patchBucketEntry dialed the bucket's ring owner directly, so a restarted
filer's stale ring address hung every bucket-config write (versioning,
lifecycle, object lock, ownership, ACL, policy, CORS) - the same failure
as routed object writes. Route it through objectTxnOnFiler so it skips an
unreachable owner and a live filer forwards by route_key.
* Optimize s3api encodePath to eliminate O(n^2) allocations
encodePath built its result via string concatenation in a loop, which is
O(n^2) in allocations. For non-ASCII (e.g. Chinese) runes it additionally
called make + hex.EncodeToString + strings.ToUpper per byte (~10 allocations
per character). Under high QPS with long non-ASCII object keys this produced
a very high allocation rate, frequent GC and long GC pauses, causing S3
request latency spikes.
Replace with a preallocated strings.Builder, a manual hex lookup table, and a
zero-allocation fast path. EncodePath now delegates to encodePath to remove
the duplicated implementation. Output is byte-for-byte identical, verified by
TestEncodePath and TestEncodePathEqual.
Benchmark (long Chinese path): 261 -> 1 allocs/op, 35810 -> 480 B/op, 7.5x faster.
Load test (50M calls): 212x fewer allocations, 76x fewer GC cycles.
* s3api: drop regexp from encodePath fast path
The unreserved-character scan already decides whether any byte needs
encoding, so reservedObjectNames.MatchString was a redundant second pass
that also ran the RE2 engine on every authenticated request. Rely on the
scan alone; output is unchanged. The ASCII fast path drops from ~188ns to
~11ns per call.
---------
Co-authored-by: LiuDoge <liudoge@LiuDogedeMacBook-Air.local>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* filer: require JWT authorization on TUS upload endpoints
filerHandler and readonlyFilerHandler run every request through
maybeCheckJwtAuthorization, but the TUS routes were registered only
behind filerGuard.WhiteList, which is a pass-through for the filer (its
guard is built with an empty whitelist), and no TUS handler called the
JWT check. So with jwt.filer_signing.key set, normal PUT/POST/DELETE
were authorized while the TUS endpoints were not.
Run the same check in tusHandler before routing: HEAD uses the read key,
POST/PATCH/DELETE use the write key, and creation scopes a
prefix-restricted token against the resolved target path. OPTIONS stays
open for capability discovery.
* filer: enforce read-only and WORM rules on TUS completion
completeTusUpload wrote the final entry with CreateEntry directly,
skipping the read-only and WORM checks the normal write path applies.
Reject completion when the target prefix is read-only or the existing
entry at the target is WORM-enforced.
* filer: align TUS write path with the normal write path
Resolve the create target with a guaranteed leading slash so a
prefix-restricted token and the stored path stay absolute even if
TusBasePath were set with a trailing slash. Reject read-only prefixes at
session creation before any chunk is written, and map read-only and WORM
rejections at completion to 507 and 403 instead of a generic 500.
* filer: cover method-restricted TUS tokens in the auth test
With replication 001 and one node down, ~2/3 of volumes have their
replica on the dead node. The primary write lands locally but the
replica forward fails, so the volume returns 500 "failed to write to
replicas". #9744 made that upload fail fast so the client re-assigns,
but the client retry never fired: the reassign gate matched a fixed
list of error substrings that didn't include this one. The mount
surfaced I/O error and dropped the chunk, leaving missing lines and
null-byte gaps in an append workload while a node rebooted.
Decide reassignment by HTTP status instead of matching message text.
On the write path a volume server only 5xxs on a ReplicatedWrite
failure (local disk, replica peer down, under-replication) — all of
which a different volume dodges — so any 5xx reassigns; a no-response
transport failure (the assigned target itself is down) reassigns too;
a 4xx is a genuine client error and is surfaced. doUploadData tags its
errors with the response status via uploadStatusError, and the gate
moves from util.MultiRetry(errList) to util.RetryOnError(predicate).
This drops the fragile substring list (and the message-prefix
constants and guard test it needed); store_replicate.go is untouched.
Chris Lu
Eliah Rusin
Jaehoon Kim
github-actions[bot]
Lars Lehtonen
qzhello
Aleksey
Chris Lu
Chris Wydra
jay
dependabot[bot]
dongle-code