The maintenance operations built manifest, manifest-list, data-file and
metadata-log paths with a bare path.Join("metadata", ...), so a single
maintenance run wrote scheme-less relative paths into the new snapshot,
the metadata-log and the table xattr. SeaweedFS reads those back fine
because normalizeIcebergPath accepts both forms, but strict readers
resolve every location through S3FileIO and fail with "Invalid S3 URI,
cannot determine scheme", leaving the whole table unreadable after any
maintenance commit.
Add absoluteIcebergPath, the inverse of normalizeIcebergPath, and apply
it at every site that authors locations: rewrite_manifests, compact,
rewrite_position_delete_files, and the shared commit path. The base is
derived from the bucket and table path since that is where the worker
physically writes, matching the locations the REST catalog generates.
The pre-move stale check compared the master's location urls (host:port)
literally against the proposal's node addresses, which carry the grpc
suffix (host:port.grpcPort). Every move failed as "stale move: volume no
longer on source" even though the volume was still there. Normalize both
sides through pb.ServerAddress before comparing.
* regenerate master_grpc.pb.go with protoc-gen-go-grpc v1.6.2
The other generated pb files are already on v1.6.2; this one was stale.
* shell: keep unlock from racing the lease renewal
A renewal RPC in flight while ReleaseLock runs re-creates the lock on the
master after the release deletes it, and can blank the client name if the
renewal reads it mid-release. The stale-token release is then ignored, so
the lock stays held (sometimes anonymously) until it expires. Serialize
the renew and release RPCs, and set the client name before flipping
isLocked so the renewal never sends a partial acquisition.
* shell: restart lease renewal after a failed renewal
The renewal goroutine exits on error but never cleared its running flag,
so later locks in the same process were never renewed and silently
expired after ten seconds.
* shell: show who holds the cluster lock
A blocked lock command gave no hint that another client holds the lock
(the refusals only surfaced at -v=2), and cluster.status reported the
shell's own lock state as if it were the cluster's. Add a
GetAdminLockStatus RPC to the master so lock prints the holder before
blocking and cluster.status shows the actual cluster-wide holder. Both
degrade silently against masters without the RPC.
* shell: bound admin lock RPC attempts with timeouts
The lease, renew, release, and holder-status calls all ran without a
deadline, so an unresponsive master could hang the renewal goroutine,
an unlock (which now waits on the renewal mutex), or the shell prompt.
Give each attempt its own short context; the retry loops still resolve
a fresh leader on the next try.
* master: reject admin token release on non-leaders
A follower holds no lock state, so it answered a release with success
while the leader kept the lock until expiry. Refuse like LeaseAdminToken
does so the client can try the leader instead.
* shell: leave the lock release call unbounded
A release cut short by a deadline leaves the lock held on the master
until it expires, so a slow master would turn every unlock into a
ten-second ghost lock. Restore the single fire-and-forget attempt;
the timeouts stay on the lease and renew paths, where a stalled call
forfeits the lease anyway.
* shell: release only the token unlock started with
A RequestLock racing a slow release (the admin presence lock does this
on shutdown) could have its freshly acquired token sent in the release
request or zeroed by the trailing stores. Capture the token once under
the mutex and compare on clear so a concurrent acquisition survives an
in-flight unlock.
Reads of remote-backed entries now record hit or miss in
SeaweedFS_remote_cache_read_total{source,bucket,result} on the filer HTTP
path and the S3 gateway, so cache effectiveness of mounted buckets can be
graphed. Inline-content entries count as hits since they are served
locally without chunks. The filer purges the per-bucket series when the
bucket directory is deleted, so a standalone filer does not accumulate
series across bucket delete/recreate churn.
Only the heartbeat path read the .dat mtime; collectStatForOneVolume left
the field at 0, so /status consumers could not tell how long a volume had
been idle.
The initiator's shed check (initiatedGrow != HasGrowRequest) compares
against an err from a PickForWrite that may predate the growth
concluding: the grower registers its volumes before clearing the flag,
so when growth lands between the failed pick and the check, the assign
shed ResourceExhausted even though a writable volume was already
registered. Re-pick once after observing the conclusion and shed only
if the volume layout still has nothing writable. Applies to both the
gRPC Assign and the HTTP dirAssign paths, which share the shed logic.
Flaked in CI as TestAssignInitiatorWaitsForItsOwnGrowth; reproduced
deterministically by widening the enqueue-to-check window.
* fix: report short S3 ReaderAt reads
Problem: S3BackendStorageFile.ReadAt returned a short buffer with a nil error, hiding truncated remote data.
Root cause: Every terminal io.EOF was cleared regardless of how many bytes were read.
Fix: Clear io.EOF only when the requested buffer was completely filled.
Validation: go test ./weed/storage/backend/...
Co-authored-by: Codex <noreply@openai.com>
* fix: validate S3 ReaderAt requests
Co-authored-by: Codex <noreply@openai.com>
* s3 ReadAt: simplify negative offset error
* s3 ReadAt: stop reading once the buffer is full
Skips the extra Read that only collected the terminal EOF, and bounds
the loop if the server ignores the Range header and returns more data
than requested, where Read on an empty slice can spin forever.
* s3 ReadAt: cover full reads that arrive with io.EOF
---------
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix: reject overflowing needle ID deltas
Problem: Parsing a file ID with a delta can wrap a valid maximum needle ID back to zero without returning an error.
Root cause: Needle.ParsePath added the parsed uint64 delta without checking whether the sum exceeded the needle ID range.
Fix: Compare the delta with the remaining uint64 capacity before addition and return a contextual overflow error when it does not fit.
Validation: go test ./weed/storage/needle -run ^TestNeedleParsePathRejectsDeltaOverflow$ -count=1; go test ./weed/storage/needle -count=1; git diff --check 10cdaf381875492a2c752d1038797e96ff18208f..HEAD
Co-authored-by: Codex <noreply@openai.com>
* fix: propagate needle ID delta parse errors
Co-authored-by: Codex <noreply@openai.com>
* print the needle id in hex in the delta overflow error
* batch delete: keep processing after a cookie mismatch
* rust volume: reject overflowing needle id deltas
---------
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
When every volume is full, a chunk upload fails only after the assign
retry budget, the failed chunk's data is dropped, and the mount kept
accepting writes anyway. cp would crawl for hours pushing the rest of
the file through a pipeline that could not persist it, and only close()
reported an error - a generic EIO.
Poison the file handle on the first failed chunk upload so subsequent
writes fail immediately, and map "no writable volumes" / "no free
volumes" upload errors to ENOSPC so the writing process aborts with
"No space left on device". Also guard lastErr with a mutex: it was
written concurrently by uploader goroutines, and keep the first error
so later failures do not mask the root cause.
* remote storage: carry Content-Encoding into mounted entries
A RemoteEntry now records the remote object's Content-Encoding, and every
path that materializes a local entry from remote metadata (lazy fetch, lazy
listing, remote.mount, remote.meta.sync, remote.cache) stamps it into the
entry extended attributes, so HTTP and S3 HeadObject/GetObject return the
header. GCS and Azure populate it on listing and stat; S3 only exposes it
via HeadObject, so listings leave it empty.
* remote storage: set Content-Encoding when uploading to the remote
An entry carrying Content-Encoding in its extended attributes (a native S3
upload, or a value pulled from the remote) now keeps it when
filer.remote.sync or remote.copy.local writes the object to GCS, S3, or
Azure, instead of silently dropping it.
* gcs: read remote objects without decompressive transcoding
GCS transparently decompresses gzip-encoded objects on download, which
ignores range requests and returns byte counts that disagree with the
tracked RemoteSize. Request the stored bytes instead; chunked reads of
gzip-encoded objects then behave like any other object.
* remote storage: track Content-Encoding presence so removals propagate
A listing that does not report encodings (S3) leaves the field unset and
the local header untouched, while an authoritative report of no encoding
(GCS, Azure, any stat) now clears a previously stamped header instead of
leaving it stale. remote.cache also schedules a metadata update when only
the reported encoding changes.
* remote storage: propagate Content-Encoding on metadata-only updates
filer.remote.sync routes same-content changes through UpdateFileMetadata,
which only touched custom metadata (GCS, Azure) or tags (S3), so a
Content-Encoding change in the extended attributes never reached the
remote object's real header. GCS now patches contentEncoding alongside
the metadata, and Azure reissues the blob's HTTP headers with the new
value, carrying the others over since the call replaces the full set.
S3 stays tags-only: changing the header there means rewriting the
object, which the sync already does whenever content changes.
* remote.meta.sync: optional per-file stat for listing-omitted metadata
S3 listings carry no Content-Encoding, so entries synced from them never
learn it and the lazy-stat path never runs once an entry exists. With
-statFiles, each new or changed file whose listing left the encoding
unreported is stat-ed before reconciling, and the stat-derived value is
persisted so the next run only stats files that changed. Off by default:
it costs one remote request per file, and GCS and Azure listings already
carry the encoding.
* s3: apply metadata-only Content-Encoding changes with an in-place copy
Content-Encoding is S3 system metadata, so the tags-only metadata update
silently left the object's real header untouched. When the encoding
differs, reissue the object as a self-copy with replaced metadata,
carrying the content type and configured storage class like a fresh
write does. CopyObject caps at 5 GiB; beyond that the change is logged
and applies on the next content write.
* azure: skip the metadata call when user metadata is unchanged
An encoding-only change reissues the blob's HTTP headers; sending the
unchanged user metadata alongside it wastes a round trip and bumps the
blob's ETag once more than needed.
* s3: carry existing object metadata through the encoding copy
The replace directive drops everything not resent, and a mounted entry
usually has no local mime or user metadata, so the in-place copy wiped
the object's Content-Type, Cache-Control, user metadata, encryption
settings, and storage class. Read them back with a HeadObject first and
carry them over, overriding only what SeaweedFS manages: the encoding,
a locally set mime, and the configured storage class. S3 reports
Expires as a string while the copy input wants a time, so it is parsed
and skipped when malformed.
* generate vtproto marshalers for filer_pb and use them on the metadata log path
Reflection-based proto.Unmarshal allocates a fresh message tree through
reflect.New on every call. On the metadata subscription fan-out the same
event is decoded once per subscriber, so reflect.New tops the decode
churn under many mounts.
Generate MarshalVT/UnmarshalVT/SizeVT for filer.proto (a separate
filer_vtproto.pb.go, filer.pb.go untouched) and call them on the log
entry marshal and the subscribe/replay decode paths. UnmarshalVT
allocates message structs directly and copies byte and string fields, so
it stays wire-compatible with proto.Unmarshal and preserves the
non-aliasing the persisted-log cache depends on.
For SubscribeMetadataResponse this cuts decode allocations 69 -> 50 and
~4.5us -> ~2.1us per event; the win scales with subscriber overlap.
* marshal log entries directly into the buffer
SizeVT is allocation-free and MarshalToSizedBufferVT writes into a
pre-sized slice, so the log entry can be marshaled straight into
logBuffer.buf. This drops the per-entry MarshalVT allocation and the
follow-up copy on the write path.
* expand vtproto benchmarks: marshal, decode, and marshal-into-buffer by chunk count
Parametrize by nested-message count (chunks per event) and add encode +
zero-alloc marshal-into-buffer benchmarks alongside the decode one, so
the write-path win from MarshalToSizedBufferVT is measurable too.
* keep proto.Unmarshal for metadata events to preserve UTF-8 validation
UnmarshalVT skips proto3's UTF-8 validation of string fields, so a
SubscribeMetadataResponse with an invalid-UTF-8 string (e.g. Directory
"\xff") that proto.Unmarshal rejects would decode and reach path
filtering and subscribers. Decode events with proto.Unmarshal again;
UnmarshalVT stays on the log entry paths, whose only variable-length
fields are bytes and so carry no UTF-8 constraint.
Tests cover the codec difference and that a malformed event is skipped
before delivery.
Put/DeleteObjectTagging set the update directory to the bucket root for a
null version, so a tag change on allowed/protected.txt landed on
protected.txt at the bucket root instead. A principal scoped to one
nested key could overwrite a different object sharing the basename, the
same class of scope bypass fixed for PutObjectAcl. Share the object's
parent-directory resolver across both paths.
* route log buffer flush copies through the shared slab pool
Each flush copied the sealed window into a bytes.Buffer drawn from a
package-local sync.Pool. GC drains that pool, so once collections run
often (e.g. under GOMEMLIMIT) most flushes miss and Write grows a fresh
window-sized array, the dominant bytes.growSlice source under write load.
Copy into a size-classed slab from weed/util/mem instead. Slabs are
reused process-wide and returned to their exact size class after the
flush, so variable-sized flushes across many partitions stop churning
mismatched buffers.
* guard nil slab in flush releaseMemory
mem.Free(nil) resolves to the smallest slot pool and stores a zero-cap
slice, so a later mem.Allocate could return nil and panic. The current
call sites never pass nil, but the guard keeps a double release harmless.
* Fix PutObjectAcl writing back to the wrong object for nested keys
PutObjectAcl set the update directory to the bucket root, so an ACL
change on allowed/protected.txt landed on protected.txt at the bucket
root instead. A principal scoped to one nested key could overwrite a
different object sharing the basename. Target the object's own parent
directory.
* Test object ACL update directory resolution for nested keys
* Introduce weed shell command `ec.check.replication`.
This command performs a quick check of EC volume shard replication, reporting
volumes whose shards are over- or under-replicated. Each volume is checked
against its own data+parity ratio, obtained via
erasure_coding.EcShardsVolume{Data,Parity}Shards, so builds that derive the EC
ratio per volume report custom ratios correctly.
The name follows the shell's convention (cluster.check, volume.check.disk); the
closest normal-volume counterpart is volume.fix.replication.
* shell: ec.check.replication reports mixed under+over-replication in both lists
StatFs on a mount reported cluster-wide total capacity but used size from
a single volume layout keyed by collection, replication, ttl, and disk
type. A mount without -collection therefore showed only the default
collection's usage, hiding data in named collections, and even a
collection-scoped mount missed volumes with a different replication,
ttl, or disk type.
Aggregate used size and file count across all layouts of the requested
collection, and across every collection when the collection is empty,
matching how Topology.Lookup treats an empty collection. Looking up
stats no longer creates a phantom collection as a side effect.
An absent needle is normally indistinguishable from a vacuumed delete,
so check.disk skips it and replicas diverged by replication failures
cannot be reunited. But a replica with compaction revision 0 has never
been vacuumed: every delete it processed still holds its tombstone, so
a needle absent there is provably a missing write.
The new flag resurrects absent needles in exactly that case. The
receiving replica's live compaction revision is read after the index
snapshot and must be 0; vacuumed replicas keep the skip, now with the
revision in the message. The decision is passed per direction since
pass 2 runs pairs concurrently. Resurrected needles count toward
-nonRepairThreshold as before.
* filer: authorize TUS existing-session verbs against the validated stored target
Scope-check on TUS HEAD/PATCH/DELETE only populated a resource path for POST,
so a prefix-restricted token that learned another tenant's session id could act
on that session and land content at a TargetPath its own AllowedPrefixes forbid.
Split the filer JWT check into authenticateFilerJwt (signature and method) and
authorizeFilerJwtPaths (resource scope), and make the scope check fail closed:
a prefix-restricted token with no resolved resource path is denied instead of
authorized on signature alone. The TUS handler now authenticates first, reads
and validates the session once, authorizes the stored TargetPath, then operates
on that single pinned snapshot. readTusSessionInfo rejects a session whose id,
target or size is unusable, and getTusSession is split so the authorization
lookup no longer lists chunks.
* filer: reject non-canonical TUS upload ids
The uploads route took the first path component as the session id, so a trailing
path or other non-canonical spelling aliased one session under several URLs.
Require the id to be a canonical UUID, the only form the server mints, both at
routing and when reading a session's metadata, so one URL maps to one resource.
* filer: revalidate the pinned TUS session before completing an upload
Completion re-read chunks but not the session identity, so a PATCH finishing
after a concurrent DELETE or metadata replacement could still land at the id's
stored path. Before completing, confirm the session still exists and its target,
size and creation time are unchanged from the authorized snapshot; otherwise the
completion fails instead of writing to a path the request never authorized.
* filer: log TUS session lookup failures before returning not-found
readTusSessionInfo and loadTusSessionChunks failures answered "not found" with
no log line, so a transient filer or listing error was indistinguishable from a
genuinely missing session. Log the lookup at V(1) (a missing session is common
and benign) and the chunk-load error at Errorf (the session already resolved).
* volume: copying a remote-backed volume only needs space for the index
VolumeCopy sized its target-location check by the source .dat even when
that .dat lives in a cloud tier and only .idx/.vif land locally, so
re-replicating a tiered volume demanded the full remote size in free
disk. Require the index size instead.
* shell: volume.tier.upload keeps volume replicas
Tiering a replicated volume deleted every replica but the upload
source, leaving one server holding the only .idx and the only .vif
that knows the remote object key — losing that server orphaned the
volume even though its data sat intact in the cloud.
Replicate the uploaded .idx/.vif onto the other replica servers
instead (VolumeCopy skips the .dat for remote-backed volumes), so all
replicas serve reads from the same remote object and the volume keeps
its replica count. An already-tiered replica is preferred as the
upload source, so a rerun after a partial failure reuses the existing
remote object instead of uploading a second copy under a new key.
* shell: group tier upload locations instead of re-prepending
* rust volume: copying a remote-backed volume only needs space for the index
Mirror the Go VolumeCopy change: size the free-location check by the
source .idx when the .dat lives in a cloud tier, since only .idx/.vif
land locally.
* admin: surface lifecycle rule counts in bucket listing
* admin: add bucket lifecycle JSON endpoint
* admin: show lifecycle rules on the buckets page
* admin: make the lifecycle count badge keyboard-accessible
* admin: match buckets empty-state colspan to the column count
* admin: drop stale lifecycle modal responses
* make
* sftp: match permission paths on path-component boundaries
Permission checks compared paths with raw string prefixes, so a
permission entry for /tenants/alice also matched sibling paths such as
/tenants/alice-archive, letting a scoped user read and overwrite
another tenant's files. The home directory containment check had the
same flaw. Route both through pathWithin, which cleans the paths and
requires exact equality or a separator-delimited descendant.
* sftp: clean permission path into a local when ranking matches
* filer: allow clearing a bucket read-only flag stuck after quota removal
Once s3.bucket.quota.enforce marks a bucket read-only, removing or
disabling the quota orphans the flag: enforcement skips buckets with a
non-positive quota, and fs.configure merges booleans with OR so
-readOnly=false could never turn it off. The only way out was deleting
the whole path rule.
- s3.bucket.quota -op=remove/disable and the admin server quota update
now lift the read-only flag on the bucket's path rule
- fs.configure now honors explicitly passed false boolean flags
(-readOnly, -fsync, -worm) instead of OR-merging them away
* filer: ClearBucketReadOnly reports unchanged when the save fails
* fix(rust-volume): remove .ecsum sidecars on EC destroy / shard delete
Rust EcVolume::destroy removed shards and .ecx/.ecj/.vif but left bitrot
checksum sidecars (.ecsum / .ecsum.v*). On clusters that run weed-volume
(not Go weed volume), collection.delete therefore orphans every sidecar
while correctly wiping shards — observed live on 4.39 (14/14 .ecsum
survived after collection.delete on a freshly encoded EC volume).
Go Destroy already calls RemoveBitrotSidecars; this brings Rust to parity:
- hoist remove_bitrot_sidecars into ec_bitrot (shared helper)
- call it from EcVolume::destroy for dir / dir_idx / ecx_actual_dir
- call it from Store::delete_ec_shards when a disk has no remaining shards
- unit test: test_destroy_removes_bitrot_sidecar
* rust volume: gate the shard-delete sidecar sweep on a local shard removal
Only sweep a disk's .ecsum when this delete actually removed a shard file
there, matching Go's found gate: a delete that never touched a disk must not
strip a sidecar it does not own — a shared -dir.idx sibling with surviving
shards, or an ec.rebuild index-prep copy that lands .ecx/.ecsum before any
shard. The shard-presence probe now treats unexpected stat errors as
"exists" so a transient failure cannot orphan-classify live shards, and
check_all_ec_shards_deleted reuses it.
* rust volume: destroy() sidecar sweep needs only the data and idx bases
ecx_actual_dir is always one of the two, so the third branch could never
run; this is now exactly Go Destroy()'s two-base sweep.
* rust volume: call the shared sidecar removal helper directly
* rust volume: unit-test remove_bitrot_sidecars
Mirrors Go's TestRemoveBitrotSidecars: legacy and versioned sidecars are
removed, a shard file and a longer-vid sidecar survive, absent is success.
* rust volume: keep the shared idx-base sidecar while a sibling disk has shards
One -dir.idx serves every location, so emptying one disk must not sweep
<idx>/<vol>.ecsum out from under a sibling that still holds shards. Nothing
reads the idx-base sidecar today, but .ecx shows index-dir files are real;
this keeps the defensive sweep safe if a writer ever lands one there.
* ec shard delete: keep the shared idx-base sidecar while a sibling disk has shards
One -dir.idx serves every disk, so emptying one disk must not sweep
<idx>/<vol>.ecsum out from under a sibling that still holds shards of the
volume — the same gate the Rust volume server applies. A status error counts
as in-use so a transient failure never strips it early.
* rust volume: drop a shard-only disk's stale .vif with the node's last shard
Go's removeEcSharedIndexFiles also clears the data-base .vif in the
all-shards-gone pass, gated on .idx absence so a disk still hosting the
source volume keeps its live .vif; the Rust delete path left it behind.
Unexpected stat errors count as .idx-present so a transient failure never
strips a live volume's .vif.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* filer: scope TUS HEAD/PATCH/DELETE against the session target path
checkTusJwtAuthorization only populated the scoped-path list for POST, so
a prefix-restricted token was scope-checked at session creation but not on
HEAD/PATCH/DELETE, which act on an existing session addressed by id. A
low-privilege tenant holding a valid write token who learns another
upload's session id could PATCH attacker bytes into that session, or
DELETE/HEAD it, landing content at a target path its own AllowedPrefixes
forbids. The session id is unguessable to an unauthenticated attacker but
is not an authorization boundary against a legitimate tenant.
Resolve the effective target for HEAD/PATCH/DELETE from the session's
stored TargetPath and scope the prefix check against it, the same way POST
scopes the create target. The scoped paths are resolved lazily so the
session read runs only for a prefix-restricted token; unrestricted tokens
and deployments without a signing key touch no extra state. An unknown
session stays unscoped so the handler still answers 404.
* filer: fail closed when a TUS session target cannot be resolved
The lazy scope resolver swallowed every readTusSessionInfo error and left
the request unscoped, so a filer read error or a corrupt session .info
would authorize a prefix-restricted token against a target the server
never resolved. Only a genuinely missing session should stay unscoped (so
the handler answers 404); any other failure now propagates and denies the
request. checkJwtAuthorizationScoped's resolver returns an error and a
failed resolution is treated as not-authorized.
* filer: fold the scoped-path resolver into checkJwtAuthorization
checkJwtAuthorization now takes the lazy resolver directly instead of a
separate checkJwtAuthorizationScoped wrapper, and the surrounding comments
are trimmed to the load-bearing why.
* shell: ec.encode batches 10 volumes by default
Encoding a whole collection as one batch means a late failure leaves
everything half-converted. Default -batchSize to 10 so each batch is
encoded, rebalanced, verified, and its originals deleted before the
next starts. -batchSize=0 keeps the all-at-once behavior. Batch
progress messages now print only when there is more than one batch,
so small runs read as before.
* shell: ec.decode decodes in batches, refreshing topology in between
ec.decode walked every volume off the single topology snapshot taken
at startup, which goes stale as earlier decodes move shards around and
create volumes. Decode 10 volumes per batch by default, re-collecting
the topology and rebuilding the free-space accounting between batches.
-batchSize=0 keeps the single-snapshot behavior.
Mounting freshly generated shards notifies the master asynchronously, while the post-encode rebalance plans from a fresh VolumeList snapshot; a snapshot that predates the delta shows zero shards, so the balance plans no moves and silently succeeds, and the original .dat is deleted with all shards on the generation host. Poll until every encoded volume reports a full shard set before balancing — across all disk types, since fresh shards register under the source volume's disk type, and scoped to the newest encode generation so an orphaned older generation neither satisfies the wait nor defeats the clump check. As defense in depth, refuse to delete originals when a volume's shards still sit on one node while another node has free EC slots.
* s3: accept raw semicolons in query strings
Go's url.ParseQuery drops any key=value pair containing a raw ';'. A
presigned PUT that signs content-type carries
X-Amz-SignedHeaders=content-type%3Bhost; when a client or proxy decodes
the %3B, the parameter vanished and the upload failed with
MissingFields, while AWS accepts the raw ';' as query data. Re-encode
it before routing so the pair survives parsing and signature
verification.
* iam, iceberg: recover raw-semicolon query pairs on the other listeners
The standalone IAM API verifies SigV4 with a canonical query recomputed
from the parsed query, and Iceberg REST warehouse/parent values may
legally contain ';'. Move the normalization middleware to util/http and
attach it to both routers.
A raw OIDC token sent as Authorization: Bearer was validated and mapped
to a role through the provider's roleMapping, then authorized against
the role's attached policies without ever consulting the role's trust
policy. A federated user could act as a role that
AssumeRoleWithWebIdentity would refuse to issue a session for with the
same token. Run the same trust-policy validation before returning the
principal on the bearer path.
* fix(filer.backup): stop silently dropping events on transient not-found
Under a write burst, filer.backup could consume a metadata event (advancing
the persisted offset) without replicating the file, with no error logged:
1. filersink CreateEntry/UpdateEntry swallowed replicateChunks errors
(glog.Warningf + return nil), so the offset advanced past entries that
were never written.
2. The manifest-chunk branch of replicateChunks resolved via LookupFileId
with no retry, unlike the data-chunk branch — transient lookup races
dropped exactly the large manifest-backed files while small
inline-content siblings landed.
3. isIgnorable404 matched "LookupFileId" / "volume id ... not found",
misclassifying those races as genuine source 404s at the backup layer.
Fix: on a replicateChunks failure the filer sink now skips only when the
live source has moved past the replayed version (deleted or strictly-newer
mtime) — lossless, a later event carries the current content — and
propagates otherwise so the event is retried. The manifest resolve retries
transient errors like the data-chunk path. isIgnorable404 is narrowed to
genuine 404s; non-filer sinks and the initial-snapshot walk, which relied
on the broad match as their only lossless-skip valve, now make the same
live-source decision (filersink.SourceSupersedes) instead of retrying
forever on a permanently gone volume.
Tests cover propagation of unconfirmed lookup failures, the narrowed 404
classification, and the supersession guards.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(filer.backup): derive supersession path with directory fallback
eventSourceSuperseded built the source path from NewParentPath alone.
Legacy metadata events (persisted by older filers) carry an empty
NewParentPath, so the probe looked up "/<name>", read the miss as
"source gone", and skipped a live file on a transient lookup error —
the silent drop this change is meant to eliminate.
Derive the path via MetadataEventTargetFullPath (the same directory
fallback genProcessFunction uses) and cover both event shapes with
TestEventSupersessionProbe_PathDerivation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(filersink): retry manifest resolve only for transient errors, bounded when unverifiable
The manifest-resolve retry stopped only when hasSourceNewerVersion proved
the source moved past the replayed version, which wedged the sink in two
cases: incremental sinks use dated target keys that cannot map back to a
source path (supersession never provable), and permanent resolve errors
(corrupt manifest data, bad file ids) fail forever while the source entry
stays live.
Gate the retry instead: keep retrying only transient errors (volume-lookup
races, network interruptions), stop after a few attempts when supersession
cannot be checked, and propagate everything else immediately so the
configured metadata error policy applies (-disableErrorRetry included).
Propagation is lossless: filer.backup's fallback decides with the event's
real source key, and both filer.backup and filer.sync re-deliver the event
(RetryForeverOnError) without advancing the offset.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(filer.backup): make error classifiers nil-safe
isIgnorable404, isSourceLookupError, and isTransientResolveError called
err.Error() without a nil guard. All current call sites pass a non-nil
error, but the guard is free and matches isRetryableNetworkError.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* admin: let waiting shell clients win the admin lock
The admin lock manager re-acquired the cluster shell lock immediately
after releasing it, while a waiting weed shell client only polls the
master once a second, so an operator could lose the race indefinitely.
Leave the lock free for slightly more than one poll interval before
re-acquiring, and once overlapping reference-counted holds have kept
the lock continuously held for over a minute, make new admin-side
acquires wait for a full release before piggybacking.
* admin/plugin: take the shell lock per detection and per job, not per batch
The default-lane scheduler acquired the cluster shell lock once and
held it across the whole pass: every due job type's detection plus all
of its dispatched jobs, each drained to completion. A manual weed
shell operation sharing that lock could stall behind the batch for up
to the extended execution window.
Hold the lock only while it protects something: around each detection
scan and around each dispatched job. Manual shell operations now wait
for at most one in-flight job. The admin UI detect+execute path stops
wrapping dispatch in an outer hold, since a nested acquire would
deadlock against the lock manager's fairness window; detection and
per-job dispatch take the lock themselves.
* admin/plugin: stop extending the dispatch window to the largest job estimate
When any proposal's estimated runtime exceeded the remaining
JobTypeMaxRuntime, the whole dispatch context was replaced with a
fresh one capped at eight hours, so a single balance backlog could
hold the default lane (and with it erasure_coding and vacuum
detection) for that long.
Keep the dispatch window at JobTypeMaxRuntime and instead detach each
started attempt onto its own estimated-runtime deadline. Large jobs
still get their full time once started; jobs not yet started when the
window closes are canceled and re-proposed by a later detection, so
sibling job types get a turn every window.
* worker/balance: re-check each planned move against the master before executing
A balance plan is computed at detection time, but the admin lock is
released between detection and execution, so a manual shell operation
can rearrange the volume in the gap. The task's own guards catch a
vanished source, but a target that gained a replica in the meantime
would be silently overwritten by VolumeCopy and the source delete
would then reduce the volume to a single copy.
Before executing each move, ask the master for the volume's current
locations (uncached) and skip the move if the volume has left the
source or the target already holds a replica. Skipped moves fail with
a stale-move error and the next detection replans them. Without master
addresses in the cluster context the check is skipped, preserving the
old behavior with older admins.
* admin: block re-acquire while the final lock release is in flight
Release dropped the manager mutex before calling ReleaseLock, so a
concurrent Acquire could see hold count zero and call RequestLock
while the locker still considered itself locked. That request no-ops,
leaving a hold with no live master lease. Track the in-flight release
and make Acquire wait for it.
Also normalize a nil release function from lock manager
implementations, and make the yield and fairness windows per-instance
fields so tests stop mutating globals.
* rust volume: accept odd-length needle id hex in file ids
Go formats the needle id with strconv.FormatUint and parses it back with
strconv.ParseUint, neither of which pads to an even number of hex digits.
hex::decode rejected such file ids with "Odd number of digits", so
volume.fsck could not purge orphans from a rust volume server. Parse the
needle id and cookie with from_str_radix, matching Go's ParseNeedleId
and ParseCookie.
* storage: emit even-length needle id hex in NeedleId.FileId
volume.fsck and volume.check_disk build purge file ids here with unpadded
FormatUint hex, while every other fid formatter strips whole leading zero
bytes. Pad to even length so the output matches the canonical fid format
and strict hex parsers accept it.
* s3: verify SigV4 against each plausible reverse-proxy host
A portless X-Forwarded-Host leaves the client's true port ambiguous: a
proxy that kept the Host header makes the backend Host port right, one
that rewrote it makes X-Forwarded-Port right, and a client on the
scheme's default port signed no port at all. The verifier bet on the
backend Host port whenever the hostnames matched, so nginx-style
$host/$server_port forwarding got SignatureDoesNotMatch whenever the
proxy and backend share a hostname. Try each plausible host value in
likelihood order instead of guessing one.
* s3: unbracket IPv6 X-Forwarded-Host before matching the request host
net.SplitHostPort strips brackets from the request host, so a bracketed
portless X-Forwarded-Host like [::1] never matched and lost its port
candidate.
* s3: cover unbracketed IPv6 forwarded-host candidates; compare with slices.Equal
TestSharedSnapshotConcurrentIntegrity writes as fast as one core can
marshal for two seconds, so its transient heap scales with host speed.
On linux/386 a fast runner walks the heap into the 4GB address-space
ceiling and the run dies with out of memory. Set a 1GB memory limit for
the duration of the test so the GC paces the writer instead of the
address space; the seal-and-recycle race being tested is unaffected.
* s3: keep listing when empty directories fill the listing window
doListFilerEntries issued a single ListEntries request per directory with
Limit = maxKeys+2. Entries that emit nothing - empty directories, the
.uploads folder, the marker echo - consume that window without consuming
maxKeys, so a bucket whose first window held only empty directories was
reported empty and not truncated, and a subdirectory whose window filled
up silently dropped the entries behind it. Keep requesting from the last
received entry until the quota is filled or a short window shows the
directory is exhausted.
* s3: test listing across windows of empty directories
The test filer client now honors StartFromFileName so repeated windows
advance like the real filer.
* s3: propagate the request context into directory listing RPCs
A disconnected client now cancels the ListEntries streams instead of
letting the listing keep issuing requests against the filer.
* s3: group per-directory listing parameters into a request struct
doListFilerEntries took ten positional parameters; call sites read as
string and bool soup. Wrap the per-directory arguments in
listDirectoryRequest so recursions and tests name what they pass.
* s3: group list request parameters into a struct
listFilerEntries took eight positional parameters ending in two bare
booleans. Wrap them in listObjectsRequest so the V1 and V2 handlers
name what they pass.
Move pickOneReplicaToCopyFrom, pickOneReplicaToDelete,
pickOneMisplacedVolume, and isMisplaced into weed/topology/balancer,
following satisfyReplicaPlacement. Selection functions take the shared
balancer.Replica shape and return indices, so the shell keeps its own
replica type; behavior is unchanged and covered by the existing shell
tests.
* volume.fix.replication: parallelize under-replicated volume copies
Fan out fixOneUnderReplicatedVolume up to -maxParallelization at a time.
Destination selection and free-slot accounting stay atomic behind a
scheduler mutex, so concurrent fixes see each other's reservations; a
failed copy returns its reserved slot. A per-server in-flight cap
(-maxParallelizationPerServer, default 1) keeps many simultaneous copies
from swamping a single destination server; when every eligible
destination is at the cap the fix waits for a slot instead of failing.
* volume.fix.replication: isolate the scheduler's location ordering
Clone allLocations before the per-volume fan-out so the scheduler's
re-sorting cannot alias the slice shared with the concurrent delete
phases, and make the test's per-iteration location copy explicit.
* s3: routed last-version delete removes the emptied .versions directory
The routed versioned delete (routedDeleteSpecificVersion) repoints the
latest pointer and deletes the version file, but unlike the lock-path
fallback (updateLatestVersionAfterDeletion) it never tears down the
.versions/ directory it just emptied. The residue keeps the key's read
path in the self-heal rescan loop: every GET of the deleted key logs
event=surfaced plus a GetObject error until the background
EmptyFolderCleaner gets to the directory — at least two minutes away on
its delay queue, and possibly never (the queue is in-memory, bounded,
and gated on the bucket's allow-empty-folders policy). Veeam's lock
arbitration probes deleted lock keys continuously, so those windows are
always open and the log spam is chronic.
ObjectMutation DELETE gains remove_empty_parent: after the child delete,
the filer best-effort removes the parent directory in the same locked
transaction. Non-recursive on purpose — a concurrent write that lands a
new version fails the removal instead of being lost with it. The routed
last-version delete sets it on the version-file DELETE, matching the
lock-path fallback's contract.
Claude-Session: https://claude.ai/code/session_014mMYAHXZySkCCUfpRFNtSv
* s3: drain empty .versions residue on read heal and in s3.versions.audit
Directories already stranded by pre-teardown deletes (or dropped from
the EmptyFolderCleaner's bounded in-memory queue) previously re-entered
the self-heal rescan on every GET forever: the heal only cleared the
pointer and nothing ever removed the directory, and s3.versions.audit
counted the state as clean.
When the heal rescan finds no remaining version, remove the directory
outright (non-recursive, so orphan children still block and fall back to
the pointer clear) and log event=healed mode=empty_dir_removed; the next
GET takes the clean not-found path. The audit gains an empty category so
the residue is visible, and -heal removes such directories in bulk.
Claude-Session: https://claude.ai/code/session_014mMYAHXZySkCCUfpRFNtSv
Align the usage percentage with s3.bucket.quota.enforce and the admin
UI, which measure quota against the live data size (size minus
un-vacuumed deleted bytes). Also print the logical size so both the
raw and live views stay visible.
* s3: keep empty-folder cleanup out of the multipart .uploads staging tree
The async EmptyFolderCleaner deleted <bucket>/.uploads once it looked
empty. A concurrent CreateMultipartUpload inserting its marker between
the cleaner's emptiness check and the bulk child delete had its row
wiped, so the upload silently vanished: ListMultipartUploads then omits
it and the following part/complete/copy calls fail. Skip the .uploads
subtree in both the queue and the delete path (including the eager
parent cascade); the multipart upload lifecycle owns it.
* s3: slice the bucket-relative path instead of reallocating it
* volume.balance: rank by physical disk usage
* volume.balance: keep -byDiskUsage ranking on one scale across the fleet
A server that does not report disk bytes ranks at whole volume-equivalents
while reporting servers stay below 1.0, so during a rolling upgrade the
balancer drains nearly empty old-build servers onto physically fuller ones.
Decide the scale once: rank by physical used percent only when every server
reports disk bytes, otherwise fall back to the data-size ranking for all.
Normalizing the fallback by maxVolumeCount instead would reintroduce the
over-configured-maxVolumeCount distortion this flag exists to avoid.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Drives the full path: AssumeRoleWithWebIdentity embeds the claims in the
session JWT, AuthenticateJWT restores them, and AuthorizeAction evaluates
ForAnyValue:StringEquals on jwt:groups / jwt:roles per bucket. The mock
OIDC provider now carries token claims through and surfaces roles,
matching the real provider.
* iam: surface OIDC groups into STS session request context for resource-policy ABAC
The groups claim was added to ExternalIdentity.Groups but excluded from Attributes
(processedClaims), so it never reached the session RequestContext. As a result
group membership was usable only in role trust policies (assume-time), not in
resource/permission policies (request-time) - so a single role could not scope
access by the caller's groups (aggregate ABAC). Surface Groups as a []string in
the request context; the string-condition evaluator already handles multi-valued
context keys, and the S3 middleware exposes it as jwt:groups.
* iam: also surface OIDC roles into the STS request context (companion to groups)
The 'roles' claim is excluded from the OIDC attributes by the same processedClaims
set that excluded 'groups', so it never reached the session request context and was
usable only via provider-configured role mapping - not a raw token roles claim. Add
ExternalIdentity.Roles, populate it from the token's roles claim, and surface it as a
[]string in the request context so resource policies can gate on jwt:roles, exactly
as the previous commit did for jwt:groups.
github-actions[bot]
Chris LuandGitHub
Lars LehtonenandGitHub
qzhelloandGitHub
Chris WydraandGitHub