14442 Commits

Author SHA1 Message Date
github-actions[bot] db42bb4975 4.39 4.39 2026-07-10 04:08:14 +00:00
Chris Lu 9a7cfaf371 admin: stop holding the shell admin lock across whole scheduler batches (#10290)
* admin: let waiting shell clients win the admin lock

The admin lock manager re-acquired the cluster shell lock immediately
after releasing it, while a waiting weed shell client only polls the
master once a second, so an operator could lose the race indefinitely.
Leave the lock free for slightly more than one poll interval before
re-acquiring, and once overlapping reference-counted holds have kept
the lock continuously held for over a minute, make new admin-side
acquires wait for a full release before piggybacking.

* admin/plugin: take the shell lock per detection and per job, not per batch

The default-lane scheduler acquired the cluster shell lock once and
held it across the whole pass: every due job type's detection plus all
of its dispatched jobs, each drained to completion. A manual weed
shell operation sharing that lock could stall behind the batch for up
to the extended execution window.

Hold the lock only while it protects something: around each detection
scan and around each dispatched job. Manual shell operations now wait
for at most one in-flight job. The admin UI detect+execute path stops
wrapping dispatch in an outer hold, since a nested acquire would
deadlock against the lock manager's fairness window; detection and
per-job dispatch take the lock themselves.

* admin/plugin: stop extending the dispatch window to the largest job estimate

When any proposal's estimated runtime exceeded the remaining
JobTypeMaxRuntime, the whole dispatch context was replaced with a
fresh one capped at eight hours, so a single balance backlog could
hold the default lane (and with it erasure_coding and vacuum
detection) for that long.

Keep the dispatch window at JobTypeMaxRuntime and instead detach each
started attempt onto its own estimated-runtime deadline. Large jobs
still get their full time once started; jobs not yet started when the
window closes are canceled and re-proposed by a later detection, so
sibling job types get a turn every window.

* worker/balance: re-check each planned move against the master before executing

A balance plan is computed at detection time, but the admin lock is
released between detection and execution, so a manual shell operation
can rearrange the volume in the gap. The task's own guards catch a
vanished source, but a target that gained a replica in the meantime
would be silently overwritten by VolumeCopy and the source delete
would then reduce the volume to a single copy.

Before executing each move, ask the master for the volume's current
locations (uncached) and skip the move if the volume has left the
source or the target already holds a replica. Skipped moves fail with
a stale-move error and the next detection replans them. Without master
addresses in the cluster context the check is skipped, preserving the
old behavior with older admins.

* admin: block re-acquire while the final lock release is in flight

Release dropped the manager mutex before calling ReleaseLock, so a
concurrent Acquire could see hold count zero and call RequestLock
while the locker still considered itself locked. That request no-ops,
leaving a hold with no live master lease. Track the in-flight release
and make Acquire wait for it.

Also normalize a nil release function from lock manager
implementations, and make the yield and fairness windows per-instance
fields so tests stop mutating globals.
2026-07-09 20:43:55 -07:00
Chris Lu 6fd82b7e6f fix: merge filer service annotations to avoid duplicate keys in Helm (#10293) 2026-07-09 18:25:34 -07:00
Lars Lehtonen 3a0021e144 fix(weed/worker/tasks/balance): dropped test error (#10291) 2026-07-09 18:20:14 -07:00
Chris Lu 6f816c955d volume.fsck: fix orphan purge against the rust volume server (#10289)
* rust volume: accept odd-length needle id hex in file ids

Go formats the needle id with strconv.FormatUint and parses it back with
strconv.ParseUint, neither of which pads to an even number of hex digits.
hex::decode rejected such file ids with "Odd number of digits", so
volume.fsck could not purge orphans from a rust volume server. Parse the
needle id and cookie with from_str_radix, matching Go's ParseNeedleId
and ParseCookie.

* storage: emit even-length needle id hex in NeedleId.FileId

volume.fsck and volume.check_disk build purge file ids here with unpadded
FormatUint hex, while every other fid formatter strips whole leading zero
bytes. Pad to even length so the output matches the canonical fid format
and strict hex parsers accept it.
2026-07-09 11:19:31 -07:00
Chris Lu e6b2849381 s3: verify SigV4 against each plausible reverse-proxy host (#10284)
* s3: verify SigV4 against each plausible reverse-proxy host

A portless X-Forwarded-Host leaves the client's true port ambiguous: a
proxy that kept the Host header makes the backend Host port right, one
that rewrote it makes X-Forwarded-Port right, and a client on the
scheme's default port signed no port at all. The verifier bet on the
backend Host port whenever the hostnames matched, so nginx-style
$host/$server_port forwarding got SignatureDoesNotMatch whenever the
proxy and backend share a hostname. Try each plausible host value in
likelihood order instead of guessing one.

* s3: unbracket IPv6 X-Forwarded-Host before matching the request host

net.SplitHostPort strips brackets from the request host, so a bracketed
portless X-Forwarded-Host like [::1] never matched and lost its port
candidate.

* s3: cover unbracketed IPv6 forwarded-host candidates; compare with slices.Equal
2026-07-09 02:34:13 -07:00
qzhello 95023af489 fix(shell): print markVolumeWritable/markVolumeReadonly based on the writable flag (#10283) 2026-07-09 00:30:21 -07:00
Chris Lu fd5a6ff427 log_buffer: cap the shared-snapshot race test's heap (#10281)
TestSharedSnapshotConcurrentIntegrity writes as fast as one core can
marshal for two seconds, so its transient heap scales with host speed.
On linux/386 a fast runner walks the heap into the 4GB address-space
ceiling and the run dies with out of memory. Set a 1GB memory limit for
the duration of the test so the GC paces the writer instead of the
address space; the seal-and-recycle race being tested is unaffected.
2026-07-09 00:29:17 -07:00
Chris Lu 399f8033f8 s3: keep listing when empty directories fill the listing window (#10280)
* s3: keep listing when empty directories fill the listing window

doListFilerEntries issued a single ListEntries request per directory with
Limit = maxKeys+2. Entries that emit nothing - empty directories, the
.uploads folder, the marker echo - consume that window without consuming
maxKeys, so a bucket whose first window held only empty directories was
reported empty and not truncated, and a subdirectory whose window filled
up silently dropped the entries behind it. Keep requesting from the last
received entry until the quota is filled or a short window shows the
directory is exhausted.

* s3: test listing across windows of empty directories

The test filer client now honors StartFromFileName so repeated windows
advance like the real filer.

* s3: propagate the request context into directory listing RPCs

A disconnected client now cancels the ListEntries streams instead of
letting the listing keep issuing requests against the filer.

* s3: group per-directory listing parameters into a request struct

doListFilerEntries took ten positional parameters; call sites read as
string and bool soup. Wrap the per-directory arguments in
listDirectoryRequest so recursions and tests name what they pass.

* s3: group list request parameters into a struct

listFilerEntries took eight positional parameters ending in two bare
booleans. Wrap them in listObjectsRequest so the V1 and V2 handlers
name what they pass.
2026-07-09 00:22:34 -07:00
Chris Lu 02b5b66a3f topology/balancer: share replica selection between shell and workers (#10276)
Move pickOneReplicaToCopyFrom, pickOneReplicaToDelete,
pickOneMisplacedVolume, and isMisplaced into weed/topology/balancer,
following satisfyReplicaPlacement. Selection functions take the shared
balancer.Replica shape and return indices, so the shell keeps its own
replica type; behavior is unchanged and covered by the existing shell
tests.
2026-07-08 20:06:15 -07:00
Chris Lu 78e428758b volume.fix.replication: parallelize under-replicated volume copies (#10275)
* volume.fix.replication: parallelize under-replicated volume copies

Fan out fixOneUnderReplicatedVolume up to -maxParallelization at a time.
Destination selection and free-slot accounting stay atomic behind a
scheduler mutex, so concurrent fixes see each other's reservations; a
failed copy returns its reserved slot. A per-server in-flight cap
(-maxParallelizationPerServer, default 1) keeps many simultaneous copies
from swamping a single destination server; when every eligible
destination is at the cap the fix waits for a slot instead of failing.

* volume.fix.replication: isolate the scheduler's location ordering

Clone allLocations before the per-volume fan-out so the scheduler's
re-sorting cannot alias the slice shared with the concurrent delete
phases, and make the test's per-iteration location copy explicit.
2026-07-08 20:03:01 -07:00
Chris Lu a9cfbd8d3a s3: tear down the emptied .versions directory on last-version delete; drain existing residue (#10278)
* s3: routed last-version delete removes the emptied .versions directory

The routed versioned delete (routedDeleteSpecificVersion) repoints the
latest pointer and deletes the version file, but unlike the lock-path
fallback (updateLatestVersionAfterDeletion) it never tears down the
.versions/ directory it just emptied. The residue keeps the key's read
path in the self-heal rescan loop: every GET of the deleted key logs
event=surfaced plus a GetObject error until the background
EmptyFolderCleaner gets to the directory — at least two minutes away on
its delay queue, and possibly never (the queue is in-memory, bounded,
and gated on the bucket's allow-empty-folders policy). Veeam's lock
arbitration probes deleted lock keys continuously, so those windows are
always open and the log spam is chronic.

ObjectMutation DELETE gains remove_empty_parent: after the child delete,
the filer best-effort removes the parent directory in the same locked
transaction. Non-recursive on purpose — a concurrent write that lands a
new version fails the removal instead of being lost with it. The routed
last-version delete sets it on the version-file DELETE, matching the
lock-path fallback's contract.

Claude-Session: https://claude.ai/code/session_014mMYAHXZySkCCUfpRFNtSv

* s3: drain empty .versions residue on read heal and in s3.versions.audit

Directories already stranded by pre-teardown deletes (or dropped from
the EmptyFolderCleaner's bounded in-memory queue) previously re-entered
the self-heal rescan on every GET forever: the heal only cleared the
pointer and nothing ever removed the directory, and s3.versions.audit
counted the state as clean.

When the heal rescan finds no remaining version, remove the directory
outright (non-recursive, so orphan children still block and fall back to
the pointer clear) and log event=healed mode=empty_dir_removed; the next
GET takes the clean not-found path. The audit gains an empty category so
the residue is visible, and -heal removes such directories in bulk.

Claude-Session: https://claude.ai/code/session_014mMYAHXZySkCCUfpRFNtSv
2026-07-08 18:50:54 -07:00
Chris Lu 11fd20e2b0 s3.bucket.list: measure quota usage by logical size (#10274)
Align the usage percentage with s3.bucket.quota.enforce and the admin
UI, which measure quota against the live data size (size minus
un-vacuumed deleted bytes). Also print the logical size so both the
raw and live views stay visible.
2026-07-08 14:32:04 -07:00
Chris Lu b43089f721 s3: keep empty-folder cleanup out of the multipart .uploads staging tree (#10273)
* s3: keep empty-folder cleanup out of the multipart .uploads staging tree

The async EmptyFolderCleaner deleted <bucket>/.uploads once it looked
empty. A concurrent CreateMultipartUpload inserting its marker between
the cleaner's emptiness check and the bulk child delete had its row
wiped, so the upload silently vanished: ListMultipartUploads then omits
it and the following part/complete/copy calls fail. Skip the .uploads
subtree in both the queue and the delete path (including the eager
parent cascade); the multipart upload lifecycle owns it.

* s3: slice the bucket-relative path instead of reallocating it
2026-07-08 14:30:57 -07:00
Aleksey cfb46ee19f volume.balance: rank by physical disk usage (#10271)
* volume.balance: rank by physical disk usage

* volume.balance: keep -byDiskUsage ranking on one scale across the fleet

A server that does not report disk bytes ranks at whole volume-equivalents
while reporting servers stay below 1.0, so during a rolling upgrade the
balancer drains nearly empty old-build servers onto physically fuller ones.
Decide the scale once: rank by physical used percent only when every server
reports disk bytes, otherwise fall back to the data-size ranking for all.
Normalizing the fallback by maxVolumeCount instead would reintroduce the
over-configured-maxVolumeCount distortion this flag exists to avoid.

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-07-08 09:29:59 -07:00
Chris Lu 65f2f1488a iam: test that groups and roles claims reach request-time policy evaluation
Drives the full path: AssumeRoleWithWebIdentity embeds the claims in the
session JWT, AuthenticateJWT restores them, and AuthorizeAction evaluates
ForAnyValue:StringEquals on jwt:groups / jwt:roles per bucket. The mock
OIDC provider now carries token claims through and surfaces roles,
matching the real provider.
2026-07-08 01:53:37 -07:00
Chris Wydra 63db56ff7d iam: surface OIDC groups and roles into the STS session request context for resource-policy ABAC (#10263)
* iam: surface OIDC groups into STS session request context for resource-policy ABAC

The groups claim was added to ExternalIdentity.Groups but excluded from Attributes
(processedClaims), so it never reached the session RequestContext. As a result
group membership was usable only in role trust policies (assume-time), not in
resource/permission policies (request-time) - so a single role could not scope
access by the caller's groups (aggregate ABAC). Surface Groups as a []string in
the request context; the string-condition evaluator already handles multi-valued
context keys, and the S3 middleware exposes it as jwt:groups.

* iam: also surface OIDC roles into the STS request context (companion to groups)

The 'roles' claim is excluded from the OIDC attributes by the same processedClaims
set that excluded 'groups', so it never reached the session request context and was
usable only via provider-configured role mapping - not a raw token roles claim. Add
ExternalIdentity.Roles, populate it from the token's roles claim, and surface it as a
[]string in the request context so resource policies can gate on jwt:roles, exactly
as the previous commit did for jwt:groups.
2026-07-08 01:52:39 -07:00
Chris Lu e873e671b6 filer: share one log-buffer window snapshot across all subscriber reads (#10267)
* log_buffer: share one window snapshot across all subscriber reads

Every in-memory read handed each subscriber a private pooled copy of the
window it wanted, so N subscribers reading the same data cost N copies of
up to 8MB each -- and slow consumers (grpc send backpressure) held those
copies live for their whole iteration. With hundreds of mount subscribers
that multiplied into gigabytes of live heap on the filer.

Share the bytes instead of copying per reader:

- Sealed windows get a lazily created GC-owned snapshot, made once by the
  first reader and handed out zero-copy to the rest. The snapshot travels
  with its window when SealBuffer shifts slots, so recycling the sealed
  array never invalidates it.
- The current window keeps a shared snapshot of its append-only prefix
  buf[:pos], extended on demand; each byte is copied once per window
  (writer-rate-bound) instead of once per reader. At seal a fully
  extended prefix becomes the sealed window's snapshot.

ReadFromBuffer now reports whether the returned buffer is a pooled copy
(flush path) or a shared view that must not be released; the read loops
only recycle pooled buffers.

With 200 subscribers consuming at grpc pace over sealed and current
windows, peak live heap drops from 5.2GB to 178MB.

* log_buffer: clear released read buffer so a panic cannot double-free it

The read loops release the previous iteration's pooled buffer and then
call ReadFromBuffer. If that call panicked before reassigning bytesBuf,
the deferred cleanup would put the same buffer into the pool a second
time, letting two future readers share one backing array. Nil the
pointer at the release site so the defer sees nothing to free.
2026-07-08 01:50:51 -07:00
Chris Lu 24fb7e47f3 java client: fail cleanly when a chunk read returns short or empty data (#10269)
* java client: fail cleanly when a chunk read returns short or empty data

readChunkView copied chunkView.size bytes out of the fetched chunk without
checking the fetched length, so an empty body (seen transiently right after an
append) threw an opaque IndexOutOfBoundsException. Retry empty chunk fetches and
bound the copy to the data actually returned.

* overflow-safe bounds check and null guard on fetched chunk data
2026-07-08 01:50:42 -07:00
Chris Lu 0ae1fdcad2 volume: reload a remote-tiered volume without re-entering the data lock (#10266)
LoadRemoteFile now takes dataFileAccessLock (so a live tier upload does
not race the heartbeat's DataBackend read). But load() also runs it, and
CommitCompact calls load() while already holding that lock, so reloading
a remote-tiered volume during a compaction commit re-enters the
non-reentrant lock and deadlocks.

Split the locked bodies out: swapDataBackendLocked and loadRemoteFileLocked
assume the caller holds dataFileAccessLock. load() uses loadRemoteFileLocked;
the public LoadRemoteFile keeps taking the lock for the live tier-upload
handler that does not hold it.
2026-07-08 01:27:10 -07:00
jay a16194f5b4 refract: reduce mem alloc while building str (#10261)
Signed-off-by: jayl1e <jayl1e@outlook.com>
2026-07-08 01:24:45 -07:00
dependabot[bot] c000addfc9 build(deps): bump golang.org/x/crypto from 0.51.0 to 0.52.0 in /test/kafka/kafka-client-loadtest (#10265)
build(deps): bump golang.org/x/crypto

Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.51.0 to 0.52.0.
- [Commits](https://github.com/golang/crypto/compare/v0.51.0...v0.52.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-version: 0.52.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-08 01:23:00 -07:00
Chris Lu 6413b774df filer: cut metadata-subscription allocation churn (issue #10253) (#10260)
* log_buffer: read only ts_ns during metadata-read binary search

readTs ran a full proto.Unmarshal of each LogEntry just to compare
timestamps during ReadFromBuffer's binary search, allocating fresh
slices for the data and key byte fields on every probe. Under
metadata-subscription fan-out this decode churn dominated allocations
and drove heavy GC, inflating filer RSS well past the live heap.

Scan the wire format for ts_ns (field 1) instead and skip the data/key
payloads without copying them. readTs is now zero-alloc; the binary
search no longer touches the event payloads at all.

* log_buffer: alias event payloads instead of copying them on delivery

The subscribe read loops decoded each LogEntry with a full proto.Unmarshal,
allocating fresh slices for the data and key byte fields on every entry
(protobuf consumeBytesNoZero). When many metadata subscribers each re-read
the in-memory log window under a reconnect storm, that per-entry copy was a
large share of allocation churn -- and for entries a subscriber filters out
by prefix, the copied payload was never even looked at.

Decode with a wire scan that points data/key at sub-slices of the source
buffer instead of copying. The aliased slices are valid only for the
duration of the eachLogEntryFn callback; every current subscriber either
re-decodes the event into its own struct or hands it to a synchronous grpc
Send, so none retain them. Per delivered entry the loop decode drops from
176 B / 2 allocs to zero.
2026-07-08 00:05:20 -07:00
github-actions[bot] 274a41fd26 java 4.38 2026-07-08 06:43:07 +00:00
Chris Lu 218f23fdf0 ci: bump wiki java client versions after publish 2026-07-07 23:41:58 -07:00
dependabot[bot] 1262147d9f build(deps): bump golang.org/x/crypto from 0.45.0 to 0.52.0 in /test/sftp (#10264)
build(deps): bump golang.org/x/crypto in /test/sftp

Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.45.0 to 0.52.0.
- [Commits](https://github.com/golang/crypto/compare/v0.45.0...v0.52.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-version: 0.52.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-07 23:41:47 -07:00
Chris Lu 56fec510e4 ci: make java release version-bump push rebase-safe 2026-07-07 23:36:44 -07:00
Chris Lu 94078b6e3f ci: default java release dry_run to false 2026-07-07 23:32:15 -07:00
Chris Lu cc9549def8 ci: add java client release workflow 2026-07-07 23:26:01 -07:00
Chris Lu 60d9c7ef4c java: bump maven-gpg-plugin to 3.2.4 for headless signing 2026-07-07 23:26:01 -07:00
Chris Lu 254c2a1024 volume: clear remote flag when tiering a volume back to local (#10262)
VolumeTierMoveDatFromRemote downloads the .dat, trims the .vif, and swaps
the data backend to the local file, but left hasRemoteFile set. The
volume.tier.download command masks this by unmounting and remounting
right after, which reloads the flag from the trimmed .vif — but in the
window before the remount the in-memory flag is wrong: doDeleteRequest
would skip appending the tombstone to the freshly local .dat, and the
phantom-.dat guard stays disabled.

Give SwapDataBackend a hasRemoteFile argument so the backend swap and the
flag move together under one lock, and route both tier directions through
it: the tier-down download passes false, LoadRemoteFile passes true. The
flag can no longer disagree with the live backend.
2026-07-07 23:03:17 -07:00
Chris Lu 2d2fdeac3d volume: keep tier-uploaded volume reporting to master after volume.tier.upload (#10259)
volume: keep tier-uploaded volume reporting to master

A live volume.tier.upload removes the local .dat and swaps the data
backend to remote, but v.hasRemoteFile was only ever set when a volume
is loaded from disk, so the running volume kept it false. The phantom
.dat guard then saw fileCount>0, !HasRemoteFile, and a missing .dat, and
stopped reporting the volume to the master. The volume vanished from the
topology even though the upload succeeded and the data was in cloud
storage.

Set hasRemoteFile in LoadRemoteFile, the single point where a volume's
backend becomes remote, so it is true both on disk-scan load and after
an in-process tier upload. Route the backend reassignment through
SwapDataBackend so it happens under dataFileAccessLock, closing the old
backend and never racing the heartbeat's concurrent DataBackend read.
Make the field atomic since the heartbeat now reads it concurrently with
the tier-upload handler that writes it.
2026-07-07 23:00:22 -07:00
Chris Lu bcc528a517 java: HTTP Basic Auth for filer behind an Nginx reverse proxy (#10258)
* java: HTTP Basic Auth for filer behind an Nginx reverse proxy

Attach an "Authorization: Basic" header to the filer gRPC channel through a
ClientInterceptor and to the volume read/write HTTP requests, so the Java
client can reach SeaweedFS fronted by Nginx with auth_basic enabled.

Credentials come from a [basic_auth] section in security.toml or from
FilerSecurityContext.setBasicAuth(). On writes the volume JWT still owns the
Authorization header when present, since the two cannot share it.

* java: precompute Basic Auth header and warn on plaintext channel

Cache the Base64 Authorization value once instead of re-encoding on every
chunk read/write, and log a one-time warning when Basic Auth rides a
plaintext gRPC channel where the credentials would travel in cleartext.
2026-07-07 20:52:50 -07:00
Chris Lu d35c4b3d2d s3: fail over routed object writes when the owner filer is unreachable (#10251)
* s3: fail over routed object writes when the owner filer is unreachable

A routed object write (multipart completion, PUT, delete, versioned
finalize, metadata replace) dialed the ring-selected owner filer
directly with no failover. After a filer restarts onto a new address the
lock ring can still name the old one, so every routed write hangs on the
dead address until the gateway is restarted; CompleteMultipartUpload in
particular exceeds client timeouts.

Route the transaction through withFilerClientFailover, skipping an owner
that recently failed, so a live filer forwards it to the real owner by
route_key. Mirrors the read path's getObjectEntryRoutedByKey.

* s3: fail over bucket-config writes when the owner filer is unreachable

patchBucketEntry dialed the bucket's ring owner directly, so a restarted
filer's stale ring address hung every bucket-config write (versioning,
lifecycle, object lock, ownership, ACL, policy, CORS) - the same failure
as routed object writes. Route it through objectTxnOnFiler so it skips an
unreachable owner and a live filer forwards by route_key.
2026-07-07 12:42:46 -07:00
dongle-code f58721b22f s3api: optimize encodePath memory allocations (#10252)
* Optimize s3api encodePath to eliminate O(n^2) allocations

encodePath built its result via string concatenation in a loop, which is
O(n^2) in allocations. For non-ASCII (e.g. Chinese) runes it additionally
called make + hex.EncodeToString + strings.ToUpper per byte (~10 allocations
per character). Under high QPS with long non-ASCII object keys this produced
a very high allocation rate, frequent GC and long GC pauses, causing S3
request latency spikes.

Replace with a preallocated strings.Builder, a manual hex lookup table, and a
zero-allocation fast path. EncodePath now delegates to encodePath to remove
the duplicated implementation. Output is byte-for-byte identical, verified by
TestEncodePath and TestEncodePathEqual.

Benchmark (long Chinese path): 261 -> 1 allocs/op, 35810 -> 480 B/op, 7.5x faster.
Load test (50M calls): 212x fewer allocations, 76x fewer GC cycles.

* s3api: drop regexp from encodePath fast path

The unreserved-character scan already decides whether any byte needs
encoding, so reservedObjectNames.MatchString was a redundant second pass
that also ran the RE2 engine on every authenticated request. Rely on the
scan alone; output is unchanged. The ASCII fast path drops from ~188ns to
~11ns per call.

---------

Co-authored-by: LiuDoge <liudoge@LiuDogedeMacBook-Air.local>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-07-07 12:36:47 -07:00
Chris Lu 4f1f0dcb17 filer: require JWT authorization on TUS upload endpoints (#10249)
* filer: require JWT authorization on TUS upload endpoints

filerHandler and readonlyFilerHandler run every request through
maybeCheckJwtAuthorization, but the TUS routes were registered only
behind filerGuard.WhiteList, which is a pass-through for the filer (its
guard is built with an empty whitelist), and no TUS handler called the
JWT check. So with jwt.filer_signing.key set, normal PUT/POST/DELETE
were authorized while the TUS endpoints were not.

Run the same check in tusHandler before routing: HEAD uses the read key,
POST/PATCH/DELETE use the write key, and creation scopes a
prefix-restricted token against the resolved target path. OPTIONS stays
open for capability discovery.

* filer: enforce read-only and WORM rules on TUS completion

completeTusUpload wrote the final entry with CreateEntry directly,
skipping the read-only and WORM checks the normal write path applies.
Reject completion when the target prefix is read-only or the existing
entry at the target is WORM-enforced.

* filer: align TUS write path with the normal write path

Resolve the create target with a guaranteed leading slash so a
prefix-restricted token and the stored path stay absolute even if
TusBasePath were set with a trailing slash. Reject read-only prefixes at
session creation before any chunk is written, and map read-only and WORM
rejections at completion to 507 and 403 instead of a generic 500.

* filer: cover method-restricted TUS tokens in the auth test
2026-07-06 18:25:12 -07:00
dependabot[bot] 73165203fc build(deps): bump github.com/go-redsync/redsync/v4 from 4.16.0 to 4.17.0 (#10244)
Bumps [github.com/go-redsync/redsync/v4](https://github.com/go-redsync/redsync) from 4.16.0 to 4.17.0.
- [Commits](https://github.com/go-redsync/redsync/compare/v4.16.0...v4.17.0)

---
updated-dependencies:
- dependency-name: github.com/go-redsync/redsync/v4
  dependency-version: 4.17.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-06 11:48:28 -07:00
Chris Lu e521a2a7b9 mount: re-assign to a live volume when a write can't land (#10239)
With replication 001 and one node down, ~2/3 of volumes have their
replica on the dead node. The primary write lands locally but the
replica forward fails, so the volume returns 500 "failed to write to
replicas". #9744 made that upload fail fast so the client re-assigns,
but the client retry never fired: the reassign gate matched a fixed
list of error substrings that didn't include this one. The mount
surfaced I/O error and dropped the chunk, leaving missing lines and
null-byte gaps in an append workload while a node rebooted.

Decide reassignment by HTTP status instead of matching message text.
On the write path a volume server only 5xxs on a ReplicatedWrite
failure (local disk, replica peer down, under-replication) — all of
which a different volume dodges — so any 5xx reassigns; a no-response
transport failure (the assigned target itself is down) reassigns too;
a 4xx is a genuine client error and is surfaced. doUploadData tags its
errors with the response status via uploadStatusError, and the gate
moves from util.MultiRetry(errList) to util.RetryOnError(predicate).

This drops the fragile substring list (and the message-prefix
constants and guard test it needed); store_replicate.go is untouched.
2026-07-06 11:41:26 -07:00
dependabot[bot] e5215f0785 build(deps): bump github.com/ydb-platform/ydb-go-sdk-auth-environ from 0.5.1 to 0.5.2 (#10240)
build(deps): bump github.com/ydb-platform/ydb-go-sdk-auth-environ

Bumps [github.com/ydb-platform/ydb-go-sdk-auth-environ](https://github.com/ydb-platform/ydb-go-sdk-auth-environ) from 0.5.1 to 0.5.2.
- [Changelog](https://github.com/ydb-platform/ydb-go-sdk-auth-environ/blob/master/CHANGELOG.md)
- [Commits](https://github.com/ydb-platform/ydb-go-sdk-auth-environ/compare/v0.5.1...v0.5.2)

---
updated-dependencies:
- dependency-name: github.com/ydb-platform/ydb-go-sdk-auth-environ
  dependency-version: 0.5.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-06 11:27:54 -07:00
dependabot[bot] 9d55a8bf8a build(deps): bump docker/login-action from 4.2.0 to 4.4.0 (#10241)
Bumps [docker/login-action](https://github.com/docker/login-action) from 4.2.0 to 4.4.0.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/v4.2.0...v4.4.0)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-version: 4.4.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-06 11:27:47 -07:00
dependabot[bot] d774f4abe2 build(deps): bump docker/setup-qemu-action from 4.1.0 to 4.2.0 (#10242)
Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 4.1.0 to 4.2.0.
- [Release notes](https://github.com/docker/setup-qemu-action/releases)
- [Commits](https://github.com/docker/setup-qemu-action/compare/v4.1.0...v4.2.0)

---
updated-dependencies:
- dependency-name: docker/setup-qemu-action
  dependency-version: 4.2.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-06 11:27:39 -07:00
dependabot[bot] 46b0fea62f build(deps): bump github.com/aws/aws-sdk-go-v2/credentials from 1.19.24 to 1.19.26 (#10243)
build(deps): bump github.com/aws/aws-sdk-go-v2/credentials

Bumps [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2) from 1.19.24 to 1.19.26.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/credentials/v1.19.24...credentials/v1.19.26)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/credentials
  dependency-version: 1.19.26
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-06 11:27:32 -07:00
dependabot[bot] 9c11efce6b build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob from 1.7.0 to 1.8.0 (#10245)
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob

Bumps [github.com/Azure/azure-sdk-for-go/sdk/storage/azblob](https://github.com/Azure/azure-sdk-for-go) from 1.7.0 to 1.8.0.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.7.0...sdk/azcore/v1.8.0)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
  dependency-version: 1.8.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-06 11:27:17 -07:00
dependabot[bot] ed1fbeac19 build(deps): bump google.golang.org/api from 0.278.0 to 0.287.0 (#10246)
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.278.0 to 0.287.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.278.0...v0.287.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-version: 0.287.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-06 11:27:09 -07:00
Chris Lu 6e477ae00f filer: keep the internal .system folder out of the per-bucket store path (#10248)
* filer: keep .system internal folder in the default SQL table

Bucket-table SQL stores read the first path segment under /buckets as a
bucket name. The ListBuckets owner index lives at /buckets/.system/..., so
every write there hit isValidBucket(".system") == false and returned
"invalid bucket name .system", flooding the filer log on the postgres/mysql
backends. Route dot-prefixed internal folders to the default table by their
full path, like any other non-bucket entry.

* filer: keep .system internal folder in the default leveldb3 DB

Same guard as the SQL stores: leveldb3 would otherwise open a separate DB
for the .system owner-index folder instead of keeping it in the default DB.

* filer: keep .system internal folder under the default ydb prefix

Skips a DescribeTable round trip per operation on the .system owner-index
path, which never resolves to a real bucket table.

* filer: keep .system internal folder in the default arangodb collection

Avoids creating a stray collection for the .system owner-index folder.
2026-07-06 11:25:50 -07:00
Chris Lu e98cbfc8f1 seaweed-volume: async, buffered writes in VolumeEcShardsCopy (#10237)
* seaweed-volume: async, buffered writes in VolumeEcShardsCopy

The EC-shards-copy RPC handler wrote each streamed chunk to disk with a
synchronous std::fs::File::write_all inside the async handler, blocking a
Tokio worker thread for the duration of every write — noticeable for a
large .ecx on a slow or busy disk.

Factor the five near-identical receive-and-write loops (.ec shards, .ecx,
.ecj, .vif, .ecsum) into drain_copy_stream_to_file, which uses tokio::fs +
BufWriter for async, buffered I/O. Behavior is otherwise unchanged: the
.ecj append mode, the .ecsum byte count and 0-byte-file cleanup, and all
error messages are preserved.

Claude-Session: https://claude.ai/code/session_01Ny5Rt1ph9VWeKmfY936GtF

* seaweed-volume: remove partial copy target on error in EC-shards-copy

Follow-up: drain_copy_stream_to_file now deletes the destination file on
any recv/write/flush error, so a failed VolumeEcShardsCopy no longer leaves
a truncated .ecNN/.ecx/.ecj/.vif/.ecsum on disk for a later reader to trip
on. Matches receive_file / the Go volume server. Best-effort cleanup; the
original stream error is still returned.

Claude-Session: https://claude.ai/code/session_01Ny5Rt1ph9VWeKmfY936GtF
2026-07-06 00:08:46 -07:00
github-actions[bot] 65dff4a492 4.38 4.38 2026-07-06 05:48:51 +00:00
Cole Helbling eefe634962 s3api: FULL_OBJECT checksums for CRC multipart uploads (#10236)
CRC64NVME multipart objects now emit a full-object checksum instead of a composite base64-N value, matching AWS (CRC64NVME supports full-object only). Adds x-amz-checksum-type handling (COMPOSITE/FULL_OBJECT) for CRC32/CRC32C/CRC64NVME via CRC combination, resolved at CreateMultipartUpload and applied at completion.
2026-07-05 20:19:20 -07:00
Chris Lu 39ff5ae767 filer: default cassandra2 timeout and ydb prefix so env-var configs match the scaffold (#10234)
* cassandra2: default connection_timeout_millisecond so env-var configs keep the 600ms timeout

The scaffold documents a 600ms default, but the value has no SetDefault. A
config supplied purely through env vars (or a minimal toml) that omits the
key read back 0, silently overriding the client timeout with no bound.

* ydb: default the table path prefix so env-var configs keep the seaweedfs sub-path

The scaffold documents prefix = "seaweedfs", but with no SetDefault a config
that omits it lands tables at the database root instead of under seaweedfs/.
2026-07-05 10:19:20 -07:00
Chris Lu 873705baf9 filer: default the filemeta CREATE TABLE for postgres2/mysql2 when createTable is unset (#10232)
* postgres2: default the filemeta CREATE TABLE when createTable is unset

An empty createTable rendered through fmt.Sprintf produced
%!(EXTRA string=filemeta), which Postgres rejected with a syntax error at
init. Fall back to a working template so a minimal config bootstraps.

* mysql2: default the filemeta CREATE TABLE when createTable is unset

Same empty-template failure the postgres2 path had: an unset createTable
rendered to %!(EXTRA string=filemeta) and MySQL rejected it at init.
Fall back to a working template.
2026-07-05 10:17:40 -07:00