mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-30 03:35:37 +00:00
110b485bae49165aee788e052eb4174b4de9b136
9932
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  hsdfatandGitHub
|
110b485bae |
fix(volume): stop ScanVolumeFileFrom at a header it cannot advance past (#11398)
fix(volume): stop scans at a header they cannot advance past A corrupt .dat header with a very negative size gives a record length (NeedleHeaderSize + NeedleBodyLength) of zero or less: v3 sizes -43..-36 and v2 sizes -35..-28 give exactly zero, and smaller sizes give a negative length. ScanVolumeFileFrom advanced by that length, so it re-read the same header forever or stepped back into the record before it. weed fix, weed export, weed compact, incremental weed backup and the tail sender behind volume.move and volume.merge could hang on such a volume, and weed compact could also finish with a .cpx that had dropped every needle after the header. Return an error wrapping needle.ErrorCorrupted instead. The check runs after the visitor has seen the record, so the rebuild scanner still stops quietly with io.EOF. Smaller negative sizes whose record length is positive are still stepped over, preserving the salvage behavior compaction relies on. Mirror the guard into the Rust volume scans: DatScanPlan::scan and read_all_needles fail on a non-positive record length, as does scan_dat_head, so a corrupt header cannot stall a tail pass or leave the repair scan walking stale offsets. |
||
|
06dda12e4b |
fix(volume): validate sizes in ReadNeedleBlob and WriteNeedleBlob (#11399)
* fix(volume): reject negative sizes in ReadNeedleBlob and WriteNeedleBlob A ReadNeedleBlob RPC with a size of -44 or below (-36 on v2 volumes) panics in makeslice inside needle.ReadNeedleBlob. The volume gRPC server has no recovery interceptor, so one request kills the process. Smaller negative sizes return bytes that are not a record. WriteNeedleBlob accepted a negative size whenever the blob header carried the same value: it appended the blob to .dat and indexed the needle with that size, which reads as deleted. Reject size < 0 in both Volume methods. Size 0 still passes, since delete records carry it. The Rust volume server got the same storage guards in #11345. * fix(volume): reject needle blobs whose length does not match their size WriteNeedleBlob appends the blob as is. A blob that is not the length its size implies leaves .dat off the 8-byte grid, and every later ordinary write to the volume is indexed at a truncated offset and reads back as EOF. A blob off by 8 bytes keeps the grid but leaves bytes that a .dat scan reads as the next record. The in-tree callers already send exact lengths. The one case this newly refuses is a copy between volumes of different needle versions, and that case already writes a broken record: a v3 record lands on a v2 volume with 8 extra bytes, and a v2 record on a v3 volume either fails the timestamp check or lands 8 bytes short. This is separate from the negative-size guards, whose Rust counterpart is #11345. The Rust server does not check the length yet. * fix(volume): guard the blob buffer allocation in needle.ReadNeedleBlob Volume.ReadNeedleBlob rejected negative sizes, but needle.ReadNeedleBlob still sized its buffer from the size and is called directly by vacuum and other paths. Reject a deletion marker before make() there too, and use size.IsDeleted() in the volume-level checks. * fix(volume): mirror the blob length check in the rust volume server write_needle_blob_and_index checked the size against the blob header but appended the blob verbatim, so a blob that is not the length its size implies still leaves .dat off the record grid. Match the Go check. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
|
a93a1ab2eb |
fix(volume): return an error instead of 201 when a write lands on no volume (#11397)
* fix(volume): return an error instead of 201 when a write lands on no volume ReplicatedWrite only writes locally when this server holds the volume. For a volume id no server holds, the master lookup returns no locations, so the write went nowhere and the upload still got 201 Created. The same happened for a type=replicate write to a server without the volume, so the primary, or the S3 chunk fan-out, counted a replica that was never written. A server without the volume still forwards the write to the replicas the master lists. When there is nothing to forward to, fail with "volume N not found on host:port". PostHandler returns that as 500, the status the Rust volume server already returns here, and uploaders re-assign on 5xx. Fixes #6609 * volume: reuse Store.HasVolume, drop issue ref from test comment --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
cd1e738422 |
[Volume] Scrub local deletion tombstones during FULL scrub (#11396)
* fix 11388 * fix(volume): scrub validates local deletion tombstones TombstoneFileSize (-1) is an .idx-only sentinel; the physical record it points at carries a zero-sized body. Normalize deleted index sizes to 0 via onDiskSize before computing disk usage and calling ReadData, so corrupted or truncated tombstone records are detected instead of skipped. Offset-zero entries (remote logical deletes, no .dat record) remain skipped, and the physical needle id is checked against the index key. Mirror the behavior in the Rust volume server. * fix(volume): scrub preserves physical size of deleted non-tombstone entries Size.Raw()/raw() already encodes the index-to-disk mapping: tombstone (-1) -> 0, other negative sizes -> their absolute value (the offset then points at the original record, per the ReadDeleted path). Use it instead of mapping every deleted size to 0. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
 Chris LuandGitHub
|
f6a3286b32 |
fix(volume): derive needle body tail bound from the version layout (#11395)
* fix(volume): derive needle body tail bound from the version layout The size guard in ReadNeedleBodyBytes computed the tail length as checksum, plus timestamp only for Version3. Forks and future on-disk formats whose tail carries more fields would silently under-check and still panic in readNeedleTail on a truncated body. Derive the tail from NeedleBodyLength minus data and padding so the bound stays exact for every version. Iterate IsSupportedVersion in the new tests instead of hardcoding v1-v3 so downstream formats get covered automatically, and skip versions the build cannot write rather than failing on them. * test: skip needle write only on the unsupported-version error A blanket skip would hide a real writer regression. Skip the version subtest only when the writer reports the version is not supported in this build (the error text differs between builds), and fail on any other write error. |
||
|
Chris LuandGitHub
|
01bb3b3053 |
s3api: add Snowflake s3compat API integration tests (#11394)
* s3api: add Snowflake s3compat API integration tests Run the upstream snowflakedb/snowflake-s3compat-api-test-suite against a local SeaweedFS server in CI. test/s3/snowflake/run.sh starts weed server with S3 (-s3.autoCreateBucket=false so missing-bucket PUTs return NoSuchBucket), prepares the fixtures the suite needs (versioned bucket, deny-all-policy bucket, >1000-object prefix), clones the suite, patches it to path-style addressing, and runs mvn -Dtest=S3CompatApiTest. The suite also exposed that GetBucketLocation returned 404 NoSuchBucket for a malformed bucket name; validate the name first and return 400 InvalidBucketName like AWS. * test: harden snowflake s3compat runner per review - Pin the upstream suite to a tested commit (SUITE_REV) instead of the moving default branch - Bind the test server to loopback only - Require the AccessDenied error code when verifying the denied bucket - Fix README so go install runs in a subshell - checkout with persist-credentials: false - Make the concurrency group unique per PR, and widen path filters to the storage/operation/wdclient/cluster/pb packages the S3 stack uses * test: advertise loopback ip for snowflake test server -ip.bind 127.0.0.1 alone left the volume server advertising the host's primary address, so chunk uploads were refused. Also set -ip 127.0.0.1 and disable the Iceberg/Lance listeners so the harness is loopback-only and does not collide with other local services. |
||
|
hsdfatandGitHub
|
5769057af3 |
fix(volume): return an error instead of panicking on a corrupt needle size (#11393)
ReadNeedleBodyBytes sliced the needle body with the size from the needle header without checking it. A corrupted .dat header carrying size -1 still gets a positive body length (16 bytes on v3), so vacuum compaction read that body and panicked with "slice bounds out of range [:-1]". Writers never put a negative size in a .dat header: a delete appends a size-0 record, and TombstoneFileSize only lives in the .idx. Reject a size that is negative or leaves no room for the checksum/timestamp tail with an error wrapping ErrorCorrupted. ScanVolumeFileFrom already logs body read errors and moves on, so compaction now skips the record like any other corrupt needle. Fixes #6763 |
||
|
Chris LuandGitHub
|
37bf1cd91d |
volume: validate copy/tail source addresses before dialing (#11390)
* pb: stop exiting the process on malformed server addresses ServerToGrpcAddress and GrpcAddressToServerAddress called glog.Fatalf when hostAndPort could not parse the port, which os.Exit(255)ed the whole process. A caller-supplied copy or tail source address reached this path synchronously in the serving goroutine, so one anonymous VolumeCopy with a non-numeric port terminated the volume server. Log the parse error and return the input unchanged instead: the dial or request that consumes the address then fails as an ordinary error. * volume: validate copy and tail source addresses before dialing VolumeCopy, VolumeEcShardsCopy and VolumeTailReceiver dial a caller-supplied source address (SourceDataNode / SourceVolumeServer) with no endpoint validation, so an anonymous caller could aim the volume server at loopback, link-local (cloud metadata) or other unintended destinations and read dial behavior back as a connectivity oracle. Apply the same peer-target deny list FetchAndWriteNeedle uses for replica targets: the source must be a bare host:port whose host is not loopback, link-local or unspecified; cluster peers stay reachable on private networks, and -volume.allowUntrustedRemoteEndpoints opts out. The loopback-using copy tests set the flag to keep exercising the copy path in process. * rust volume: validate copy and tail source addresses before dialing Mirror the Go guard on the Rust volume server: volume_copy, volume_ec_shards_copy and volume_tail_receiver dial a caller-supplied source address, so run it through validate_replica_target first (bare host:port; no loopback, link-local or unspecified hosts; private peers stay allowed). --volume.allowUntrustedRemoteEndpoints opts out; the test fixture and the Rust test-cluster launcher set it so loopback sources in tests keep working. * volume: pin validated copy/tail source addresses at dial time validateReplicaTarget resolves the source hostname once, but the gRPC client resolved it again at connect, leaving a DNS-rebinding window for hostname sources. The copy and tail source dials now run through the same guardedDialerPolicy the remote-storage path uses, so every resolved address is re-checked against the replica deny list (private peers allowed) immediately before the TCP connect. guardedDialerPolicy also moves to util.OutboundDialContext so the guarded path keeps the -ip.bind source binding the default gRPC dialer had. The Rust volume server mirrors this with connect_guarded, a tonic connector that resolves, re-checks each address, and connects to the first passing IP; handlers use it whenever the untrusted-endpoint opt-out is off. A handler-level test now exercises the enabled validation branches for all three source-taking RPCs. * pb: return empty server address for malformed grpc addresses GrpcAddressToServerAddress used to return the unparseable input on a hostAndPort failure, so a malformed raft address (e.g. "host:abc") flowed into admin dashboard master maps unchanged. Return an empty string instead, skip empty conversions at the two raft-cluster merge sites, and drop the now-stale comment about the fatal exit the earlier commit removed. * test: opt erasure-coding loopback clusters out of the remote endpoint guard The erasure-coding suites drive VolumeEcShardsCopy / VolumeCopy between volume servers bound to 127.0.0.1, which the copy/tail source guard now rejects by default. Pass -volume.allowUntrustedRemoteEndpoints to the test volume launches, matching what the volume_server framework harnesses already do. * admin: only claim fallback master leadership on an empty raft response A nonempty RaftListClusterServers response whose entries were all rejected left masterMap empty, so the fallback marked the reachable current master as leader the same way a genuinely empty (non-raft) response does. Track whether the successful response returned zero servers and only promote the fallback master then. |
||
|
Chris LuandGitHub
|
a6d72bc272 |
s3api: delete orphaned chunks only when the entry is confirmed absent (#11389)
* s3api: test for chunks deleted under an entry the filer committed Issue #11387: the filer can report a create failure after inserting the entry (e.g. a parent-directory creation failing post-insert). The error arrives in the response rather than as a transport status, so it maps to a definitive error and putToFiler deletes the chunks of the live entry. * s3api: confirmCreateLanded also reports a confirmed-absent entry The verification a failed create runs can answer both directions: the entry matching the uploaded chunks proves the write landed, and an authoritative not-found proves the uploaded chunks are orphaned. Return both outcomes so the cleanup path can gate on the fact rather than the error class. An empty upload can never prove a landing, so a zero-chunk entry match no longer upgrades the outcome. * s3api: delete orphaned chunks only when the entry is confirmed absent A failed create no longer skips verification based on the error class: the filer can fail after inserting the entry (issue #11387) and a partially-applied routed transaction can leave it behind too, both surfacing as definitive errors. Every failed create now resolves the entry's fate, and the uploaded chunks are deleted only when the entry is confirmed absent; anything unverifiable keeps them for vacuum. * s3api: confirm absence on every filer the create could have committed on A lock-path create fails over across filers, so the entry can live on a replica the routed owner has not caught up to; one not-found does not prove absence. The confirmation now queries the owner, the prior owner, and the failover set, declaring absent only when none of them has the entry. * s3api: bound the reconciliation lookups confirmCreateLanded runs The lookups ran on context.Background() under the object write lock, so a connected filer that never replies could stall the write path. One timeout now covers the whole enumeration; an expired budget fails the remaining lookups as uncertain, which keeps the chunks. |
||
|
Chris LuandGitHub
|
c72eda50a8 |
s3: drop implicit reader cache budget that throttled S3 GETs (#11384)
* fix(filer): leave reader cache unbounded without an explicit budget NewReaderCache silently installed a 256MiB ReaderCacheBudget when the caller passed none. Only weed mount opts into a budget; every other caller (S3 gateway, WebDAV, query engine, mq logstore) inherited the cap. Under ~90 concurrent S3 GETs of medium objects, prefetch wants far more than 64 chunk buffers, so reserve() serialized chunk fetches, clients timed out and retried, and the retry re-downloaded chunks the cancelled request had already fetched. A nil budget now means unbounded, restoring the pre-4.47 behavior for callers that never asked for a memory cap; reserve/complete/release are nil-safe. The mount path is unchanged and still enforces -readerCacheSizeMB. Fixes #11380 * feat(s3): expose -s3.readerCacheSizeMB reader buffer budget Operators who want the S3 gateway read path memory-bounded can now opt in: -s3.readerCacheSizeMB on weed filer/server/mini and -readerCacheSizeMB on standalone weed s3, matching the mount flag. The default 0 keeps the unbounded pre-4.47 behavior; a positive value installs a shared ReaderCacheBudget across in-flight and retained chunk buffers for all S3 GETs. * fix(filer): validate chunk size before consulting the reader budget A nil budget returned early and skipped the negative chunkSize check, letting a corrupted size reach mem.Allocate and panic. Also drop the command-specific flag prefix from the S3 validation error since standalone weed s3 exposes the option as -readerCacheSizeMB. * filer: drop chunk buffers once fully consumed ReaderCache retained every completed chunk buffer in the downloaders map until the slot limit evicted it, so buffers lingered after all readers finished with them. Track attached readers on each SingleChunkCacher and remove the cacher when the last reader consumes the buffer to its end. In-flight download deduplication and the prefetch handoff are unchanged: a buffer always survives until fully read, partial reads keep it available, and an attached reader pins a consumed buffer until it detaches. Repeat reads now go through the chunk cache where enabled, or refetch. * filer: drop consumed buffers on last detach, rechecked under cache lock Two review findings on the drop-on-consume change: - Removal only fired when the detaching reader itself reached the chunk end. If the end-reaching reader finished first and the last remaining reader did a partial read or cancelled, the consumed buffer and its budget reservation lingered until eviction. Track a persistent consumed flag instead, so any end-reaching read marks the buffer and the last detach drops it. - remove() checked only map identity, so a reader attaching between the reader count hitting zero and removal could attach to a cacher that was then deleted underneath it. removeConsumed() re-checks identity, readers == 0, and consumed under the ReaderCache lock; a raced attach keeps the cacher and its own detach retries the removal. |
||
|
Chris LuandGitHub
|
87ee3b63a2 |
s3: abort completed multipart uploads metadata-only (#11385)
* s3: abort a completed upload's leftover directory metadata-only A .uploads/<id> directory can outlive the object it completed into when the commit's metadata-only removal failed or the gateway died in between; the restored part entries then share chunks with the published object. AbortMultipartUpload deleted the directory recursively, chunks and all, so aborting such a leftover destroyed a committed object (#11382). Run the same check s3.clean.uploads gained in #11375 before deleting: when the object entry or a version file under <key>.versions carries the upload id, remove .uploads/<id> metadata-only and answer the abort; when the lookup cannot decide, refuse with InternalError rather than risk live chunks. * s3: apply the completed-upload check to lifecycle MPU abort lifecycleAbortMPU ran the same destructive recursive delete on .uploads/<id>. Reuse uploadCompleted so a leftover whose object entry or version file carries the upload id is removed metadata-only, and an undecidable lookup retries later instead of freeing live chunks. * s3: serialize abort's upload-dir delete with the object's commit The completed check alone leaves a race: abort can read completed=false, then an in-flight completion publishes the object over the same part chunks before the recursive delete frees them. Run the check and delete inside the object write lock, which non-routed completions hold for their whole finalize. With an owner, send the data delete as an ObjectTransaction on the object's lock key — a routed commit then either loses its upload-exists precondition after our delete or has already stamped the object, which the transaction's IF_EXTENDED_NOT_EQUAL condition detects and falls back to a metadata-only remove. lifecycleAbortMPU shares removeUploadDir so both callers get the same ordering. * s3: check for an empty object before resolving its write owner * s3: check completion at the abort's resolved object key An upload record missing ExtMultipartObjectKey skipped the completed check entirely even though the request's Key names the object. |
||
|
Chris LuandGitHub
|
0ca1c19821 |
s3api: unify auth error handling across s3tables, iceberg and lance (#11381)
* s3api: fail closed when S3 Tables signature verification fails * s3api: avoid nil Account dereference in S3 Tables auth log * iceberg: return auth error instead of falling back to DefaultAllow * lance: return auth error instead of falling back to DefaultAllow * s3api: stop trusting client-supplied s3-account-id The header is set by the server after successful authentication; scrub inbound values alongside the other internal headers, and apply the same admin guard to the header fallback branch of getAccountID that the identity branch already has. * test: cover table-catalog auth wrappers and principal resolution * test: configure anonymous identity where catalog clients do not sign * s3api: scrub s3-account-id after signature verification |
||
|
Chris LuandGitHub
|
f40687b34e |
s3: tighten STS session token handling (#11383)
* s3api: test that a session token must not reveal its credential * sts: derive secret access key with HMAC keyed on the signing key * s3api: stop accepting STS session tokens as bearer credentials * security: reject STS session tokens on filer and admin gRPC auth * test: sign s3/iam framework requests with the session credential * s3api: exercise the real auth pipeline in the end-to-end harness |
||
|
Chris LuandGitHub
|
15520f601f |
s3: commit multipart upload and remove .uploads atomically; purge completed uploads metadata-only (#11375)
* s3: commit versioned multipart upload in one transaction CompleteMultipartUpload wrote the version file, flipped the .versions pointer, then removed .uploads/<id> metadata-only as a best-effort post-commit step. A filer error or gateway crash in that window left the upload directory referencing the same chunks as the published object, and the next s3.clean.uploads run purged it with data -- corrupting a committed object. Put the version file, remove the upload directory metadata-only (its chunks are the object's chunks), and recompute the latest pointer in one ObjectTransaction under the object's per-path lock on the owner filer. The mutation order keeps every partial state safe: the chunks stay referenced at all times, and a published object never coexists with the upload directory the cleaner would purge. Unused part entries are freed before the transaction, since the metadata-only directory delete would otherwise leak their chunks. * s3: remove upload directory inside the multipart object PUT The same committed-object/stranded-upload window existed on the suspended and non-versioned paths: writeMultipartObject committed the object, then a best-effort rm dropped .uploads/<id>. Ride the metadata-only removal on the routed PUT itself so the two land in one transaction; the unrouted mkFile fallback keeps post-commit cleanup. * shell: purge completed uploads metadata-only in s3.clean.uploads A leftover .uploads/<id> can outlive a committed object when the completion's metadata-only delete fails or the gateway dies in between; its part entries then share chunks with the live object, and a recursive purge frees them out from under it. Before purging a stale upload, check whether it completed: the object entry or any version file under <key>.versions carrying the upload id. If so, delete with skipChunkDeletion. If the lookup fails, skip the upload for this run rather than risk live chunks. * s3: abort multipart completion when unused part cleanup fails Deleting the upload directory metadata-only erases the only metadata pointing at part entries whose deletion failed, orphaning their chunks. Propagate the error so the completion fails while the upload directory still exists and the request remains retriable. * s3: require the upload directory to exist at multipart commit A delete that does not take the object lock (abort, lifecycle, s3.clean.uploads) can remove .uploads/<id> and its chunks between the prepare step and the commit transaction. The commit now carries an IF_EXISTS precondition on the upload directory so the race fails the request with NoSuchUpload instead of publishing an object over freed chunks. * s3: keep the version file when the upload directory is gone The finalize transaction has no rollback, so a failure at the latest-pointer recompute leaves the version written and .uploads/<id> removed. Deleting the version then destroys the only remaining record of the upload, making a retried CompleteMultipartUpload return NoSuchUpload while the version's chunks leak. Roll back only while the upload directory survives; otherwise keep the version, which a retry resolves through SeaweedFSUploadId and the version reconciler promotes. * s3: keep manifests when a routed object write partially commits For non-versioned and suspended completions the object PUT precedes the upload-directory DELETE, so an error can mean the object entry exists while the response reports failure. Freeing this attempt's manifest chunks then destroys the committed object. Keep them when the object entry survived, and after a failed null-marker finalize which always follows a committed write. * s3: skip the keep-version path on precondition failure A rejected precondition means no mutation ran, so there is no version file to preserve and this attempt's manifests are orphans the error cleanup should free. * s3: keep manifests when the object-existence check itself fails A transient lookup error previously read as absent, letting the error cleanup free manifest chunks a committed object still references. * s3: keep the upload directory when post-commit part cleanup fails Removing it metadata-only after a failed entry delete erases the only reference to the leftover chunks. Leave the directory so the entries keep their chunk references for s3.clean.uploads or manual recovery. * pb: fix filer list entry counting on 32-bit int(limit) wraps to -1 on 386 when limit is math.MaxUint32, so the beyond-limit check discarded every streamed entry. Compare in uint64 instead; the semantics are unchanged on 64-bit platforms. * shell: resolve trailing-slash object keys in s3.clean.uploads Completion stores a key ending in / inside the directory it names (<bucket>/dir/dir), but FullPath+DirAndName on the normalized key looked one level too high. Deriving dir and name with path.Dir and path.Base mirrors getEntryNameAndDir so the completed-upload check finds the entry instead of purging its chunks. * s3: heal a suspended completion hidden behind a delete marker Removing .uploads/<id> inside the commit transaction means a failed finalizeSuspendedNullWrite leaves nothing to retry against: the object entry is committed but the marker still makes the key read as deleted, and a retried CompleteMultipartUpload can only report NoSuchUpload. When the upload directory is gone, check the regular path for an entry carrying the upload id and re-run the marker finalize, so the retry both succeeds and repairs the key. Only suspended buckets can hold this state; anything newer owns the key. * s3: report store errors when resuming a committed multipart object |
||
|
Chris LuandGitHub
|
bdc37a1e86 |
mount/shell: bucket allow-empty-folders toggle, mount keeps explicit false (#11370)
* mount: keep a deliberate bucket allow-empty-folders setting * shell: s3.bucket.allowEmptyFolders toggles the empty folder cleaner * shell: guard allow-empty-folders toggle with expected extended attrs * filer: drop cached empty-folder policy on bucket entry update * mount: guard allow-empty-folders write with expected extended attrs * filer: skip caching a stale cleanup policy read across an update * mount, shell: snapshot the full extended attributes for update preconditions * filer: fail closed and invalidate on all bucket entry events for cleanup policy * filer: key the cleanup policy generation by bucket * filer: requeue cleanup when the bucket policy cannot be loaded * filer: expire idle cleanup policy generations * filer: skip requeueing cleanup after the cleaner stops * filer: bound cleanup retries on repeated policy failures * filer: cover cleanup requeue on repeated policy failures * filer: keep cleanup policy generations while reads are in flight * filer: exercise the cleanup queue lifecycle in the retry-cap test |
||
|
Chris LuandGitHub
|
ce1e0dc30a |
s3api: don't delete chunks when CreateEntry outcome is ambiguous (#11376)
* s3api: map ambiguous filer transport errors to retryable 503 Canceled, DeadlineExceeded and Unavailable can be returned after the filer applied the write, so the outcome is ambiguous. Reporting them as a 4xx tells the client not to retry; report ServiceUnavailable instead. * s3api: verify entry existence before deleting orphaned chunks A failed CreateEntry can still have landed on the filer when the error is a transport failure, and entryCreated=false would tombstone chunks a live entry references, leaving a dangling pointer that survives only because reads pass readDeleted=true until vacuum reclaims the needle. Before deleting, look the entry up: if it is stored with the same chunks, the write succeeded; if the lookup cannot be answered, keep the chunks for vacuum to reclaim; only a confirmed absence still cleans up. * s3api: regression tests for ambiguous CreateEntry outcomes Covers the three post-create-failure cases in putToFiler: the entry landed despite the error (treat as success, keep chunks), the entry is confirmed absent (delete orphans), and the outcome is unverifiable (keep chunks, return error). * volume: count reads served from deleted needles A readDeleted read succeeding on a tombstoned needle is the signal that metadata still points at deleted data. Count it under a readDeletedNeedle handler label in both the Go and Rust volume servers so the condition is visible before vacuum turns it into a 404. * s3api: never delete chunks on an ambiguous create error Review feedback on the first fix showed verification could still go wrong in both directions: a stale or lagged lookup could report not-found for a committed entry, a prefix object stores its chunks on a directory entry, and filer-side manifestization rewrites the top-level chunk ids the comparison relied on. Rework the rule so the outcome classes are asymmetric: - A transport-level error (anything filerErrorToS3Error maps to a retryable 503) is ambiguous and never deletes chunks; the lookup can only upgrade the write to success. - Any other error is a definitive filer refusal and still cleans up. confirmCreateLanded asks the write owner first, resolves the stored entry through chunk manifests, requires an exact match of the uploaded file ids, and on success runs the finalize callback the failed create skipped (under the object write lock, with the same rmObject undo the create path uses). Zero-chunk writes stay ambiguous since they cannot be told apart by chunks. * s3api: cover definitive refusals and stale entries in put tests The confirmed-failure case now uses a definitive refusal so it still exercises orphan cleanup, and a new case keeps chunks when the stored entry belongs to an older object rather than this PUT. * volume: count deleted-needle reads once per request Streamed Go reads ran the deleted check in readNeedle and again in readNeedleDataInto, and non-streamed Rust reads in stream_info and the full-read fallback, double-counting one request. Count at the single entry probe each implementation takes per GET: readNeedle in Go, read_needle_stream_info in Rust. * s3api: run recovered-write rollback under the object lock Two follow-ups from review: ResolveChunkManifest returns traversed manifest blobs in its manifestChunks output, so requiring it empty rejected every manifestized landing; and the rmObject undo ran after the object write lock was released, so a concurrent newer write could be deleted between finalize failure and rollback. Compare only the resolved data chunks and keep the undo inside the lock. * s3api: verify, finalize and roll back recovered creates in one lock A lookup done before the object write lock let a concurrent PUT replace the entry between the chunk comparison and the finalize/rollback section, so a failed afterCreate could rmObject a newer write. Run the owner lookup, manifest resolution, chunk comparison, afterCreate and the conditional undo inside a single withObjectWriteLock section. |
||
|
Chris LuandGitHub
|
4ec564469a |
s3tables: hide inaccessible catalog resources (#11365)
* s3tables: hide inaccessible table buckets * s3tables: hide inaccessible namespaces * s3tables: hide inaccessible tables * s3tables: hide inaccessible resources in rename and namespace delete RenameTable/RenameView denied on the source now report the same not-found as a missing source, and the destination name conflict is checked only after destination authorization so a denied caller cannot distinguish an existing destination namespace or name from a missing one. DeleteNamespace denials use the same formatted message as a missing namespace. |
||
|
Chris LuandGitHub
|
66f1754896 |
s3: enforce dedicated Object Lock actions (#11362)
s3: enforce dedicated object lock actions |
||
|
Chris LuandGitHub
|
994e1f7d64 |
admin: replace Font Awesome with MIT-licensed icons (#11364)
admin: replace Font Awesome with MIT icons |
||
 David ChristopherandGitHub
|
1a285c1334 |
filer: join shutdown paths before closing metadata store (#11363)
Serve can return when its listener closes while HTTP requests are still draining. The main path could then close the metadata store before those requests finish. Make signal, context, and Serve-exit paths join one shutdown sequence. Drain gRPC and HTTP concurrently with 15-second default limits, then close the store. Test both completion orders. |
||
|
Chris LuandGitHub
|
3ebc05930d |
s3: separate Object Lock configuration permission (#11361)
* s3: separate object lock configuration permission * test: synchronize manifest cancellation setup |
||
|
Chris LuandGitHub
|
0c7beec697 | server: add filer-specific disableHttp flag (#11360) | ||
|
David ChristopherandGitHub
|
a859f0a019 |
filer: preserve accepted metadata log records on shutdown (#11359)
fix: flush metadata log before closing filer store Serialize sealed-batch handoffs with shutdown, reject late appends, and wait for log-buffer workers before closing the filer metadata store. Cover queued writes, interval and explicit flushes, late-write rejection, and pending persistence with shutdown tests. |
||
|
71f8128d75 |
shell: fs.verify -pruneEntries deletes entries whose needles are lost (#11338)
* shell: fs.verify -pruneEntries deletes entries whose needles are lost
* shell: harden fs.verify -pruneEntries guards; VolumeNeedleStatus returns NotFound for absent needles
* shell: resolve chunk manifests in fs.verify metadata path; require confirmed deletion before counting prunes
* shell: anchor fs.verify legacy missing-needle error matching
* shell: keep fs.verify metadata scan alive on manifest resolution failures
* shell: classify EC missing needles and keep manifest failures unverified
VolumeNeedleStatus now canonicalizes erasure_coding.NotFoundError to
codes.NotFound, so absent needles in EC volumes reach the prune path
through the same stable contract as regular volumes. The client-side
isNeedleMissingError keeps recognizing the legacy wrapped EC shape
("locate in local ec volume: ... needle not found") for mixed-version
clusters.
A chunk manifest that fails to resolve is now an entry-level
verification failure even when the raw top-level chunks are healthy:
the file is not fully readable without the manifest. Raw chunks are
still verified on a resolution failure so a missing top-level manifest
needle is classified and can be pruned. The per-entry logic is
extracted into resolveAndVerify for testability.
* shell: trim fs.verify prune comments
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
|
||
 
|
1f037e48f9 |
s3: a list marker that sorts before the prefix excludes nothing (#11322)
* s3: a list marker that sorts before the prefix excludes nothing ListObjects `marker` and ListObjectsV2 `start-after` are a plain key cutoff: list the keys that sort after it. A marker that sorts before the prefix and is not under it therefore excludes no key carrying the prefix, and the listing must equal the one with no marker at all. normalizePrefixMarker treated every marker that does not start with the prefix as "something wrong" and the listing came back empty. Clients send this shape routinely: docker/distribution's S3 storage driver walks prefix "<root>/<path>/" with start-after "<root>" (its rootdirectory), so on SeaweedFS a registry walk saw an empty bucket. zot read that as "no repositories": /v2/_catalog was empty, GC/scrub/retention never saw a repo, and on restart its storage parse deleted every repository's metadata as "no longer in storage". listFilerEntries now lists as if no marker were given when the marker sorts before the prefix; the response still echoes the marker the client sent. A marker that sorts after the prefix's subtree is left alone: it may legitimately sit inside a partial-name prefix's match set, which normalizePrefixMarker already handles, and otherwise correctly lists nothing. Reproduce on 4.44 and 4.47: curl -s "$S/zot?list-type=2&prefix=zot/zot/&start-after=zot/zot/" # all keys curl -s "$S/zot?list-type=2&prefix=zot/zot/&start-after=zot" # KeyCount 0 curl -s "$S/zot?list-type=2&prefix=zot/zot/&start-after=a" # KeyCount 0 * s3: keep the prefix's own key excluded by a marker that names it Fold the before-prefix marker rule into normalizePrefixMarker, which now also derives prefixEndsOnDelimiter from the effective marker instead of each cursor rebuilding the expression. A marker equal to the prefix is no longer trimmed to a subtree cutoff: start-after "a/b/" with prefix "a/b/" excludes only the "a/b/" key, so the walk starts inside that directory and its children still list. Adds a listing-level test that walks the whole path for both start-after shapes a registry sends, and covers the new normalization cases. * s3: leading slashes do not hide a marker that names the prefix * s3: echo the V1 marker the client sent, not the walk's cutoff * s3: filter only the walk's cutoff from the V1 page, not the echoed marker * s3: skip the key an exclusive marker names as it streams --------- Co-authored-by: Zuse <be9c90a8-c104-4be2-b7a4-9f92eb833ac8@forge.local> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris LuandGitHub
|
beaf96a51d |
s3: cover object lock retention on version deletes (#11335)
* s3: cover WORM guarded version deletes * s3: trim version delete comments |
||
|
Chris LuandGitHub
|
93d4a6aefd |
s3: drain request body before error response (#11334)
* s3: drain request body before error response * s3: keep oversized request bodies drainable |
||
 
|
87332eb60b |
Cloud/remote storage & tiering: configurable multipart upload/download concurrency (#11319)
* pb: add multipart concurrency fields to RemoteConf and tier move requests RemoteConf gains upload_concurrency/download_concurrency (0 = client default); VolumeTierMoveDatToRemote/FromRemote requests gain a concurrency field (0 = backend default). * remote storage: honor RemoteConf upload/download concurrency in s3 and azure clients s3 client: ReadFile passes conf download_concurrency to the downloader, WriteFile uses upload_concurrency for the uploader; previously hard-coded 1 upload / 5 download parts. 0 keeps defaults. Same for azure client. * storage: plumb concurrency through backend interface and tier upload/download BackendStorage.CopyFile/DownloadFile take a concurrency hint (<=0 = backend configured default); s3 backend reads upload_concurrency/download_concurrency from scaffold config with parseConcurrency fallback, rclone updated to the new signature. Tier move gRPC handlers forward the request concurrency to the backend. * shell: -upload_concurrency/-download_concurrency for remote.configure, -concurrent for volume.tier remote.configure exposes upload/download concurrency persisted into RemoteConf; volume.tier move/evict commands forward -concurrent to the tier move requests. Documented in master-cloud.toml scaffold. * test: cover concurrency propagation in remote tier integration test * remote.configure: merge existing config on partial update Load the stored RemoteConf before saving so a partial update (e.g. only -upload_concurrency) preserves credentials, endpoints, and type instead of replacing them with new-config defaults. Only treat a confirmed ErrNotFound as a new configuration; propagate all other load errors so a transient filer failure does not overwrite stored settings. On a type transition, reset backend-specific fields to the destination type's new-config defaults rather than inheriting the old backend's empty values. Bound configured concurrency to a sane maximum. * remote storage: honor configured download concurrency in S3 and Azure ReadFileWithConcurrency now resolves a zero request override against the client's configured download_concurrency (new downloadConcurrency() helpers), so the remote-mount/cache read path honors RemoteConf.DownloadConcurrency instead of the hard-coded default. Azure also clamps the resolved value to math.MaxUint16 regardless of whether the fallback was used, preventing uint16 wraparound when a configured value exceeds 65535. * shell: rename -concurrent to -concurrency and validate tier transfer bounds Rename the -concurrent flag to -concurrency across volume.tier.upload, volume.tier.download, and volume.tier.compact to match the proto field and RemoteConf field names. Add validateTierConcurrency to reject values that would wrap int32 or exceed a 1024 cap before constructing the request. * server: clamp tier move concurrency in gRPC handlers Add clampTierConcurrency to both VolumeTierMoveDatToRemote and VolumeTierMoveDatFromRemote handlers so a direct gRPC caller cannot spawn an unbounded number of network workers. * trim verbose comments added with concurrency feature Remove redundant doc comments on the backend interface, rclone backend, s3_backend parseConcurrency, and test helpers that restated the obvious. * remote.configure: apply type defaults before re-parse so explicit flags win applyTypeDefaults ran after the second flag parse, overwriting explicit destination flags (e.g. -s3.region=eu-west-1) with new-config defaults. Move the type-transition default reset before the re-parse so user-supplied flags override the destination defaults. * remote.configure: only treat explicit -type as a type transition The first parse defaults -type to s3, so a concurrency-only update on an existing non-S3 config captured requestedType=s3 and wrongly triggered a type transition, resetting the stored backend to S3. Use fs.Visit to detect whether -type was explicitly supplied; an omitted -type keeps the stored backend. --------- Co-authored-by: Jack Meredith <9480542+jackusm@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
|
e4ca0d09e7 |
s3: preserve versions for POST policy uploads (#11316)
* s3: preserve versions for POST policy uploads Route POST policy uploads through the existing version-aware write helpers and validate promoted Object Lock headers before writing. Return the generated version ID when versioning is enabled, return x-amz-version-id: null when versioning is suspended, and omit the header when versioning has never been enabled. * s3: reuse versioning helpers in POST policy handler Route the POST policy handler through the existing getVersioningState and isObjectLockEnabled helpers instead of open-coding the object-lock forces-versioning-enabled rule, matching the PUT path. Drop the x-amz-version-id: null response header for suspended versioning; the PUT handler omits it and the S3 PutObject sample response for suspended buckets does not include it. Trim the moved fileSize comment. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
  |
01433e801d |
build(deps): bump github.com/redis/go-redis/v9 from 9.21.0 to 9.22.0 (#11306)
* build(deps): bump github.com/redis/go-redis/v9 from 9.21.0 to 9.22.0 Bumps [github.com/redis/go-redis/v9](https://github.com/redis/go-redis) from 9.21.0 to 9.22.0. - [Release notes](https://github.com/redis/go-redis/releases) - [Changelog](https://github.com/redis/go-redis/blob/master/RELEASE-NOTES.md) - [Commits](https://github.com/redis/go-redis/compare/v9.21.0...v9.22.0) --- updated-dependencies: - dependency-name: github.com/redis/go-redis/v9 dependency-version: 9.22.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * test(redis_conf): track go-redis 9.22.0 default read timeout of 5s go-redis 9.22.0 raised the default ReadTimeout from 3s to 5s (part of the cross-SDK configuration alignment). Update TestUnsetKeepsGoRedisDefaults to expect the new default so the bump in #11306 stops failing CI. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris LuandGitHub
|
0e82b4e351 |
s3: populate Initiated timestamp in ListMultipartUploads (#11313)
* s3: populate Initiated timestamp in ListMultipartUploads ListMultipartUploads returned each upload with only Key and UploadId, omitting the Initiated timestamp. Clients such as GeeseFS rely on this field to expire stale uploads and crash on its absence. Set Initiated from the upload directory entry creation time so repeated listings preserve the original initiation time. * test/s3: verify Initiated timestamp in ListMultipartUploads Add an integration test that initiates a multipart upload, lists it, and asserts the Initiated field is populated and preserved across repeated listings rather than reflecting the listing time. |
||
|
Chris LuandGitHub
|
c997e54096 |
admin: default to 0.0.0.0 for authenticated HTTP, keep worker gRPC on loopback (#11314)
* admin: extract isFlagExplicitlySet helper from applyViperFallback No behavior change; the inline flag-visit check becomes a reusable helper so the upcoming bind-address default can share it. * admin: default to 0.0.0.0 for authenticated HTTP, keep worker gRPC on loopback PR #11185 made the admin HTTP server default to 127.0.0.1 to stop exposing the unauthenticated admin API on the network by accident. That also locked out operators who already authenticated with -adminPassword: their UI became unreachable from the network after upgrade unless they added -ip=0.0.0.0 (see #11303). An authenticated deployment is safe to expose, so auto-upgrade the -ip default to 0.0.0.0 when -adminPassword or [https.admin] mTLS is configured. The loopback default stays for the unauthenticated case, so the unauthenticated API is never exposed on the network. An explicit -ip is always honored. The worker gRPC control plane has no password auth (only mTLS), so it must not follow the HTTP upgrade. Give it a separate bind address that stays on loopback unless -ip is explicit, so adminPassword no longer re-exposes the unauthenticated worker stream. * admin: hint loopback-only bind in startup banner When the admin server binds to loopback (the default for the unauthenticated case), print a one-line hint that it is not reachable from other hosts and how to expose it. This helps operators who, after the #11185 loopback default, can no longer reach the UI from another machine quickly see the cause and the fix without reading the docs. * admin: keep worker gRPC on loopback, decouple from https.admin mTLS The worker gRPC auto-upgrade to 0.0.0.0 was gated on hasMTLS, which reads the https.admin (HTTP) mTLS config. The worker gRPC mTLS comes from grpc.admin + grpc.ca, a separate config, so: - https.admin mTLS without grpc.admin mTLS widened the worker gRPC to 0.0.0.0 unauthenticated (re-exposing the control plane), and - grpc.admin mTLS without https.admin mTLS left the worker gRPC on loopback, blocking authenticated remote workers. Drop the worker gRPC auto-upgrade entirely. The worker gRPC keeps the raw -ip value (loopback by default), matching the pre-existing behavior; an operator who wants remote workers sets -ip explicitly. Only the HTTP admin listener auto-upgrades to 0.0.0.0 when authenticated. Addresses review feedback on #11314 from Devin and Greptile. |
||
|
Chris LuandGitHub
|
02749c1192 |
s3api: configurable trusted-proxy allowlist for aws:SourceIp (#11302) (#11315)
* s3api: add TrustedProxies allowlist helper for aws:SourceIp extraction Introduces a policy_engine.TrustedProxies type that parses a comma-separated list of bare IPs and CIDRs (mirroring Guard.UpdateWhiteList) and extracts the client IP for aws:SourceIp condition evaluation. When the direct TCP peer is in the allowlist, X-Forwarded-For is walked right-to-left skipping trusted hops (then X-Real-Ip); otherwise the direct peer address is returned. This is the building block for restoring configurable forwarded-header trust removed in |
||
|
|
ac03d3fd78 |
shell: warn when fs.mergeVolumes source holds only orphan needles (#11310)
* shell: warn when fs.mergeVolumes source holds only orphan needles fs.mergeVolumes traverses filer entries, so a source volume whose needles are all orphans — filer entries lost to a crashed write or a wiped filer store — produces only the plan header and exits 0: no move, no skip, no error. Operators read that as a successful merge while the real cleanup (volume.fsck) never runs, and dat>idx volumes keep coming back read-only after restarts. Count the source-volume needles seen during traversal and, when a plan source was never seen but its index still reports needles, print a warning pointing at volume.fsck. Dry-run warns too. * shell: make needle counting concurrency-safe and count manifest sub-chunks TraverseBfs runs its callbacks from five workers, so the plain needlesSeen map raced between source-heavy merges (fatal concurrent map writes). All increments now funnel through a mutex-guarded recordSeen closure. Manifest sub-chunks that live on planned source volumes are now recorded too — rewriteManifestChunk visits them (including dry-run and capacity-skipped ones) but previously never marked their source, which produced false 'orphan needles' warnings for sources whose chunks were all reached through manifests. * shell: extract sourceNeedleCounter so the concurrency test covers the production path The orphan-warning recording was a closure local to Do, so TestWarnUnreferencedSources_ConcurrentRecording could only exercise a test-local copy of it — a regression in the production mutex would pass the test. Lift the map and mutex into a sourceNeedleCounter type with record/count methods and use it from Do and the test, so the -race test now drives the actual recording path. Trim the verbose comments added with the warning while here. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris LuandGitHub
|
cf38c01978 |
admin: bind worker gRPC listener to -ip instead of wildcard (#11300)
* admin: bind worker gRPC listener to -ip instead of wildcard
The worker/plugin gRPC control plane called net.Listen("tcp", ":port")
directly, so it wildcard-bound every interface and ignored the -ip setting.
A cluster bound to loopback still exposed the unauthenticated
WorkerService/PluginControlService streams on 0.0.0.0. Bind through
util.JoinHostPort(bindIp, port) so the listener honors -ip like the
master, filer, and volume gRPC listeners.
* admin: warn when worker gRPC is exposed off loopback without mTLS
The worker gRPC stream has no password auth, so grpc.admin mTLS is the
only effective control once the listener leaves loopback. An operator who
sets -adminPassword and binds -ip=0.0.0.0 authenticates the HTTP API but
still exposes the unauthenticated worker control plane. Log a startup
warning naming the port and the mTLS knobs so the exposure is not silent.
* admin: address review on worker gRPC bind fix
- mini: reserve the admin gRPC port with util.JoinHostPort so an IPv6
bindIp (e.g. ::1) does not form an invalid unbracketed address and
lose the reservation.
- worker gRPC: track whether grpc.admin mTLS credentials actually loaded
rather than only whether they were configured, and gate the
non-loopback exposure warning on that. A cert/key that fails to load
now still warns instead of silently suppressing.
|
||
|
ea179963c0 |
filer: clean up manifest resolve error propagation and add webdav tes… (#11297)
filer: clean up manifest resolve error propagation and add webdav test (#78) Drop GitHub issue references from comments and trim verbose comments. Replace the viewFromChunksOrErr helper with the existing NonOverlappingVisibleIntervals + ViewFromVisibleIntervals at the stream call sites, and add a WebDavFile.Read regression test for the manifest resolution failure path. Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
 github-actions[bot]
|
c507336000 | 4.47 | ||
|
Chris LuandGitHub
|
38c14d3c13 |
filer: apply SSRF guard to the lazy-remote fetch/list/delete paths (#11294)
* filer: add guarded remote-storage client builder hook for lazy fetch The lazy-remote fetch path (maybeLazyFetchFromRemote) resolved its remote-storage client through the unguarded shared cache, bypassing the SSRF chokepoint (BuildGuardedRemoteStorageClient) that the CVE-2026-73080 remediation wired into the volume, filer stream and s3 stream dial paths. Add a RemoteStorageClientBuilder hook on Filer plus conf-only lookups on FilerRemoteStorage, and route the lazy fetch through the builder when set (endpoint deny-list + DNS-rebinding-safe dialer), falling back to the shared cache otherwise. The filer server wires the builder in a follow-up. * filer: route lazy directory listing through the guarded remote client maybeLazyListFromRemote shared the unguarded client resolution of the fetch path, so a caller-supplied remote endpoint was dialed without the SSRF deny-list or rebinding-safe dialer. Resolve the conf and build the client through buildRemoteStorageClient so the same guard covers listing. * filer: route lazy remote delete through the guarded remote client maybeDeleteFromRemote issued outbound DELETE/RemoveDirectory requests through the unguarded client, giving a write-side SSRF to a caller-chosen endpoint. Resolve the conf and build the client through buildRemoteStorageClient so the endpoint deny-list and rebinding-safe dialer apply to the delete path as well. * filer server: wire the guarded remote client builder into the filer Set Filer.BuildGuardedRemoteClient to BuildGuardedRemoteStorageClient and forward AllowUntrustedRemoteEndpoints so the lazy-remote fetch, list and delete paths apply the same SSRF endpoint checks as the volume and streaming read paths. * filer: test lazy fetch honors the guarded remote client builder Add a regression test that sets BuildGuardedRemoteClient to a rejecting builder and asserts maybeLazyFetchFromRemote returns no entry without reaching the remote, covering the SSRF guard wired in the prior commits. * filer: skip remote client for local-only lazy deletes maybeDeleteFromRemote resolved and validated the mount's remote client before checking entry.Remote, so a local-only file (no Remote entry) under a mount whose endpoint the guard rejects failed to delete: the guard error aborted the metadata deletion, leaving a file that needs no remote operation undeletable. Move the local-only check ahead of client construction so only remote-backed files and directories pay the guard. * filer: build the guarded remote client inside the lazy singleflight The lazy fetch and list paths built the guarded client before their singleflight blocks, so concurrent requests for the same key each allocated a fresh SDK client and HTTP transport even though only one remote operation ran. Move client construction inside the singleflight so the deduplicated operation builds it once, matching the per-request guard semantics of the sibling streaming paths without the duplicate transport churn. * filer: test guarded rejection for the lazy list and delete paths Add regression tests that set BuildGuardedRemoteClient to a rejecting builder and assert the lazy list does not reach the remote, a remote-backed file delete is blocked, and a local-only file under a rejected mount still deletes (covering the local-only fix). * filer: decouple lazy guarded-client build from the first caller's context Building the guarded client inside the singleflight made concurrent fetches share the first caller's context. If that caller canceled while endpoint DNS validation was running, the builder returned an error and published a not-found result to other callers whose contexts were still valid. Build with context.WithoutCancel so the guard's DNS validation is not tied to any single caller's cancellation, matching the list path's existing decoupling for the remote operation itself. * filer: reject remote-storage confs that dial blocked endpoints at load The filer's lazy-fetch / lazy-list / remote-delete paths resolve remote storage clients by name from FilerRemoteStorage.storageNameToConf and dial them via remote_storage.GetRemoteStorage, which bypasses the SSRF deny-list the volume server (BuildGuardedRemoteStorageClient) and the filer's own direct-read path apply. A RemoteConf planted under /etc/remote with a loopback / private / IMDS S3 endpoint is reloaded into storageNameToConf on the next metadata-change event and then dialed on the next cache miss — server-side request forgery from the filer. Apply the volume server's SSRF deny-list at conf load time, the single chokepoint that populates storageNameToConf: - Add RemoteStorageConfValidator, injected into FilerRemoteStorage by the filer server (the filer package cannot import the server package). A conf that fails validation is dropped from storageNameToConf, so the name-based client resolution on the lazy paths returns "not found" instead of dialing the blocked endpoint. - Add ValidateRemoteConfForLoad in weed_server, which mirrors BuildGuardedRemoteStorageClient's gcs credential + endpoint checks (validateRemoteEndpoint via guardedRemoteClient) without building a client. allowUntrusted skips the check, mirroring the volume server opt-out (-filer.allowUntrustedRemoteEndpoints). - The filer server injects the validator at construction. A conf whose type dials a fixed provider host (no caller-supplied endpoint) passes; only caller-influenced endpoints are denied. * filer: skip DNS resolution in the load-time SSRF validator ValidateRemoteConfForLoad resolved hostnames during /etc/remote reload, so a transient DNS failure (2s timeout) dropped the conf from the fresh map that replaces the live map, disabling a working mount until the next metadata event. The build-time guard (BuildGuardedRemoteStorageClient) already re-resolves and re-validates the endpoint at dial time with the rebinding-safe dialer, so DNS at load is redundant for security. Split the static checks (scheme, IMDS hostnames, IP-literal blocked addresses, gcs credentials) into validateRemoteEndpointForLoad, which does no DNS. Hostname endpoints pass at load and are caught at dial if they resolve to a blocked address. This preserves fail-fast for statically-blocked confs (loopback IPs, IMDS hostnames) without letting transient DNS failures disable mounts. * filer: accept empty S3 endpoints in the guarded remote client builder guardedRemoteClient returned ok=true with an empty endpoint for a standard AWS S3 config (no custom S3Endpoint), so BuildGuardedRemoteStorageClient and ValidateRemoteConfForLoad rejected it with "remote endpoint is empty" — breaking standard AWS S3 mounts on the lazy paths and the sibling streaming read paths that already use the guarded builder. An empty endpoint is not caller-supplied: the AWS SDK derives the regional endpoint from the region, so there is nothing for the SSRF guard to validate. Return ok=false for empty S3-compatible endpoints so the builder falls through to the shared unguarded cache, matching the historical behavior for standard AWS S3. |
||
|
Chris LuandGitHub
|
92c379e5b4 |
filer: accept gcs credentials file paths in the guarded remote client builder (#11296)
* filer: accept gcs credentials file paths in the guarded remote client builder checkGcsCredentials rejected all filesystem paths, so a gcs mount configured with remote.configure -gcs.appCredentialsFile (which stores a path in GcsGoogleApplicationCredentials) was rejected by BuildGuardedRemoteStorageClient with "gcs credentials must be inline JSON". This broke existing gcs mounts on the volume, filer, and s3 remote-mount read paths that use the guarded builder. Read and validate the file content instead of rejecting the path, mirroring what the gcs client itself does in MakeWithHTTPClient. A path that does not exist or does not contain valid gcs credentials is still rejected before any client is built. guardedRemoteClient now reads the file to extract the token exchange URL for the SSRF deny-list, so the rebinding-safe dialer still guards the token endpoint. * filer: resolve gcs credential paths and avoid leaking file existence loadGcsCredentialsContent passed the raw credentials string to os.ReadFile, so a documented ~/path (as written by remote.configure -gcs.appCredentialsFile=~/...) was rejected because os.ReadFile does not expand ~. It also wrapped the os.ReadFile error, which includes the file path, exposing file existence to a caller who planted a conf with an arbitrary path. Resolve the path with util.ResolvePath, matching the gcs client's own behavior in MakeWithHTTPClient. Return a generic sentinel error on read failure so the path is not reflected in the error message. The credential type validation still runs on the file content, so a path that does not contain valid gcs credentials is rejected before any client is built. |
||
|
Chris LuandGitHub
|
bea10e269f |
iceberg/s3tables: confine stored metadataLocation to the authorized table bucket (#11292)
* iceberg: confine commit/transaction/view-update write paths to authorized bucket The create, register, and createView handlers already confine the client- supplied metadata location to the caller table bucket and reject ".." segments. The commit, create-on-commit, transaction, and view-update paths read the stored metadataLocation back from the catalog and skipped the same guard, so a location poisoned via the raw S3Tables UpdateTable API (which persists metadataLocation verbatim) could escape the caller bucket through a ".." segment that path.Join collapses in saveMetadataBlob. Add confineMetadataLocation and apply it after parseS3Location on every commit/update/transaction/view write path, mirroring the create/register/ createView check. Reject with 400 so a poisoned stored location fails the commit instead of writing into another tenant bucket tree. * s3tables: validate metadataLocation at the store layer The raw S3Tables API (CreateTable, RegisterTable, UpdateTable, CreateView, UpdateView) persisted the client-supplied metadataLocation verbatim with no bucket-confinement or traversal check, so a caller could store a location pointing outside its own bucket. The Iceberg REST gateway commit paths then read that stored value back and wrote through it. Add ValidateMetadataLocation and call it in every s3tables store handler that accepts a metadataLocation, rejecting locations whose bucket differs from the caller table bucket or whose path contains traversal segments. This prevents a poisoned location from ever being persisted, complementing the per-write-path guard added to the Iceberg commit handlers. * iceberg/s3tables: validate location before repair and after idempotency check Address review feedback: - Move the commit-path confinement check ahead of repairManifests so a poisoned stored location cannot reach manifest repair I/O before the commit is rejected. - Move ValidateMetadataLocation in CreateTable/CreateView to after the existing-resource check so idempotent retries that do not consume the requested location are not rejected for an unused bad location. - Assert HTTP 400 in the cross-tenant reproduction tests so an unrelated failure cannot satisfy them. * iceberg: confine staged metadata location before load in create-on-commit The create-on-commit path parsed the staged metadata location from the stage-create marker and called loadMetadataFile before validating that the staged bucket/path stay within the authorized bucket. Add the same confineMetadataLocation guard before the read so a tampered marker cannot direct a cross-tenant metadata read. * iceberg/s3tables: reject bucket-only metadata locations ValidateMetadataLocation and confineMetadataLocation accepted s3://bucket with an empty table path. metadataDirPath then maps every such table to the shared <TablesPath>/<bucket>/metadata directory, so tables could overwrite or read each other's metadata files. Require a non-empty table path in both validators; the empty-location case (where the catalog derives one) is unaffected. * iceberg/s3tables: reject slash-only table paths in location validation s3://bkt/// parses to tablePath="/" which passed the empty-string check but path.Join cleans it away, mapping to the bucket-level metadata directory shared across tables. Update isValidTablePath to require at least one non-empty segment and mirror the same check in ValidateMetadataLocation, closing the gap in all callers. |
||
|
Chris LuandGitHub
|
10c0857476 |
s3: gate internal LifecycleDelete gRPC behind admin Bearer auth (#11291)
* s3/lifecycle: attach admin Bearer token on internal LifecycleDelete clients Export credential.WithS3InternalAdminAuth (renamed from withIamCacheAdminAuth) and use it in the worker and shell lifecycle RPC adapters so lifecycle calls carry the same admin token the IAM-cache propagation already attaches. No-op when jwt.filer_signing.key is unset, matching the server-side checkAdminAuth. Prepares the internal clients for the server-side auth gate that follows. * s3/lifecycle: gate LifecycleDelete behind admin Bearer auth Add checkAdminAuth to LifecycleDelete, matching the SeaweedS3IamCache handlers on the same internal gRPC listener (PR #11190). No-op when jwt.filer_signing.key is unset; rejects unauthenticated callers when it is. The internal worker/shell clients already attach the token in the previous commit. |
||
|
|
c462fffce6 |
master: name the unlabeled disk layout plainly in assign errors (#11290)
* master: name the unlabeled disk layout plainly in assign errors When no volume server serves the layout an assign targets, the error named the empty disk type as "hdd" (HardDriveType is the empty string), sending operators looking for servers labeled hdd when the actual mismatch is labeled (e.g. -disk=ssd) servers versus unlabeled clients. - describe the layout as "default (unlabeled)" when the disk type is empty, keep %q naming for labeled types - log the unserved-layout condition once per option instead of letting every failing write repeat an unactionable line Observed in production: volume servers started with -disk=ssd while CSI mounts assign with the unlabeled layout; the per-write error stream pointed at a nonexistent hdd fleet. * master: bound and expire the unserved-layout warning dedupe The dedupe map retained every distinct option key permanently. Option keys embed request-derived fields (collection, disk type), so repeated assignments with distinct options would grow master memory without bound, and a retained key suppressed the warning if the same option went unserved again after the topology recovered. Remember last-warned timestamps instead, expiring after an hour, with a hard cap that resets the set when a client-driven key flood fills it. * master: silence per-retry unserved-layout log and name explicit hdd Addresses Devin Review comments on #11290. - The unserved-layout branch already rate-limits its warning via assignUnservedLayoutWarning.Do, but the common epilogue still logged lastErr at V(0) on every retry, so the flood the dedup was meant to stop continued. Skip the epilogue log when the unserved-layout branch owns the logging; the error is still returned to the client. - describeDiskLayout took the canonicalized option.DiskType, but ToDiskType folds both "" and "hdd" into HardDriveType, so an explicit disk=hdd request was mislabeled "default (unlabeled)". Pass the original request disk type instead: only an empty request is the unlabeled default; an explicit hdd is named "hdd". Adds TestAssignFailsFastNamesExplicitHdd covering the explicit-hdd wording. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris LuandGitHub
|
99d2479528 |
fix(vacuum): batch fsync in makeupDiff to prevent test timeout (#11289)
makeupDiff called dstDatBackend.Sync() (fsync) per needle in the loop over incrementedHasUpdatedIndexEntry. With 20000 entries in TestLDBIndexCompaction this resulted in up to 20000 fsync calls, which on slow CI disks exceeded the 10-minute test timeout. Batch the sync: write all needles/tombstones first, then fsync the dat file once in the defer alongside the existing idx fsync. The durability guarantee is unchanged — both files are still synced before CommitCompact writes the .cpc commit marker and swaps the files. |
||
|
|
8db41d0217 |
[Mount] Cache Chunk Manifest Resolution for Repeated File Opens (#11266)
* cache resolved chunk manifests for Mount * Address PR review: per-mount cache, singleflight, reuse ResolveOneChunkManifest - Own the manifest cache per WFS mount instead of a process-global variable, so manifests from one filer backend are never served to another (Devin/CodeRabbit major bug). - Coalesce concurrent cold misses via singleflight so only one fetch runs during a cold burst (Greptile P2). - Copy cached data after releasing the mutex so a large copy does not block concurrent hits, inserts, and evictions (CodeRabbit nitpick). - Reuse the existing ResolveOneChunkManifest function name instead of introducing a new resolveOneChunkManifest wrapper. - Validate (unmarshal) manifest bytes before caching so malformed manifests do not poison the cache. - Add TestChunkGroupManifestResolutionCoalescesColdMisses covering the singleflight cold-miss path. * Address round 2 review: coalesced-miss cancellation, test overlap - Use singleflight.DoChan in fetchOrLoad and select on ctx.Done() so a caller whose context is canceled while waiting for an in-flight fetch returns ctx.Err() promptly instead of blocking for the leader's result (Devin BUG). - Add TestResolveOneChunkManifestCanceledWaiterReturnsDuringCoalescedMiss covering the canceled-waiter path. - Delay the cold-miss fixture response so the leader's fetch is still in flight when concurrent opens join the singleflight, making the one-fetch assertions reliable (CodeRabbit Minor). * Address review: keep ResolveOneChunkManifest four-argument Restore the exported ResolveOneChunkManifest to its original four-argument signature so external callers keep compiling. Move the cache-aware resolution into an unexported resolveOneChunkManifest helper that accepts the per-mount ChunkManifestCache. The exported function delegates to the helper with a nil cache, preserving the historical uncached behavior for every non-Mount caller. The Mount path (ChunkGroup.SetChunks) now calls the unexported helper with the mount-owned cache. Tests and benchmarks that exercise the cache path call the unexported helper directly. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Bruce ZouandGitHub
|
eb6a7e93ca |
Fix mount eio on manifest resolve failure (#11287)
* mount: fail reads with error when chunk manifest resolution fails When SetChunks fails to resolve a chunk manifest (e.g. the volume is on a remote tier with reads disabled), the sections map stays empty and readDataAtSequential/readDataAtParallel zero-fill every missing section as if it were a sparse hole. Reads then return all-zero data with no error, so a plain cp of a large manifest-based file silently produces a completely zero-filled file. Remember the resolve error in ChunkGroup (guarded by sectionsLock) and return it from ReadDataAt. A later successful SetChunks clears it. Fixes the mount path of #11286. * filer: propagate manifest resolve errors in streaming read paths ViewFromChunks discards the chunk manifest resolve error returned by NonOverlappingVisibleIntervals. On failure the chunk views come back empty, and the streaming paths zero-fill the entire requested range, serving HTTP 200 / WebDAV 200 responses whose body is all zeros. Propagate the error in PrepareStreamContentWithThrottler, PrepareStreamContentWithPrefetch and the WebDAV read path so these requests fail with 500 instead. Fixes the filer HTTP and WebDAV paths of #11286. * mount: fail lseek with EIO when chunk manifest resolution fails SearchChunks still consulted the stale section map after SetChunks recorded a manifest resolution failure, so SEEK_DATA/SEEK_HOLE would describe the unresolved regions as sparse holes or return ENXIO. Return the recorded error from SearchChunks and map it to EIO in Lseek. Also add regression tests for the stream preparation error paths. Addresses review feedback on #11287. |
||
 
|
5b2fe374fc |
[Volume] Scrub every disk's EC shards for a volume id, not just the first (#11258)
* storage: add Store::find_all_ec_volumes for split-disk EC lookups Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE * ec: add merge_ec_runtimes to resolve a vid's per-disk shard set Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE * ec: replace dead slots.get(14) assertion with a width-14 pin Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE * ec: build the checksum scrub plan from every per-disk runtime Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE * ec: build the local scrub plan from every per-disk runtime Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE * ec: prove the local scrub plan reaches every runtime's slots Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE * ec: make the scrub plan tests falsifiable Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE * ec: report unverifiable protection when the sidecar predates the scrubbed encode Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE * ec: commit sidecar provenance with the sidecar it describes Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE * volume server: scrub every disk's EC shards for CHECKSUM and LOCAL Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE * ec: run the FULL/READS parity check across split-disk shards Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE * volume server: report fenced-out runtimes in FULL/READS scrubs Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE * ec: tighten verify_ec_shards ordering and missing-shard coverage Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE * volume server: visit each EC volume id once in node-wide scrubs Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE * ec: cover split-disk scrub aggregation end to end Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE * volume server: pin fenced-out disks and sibling-disk shards in EC scrubs Three scrub behaviors shipped without a test at the RPC seam. Task 8 showed the seam exists, so close them here. FULL/READS (mode 2|5) now marks a volume broken when the identity fence excludes a runtime, where it previously reported clean. Pinned against a control fixture whose two disks AGREE and scrub clean, so the test fails on the clean->broken transition, not only on the message text. That needs a structurally valid, tombstone-only .ecx (so the needle walk finds nothing to complain about) and a seeded shard-location cache (so the absent master does not short-circuit the scrub with an error of its own). LOCAL (mode 3) and CHECKSUM (mode 4) now build their plans from every per-disk runtime. Made observable by moving shard 0 -- the shard the volume's single needle spans and the one the checksum sidecar is checked against -- to the SIBLING disk, leaving shard 5 on the disk the singular find_ec_volume lookup returns. Built from that disk alone, neither scrub ever looks at shard 0. The split-disk fixture grows a config struct rather than more positional arguments; its defaults reproduce the existing layout byte for byte, so the node-wide dedupe test is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE * ec: report fenced-out disks on a malformed sidecar too `errors.extend(self.skipped)` sat below the whole status match, so only `(Some(p), On)` ever reached it. The Invalid arm already returns a non-empty error vector of its own, so the Go-parity contract that silences the Off arm (`case BitrotOff: return 0, nil, nil`) does not reach it -- appending the fence lines there costs nothing that contract protects. A volume with BOTH a malformed sidecar and a disk the identity fence excluded reported only the sidecar, hiding the unscanned disk behind an unrelated integrity error. Off stays byte-identical, and so does the `(None, On)` arm that is documented as treating a missing payload defensively as protection off. Off is now the ONLY status that drops the report, and the comment at the On-path copy says so: that is the one place the parity constraint costs us coverage. Also corrects a false claim in the FULL/READS test's doc comment. It said a fenced-out disk "is a disk this scrub did NOT read", which is true only of the merge-driven parity half. The per-needle walk still resolves `store.find_ec_volume` (store_ec.rs:281) and binds `expected_encode_ts_ns` to that runtime (:311) -- position 0, the EXCLUDED one on that fixture -- so `read_local_intervals`' generation filter (:1204) makes it read the excluded disk and treat the anchor's shards as non-local, the inverse of what `skipped` reports. The fixture's tombstone-only .ecx walks nothing, so the test cannot tell the two apart; the comment now says that rather than implying coverage it does not have. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE * ec: take CHECKSUM's bitrot protection from the disk that has the sidecar `EcChecksumScrubPlan::for_volumes` read `(prot, status)` off the ANCHOR. The anchor is the first shard-bearing runtime at the maximum `encode_ts_ns`, chosen with no regard for which disk holds the `.ecsum`. That sidecar is deliberately NOT mirrored across disks -- `ec_metadata_dirs()` exists so one authoritative copy stays reachable rather than being duplicated -- and at mount `EcVolume::new` resolves it via `load_active_bitrot_sidecar(&[])` with no sibling directories at all; only the `VolumeEcShardsMount` RPC ever passes `ec_metadata_dirs()`. So after EVERY volume-server restart, the split-disk runtime that does not physically hold the sidecar mounts `BitrotStatus::Off`. When the one copy lives on disk 1 and the anchor is disk 0, `run()` hit `case BitrotOff` and returned `(0, [], [])`: the whole volume scrubbed clean, silently. That is the steady state for roughly half of all mirrored split-disk layouts, and it is the exact failure this branch exists to remove. Source protection from the first MERGED runtime that has any -- `On` if one does, else `Invalid`, else the anchor's `Off`. Two facts make that safe, and both are load-bearing: - Every runtime that mounted `On` already passed the `geometry_matches` gate in `load_bitrot_for_generation`, so its manifest agrees with the volume's layout. A sidecar that contradicted it would have failed the mount. - All merged runtimes share the same `encode_ts_ns` by construction of the identity fence, so a sidecar from any of them describes the same encode run. The `unverifiable_sidecar` provenance rule four lines down read `anchor.bitrot_source_dir`; it now reads the SAME runtime `prot` came from. Otherwise the two would describe different sidecars and the rule would vouch for a manifest nobody is scanning against. One consequence worth naming: that source dir is now non-empty by construction (a runtime with protection found a file), where the anchor's was often "" and short-circuited the rule -- so on a fenced volume whose anchor had no sidecar, an unverifiable-protection note now surfaces where previously nothing was reported at all. `run()` is untouched, and the `BitrotStatus::Off` arm still returns `(0, [], [])` exactly, for Go parity with `case BitrotOff: return 0, nil, nil`. `parity_shards` still comes from the anchor while `prot` may come from a sibling; the geometry gate above makes them agree, and slot-width agreement is handled separately. The test drives mode 4 through the real RPC against a split-disk volume whose sidecar exists only on dir1, and asserts up front that the anchor mounted `Off` and the sibling `On` -- otherwise it would prove nothing. Reverting this commit's one-line source change makes it report `[]` instead of `[0, 5]`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE * ec: pin the slot width, contain the shard-size fallback, and cover multi-disk FULL Five findings from the whole-branch review, none of which changes what a healthy volume reports. Slot width was undefined and the two consumers disagreed (ec_volume.rs). `merge_ec_runtimes` sizes `slots` to the WIDEST merged runtime, but the identity fence keys on `encode_ts_ns` alone and never on geometry -- so two same-generation runtimes whose `.vif`s disagree do merge. The mode 2|5 arm truncates to the anchor's `data+parity` and silently drops the surplus slots, while `EcChecksumScrubPlan::for_volumes` iterated the full width and emitted "present but missing from sidecar manifest" for exactly those ids. Nothing in the volume describes them -- the sidecar manifest and the Reed-Solomon matrix are both the anchor's -- so that message was the width disagreement talking, not a finding. The `slots` field doc now states the contract (the range is the anchor's geometry; every consumer truncates to it) and CHECKSUM truncates. The LOCAL `shard_size` fallback had grown a node-wide blast radius (ec_volume.rs). `anchor.shard_file_size()` returns the anchor's FIRST held shard, not a maximum. Before aggregation the plan read only that runtime's own shards, so a truncated shard was contained to its disk; now that one value sizes every merged sibling's shards, mis-offsetting `locate_data` and manufacturing needle corruption across the node. Take the max over the merged slots, which is how `verify_ec_shards` already answers the same question (`if size > shard_size { shard_size = size }`). Only on the legacy `dat_file_size == 0` path. Multi-disk `all_local` had no end-to-end test (grpc_server.rs). The parity check is gated on every shard being present, and the one all-local fixture keeps them in a single directory, so every entry of `dirs` is the same string and a permutation or off-by-one in the `slots` -> `dirs` mapping is invisible; `test_verify_ec_shards_reads_shards_from_multiple_dirs` builds its `dirs` by hand and never goes through `merge_ec_runtimes`. The new fixture is a real 10+4 encode split 0..=6 / 7..=13 across two store locations (the `.dat`/`.idx` stay outside both, so `prune_incomplete_ec_with_sibling_dat` has nothing to act on), driven through the real RPC: clean first, then a corrupted PARITY shard on the SECOND disk -- which only the parity half can see, and only through a correct mapping. Shifting that mapping by one, or computing `all_local` from the anchor alone, both make it report `[]` instead of `[13]`. Deleted `test_ec_volume_enumeration_is_deduped` (store_ec_reconcile.rs). It built `raw` from `store.locations` and then applied its OWN inline `filter(|v| seen.insert(*v))`, asserting on that -- a property of `HashSet::insert`, never reaching the production dedupe. That path is covered by `test_scrub_ec_volume_node_wide_dedupes_a_split_disk_volume`, which does fail (2 != 1) when the dedupe is removed. Corrected `test_verify_ec_shards_treats_a_none_dir_as_missing`'s docstring (ec_encoder.rs). It claimed the unmounted shard "must not drag the shards that ARE mounted down with it", but `dirs[5] = None` puts shard 5 in `broken_shards` before the block loop, so every iteration takes the `read_failed` arm and the parity comparison never runs: corrupting a mounted shard in that fixture changes nothing about the result. The assertions are unchanged; the docstring now states what they actually establish. Also refreshed two comments that cited `shard_file_size() - 1` as the reason `merge_ec_runtimes` prefers a shard-bearing anchor -- true before this commit, stale after it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE * ec: correct the Fix 1 rationale and truncate the shard-size scan The safety argument attached to `EcChecksumScrubPlan::for_volumes`'s protection selection was false as written, and it is the argument a reviewer reads first. `geometry_matches` compares a sidecar against the MOUNTING runtime's own data/parity/block size, not the anchor's, and returns true vacuously when `ec_shard_config` is `None` -- so it establishes agreement only when all merged runtimes share one geometry, which an `encode_ts_ns`-only fence does not guarantee and which `test_checksum_scrub_truncates_slots_to_the_anchors_geometry` constructs a counterexample to. The second clause was weaker than stated too: a `.ecsum` records no encode identity at all, so merged runtimes agreeing on `encode_ts_ns` does not transfer to the sidecar. Replace it with the property that is true, checkable from the selection itself, and stronger for what actually matters. `anchor` is an element of `merged`, so the `.unwrap_or(anchor)` fallback is reached only when no merged runtime is `On` and none is `Invalid` -- in which case the anchor is necessarily `Off`. The status can therefore only move `Off -> On`, `Off -> Invalid` or `Invalid -> On`; never `On -> Off`, never `Invalid -> Off`. This selection cannot stop a volume that was being scanned from being scanned, and cannot turn a reported integrity error into silence: every change it makes is toward more verification. The comment now also states what it does NOT establish -- geometry agreement is not guaranteed -- and names geometry fencing as the follow-up that would close it. Second, `EcLocalScrubPlan::for_volumes`'s `shard_size` max scanned the FULL slot width, violating the `slots` contract documented in the same commit that introduced the max: the volume's shard-id range is the anchor's geometry and every consumer must truncate to it. Pre-fix that input could not exist, because `anchor.shard_file_size()` read only the anchor's own anchor-sized vector -- so the max opened a new, narrow path to the same node-wide mis-sizing it exists to close (same-generation runtimes with disagreeing `.vif`s, the wider one holding an out-of-geometry shard larger than the in-geometry ones, `dat_file_size == 0`). `.take(anchor.data_shards + anchor.parity_shards)` mirrors the truncation already applied to the CHECKSUM shard scan. The sibling `shards:` vector is left untruncated on purpose: every access in `EcLocalScrubPlan::run` is `shards.get(sid)` with `sid < data_shards`, so the surplus entries are inert. No behavior change for any healthy volume, and no test added -- the suite is unchanged at 575 passing, 0 failing, 0 warnings. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE * ec: aggregate split-disk runtimes in Go scrubs, mirroring Rust Go volume scrubs previously used FindEcVolume (first runtime only), so a volume whose EC shards are split across multiple disks was scrubbed against just one disk's shards and the others were silently skipped. Node-wide ScrubEcVolume also appended each disk's EcVolumeIds without deduplication, scrubbing a split-disk volume once per disk. Add MergedEcRuntimes/MergeEcRuntimes (Go counterpart to Rust's merge_ec_runtimes): select the maximum EncodeTsNs as the anchor generation, fence out runtimes whose encode generation or geometry (DataShards, ParityShards, BlockSize) disagrees with the anchor, merge shard handles by shard ID, and report excluded runtimes rather than dropping them. Wire it into every scrub mode: - INDEX: scrub the anchor's index, report skipped runtimes. - LOCAL: aggregate local shards across all merged runtimes via a synthetic EcVolume built from the merged shard slots. - FULL/READS: resolve the runtime matching the anchor's encode generation (not the first match) so the needle walk and parity phase inspect one encode run; report skipped runtimes. - CHECKSUM: take bitrot protection from the first merged runtime that has a valid sidecar (On, else Invalid, else anchor's Off), preserve invalid sidecar errors from every other merged runtime, and report skipped runtimes. Deduplicate EC volume IDs in node-wide ScrubEcVolume so each volume is scrubbed exactly once. Refactor ScrubEcVolume to share the per-needle walk via scrubEcVolumeWalk, called by both the legacy first-runtime path and the new merged path. Add Go regression tests covering split-disk deduplication, encode-generation fencing, geometry fencing, sibling-disk LOCAL reach, and merge anchor selection. Rust: keep the previously-landed merge/fence/checksum changes intact; revert incidental cargo-fmt drift from unrelated files so the diff stays focused. * ec: fence merged CHECKSUM on sidecar encode generation and fix legacy shard size Address two review findings on the Go merged-runtime scrub: 1. Sidecar provenance: a merged runtime can load a bitrot sidecar from a sibling metadata directory (ReloadBitrotSidecar), and the merge fence may then exclude the runtime owning that directory. Generation-0 sidecars do not identify the encode run, so geometry validation alone cannot prove the borrowed manifest describes the anchor shards. If the sidecar records a non-zero EncodeTsNs that disagrees with the anchor, refuse the scan instead of applying stale checksums to current shards and reporting false corruption. 2. Legacy shard size: for volumes without datFileSize in .vif, LocateEcShardNeedleInterval derives the shard size from Shards[0].ecdFileSize. The merged shard set is compacted in shard-ID order, so a truncated lowest-ID shard would shrink every interval and misread intact sibling shards. Synthesize a datFileSize from the maximum mounted shard size when the anchor lacks one, so the datFileSize>0 path uses the largest shard size across all merged runtimes. * ec: fix copylocks, legacy shard boundary, and encode-aware Rust lookups Address review findings from CodeRabbit and Devin: Go (ec_volume_merge.go): - Remove bitrotLock copy from the synthetic EcVolume: copying a sync.RWMutex is a go vet copylocks error. The synthetic volume uses its own zero-value mutex; bitrot/bitrotStatus are set directly before ChecksumScrub reads them via BitrotProtection(), so no concurrent access occurs. - Fix legacy shard-size boundary: synthesize datFileSize from (maxShardSize - 1) * DataShards, not maxShardSize * DataShards, to match the legacy fallback in LocateEcShardNeedleInterval (ecdFileSize - 1). An exact large-block boundary is ambiguous; the unadjusted size would select an extra large row and misread intact sibling shards. Rust (store_ec.rs): - Add find_ec_volume_for_scrub helper that resolves by encode generation (not first-match find_ec_volume) and use it in scrub_snapshot_under_lock, write_back_shard_locations, and the post-refresh shard-location read. Previously the encode-aware lookup was only used for the initial runtime selection; the cache write-back and per-needle snapshot still used first-match, so a split-disk volume whose first runtime was from an older encode run would write to and read from the wrong runtime's shard-location cache and falsely abort with 'remounted as a different encode run'. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris LuandGitHub
|
c46f82d29a |
fix(master): stop goraft server on shutdown and bump raft to v1.2.1 (#11284)
MasterServer.Shutdown only stopped the Hashicorp raft implementation; when using the default goraft backend, the raft event-loop goroutine (leaderLoop/followerLoop) kept running after the master shut down. In the in-process test harness this leaked goroutines across sequential test runs, and a stale event occasionally reached a leader at term 0 and tripped the goraft "leader.elected.at.same.term" assertion, crashing the whole test binary (CI run 34670959967, PR 11279). Stop the goraft server in Shutdown() so its goroutines exit cleanly, and bump seaweedfs/raft to v1.2.1 which replaces that assertion with a graceful step-down to Follower instead of a panic. |
||
|
Chris LuandGitHub
|
5a0e017457 |
s3: reject virtual-host bucket retargeting via X-Forwarded-Host (#11281)
* s3: reject virtual-host bucket retargeting via X-Forwarded-Host SigV4 verification tries the client-supplied X-Forwarded-Host as a signed host candidate, while routing and IAM select the bucket from the actual Host header. A presigned URL for one virtual-host bucket could therefore be retargeted to another bucket accessible to the same signing identity by changing Host and adding X-Forwarded-Host. After the signature matches a host candidate, extract the bucket that the candidate implies (via the configured virtual-host domains) and compare it with the bucket the router selected. Reject when they differ, before returning success. * test(s3api): cover virtual-host presigned URL retargeting Add unit tests for bucketFromVirtualHost and end-to-end tests that reproduce the X-Forwarded-Host retargeting attack for both presigned and signed requests, plus a negative test confirming the legitimate same-bucket case still verifies. * s3: harden bucketFromVirtualHost for case and overlapping domains Compare host and domain suffixes case-insensitively so a mixed-case X-Forwarded-Host cannot bypass the consistency check. Only treat the exact path-style domain as non-virtual-host; subdomains of a path-style domain still match the virtual-host router pattern and must be checked. |
||
|
Chris LuandGitHub
|
210afacd12 |
s3: close list-type / ownership-controls routing mismatch (#11280)
* s3: reject list-type paired with another operation subresource ?list-type=2&ownershipControls= routes to ListObjectsV2 (the list-type route is registered first) while the IAM action resolver resolves the ownershipControls selector to s3:GetBucketOwnershipControls. A principal denied s3:ListBucket but allowed s3:GetBucketOwnershipControls would therefore list the bucket. list-type selects an operation just like the other keys in operationSubresources, so add it there and reject the combination before routing, matching the fix for policy&tagging (#10987). * s3: resolve list-type to s3:ListBucket ahead of bucket subresources The router registers the ListObjectsV2 route ahead of the bucket subresource routes, so the action resolver should resolve list-type the same way. Without this, a request carrying list-type and another operation selector resolves to the subresource action (e.g. s3:GetBucketOwnershipControls) while being served by ListObjectsV2. The ambiguity guard rejects such combinations before routing, but resolving list-type to s3:ListBucket keeps the resolver aligned with the router, mirroring how versions is handled. * s3: match list-type=2 exactly in action resolver The router selects ListObjectsV2 only for list-type=2; other values fall through to the subresource routes. Resolve the same way so the action matches the handler for every list-type value, not just 2. |
||
|
Chris LuandGitHub
|
9f6feef299 |
feat(s3api): add bucket quota S3 extension via ?seaweedfs-quota (#11279)
* feat(s3api): add bucket quota S3 extension via ?seaweedfs-quota
Add a SeaweedFS-specific S3 subresource for bucket quota management:
PUT /{bucket}?seaweedfs-quota — set bucket quota (s3:PutBucketQuota)
GET /{bucket}?seaweedfs-quota — get bucket quota (s3:GetBucketQuota)
The request/response body is JSON:
{"quota_size": 100, "quota_unit": "GB", "quota_enabled": true}
Quota is stored on the bucket's filer entry (positive = enabled,
negative = disabled but retained, zero = no quota), matching the
existing admin REST API behavior. When quota is cleared, the bucket's
read-only flag is also lifted.
Authentication uses the existing S3 SigV4 flow — no new global secret
is needed. Authorization uses two new dedicated IAM permissions:
s3:PutBucketQuota
s3:GetBucketQuota
This allows integrations like Apache CloudStack to manage per-bucket
quotas through the S3 endpoint with a scoped credential, without
exposing the broad admin REST API or requiring a separate admin token.
The credential can be limited to s3:PutBucketQuota/s3:GetBucketQuota
only, preventing bucket deletion, user management, or cluster topology
changes.
The coarse-grained ACTION_PUT_BUCKET_QUOTA/ACTION_GET_BUCKET_QUOTA
constants are added to s3_constants, and the action resolver maps the
seaweedfs-quota query parameter to the fine-grained s3: actions for
policy evaluation.
* docs: update design for S3 ?seaweedfs-quota extension approach
Replace the broad admin REST API + bearer-token design with the narrow,
scoped S3 ?seaweedfs-quota extension. Update quota, usage reporting, and
SeaweedFS-side changes sections to reflect PR #11279.
* fix(s3api): address review comments on quota handler
Fix four issues identified by Devin, Greptile, and CodeRabbit reviews:
1. Integer overflow in convertQuotaToBytes: large quota_size values
(e.g. 8388608 TB) could overflow int64, wrapping to negative and
being silently treated as zero quota. Now returns an error when
size * multiplier would exceed math.MaxInt64.
2. Disabled quotas returned negative sizes in GET: the GET handler
returned entry.Quota directly, which is negative for disabled-but-
retained quotas. Now returns the absolute magnitude as quota_size
and derives quota_enabled from the sign, making the response
round-trippable.
3. Missing buckets returned 500 instead of NoSuchBucket: the PUT
handler treated all lookup failures as internal errors. Now
distinguishes filer_pb.ErrNotFound and returns ErrNoSuchBucket.
4. Trailing JSON was silently accepted: the decoder read only the
first JSON object without checking for trailing data. Now
requires EOF after the object, rejecting malformed payloads.
Also add tests for overflow detection and trailing data rejection.
* fix(s3api): cast math.MaxInt64 to int64 for 32-bit vet
On 32-bit platforms, math.MaxInt64 is an untyped int constant that
overflows int (32-bit) when used directly in fmt.Errorf with %d.
Cast to int64 explicitly to fix Go Vet 32-bit.
* docs: reconcile design doc with implementation and add AWS tools note
- Resolve open question about IAM endpoint path: driver accepts optional
iamUrl and defaults to <s3Url>/iam
- Add note explaining ?seaweedfs-quota is not callable by standard AWS tools
(aws s3api, s3cmd, rclone), and how this compares to MinIO and Ceph quota
APIs which also live outside the standard S3 API
* docs: fix IAM endpoint default — SeaweedFS IAM is at POST / on S3 endpoint
SeaweedFS registers its embedded IAM API at POST / on the same S3
endpoint (UnifiedPostHandler), not under /iam. The design doc
previously said the driver defaults iamUrl to <s3Url>/iam, which would
send IAM operations to an unregistered path. Correct the default to
s3Url.
Found by Greptile review on PR #11279.
* docs: fix credential model, signer, and GET response shape in design doc
Three issues found by CodeRabbit review on PR #11279:
1. Credential-scope contradiction: the doc claimed the service credential
is scoped to only s3:PutBucketQuota/s3:GetBucketQuota, but the
implementation uses it as the admin credential for all operations
(bucket CRUD, IAM user provisioning, quota). Document the actual
model.
2. S3Signer -> AWSS3V4Signer: the doc said 'S3Signer for SigV4 signing'
but S3Signer is legacy SigV2. Correct to AWSS3V4Signer.
3. GET response shape: the doc showed a single JSON example with 'GB'
for both PUT and GET, but GET always returns quota_unit 'B' and the
absolute byte count. Document PUT input and GET response separately.
|