mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-08-16 12:16:36 +00:00
2bea4dd610fec703672e82d97e1560ab100ea6ab
14560
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Lars LehtonenandGitHub
|
2bea4dd610 | chore(weed/s3api): prune dead code (#10462) | ||
  |
fe8d48b12e |
build(deps): bump cloud.google.com/go/pubsub from 1.50.2 to 1.51.0 (#10453)
Bumps [cloud.google.com/go/pubsub](https://github.com/googleapis/google-cloud-go) from 1.50.2 to 1.51.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.50.2...pubsub/v1.51.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/pubsub dependency-version: 1.51.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
 Chris LuandGitHub
|
84d3d62697 |
rust volume: mark-readonly notifies the live leader, not the static seed (#10461)
VolumeMarkReadonly mutates raft-replicated master topology, so it must reach the leader. notify_master_volume_readonly targeted the static seed (config.masters.first()), so after any master failover it hit a follower and failed "not current leader". Prefer current_master_url (the live leader the heartbeat tracks), fall back to the seed before the first heartbeat, mirroring store_ec.rs and Go's vs.GetMaster(). Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu |
||
|
Chris LuandGitHub
|
62c4333074 |
s3: list the buckets an attached IAM policy grants (#10458)
* s3: list the buckets an attached IAM policy grants ListBuckets served an identity authorized by an attached IAM policy only the buckets it had created itself. A user granted s3:ListBucket on a bucket someone else provisioned could GetObject and ListObjectsV2 against it, but the bucket never showed up in the listing any S3 client uses to build its bucket picker. The owner-index fast path is only valid for an identity whose grants name every bucket it can reach, and the routing check assumed a policy could never be enumerated. Read the names out of the policy instead: statements that allow s3:ListBucket on a concrete bucket ARN become candidates, and the per-bucket permission re-check still decides what is listed. A policy that can reach a bucket it does not name -- a wildcard resource, a policy variable, a NotResource, an STS session policy -- falls back to the full scan, which evaluates the policy per bucket. * s3: share the attached policy name lookup authorizeWithIAM and the ListBuckets enumeration both built an identity's policy names the same way, its own plus the ones from its enabled groups. Pull that into one helper so group eligibility is decided in a single place. * s3: read policy actions the way the IAM authorizer matches them The IAM authorizer matches action names case-insensitively, so a policy granting "S3:LISTBUCKET" or "S3:*" authorizes a list. The ListBuckets classifier read those actions with the local case-sensitive matcher and found no grant, so once the owner index was ready the buckets that policy allows dropped out of the listing. Match the action the looser way in the classifier: case-insensitive, and true for any pattern holding a policy variable. Over-matching only costs a candidate the per-bucket permission check then rejects, while under-matching hides a bucket the caller can read. * s3: infer a multipart grant in any case The classifier matches action patterns case-insensitively but looked the requested action up in a canonical-cased set, so "S3:UPLOADPART" missed the s3:PutObject inference that the authorizer makes. Key the set for lookup in lower case, matching how the IAM authorizer holds it. |
||
|
Chris LuandGitHub
|
5536d88fbb |
azure: let the blob endpoint be configured (#10460)
* azure: let the blob endpoint be configured The service url was always derived as <account>.blob.core.windows.net, which leaves out Azure Government, Azure China, and private endpoints. Name the blob service url instead and those accounts become reachable. The url has to be https, since the account key or the bearer token would otherwise travel in the clear. * azure: reject an endpoint that carries no hostname A url like https://:443/ has a host of ":443", so the emptiness check on Host let it through and the request only failed once it reached Azure. The hostname is what has to be there. |
||
|
Chris LuandGitHub
|
fee3fcb55a |
mount: report data sizes to df with -df.logical (#10459)
df on a mount shows the space the cluster gives up to the data: every replica of a regular volume, every shard of an ec one. That is the honest answer for capacity planning, but it is not the question a user asks when they want to know how much of their data is stored. Add -df.logical. The master reports the logical sizes alongside the raw ones: one replica per regular volume, the data shards of each ec volume counted once. Free space is divided by the copies the requested replication makes, so used plus available stays the amount of data the mount can still write, and it comes off the cluster-wide usage rather than one collection's, since capacity is cluster-wide too. Statistics through a filer resolves an unset replication to the filer's default rather than the master's, matching where the writes it is sizing for actually land. The flag governs the quota check too, so a mount has one notion of how much it is using. A filer that predates the new fields sends zeros, and the mount keeps reporting the raw sizes. |
||
|
Chris LuandGitHub
|
152f1a2096 |
master: count EC volumes in statistics used size (#10457)
Statistics aggregates the volume layouts of a collection, but EC volumes are tracked outside collectionMap, so they were reported as nothing. A mount over a cluster whose volumes have mostly been encoded showed a df used size of a few GiB against terabytes of EC data. Walk the data nodes and add the EC volumes of the requested collection. Every shard copy counts, parity included, the way a regular volume's used size counts every replica, so used size stays the space the cluster actually occupies. File count comes from the volume-wide .ecx and .ecj counts, taking the largest a holder reports rather than summing them: both files travel with the shards on a move, so several nodes can report the same tombstones. |
||
|
Chris LuandGitHub
|
3ae4e9c563 |
azure: authenticate with Entra ID instead of a storage account key (#10456)
* azure: authenticate the blob sink with Entra ID Shared account keys have to be distributed and rotated everywhere a sink runs. Leaving account_key empty now falls back to the identity chain, so a workload identity or managed identity carries the authorization instead. * azure: authenticate remote storage with Entra ID The remote storage client demanded an account key and refused to start without one. Fall back to the identity chain when it is absent, and let azure.client_id pin a user-assigned identity. * azure: reject a malformed storage account name The account name is interpolated into the service URL, so a name carrying a "/", "?" or "@" moves the authority elsewhere and an authenticated request follows it. Hold callers to Azure's own naming rule instead. * azure: keep a leftover environment key off the identity path A configured client id asks for Entra ID, but AZURE_STORAGE_ACCESS_KEY still filled in the account key behind it. An old mounted secret would go on authenticating until it rotated, and the failure then blamed the key. * azure: say what the identity path reads from the environment A pinned client id alone is not enough for workload identity: the tenant and the projected token come from the environment, and missing them only surfaces later, when a token is first requested. |
||
|
Chris LuandGitHub
|
6b6e6d8547 |
s3: apply filer identity changes despite a static config file (#10392)
* s3: apply filer identity changes despite a static config file A -config file with inline identities disabled the metadata-subscription reload entirely, leaving the best-effort filer->s3 push as the only way s3.configure changes could reach a running gateway. Reload on IAM events regardless: the merge keeps the file's identities protected, and a full credential-manager snapshot now also drops dynamic identities the store no longer has, so revocation works without a restart. * s3: log identity propagation failures as warnings * s3: retry failed IAM reloads and reconcile policies and groups An event-driven reload that fails now hands off to a coalescing retry loop, so a transient filer error cannot strand a revoked credential until the next IAM event. Full-state merges also drop dynamic policies the store no longer has, keeping the static file's, and treat the group snapshot as authoritative even when empty. * s3: serialize IAM configuration loads The SIGHUP file reload, subscription reloads, the retry loop, and the postgres poll run on different goroutines. Without an end-to-end lock a load holding an older store snapshot can commit after a newer one and revert it. Hold reloadMu from snapshot through commit in both load entry points; partial merges from pushed updates stay lock-free and self-heal through the next event-driven reload. * s3: keep static-file groups through full-state reconciliation Group names from the static config file are tracked like identities and policies, and a full snapshot that does not carry them keeps the current definition and its memberships instead of dropping them. * credential: include groups in postgres configuration snapshots Full-state reconciliation treats absent groups as deleted, so a snapshot that never carries them would erase every dynamic group. * s3: revoke static-file groups dropped from the config file A file reload is authoritative for the file's group set while keeping dynamic groups, mirroring how full snapshots are authoritative for dynamic groups while keeping the file's. * credential: fail filer snapshots on unreadable entries A skipped identity or policy file made the load report success with an incomplete snapshot, which reconciliation reads as deletion and the retry loop never sees. Unparseable content is still skipped: it is durable, matches boot behavior, and must not block reloads forever. * s3: ignore groups in static config files Groups are managed through the IAM API and the dynamic store; no deployment defines them in a bootstrap config file. Ignoring them with a warning removes the two-directional group merge: full snapshots are plainly authoritative and file reloads never touch groups. |
||
|
|
c917edde90 |
build(deps): bump github.com/aws/aws-sdk-go-v2/credentials from 1.19.28 to 1.19.30 (#10455)
build(deps): bump github.com/aws/aws-sdk-go-v2/credentials Bumps [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2) from 1.19.28 to 1.19.30. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/credentials/v1.19.28...credentials/v1.19.30) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/credentials dependency-version: 1.19.30 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
|
972edc092b |
build(deps): bump github.com/ThreeDotsLabs/watermill from 1.5.1 to 1.5.2 (#10454)
Bumps [github.com/ThreeDotsLabs/watermill](https://github.com/ThreeDotsLabs/watermill) from 1.5.1 to 1.5.2. - [Release notes](https://github.com/ThreeDotsLabs/watermill/releases) - [Changelog](https://github.com/ThreeDotsLabs/watermill/blob/master/RELEASE-PROCEDURE.md) - [Commits](https://github.com/ThreeDotsLabs/watermill/compare/v1.5.1...v1.5.2) --- updated-dependencies: - dependency-name: github.com/ThreeDotsLabs/watermill dependency-version: 1.5.2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
|
0153755287 |
build(deps): bump cloud.google.com/go/storage from 1.62.3 to 1.64.0 (#10452)
Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.62.3 to 1.64.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-cloud-go/compare/storage/v1.62.3...compute/v1.64.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/storage dependency-version: 1.64.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
|
cb2eb9b4f8 |
build(deps): bump actions/checkout from 6 to 7 (#10451)
Bumps [actions/checkout](https://github.com/actions/checkout) from 6 to 7. - [Release notes](https://github.com/actions/checkout/releases) - [Commits](https://github.com/actions/checkout/compare/v6...v7) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
|
ddae030497 |
build(deps): bump github.com/shirou/gopsutil/v4 from 4.26.5 to 4.26.6 (#10450)
Bumps [github.com/shirou/gopsutil/v4](https://github.com/shirou/gopsutil) from 4.26.5 to 4.26.6. - [Release notes](https://github.com/shirou/gopsutil/releases) - [Commits](https://github.com/shirou/gopsutil/compare/v4.26.5...v4.26.6) --- updated-dependencies: - dependency-name: github.com/shirou/gopsutil/v4 dependency-version: 4.26.6 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
|
eaf266b9df |
build(deps): bump docker/login-action from 4.4.0 to 4.5.1 (#10449)
Bumps [docker/login-action](https://github.com/docker/login-action) from 4.4.0 to 4.5.1. - [Release notes](https://github.com/docker/login-action/releases) - [Commits](https://github.com/docker/login-action/compare/v4.4.0...v4.5.1) --- updated-dependencies: - dependency-name: docker/login-action dependency-version: 4.5.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Chris LuandGitHub
|
a7f4b88a61 |
s3: require a bucket-policy action to write a bucket policy (#10444)
* s3: require a bucket-policy action to write a bucket policy PutBucketPolicy and DeleteBucketPolicy were gated on ACTION_WRITE, the same action that grants object writes. An explicit Allow in a bucket policy short-circuits IAM entirely -- authRequestWithAuthType sets policyAllows and skips VerifyActionPermission -- so anyone who could write an object could author a policy granting itself, or anonymous, anything on the bucket. That is what separates a bucket policy from the sibling bucket controls also gated on ACTION_WRITE: rewriting cors or lifecycle can destroy data, but only a policy hands out access. Give the two verbs their own actions, mapped to the AWS names that were already defined but unrouted. ACTION_ADMIN would also have closed it, but it resolves to s3:* for IAM identities, forcing a blanket grant on a user holding a precise s3:PutBucketPolicy. Admins are unaffected, since isAdmin short-circuits CanDo, and an operator can delegate with PutBucketPolicy:bucket. The route binding is asserted from the router source: checking the action constants alone still passes when the route says ACTION_WRITE. * s3: also read the action from a direct iam.Auth call in the route test Routes read iam.Auth(cb.Limit(handler, ACTION)), a multi-value pass-through: Limit returns (http.HandlerFunc, Action) and those become Auth's parameters, so the action Auth authorizes on is Limit's second argument and the two cannot disagree -- Auth(Limit(h, X), Y) does not compile. A route that skipped Limit and called Auth with its own action would compile, though, and the test reported that as a missing route rather than as the wrong action. Recognise the two-argument Auth form so it names the action instead. * s3: make the bucket-policy actions grantable through an IAM policy The new actions close the escalation only if an operator can grant them, and they were not reachable: MapToStatementAction had no entry for PutBucketPolicy, so an IAM policy naming s3:PutBucketPolicy was rejected outright with "not a valid action". GetBucketPolicy was unmapped the same way. DeleteBucketPolicy was mapped, but to ACTION_ADMIN -- granting an identity permission to delete a bucket policy handed it full administrative access. Map all three to the actions the router now uses, and add the reverse direction so an identity holding them renders back as a policy statement instead of a bare "s3:". * admin: offer the bucket-policy permissions in the user editor The two new actions are otherwise only grantable by hand-editing identity JSON or by calling the IAM API, so an operator using the UI cannot delegate bucket policy management without granting Admin. Regenerating this file also picks up codegen the repo has not taken yet: the checked-in _templ.go files were produced by templ v0.3.1001 while go.mod pins v0.3.1020, so the generator rewrites the attribute-value calls. That churn is confined to this one file; running `make generate` in weed/admin reproduces it across all 36. |
||
|
Chris LuandGitHub
|
c7d0477117 |
volume: widen the gRPC admin gate and stop it drifting (#10443)
* volume: gate the admin RPCs that only shell and workers call checkGrpcAdminAuth covered 19 of the 48 VolumeServer RPCs, so an operator who sets -whiteList expecting it to cover the gRPC surface gets partial coverage. Extend it to ten that mutate state and are only ever called by the shell or a worker: SetState, VolumeCopy, the EC generate/rebuild/copy/unmount/to-volume pair, both tier moves, and VolumeTailReceiver. That is safe because the same callers already reach gated RPCs today -- VolumeMarkReadonly, VacuumVolume*, VolumeEcShardsDelete, VolumeDelete -- so a whitelist deployment already lists those hosts. Nothing here is on a master or peer path, which is what made the earlier fail-closed gate break multi-host clusters. The split is by caller rather than by blast radius: the guard matches a peer IP against the whitelist, and a whitelist holds masters, shell hosts and workers, not every peer volume server. Gating a call one volume server makes to another would break replication, EC and tiering, so those stay open. Two test fakes embedded a nil grpc.ServerStream and only implemented Send; they now implement Context, which the streaming RPCs read to authorize. * volume: fail the build when a gRPC method skips the admin gate The admin gate is an opt-in list in a 48-method service, which is how it drifted down to covering 19 of them: nothing tied adding an RPC to deciding whether it needed the gate. Parse volume_server.proto, walk the AST of every *VolumeServer method, and require each RPC to either call checkGrpcAdminAuth or appear in ungatedVolumeServerRPCs with the reason it stays open. A stale entry naming an RPC that no longer exists fails too, so the list can't quietly stop exempting anything. The exemptions are the cluster-internal calls -- replica sync, EC shard distribution, vacuum reads, backup, tailing -- plus the read-only and liveness RPCs. Closing the cluster-internal ones needs a peer identity rather than an IP whitelist; recording them here makes that a visible decision instead of an omission. The AST walk also corrects the count: a line-window scan credits VacuumVolumeCheck and VolumeServerStatus with a neighbouring function's guard. |
||
|
Chris LuandGitHub
|
be81b9d5d7 |
volume: fix EC decode/reconstruct index locality under -dir.idx (#10442)
* volume: fix EC decode/reconstruct index locality under -dir.idx EC->replicated decode failed under -dir.idx and on multi-disk with "volume not found on disk". The reconstruct rebuilds the .dat on the data disk but the on-demand VolumeMount scans only the data directory, matching on .idx/.vif; with the rebuilt .idx off in the index directory it matched the volume's leftover EC .vif and skipped the volume as EC metadata. - Resolve the EC .ecx local-first: prefer the copy co-located with the shards over the shared -dir.idx copy, with a non-empty preference so a 0-byte local stub still yields to a valid sibling (the cross-disk fallback). - Co-locate the rebuilt .idx with the .dat at the end of the reconstruct so the mount finds it; sweep .ecx/.ecj from both the data and index directories on Destroy so a stale copy cannot re-mount as a phantom EC volume. - Add VolumeConsolidateIndex: once the EC shards are deleted, unmount, move the .idx/.sdx from the data disk back to the -dir.idx directory (copy fallback across filesystems), and remount. A no-op without -dir.idx. * volume: tests for EC index locality (local-first .ecx, sweep, consolidate) - NewEcVolume prefers a non-empty local .ecx over the shared index dir, and a 0-byte local stub yields to a non-empty shared copy (the #9212 fallback). - Destroy sweeps .ecx/.ecj from both the data and index directories. - ConsolidateVolumeIndex moves a co-located index back to the -dir.idx dir and keeps the volume mounted; no-op without a separate index dir. - RenameOrCopyFile moves a file and drops the source. * volume: relocate the decoded index in place, without a read gap ConsolidateVolumeIndex previously unmounted the volume, moved the index, and remounted it. Between the EC-shard delete and the remount the volume had neither a normal nor an EC form mounted, so a read landing in that window got a not-found (or was proxied away). Move the index in place instead: RelocateIndexTo takes the data-file write lock, closes the needle map and data backend, moves the .idx (and derived .sdx), then retargets dirIdx and reloads — the same close-swap-load CommitCompact uses. The volume never leaves the mounted set, so a concurrent read blocks briefly on the lock rather than failing. The test now writes a needle before consolidating and reads it back after, proving the in-place reload keeps the volume serving. * volume: address review — maintenance guard, no orphan on copy failure - VolumeConsolidateIndex now rejects the request under maintenance mode, like VolumeConfigure and the other mutating volume RPCs. - RenameOrCopyFile rolls the cross-device copy back when the source cannot be removed, so a failed move never leaves two divergent copies (the loader would keep the data-dir one while the idx-dir orphan goes stale). - RelocateIndexTo logs a failed reopen-after-failed-move instead of swallowing it, since that leaves the volume unusable until the next load. |
||
|
Chris LuandGitHub
|
83f754763e |
filer: make the redis connection settings configurable (#10441)
The sentinel stores hardcoded a 30s read timeout and a 1m retry backoff. After a sentinel failover every request that picked a pooled connection to the old master sat there for 30s before the connection was retired, and the pool timeout derived from it (read timeout + 1s) queued the rest behind them. The other redis stores took the go-redis defaults with no way to tune anything. Read the dial, timeout and pool knobs from each redis store section instead, keeping the go-redis default for every key left unset. |
||
|
Chris LuandGitHub
|
5cac980b32 |
filer: drop a caller's jwt query param on a proxied read (#10440)
security.GetJwt reads the "jwt" query parameter before the Authorization header, and the proxy forwarded the caller's whole query apart from proxyChunkId. So on a read, where the filer mints a volume token and sets the header itself, a caller-supplied ?jwt= silently outranked it: the volume server validated a credential the caller chose rather than the one the filer attached, and the read failed with a 401 the filer could not explain. Drop it on the read path, where the filer owns the credential. Writes keep theirs -- the proxy forwards a writer's own AssignVolume token either way, so the query parameter is just a second channel for the same credential and stripping it would break a caller that presents it that way. Nothing in the tree passes a jwt by query; maybeAddAuth always sets the header. |
||
|
Chris LuandGitHub
|
6824619c16 |
s3: chunk uploads at the filer's maxMB (#10439)
The S3 write path cut fixed 8MB chunks, so an object stored through S3 chunked differently from the same bytes stored through the filer, WebDAV or a mount, and -maxMB had no effect on it. Read maxMB from the filer configuration at startup and use it, falling back to 8MB when the filer reports none. |
||
|
Chris LuandGitHub
|
0f718f8509 |
filer: add a placement overlay seam for the write path (#10437)
* filer: add a placement overlay seam for the write path New volumes take their disk type, replication, and data center from the explicit request or the matched filer.conf rule. That leaves no way for a feature to steer a whole collection onto a medium without an operator writing an fs.configure rule by hand. Add a generic PlacementOverlay hook on the filer: a func that maps a collection to a placement override, installed by a factory the way the plugin-worker handlers register. detectStorageOption consults it between the explicit request value and the filer.conf rule, so it overrides the rule but yields to a value the caller asked for. The seam names no feature concepts, so it stays generic; a downstream build registers the overlay it wants (e.g. a storage-class Landing tier). Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu * filer: address review on the placement overlay seam Honor ResolvePlacement's ok flag explicitly rather than relying on empty values falling through the util.Nvl chain, and log at V(4) when the overlay steers a collection. Document that RegisterPlacementOverlay is init-only, so the unsynchronized read in NewFiler cannot race the write. Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu |
||
|
Chris LuandGitHub
|
91979ec51e |
filer: fix credential handling on the proxyChunkId path (#10434)
* filer: claim the base fid when minting a volume read token GenJwtForVolumeServer stamped the fid verbatim, but the volume server strips a trailing _N delta suffix before comparing the claim, so a token minted for a batch-assigned fid like 3,01637037d6_1 was checked against 3,01637037d6 and never matched. Reading such a chunk through the filer returned 401 wherever jwt.signing.read.key was configured. Strip the suffix before minting, via a helper shared with the proxyChunkId validation that was already doing the same thing inline. * filer: don't mint a volume write token for an anonymous proxy caller The ?proxyChunkId= branch dispatches and returns before the JWT gate, so whatever credential the proxy attaches is reachable without authentication. It attached a token from maybeGetVolumeReadJwtAuthorizationToken, which fell back to the write signing key when jwt.signing.read.key was unset -- the configuration scaffold/security.toml recommends for a filer, since read JWTs are only supported in a master+volume setup. An anonymous DELETE /?proxyChunkId=<fid> therefore arrived at the volume server holding a write-key token scoped to that fid, and the volume server honored it. Sign read tokens with the read key only. The fallback bought nothing on a read anyway: a volume server enforces read JWTs solely when that same key is set, so when the fallback fired the read was unchecked regardless. Mint only for reads. Writers proxied through the filer carry their own volume JWT from AssignVolume, forwarded with the rest of the caller's headers, so weed mount -filerProxy uploads are unaffected. Moving the dispatch below the JWT gate instead would have broken them, since that token is signed with jwt.signing rather than jwt.filer_signing. On a read with nothing to mint, drop the caller's Authorization rather than relaying it: there it is a filer credential, and forwarding it would hand a volume server a token it never used to see. * filer: keep proxied writes out of the read concurrency semaphore The semaphore is named and documented for reads -- it exists so replication bursts can't open hundreds of connections to one volume server -- but it was applied to every proxied method. A write queued behind sixteen in-flight reads can wait past the 10s default expiry of the AssignVolume token it carries, and the volume server then answers 401. shouldReassignUpload treats a 4xx as final, so the uploader replays the same expired token instead of re-assigning and the write fails up to the caller. This only became reachable once the filer stopped re-minting a fresh token after the wait. |
||
|
Chris LuandGitHub
|
4f43153918 |
filer: reject a proxyChunkId that isn't a well-formed fid (#10436)
LookupFileId only requires the fid to contain a single comma, and the value is pasted straight into the volume server URL path, so ?proxyChunkId=3,x/../../status resolves to volume 3 and then addresses an endpoint the caller never named: Go sends the dot segments verbatim, the volume server's mux cleans the path and redirects to /status, and the filer follows the redirect and relays the body. That reaches any handler on the volume server -- status, stats, the UI -- past a filer that operators expect to be the only exposed surface. Parse the fid before the lookup and answer 400 when it doesn't parse. A trailing _N delta suffix from batch assigns is legal, so it is stripped first, but only when it is a non-empty run of digits. The volume server strips at the last "_" unconditionally, which is safe there because its fid came out of a path the mux parsed and so cannot hold a "/"; here the value is raw query input, and an unguarded strip would reduce "3,01637037d6_1/../../status" to a valid fid and wave the traversal through. |
||
|
Chris LuandGitHub
|
d867b6e739 |
log_buffer: bound the flush queue in bytes, not in copies (#10433)
The queue holds sixteen sealed windows, which is a memory bound only while a window is BufferSize. An entry larger than that grows its window to fit, and the depth then multiplies straight through: sixteen queued copies of a 100 MB window is 1.6 GB of flush data alone. Account the queued bytes and make producers wait once they pass the ceiling the depth was chosen for. What is charged is the pooled slab rather than the window length, since mem.Allocate rounds up to a size class and the queue holds the whole slab. A window larger than the whole budget still goes through on its own, so an oversized entry is never stuck. Windows are admitted in the order they were sealed. A producer can now park here for seconds, and letting a later window overtake an earlier one would persist them out of order and walk lastFlushedOffset and lastFlushTsNs backwards. A window is copied into its slab under the write lock, before the reservation is taken, so a burst of concurrent oversized writers would each hold a full copy in hand while queueing up -- memory the budget never sees. Large writers wait for queue headroom before they take the lock, which throttles the burst; it does not bound it, since a writer that passes the check still seals unconditionally. Take any room in the queue before the shutdown escape, too: the window is already sealed by then, so dropping it loses records the caller was told were accepted. A shutdown that races a full queue can still drop one -- that predates this change and needs the flush loop's lifetime reworked. Size a grown window to the entry rather than to twice it: the extra room only bought space for a second oversized record in the same window, which doubles the flush copy and the snapshot taken of it. The overflow guard halved its bound for that doubled allocation, so raise it to match what is now allocated and what maxBufferSize documents. |
||
|
Chris LuandGitHub
|
2d9227747a |
volume: reject needle blob writes to read-only volumes (#10435)
* volume: reject needle blob writes to read-only volumes WriteNeedleBlob appends the blob to .dat and only then calls nm.Put. On a read-only volume the needle map is a SortedFileNeedleMap whose Put always fails, so the append is never indexed and never rolled back. Nothing upstream stops this: volume.check.disk picks its targets from the master's cached topology, which goes stale the moment a volume server marks a replica read-only itself — a failed data integrity check at load, or an EIO quarantine. Each sync attempt then grows the .dat of a replica that is supposed to be frozen by one unindexed needle, and reports it as "invalid argument", the bare os.ErrInvalid the needle map returns. Check IsReadOnly before touching .dat, same as the upload path does. * volume: say which needle and volume failed to index An index write that fails surfaced as a bare errno with no volume, no needle and no file — "invalid argument" for a read-only needle map, or a plain ENOSPC when .idx lives on its own filesystem via -dir.idx. Both were logged at V(4), so by default the operator saw only the errno the client got back. |
||
 |
186a72c39d |
build(deps): bump rand from 0.8.5 to 0.10.2 in /seaweed-volume (#10428)
* build(deps): bump rand from 0.8.5 to 0.10.2 in /seaweed-volume Bumps [rand](https://github.com/rust-random/rand) from 0.8.5 to 0.10.2. - [Release notes](https://github.com/rust-random/rand/releases) - [Changelog](https://github.com/rust-random/rand/blob/master/CHANGELOG.md) - [Commits](https://github.com/rust-random/rand/compare/0.8.5...0.10.2) --- updated-dependencies: - dependency-name: rand dependency-version: 0.10.2 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * rust volume: follow the rand 0.10 renames thread_rng is now rng, the Rng extension trait is RngExt, and RngCore is Rng. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Lars LehtonenandGitHub
|
b3f35ed338 | chore(weed/worker): prune unused Registry (#10431) | ||
|
Chris LuandGitHub
|
7b3462be6a |
filer: stop an oversized metadata log flush from wedging the change feed (#10430)
* filer: write the metadata log in pieces a volume server will accept A single oversized metadata event grows the log buffer past the volume server's fileSizeLimitMB, and the flush of that buffer is then rejected forever: the retry loop has no exit, so the blob at the head of the queue blocks every later flush and the metadata feed stalls until restart. Split the flushed buffer into BufferSize pieces, on record boundaries where possible so each piece still decodes on its own, and retry each piece separately so a partial success is not replayed. Log files are already read as a chunk stream, with a whole-file fallback when a chunk does not decode standalone, so a record may cross a piece boundary. * log_buffer: let go of a window array grown for an oversized entry An entry larger than BufferSize grows the window array to 2*size+4, and window arrays cycle through SealBuffer rather than being freed. One such entry therefore leaves every later window carrying its size, and currentSnapshotView allocates a snapshot as wide as the array on each window, so a few KB of metadata keeps paying for it. Drop the array when SealBuffer hands it back. Growth is on demand, so the next oversized entry just reallocates. * iceberg maintenance: store merged data files as chunks, not inline saveFilerFile had no size threshold, so compaction wrote whole merged parquet files -- hundreds of MB -- as Entry.Content. That puts the parquet bytes verbatim in the filer store and sends them through the metadata change log again as one event. Keep manifests and metadata JSON inline, upload anything larger to volume servers in chunks, assigning through the filer so the path's storage rules apply. * filer: follow the file size limit the volume servers report The starting piece size is a constant, so a cluster whose -fileSizeLimitMB is set below it would reject every piece and wedge just the same. The rejection names the limit, so take it from there and re-cut the rest of the flush to fit. Piece the buffer one at a time rather than up front, since the size can change partway through a flush. |
||
|
Chris LuandGitHub
|
cba2e5150c |
plugin: fix flaky scheduler lock test (#10432)
plugin: stop the scheduler lock test racing its own background loops TestRunLaneSchedulerIterationLockBehavior constructed the plugin with a cluster-context provider, which makes New start a background scheduler loop per lane. Those loops call runLaneSchedulerIteration on the same lane the test then drives by hand, so a loop could consume the due job — running detection and pushing the next-detection time forward — before the manual call observed the lock. The Default case then saw the lock acquired zero times and failed intermittently. Construct without the provider so no loops start, and set the provider afterward so the manual iteration can still detect. This is the pattern scheduler_status_test.go already uses for the same reason. Reproduced under -race -count=100 -p 4 before, green after. Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu |
||
|
Chris LuandGitHub
|
19ce7c0b6f |
consolidate the duplicated transient-error classifiers onto util.IsTransientError (#10429)
* util: match transient error messages case-insensitively, expose the message form The same condition reaches different layers capitalized differently -- a volume server relays its idle timeout as "I/O timeout" inside a JSON string -- and the callers that grew their own substring lists all lower-case first. Also split out IsTransientErrorMessage for the paths that carry only the text, such as the per-file status strings in a batch delete response, and pick up "no route to host" and "network is unreachable" from the gRPC classifier. * filersink: classify transient network errors through util.IsTransientError The local list caught i/o timeout, connection reset, and broken pipe but not connection refused, no such host, unexpected EOF, the syscall errnos, or the gRPC and S3 overload codes. Keep only the bare io.EOF case, which is transient here -- a truncated chunk read -- but a clean stream end elsewhere. * filer deletion: reuse util.IsTransientErrorMessage for the network patterns Six of the sixteen patterns were already covered. Keep the ones specific to this pipeline -- read-only volumes, lookup failures, backpressure -- and note why context cancellation stays retryable here: it decides whether to requeue the deletion, not whether to retry a call. * wdclient: fold the shared classifier into the volume lookup retry check The string tail duplicated the shared list and missed the syscall errnos and net.Error timeouts. Keep "connection" and "timeout", which are broader than the shared classifier on purpose: a volume lookup is a cheap read-only call. |
||
|
Chris LuandGitHub
|
c438c5ef94 |
filer.replicate: acknowledge notifications after the sink write, not on receipt (#10427)
* filer.replicate: commit the kafka offset after replicating, not on receipt The partition consumer committed the offset as soon as it handed the message to the channel, so a sink write that failed was logged and the message was already behind the committed offset -- never redelivered, permanently missing from the sink. Commit in onSuccessFn instead, and hold the committed offset behind the oldest offset that failed to replicate so a restart redelivers from there. * filer.replicate: delete the sqs message after replicating, not on receipt ReceiveMessage deleted the message before the replicator had a chance to run, so a failed sink write dropped it for good. Move the delete into onSuccessFn and leave the message in the queue otherwise, letting the visibility timeout redeliver it. |
||
|
|
79b7356a52 |
build(deps): bump quinn-proto from 0.11.14 to 0.11.16 in /seaweed-volume (#10426)
Bumps [quinn-proto](https://github.com/quinn-rs/quinn) from 0.11.14 to 0.11.16. - [Release notes](https://github.com/quinn-rs/quinn/releases) - [Commits](https://github.com/quinn-rs/quinn/compare/quinn-proto-0.11.14...quinn-proto-0.11.16) --- updated-dependencies: - dependency-name: quinn-proto dependency-version: 0.11.16 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Chris LuandGitHub
|
652273301e |
filer sync: do not advance the sync offset past a failed event (#10424)
* util: retry transient errors, not just the ones containing "transport" util.Retry only retried when the error string contained "transport", so a plain "read: connection reset by peer" from S3 got zero retries. Classify the error instead: net timeouts, connection resets, and the throttling and overload replies S3 and gRPC return are all worth another attempt, while a cancelled or expired context is not. * filer sync: hold the sync offset behind a failed event A sync job that returned an error was logged and forgotten, and the watermark advanced past it anyway. The offset is the durable resume point, so the event was never replayed: for filer.remote.sync that left the file present locally, absent on the remote, with no RemoteEntry and nothing to retry it. Pin the watermark at the oldest failed event. Later events keep flowing, but the persisted offset stays behind the failure, so a restart replays it. |
||
|
Chris LuandGitHub
|
f18ad39142 |
filer: honor the documented TLS options in every redis store (#10425)
The scaffold advertises enable_tls, ca_cert_path, client_cert_path and client_key_path under redis2, redis2_sentinel and redis_cluster2, but only the plain redis2 and redis3 stores ever read them, and under a different name, enable_mtls. Sentinel and cluster setups quietly connected in plaintext. Build the TLS config in one place and use it from all six stores. enable_mtls still works. The CA and the client key pair are optional now, so enable_tls alone verifies against the system roots, and ServerName is left unset so go-redis validates each address the sentinel and cluster clients dial. |
||
 TJDawson10andGitHub
|
b50116ccae |
fix(redis2/redis3): support separate sentinel auth credentials (#10412)
redis2_sentinel and redis3_sentinel stores only passed Username/Password into redis.FailoverOptions, which authenticates against the Redis master/replica servers. When Sentinel itself requires auth (requirepass set in sentinel.conf), go-redis had no credentials to send to it, causing a NOAUTH error before ever reaching the master. Add sentinel_username/sentinel_password config options that map to go-redis's SentinelUsername/SentinelPassword fields, distinct from the existing master auth credentials. |
||
|
Chris LuandGitHub
|
c392f45705 |
s3: stop listing prefixes whose objects are all delete-marked (#10419)
Deleting the only object under a prefix in a versioned bucket writes a delete marker and keeps the version history, so the filer directory survives with nothing a current-version listing would return. A delimited ListObjects kept reporting that path in CommonPrefixes, because the prefixes come from the directory tree rather than from the keys, while a listing scoped inside the prefix correctly came back empty. Probe a directory before reporting it: one that holds entries but no key the listing returns is neither a CommonPrefix nor a path the trailing-slash probe answers for. Empty directories keep the meaning they have today, and the probe only runs for buckets with versioning configured, the only ones that can reach this state. |
||
|
|
bee5fbc05d |
build(deps): bump google.golang.org/grpc from 1.81.1 to 1.82.1 in /seaweedfs-rdma-sidecar (#10407)
build(deps): bump google.golang.org/grpc in /seaweedfs-rdma-sidecar Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.81.1 to 1.82.1. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](https://github.com/grpc/grpc-go/compare/v1.81.1...v1.82.1) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-version: 1.82.1 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
|
0e78031fa8 |
build(deps): bump google.golang.org/grpc from 1.82.0 to 1.82.1 (#10409)
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.82.0 to 1.82.1. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](https://github.com/grpc/grpc-go/compare/v1.82.0...v1.82.1) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-version: 1.82.1 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
|
9ad19c0ca4 |
build(deps): bump io.netty:netty-codec-http from 4.2.15.Final to 4.2.16.Final in /test/java/spark (#10408)
build(deps): bump io.netty:netty-codec-http in /test/java/spark Bumps [io.netty:netty-codec-http](https://github.com/netty/netty) from 4.2.15.Final to 4.2.16.Final. - [Release notes](https://github.com/netty/netty/releases) - [Commits](https://github.com/netty/netty/compare/netty-4.2.15.Final...netty-4.2.16.Final) --- updated-dependencies: - dependency-name: io.netty:netty-codec-http dependency-version: 4.2.16.Final dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Chris LuandGitHub
|
1e1b2bb2f9 |
iceberg maintenance: record file counters on the snapshots it commits (#10420)
A maintenance snapshot carried only its own labels — merged-files, delete-groups and friends — and no summary counters, so every engine that reads a table's size out of the current snapshot summary reported nothing for it: PyIceberg's inspect.snapshots, Trino's $snapshots and Spark's DESCRIBE all read total-records, total-data-files and total-files-size verbatim, and a table lost them the moment compaction touched it. Accumulate the files each operation adds and removes, and render them the way the spec defines: the added-*/removed-* counters from the files themselves, then the totals carried over from the parent snapshot. Carry a total only when the parent recorded it. Iceberg treats a missing total as zero, which turns a compaction replacing two files with one into a negative total-data-files, or a table with millions of rows into total-records: 0. Leaving the field out lets a reader fall back to the manifests instead of believing a made-up number. Compaction also accounts for the delete files it consumes, so a run that folds every delete into the rewritten data reports them as removed. |
||
|
Chris LuandGitHub
|
b4b0346f95 |
iceberg maintenance: resolve table files from the recorded location (#10418)
The worker assumed every file of a table sits under its catalog path, so loadFileByIcebergPath stripped the scheme off a recorded location and joined the remainder onto /buckets/<bucket>/<ns>/<table>. A table the REST catalog placed elsewhere in the bucket — which is what a client gets whenever the catalog path is already occupied — then resolves to a doubled path: lookup /buckets/lake/source/t/lake/source/t-0cd81bca-.../metadata/snap-.avro so the very first manifest list read fails and the job fails again on every scan interval, indefinitely. Resolve absolute references (s3:// URIs and /buckets paths) from the bucket root and keep relative ones under the table's own directory; the bucket-relative form is now the canonical key everywhere references are compared. That directory comes from the metadata location the catalog stores, so reads, writes and deletes all land where the table's other files are instead of splitting it across two trees. References outside the table's bucket are rejected rather than silently misresolved. Rewritten position-delete files now name their data file by absolute URI, the way the table itself names it, instead of a path relative to the table. |
||
|
Chris LuandGitHub
|
c194924d13 |
telemetry: per-cluster size over time on the dashboard (#10417)
The dashboard charted one summed disk-usage line, so a step in the total gave no hint which cluster moved. A new panel stacks each cluster's daily size as its own band: the top of the stack is the fleet total, each band is one cluster, and the clusters past the twentieth are summed into an "other" band so the stack still adds up to the total. The series is built from the per-cluster daily histories and served by /api/cluster-sizes. Clusters report roughly once a day at no fixed hour, so a day with no report carries the previous value forward — dropping it to zero would sag the total every day as the clusters that have not reported yet fall out from under it. A cluster that stops reporting past the active window ends at its last sample instead of holding capacity forever. Ranking is by the most recent day, tie-broken on cluster id so the colors do not shuffle between refreshes. Hover and click resolve to the band under the pointer: Chart.js's builtin interaction modes match the nearest line, which on a stack of thin bands is rarely the band being pointed at. Clicking one fills the per-cluster history lookup below it. |
||
|
Chris LuandGitHub
|
6e6255b58e |
shell: accept a context in the volume move helpers (#10415)
LiveMoveVolume and the copy, tail, delete, mark, replicate, and configure helpers around it issued every RPC on context.Background(), so a caller had no way to bound or abort a move once it started. They now take a context, which the exported LiveMoveVolume in particular needs: callers outside the shell drive long moves and want to stop them. The deferred restore in copyVolume runs on a detached, bounded context rather than the caller's. Marking the source writable again is cleanup, and cancelling the copy must not skip it and leave the volume readonly — the same guard balance_task.go already applies for the same reason. Shell commands pass context.Background(): their Do signature carries no context, and changing it would touch every command in the package. Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu |
||
|
Chris LuandGitHub
|
47b491b53c |
mount: version open file handles by filer log position (#10403)
* filer: stamp a log position on lookup and remote-cache responses Metadata events are logged after their store write and stamped with the filer clock. Reading that clock before serving an entry therefore gives a timestamp with a causal guarantee: every event at or below it is reflected in the returned entry. Clients caching filer state can use it as the entry's version to order the response against subscription events, including events committed before the call but delivered after it. * mount: version open file handles by filer log position A subscription event refreshing an open handle did a second lookup; a transient failure left the handle pinned to its old entry with no retry, since the subscription cursor had already advanced. The deeper problem is ordering: the handle is a cache written by three unordered channels — the async invalidation worker, local mutation acks, and open-time lookups — and overwriting cached state safely requires knowing which write is newer. The filer log timestamp is that order, and it now travels with every value instead of being derived out of band. Events carry it natively; lookup and remote-cache responses carry the log position stamped before the serving read; mutation acks carry it in their returned event; and the local store pairs each read with a version cursor advanced under the same lock as the store write. Each handle records the version its entry reflects, and one rule replaces the per-site reasoning: state at or below the handle's version is old news and must not be installed. The invalidation itself applies the event's own entry — no lookup, so no transient-failure window — except under a cached parent, where the store entry is the ordered merge of the event and anything applied since, and its version outranks the event's. An uncached parent receives no store writes, so a hit there would be a stale leftover masking the event. A vacated path (delete, rename away) keeps the last entry so unlinked-but-open reads still work. Directory builds version the completed directory at the listing snapshot and re-invalidate buffered events at that version, since their mid-build refresh ran against an incomplete store. The tests replay every race this replaces machinery for: rollback of a newer local flush (queued, cached, and read-through), stale leftovers under uncached parents, the build window including abort, handles opened after an event was queued, events landing mid-lookup, and undelivered events at remote-cache time across a filer failover. * filer: serialize the log position fence with mutations, stamp mutation acks The fence stamped before an unlocked entry read could precede state the read returned: a mutation writes storage first and assigns its event timestamp only at notify time, so a lookup racing that window handed the mount an entry newer than its fence, and the event's later delivery looked like fresh news — destroying dirty pages for a change the handle already had. The mutation handlers already hold an exclusive per-path lock across read, write, and notify; the lookup and remote-cache reads now take it shared around the stamp and the read, making the fence exact: everything at or below it is in the entry, nothing above it is. A no-change update returns success without an event, leaving the mount nothing to fence with even though the response confirms current state. Create and update acks now carry a log position stamped under the same lock, and the mount falls back to it whenever the ack has no event. Also regenerate the VT marshalers, which the earlier generation missed: without them a VT round-trip silently zeroed every log position. * java: sync filer.proto * mount: scope store versions to what they vouch for; atomic handle install The store's version cursor claimed too much. Advanced by local mutation acks and directory listing snapshots, it inflated the version of store reads for unrelated paths whose events the subscription still owed, and those events were then fenced out permanently. The cursor now tracks subscription progress only — events arrive in log order, so everything at or below it has been delivered for every path — and a completed listing records its snapshot as a per-directory floor instead of a global claim. Local acks never touch it: they version their own handle directly. Buffered build events advance the cursor at delivery, since their store write may never happen (abort) while their invalidation is already queued; their read-through directory pairs no store read with it, and rename fragments are applied first. Concurrent first opens raced: a slower opener's older lookup could overwrite the newer entry a faster opener had installed, while the monotonic version kept the newer timestamp — an old entry fenced at a new version, immune to every correcting event. Entry and version are now installed as one decision under the handle map lock, and an install that does not outrank the handle's version is dropped. The remote-cache commit also escaped the fence: it wrote storage and notified without the path lock, so a lookup's shared-locked fence and read could land between the two and hand out the cached state under-versioned. The commit now re-reads and writes under the exclusive path lock, and backs off entirely when the entry changed during the download — the concurrent writer supersedes the cached content. * mount: floors gate store applies; installs respect handle users; renames join the fence A directory floor certifies the listing state as of its snapshot, but a delayed event at or below the floor was still applied to the store — rolling the content back to pre-snapshot state while the floor kept claiming the snapshot version, so the correcting events were fenced out of every future read. Events are now gated against the affected directory's floor, each half of a rename independently. Fences are lower bounds: a listing or lookup can include a mutation whose event has not been delivered yet, and that event later passes every gate carrying state the handle already holds. Such a re-delivery now advances the version without destroying dirty pages or reinstalling the entry — invalidating local writes over a no-op was the real damage in every remaining under-fence window, including the unlocked listing snapshot, which no per-path lock can serialize. The concurrent-open install moved from the map lock to the handle lock every reader, writer, and invalidation synchronizes on, and rejects what cannot improve the handle: dirty state (local writes would be lost), unversioned lookup responses (they cannot outrank anything, and two zero-version opens must not overwrite each other), and anything not strictly newer. New handles are still fully initialized before the map exposes them. Renames committed metadata and emitted events with no path lock, so a lookup could read the renamed state under a fence preceding its events. Both rename handlers now hold the source and destination locks, ordered by path, across commit and notification; descendants of a renamed directory are not individually locked and rely on the no-op re-delivery handling above. * mount: per-entry store versions replace the cursor and directory floors The store's aggregate versions — a global subscription cursor and per-directory listing floors — were versions at coarser granularity than the values they described, and every over-claiming bug in this series traced to that gap: an aggregate vouching for state its source never saw. Each store entry now carries the filer log position of the write that produced it — the event that applied it, or the listing snapshot that inserted it, recorded in the store's key-value space under the same lock as the entry write. The store becomes what the handle already is: a last-writer-wins register with one rule, install only what outranks the current claim. The cursor, the floors, their advancement rules, the pairing ordering constraint, and the floor gating all collapse into that rule. Applies are gated per entry, each half of a rename independently; an unversioned local write clears the claim its content no longer proves; version records lingering after a bulk folder wipe cannot fence a recreate, since a claim only blocks while its entry exists. Listing inserts are stamped at build completion, before the buffered replay so newer replayed events override the stamp. Filer side, the fence dance every versioned read must perform is now a single choke point, fencedFindEntry, so a future read RPC gets the lock-serialized stamp by construction rather than by convention. * mount: judge no-op re-deliveries against an immutable base, not the live entry The equal-state skip compared the incoming event to the live handle entry, but local writes mutate the live entry — size, timestamps, chunks — so a delayed event re-delivering the base the handle was opened with no longer matched, and the installer destroyed the dirty pages and rolled the entry back over nothing new. The handle now keeps an immutable snapshot of the filer state it last installed or acknowledged, refreshed at every install and mutation ack (flush acks snapshot the request entry before the id mapping mutates it), and the no-op judgment runs against that base: an event carrying the base brings nothing, whatever the live entry has diverged to since. * mount: tombstones for versioned deletes, absence floors, copy enrollment Four gaps in the per-entry version protocol, all the same shape: a versioned fact with nothing carrying its version. A deletion is a fact about a path with no entry left to hold it — clearing the record let a delayed older event resurrect the deleted path, permanently, since the deletion's own redelivery is dedup-suppressed. Versioned deletes now leave a tombstone record that fences without an entry; renames tombstone their source the same way. Plain records still only block while their entry exists, so records lingering after a bulk folder wipe cannot fence a recreate. A completed listing proves absences as well as presences: a name it omitted was deleted as of the snapshot, and a delayed create below the snapshot re-creates it. The snapshot is kept per directory strictly as an absence fence, consulted only when a path has neither an entry nor a version record — present entries carry their own versions and never touch it, which is what separates this from the over-claiming floor it replaces. A rebuild against a pre-upgrade filer returns no snapshot; stamping now clears the children's records in that case, so a reinserted entry cannot reactivate the stale claim its previous incarnation left behind and reject valid events below it. Server-side copies installed the copied entry without enrolling in the base protocol, so the copy's own event differed from the stale pre-copy base and destroyed writes made to the destination after the copy. The install now refreshes the base and takes its version from the fenced readback. * mount: deletion facts outlive the cache's knowledge of the entry A versioned delete of a path the store held no entry for recorded nothing, so a delayed older event recreated the path — permanently, with the deletion's redelivery dedup-suppressed. The tombstone is now written whenever a versioned event vacates a path: the deletion is a fact about the path, not about what this cache happened to hold. For an absent entry, the listing's absence floor now speaks whatever older record remains: a tombstone at one position does not exhaust what is known about the path when a newer snapshot has confirmed the name still absent, and an event between the two was slipping past both. A committed copy whose readback failed installed a synthesized base with local timestamps; the copy's real event legitimately differs from it, and was read as foreign state — destroying writes made to the destination after the copy. The handle now marks that its own event is en route and adopts that event's state as the base without touching the live entry or the dirty pages; the adoption is one-shot, so a genuinely foreign event still invalidates. * mount: authoritative acks cancel pending event adoption; tombstones scoped and pruned The copy-event adoption flag could outlive its purpose: a flush after the failed readback installs a newer base and advances the version, the copy's own event is then version gated without consuming the flag, and the next genuinely foreign event was silently adopted — base advanced, live entry and dirty pages untouched — leaving the mount to later overwrite that remote change. Every local acknowledgment now installs its base through one helper that also cancels any pending adoption: the ack supersedes the mutation the adoption was waiting for. Tombstones were written for every versioned delete under the mount and survived directory eviction by design, growing LevelDB with historical deletions on delete-heavy mounts. They are now scoped to directories whose cached state the fence actually protects — an uncached parent never serves from the store nor applies the resurrecting insert — and a completed listing prunes the direct-child tombstones its absence floor supersedes, leaving only those above the snapshot. The store gains a key-prefix visitor for the sweep. * mount: acked saves install their value; trailer snapshots; direct-child prune range A version must never advance without its value. saveEntry stamped any open handle with the acknowledgment's version, but a handle opened while the save was in flight holds the pre-mutation entry — stamping it fenced out the events carrying the state it lacked, permanently, with the local apply performing no invalidation and the redelivery deduplicated. The acknowledged entry is now installed together with its version, through the same guarded install the racing-open path uses: under the handle lock, only when it outranks the handle, never over dirty local writes. Empty listings return no in-band snapshot — a snapshot-only response would be read as an entry by older consumers — so directories that end empty gained no absence floor and their tombstones were never pruned. The filer now sends the snapshot in the stream trailer, which older clients ignore, and the client reads it when no in-band snapshot arrived. Empty directories get real floors, their tombstones prune, and their buffered replays gain the snapshot filter instead of the replay-all fallback. Version records now encode the parent directory and name separated by a NUL, making a directory's direct children one contiguous key range: the tombstone prune scans exactly them under the cache lock, instead of walking every descendant record — the whole store, for root. * mount: fix dirty-page loss, uid/gid base, download race, copy adopt, leak; dedup Correctness fixes from the versioned-invalidation review: - A foreign delete/rename-away of a file held open with unflushed local writes destroyed the dirty pages unconditionally. A process may keep writing to an unlinked-but-open file and those writes were already acknowledged; preserve the pages when the handle is dirty. - downloadRemoteEntry stored the handle's base with filer-side uid/gid while every candidate it is later compared against is in local form, so under a non-identity UidGidMapper an unchanged re-delivery looked foreign and force-destroyed dirty pages. Map the base to local. - downloadRemoteEntry wrote the entry/base/version triple under only the handle's shared lock, so two concurrent reads of the same remote-only file could tear it. Serialize the install with a dedicated mutex (invalidation is already excluded by the exclusive handle lock). - A committed server-side copy whose readback failed adopted the FIRST event past the version gate as its base; a foreign write delivered first was silently swallowed. Adopt only an event whose content matches the synthesized base — the copy's own event — and install any other normally. - The deferred-create path relied on AcquireFileHandle installing the passed entry on a pre-existing handle, which the version rework dropped. Restore that install in the compat wrapper; the versioned open path keeps its gated install. Growth and hot-path cost: - Per-entry version records and tombstones leaked when a directory was evicted or read-through without a rebuild. An uncached directory gates its own inserts, so its records fence nothing; clear a directory's child version records when it is wiped for eviction. - FindEntry paid for the version KvGet on every lookup/getattr cache hit and threw it away. FindEntry now reads only the entry; the hot lookupEntry cache-hit path skips the version entirely. Cleanups: - Extract ackVersionTsNs over the shared response interface, replacing the metadata-event-else-log-ts snippet copy-pasted at four ack sites. - Extract acquireRenamePathLocks, replacing the verbatim sorted two-path lock fence in both rename handlers. * mount: no resurrection on foreign delete, version no-event acks, gate downloads, tighten copy adopt Follow-ups to the review patches: - Preserving dirty pages on a foreign delete let the next flush pass the isDeleted guard and CreateEntry, resurrecting the remotely-unlinked name. Mark the handle deleted in the vacate branch: the open fd can still read its buffered writes, but a flush no longer recreates the file. - A no-event acknowledgment (log fence only) synthesized a metadata event with TsNs 0, so the cache stored the entry unversioned and an older subscriber event rolled it back. Stamp the synthesized event with the ack's log position at all four ack sites. - downloadRemoteEntry serialized its install but did not check the version, so an older response arriving last overwrote the entry/base while the monotonic version kept the newer value, fencing corrections out. Install only when the response is at least as new as the handle. - sameEntryContent compared only size and chunks, so a foreign chmod with unchanged content was adopted as the copy's own event. Compare everything except server-assigned timestamps, so a metadata-only foreign change installs instead. * mount: trim comments to the non-obvious why The versioning work accumulated multi-line comment blocks restating what the code says. Keep the constraint a reader cannot derive — why a fence is exact, why a version must not advance without its value, why an uncached parent's records fence nothing — and drop the rest. * mount: distinguish rename from delete, tighten the download and adopt gates - A rename emits a nil old-path invalidation just like an unlink, so the vacate branch marked the handle deleted and later writes through the already-open descriptor were skipped instead of persisted. Carry the delete/rename distinction on the invalidation and mark only an actual delete. - The remote-download install accepted an unversioned response regardless of the handle's version, so during a rolling upgrade a delayed response could install stale content under a newer version. Require the response to be at least as new, with one exception: a handle still lacking local chunks takes the content anyway — it cannot read without it — but does not claim the response's log position. - Copy-event adoption returned without installing, so a foreign touch arriving before the copy's own event lost its timestamps. Content is unchanged either way, so the dirty pages stay valid; a clean handle now takes the entry, while a dirty one keeps its diverged version. * mount: one directory floor instead of a record per child; agree on TTL Review feedback: - Build completion wrote one KV record per direct child inside the cache write lock, so a large directory stalled every other cache operation for O(children) store writes. The directory's listing snapshot already covers every child it saw; make that floor the version for any child without a record of its own, and a child earns a record only when a later event touches it. One map write per build replaces the per-child writes, with the same fencing. - The presence probe read the store directly and so counted a TTL-expired entry as present, judging the path by a record describing content that has logically vanished. It now applies the same expiry the read path does, and an expired path falls back to its directory floor. - Preserve ErrNotFound identity when the commit-time re-read finds the object deleted, so callers still surface a 404. - Assert the rename-away source fence timestamp in the invalidation test. Also record the tombstone ceiling: distinct deleted names in a cached directory accumulate until it is rebuilt or evicted, which prunes everything at or below the new snapshot. * mount: pin the fence's clock domain instead of letting skew decide A log-position fence is stamped by one filer's clock under that filer's in-process lock, so comparing it to an event another filer logged is comparing two unrelated clocks. The two error directions are not equally costly: applying an event the fence already covered is a re-apply the base-equality check absorbs, while skipping one it does not cover leaves the handle holding exactly the state the event was meant to correct, with the subscription cursor already past it — the unhealable staleness this whole PR exists to remove. So refuse to guess. Fences now carry the signature of the filer that stamped them, and a handle records it alongside the position. An event is only fenced out when the filer that logged it is the one that stamped the fence — the logging filer appends its own signature, so its presence identifies the clock domain. Events from any other filer are applied. Positions taken from events keep comparing as before; the subscription already delivers those in order. The invalidation callback takes a struct now: it carries the path, entry, position, delete/rename distinction, and signatures, and was about to need a fifth positional parameter. * mount: follow a foreign rename; key page invalidation on content, not equality - A rename's old-path invalidation now carries the destination, and the handle follows the file there: an open fd tracks the inode, and leaving it on the old path made its next flush recreate that name instead of updating the renamed file. - Dirty pages overlay content, so only a content change invalidates them. Keying that on exact equality meant any timestamp-only event destroyed them, which the copy-adoption marker existed to paper over — a foreign touch could consume the marker and leave the copy's own event to drop the post-copy writes. Comparing content instead makes the marker unnecessary, so it is gone: a metadata-only event keeps the overlay, and a dirty handle keeps its diverged entry unless foreign content supersedes it. - A remote download response that is merely older is now refused even when the handle still lacks chunks; only an unversioned one is taken (and claims no position), since an older response's content predates what the handle reflects. - A refused or unversioned download no longer publishes to the metadata cache, where a zero-position event would clear the entry's version and let an older subscriber event roll the cache back. * mount: page invalidation keys on content alone; unversioned writes claim no position - sameEntryContent compared everything but timestamps, so a foreign chmod, chown, or xattr change counted as a content change and destroyed the dirty-page overlay. It was strict only to serve the copy-adoption marker, which is gone; its one caller now asks the question it actually needs — did the bytes change — so metadata-only events leave the overlay alone. - A rename over an existing file destroys that file, but its open handle was left live and still pointed at the name the renamed source now occupies, so its flush could overwrite it. MovePath already reports the displaced inode; mark that handle deleted. - An acknowledgment was refused whenever its position was numerically lower, even when a different filer stamped the fence it lost to. Two known, differing signatures mean unrelated clocks, so the comparison no longer applies there; unknown signatures still compare as before. - A local write with no log position behind it now records that explicitly instead of deleting its version record. Absence means the directory listing covers the path, which is why the snapshot floor applies; local content the listing never saw must not inherit it, or the events that would correct it are fenced out. * mount: widen the existing lookup functions instead of forking WithVersion twins The versioning work grew a parallel function for every accessor that needed to return a log position — lookupEntryWithVersion beside lookupEntry, maybeLoadEntryWithVersion beside maybeLoadEntry, FindEntryWithVersion beside FindEntry, AcquireFileHandleWithVersion beside AcquireFileHandle, advanceEntryVersion beside advanceEntryVersionTsNs, plus a getPbEntryWithVersion wrapper and an InsertListedEntriesForTest hook. Two names for one operation is two places to keep in step, and the split let callers pick the one that happened to compile. Each pair is now the single original name carrying the position, with callers that do not want it discarding it. filer_pb.GetEntry returns the fence its response already carried rather than a mount-side wrapper re-issuing the lookup, and InsertEntry takes the position its content reflects rather than a test-only twin that inserted without one. The one behavioural knot the merge exposed: AcquireFileHandle had been installing the entry on a pre-existing handle only in its unversioned form, which conflated 'the caller is authoritative' with 'the lookup had no version'. Deferred create is the only caller that means the former, so it now installs explicitly and the map function just acquires. |
||
|
Chris LuandGitHub
|
fe0a357624 |
exclusive_locks: clear renew-running flag before dropping isLocked (#10413)
On renewal failure the renew goroutine stored isLocked=false before its deferred renewGoroutineRunning.Store(false) ran. A concurrent RequestLock interleaving there reacquires the lease (sees isLocked=false), sets isLocked=true, then its CompareAndSwap on renewGoroutineRunning fails because the old goroutine's flag is still set — so no replacement renewer starts. The lock is then held locally with nothing renewing it, and silently expires on the master after the lease TTL, admitting a second holder. Clear renewGoroutineRunning before isLocked on the failure path so the reacquire path always starts a fresh renewer. Builds and vets clean; no behavior change on the success path. Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu |
||
|
Chris LuandGitHub
|
2e9b944e5c |
test(s3): aim collection force-delete at the master the suite actually runs (#10404)
The copying and tagging tests force-drop each bucket's collection at the master so volume slots are freed deterministically between tests. But the copy-tests CI job runs its master on 9336 and the tagging Makefile on 9338, while the tests default to 9333 — the cleanup dialed a dead port and quietly no-oped. Each test bucket then grows 7 volumes against -volume.max=100, and whenever async deletion lagged the data node ran out of slots and PutObject 500ed with "No writable volumes and no free volumes left". Set MASTER_ENDPOINT where the master port is non-default: the copy-tests workflow step, and the copying/tagging Makefiles (derived from MASTER_PORT). |
||
|
Chris LuandGitHub
|
5731f37a2f |
telemetry: validate reports on the collect endpoint (#10401)
* telemetry: validate reports on the collect endpoint /api/collect is anonymous, so reports can't be authenticated, but a real master can't produce a non-UUID topology_id, a version outside N.NN(-enterprise), an unknown GOOS/GOARCH, or absurd counts — reject those to keep casual junk out of the collected data, and cap the request body at 4 KB. * telemetry: integration test fixtures pass collect validation The test's topology id and version were exactly the junk shapes the new validation rejects; use a UUID and a plain version number. |
||
|
Chris LuandGitHub
|
8e8b4c4f34 |
telemetry: confirmed-cluster stats (2+ distinct days) (#10402)
telemetry: confirmed-cluster stats Count a cluster as confirmed once it has reported on >=2 distinct UTC days (per-cluster history makes this a length check). Version/OS distributions in /api/stats are computed over confirmed clusters, so a one-shot injected report can't appear in them; falls back to all active clusters while no confirmed ones exist (fresh server). Adds the seaweedfs_telemetry_confirmed_clusters gauge and a dashboard card. |
||
|
Chris LuandGitHub
|
de3ad8db12 |
telemetry: per-cluster usage history on the built-in dashboard (#10400)
telemetry: per-cluster usage history Keep one compact sample per cluster per UTC day (disk bytes, volume count, volume servers), retained for -max-age and persisted in the state file. Serve it at /api/history?cluster_id=...&days=90 and add a per-cluster lookup with disk/volume charts to the built-in dashboard. |