Compare commits

...
Author SHA1 Message Date
Chris Lu e778f13ff4 s3: count stored versions as bucket contents
A <key>.versions directory holding version entries is user data; skipping
it let bucketHasUserObjects report an empty bucket while the recursive
delete dropped every stored version.
2026-10-16 00:03:12 -07:00
Chris Lu f5e9c9cf2c test: drop redundant allowDeleteBucketNotEmpty=false in s3 integration test
False is the default now, so TestS3DeleteBucketNotEmpty exercises the
out-of-box behavior.
2026-10-15 22:59:07 -07:00
Chris Lu 1eb0c57e61 test: pin allowDeleteBucketNotEmpty=true on versioning config1
The config1 run relied on the flag's old true default to contrast with
config2's false. Pin it so the two modes stay distinct.
2026-10-15 22:51:53 -07:00
Chris Lu 87867b667d s3: default allowDeleteBucketNotEmpty to false
DeleteBucket on a non-empty bucket silently destroyed every object,
while AWS returns 409 BucketNotEmpty. Clients like 'aws s3 rb' rely on
that refusal as the 'delete only if empty' safety net.

Deployments that need the recursive delete can opt back in with
-allowDeleteBucketNotEmpty=true.
2026-10-15 22:51:20 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>Chris Lu
82f0a37efe build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob from 1.8.0 to 1.8.2 (#11740)
* build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob

Bumps [github.com/Azure/azure-sdk-for-go/sdk/storage/azblob](https://github.com/Azure/azure-sdk-for-go) from 1.8.0 to 1.8.2.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.8.0...sdk/azidentity/v1.8.2)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
  dependency-version: 1.8.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* test: restore loadBucketMetadataFromFiler stub after TestGetBucketMetadata

TestGetBucketMetadata replaced the package-level loadBucketMetadataFromFiler
with a stub that sleeps one second per load and never restored it. Every
test running after it in the same binary paid that sleep on each bucket
metadata cache miss: TestPutObjectUploadACL (~245s) and
TestCopyObjectDestinationACL (~46s) each created a fresh BucketRegistry per
subtest, so the stub fired hundreds of times. In CI this stretched the
Test job past 11 minutes and left it exposed to runner cancellation.

Restore the real loader via t.Cleanup so the slow stub is scoped to the
test that needs it.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-15 21:53:16 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
9aae0bb702 build(deps): bump helm/kind-action from 1.15.0 to 1.15.1 (#11747)
Bumps [helm/kind-action](https://github.com/helm/kind-action) from 1.15.0 to 1.15.1.
- [Release notes](https://github.com/helm/kind-action/releases)
- [Commits](https://github.com/helm/kind-action/compare/v1.15.0...v1.15.1)

---
updated-dependencies:
- dependency-name: helm/kind-action
  dependency-version: 1.15.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-15 21:53:01 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
80060e1bd6 build(deps): bump github.com/a-h/templ from 0.3.1020 to 0.3.1070 (#11742)
Bumps [github.com/a-h/templ](https://github.com/a-h/templ) from 0.3.1020 to 0.3.1070.
- [Release notes](https://github.com/a-h/templ/releases)
- [Commits](https://github.com/a-h/templ/compare/v0.3.1020...v0.3.1070)

---
updated-dependencies:
- dependency-name: github.com/a-h/templ
  dependency-version: 0.3.1070
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-15 21:52:43 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
72aa17ff02 build(deps): bump github.com/aws/aws-sdk-go-v2 from 1.47.1 to 1.47.3 (#11741)
Bumps [github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2) from 1.47.1 to 1.47.3.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.47.1...v1.47.3)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2
  dependency-version: 1.47.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-15 21:52:20 -07:00
Chris LuandGitHub e6cf190596 ec: re-plan balance moves when a target reports no space left (#11759)
* ecbalancer: add Node.SetNoReceive to exclude a node as a move target

A node that has refused a shard copy at runtime (e.g. 'no space left' on the actual disk, issue #11753) can no longer be trusted as a destination for the remainder of a balance run, but it may still hold shards that must move away. SetNoReceive excludes the node from target selection and from receiving-side capacity accounting, while leaving it eligible as a move source.

* ec: re-plan balance moves when a target reports no space left

Moves are planned from a heartbeat snapshot, but concurrent copies let the volume server's real disk state diverge from it, so a planned target can refuse the copy with 'no space left' mid-run and abort the whole balance (issue #11753). Track the refusing targets during execution, mark them no-receive, re-collect the topology, and plan the remaining moves again. The loop terminates because a marked node can never be picked as a target again.

* volume server: return ResourceExhausted when no disk can take an EC shard

A failed EcShardsCopy carried code=Unknown even though the cause is exhausted local capacity; ResourceExhausted describes the refusal and lets clients distinguish it from internal errors. The message text is kept so message-based matching still works (the Rust volume server reports the same text under Internal).

* ec: replay applied moves onto the re-collected snapshot and surface failures

Addresses review on the re-plan loop: a fresh topology read can still describe an applied move while heartbeats lag, so journal each applied move and replay it onto the next snapshot; retry only when every failure was a 'no space left' refusal (an unrelated failure aborts instead of being hidden); and exclude refused nodes' slots from the free-space precheck so a balance with no remaining receiver fails instead of reporting success for an empty plan.

* ec: carry per-disk state when replaying applied moves

A move that landed on an explicit TargetDisk loses that attribution when replayed onto a stale snapshot, leaving the disk's free slots and shard count unchanged; the next plan could pick the same disk, hit another refusal, and over-exclude the node. Replay the shard onto the target disk's entry and move the disk-level counters, and restore the source disk's counters on removal.

* ecbalancer: add Node.SetDiskNoReceive to exclude a single disk as a target

A copy with an explicit disk_id can be refused while the node's other disks still have room; a disk_id 0 refusal already means the server searched every disk, so the scopes differ. disk.noReceive removes that disk from receiving capacity and disk selection while leaving the node and its sibling disks eligible.

* ec: record explicit-disk refusals as disk-scoped no-receive

recordMoveResult now distinguishes the two refusal scopes: TargetDisk>0 marks only (node, disk), while TargetDisk==0 marks the whole node since the server's auto-pick found no usable disk. The free-slot precheck subtracts refused disk slots, and the replan log names refused destinations as node:diskN.

* ecbalancer: skip no-receive disks in pickBestDiskOnNode

The balance phases pick the destination disk through pickBestDiskOnNode, which ranked disks by score only; a refused disk with a low shard count would be re-picked, fail again, and abort the run since no new destination was rejected.

* volume_move: export IsNoSpaceLeftError for move callers

Both the ec.balance replanner and the worker ec_balance task need to recognize the volume server's no-space refusal; keep one matcher next to the move RPCs instead of a private copy in weed/ec.

* worker: retry pinned-disk EC moves with a server-picked disk on no space

ec_balance detection always pins TargetDisk, so a full disk fails the job even when a sibling disk has room. On a no-space refusal to the pinned disk, retry the move once with TargetDisk 0 so the volume server picks a disk on the same node; the planner's node choice stands.
2026-10-15 18:47:07 -07:00
ee2d98f664 mini: note mini.options and Docker filer.toml pitfalls in help (#11755)
* mini: note mini.options and Docker filer.toml pitfalls in help

Two behaviors of "weed mini" surprise users and are not mentioned in
its help text.

Flags are saved to <dir>/mini.options and applied again on the next
start to every flag that still has its default value. Removing a flag,
or passing it with its default value, therefore keeps the saved value.

The official Docker image ships /etc/seaweedfs/filer.toml, which sets
the leveldb2 store to /data/filerldb2. When filer.toml is found, the
filer ignores the default store dir derived from -dir, so moving -dir
away from /data leaves the metadata in /data. If /data is not
persisted, buckets look empty after the container is recreated.

Only the help text changes.

Signed-off-by: YoungjuKim <fjvbn2003@gmail.com>

* mini: update help notes for computed-only mini.options

---------

Signed-off-by: YoungjuKim <fjvbn2003@gmail.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-15 18:44:31 -07:00
Chris LuandGitHub 18b235da0d mini: surface saved options and warn when filer store escapes -dir (#11758)
mini: persist only computed options, and warn when filer store escapes -dir

- saveMiniConfiguration no longer writes command-line flags to
  mini.options; the file now only pins values mini computed itself
  (currently master.volumeSizeLimitMB), so a dropped flag can no longer
  resurrect on the next start. Existing files still apply once and are
  then rewritten without the saved flags.
- applyConfigFileOptions skips any flag explicitly provided on the
  command line or via WEED_* env vars, so passing a flag always wins
  over a saved value, including at its default value.
- mini prints the saved options it applies at startup (secret values
  redacted), since quietMiniLogs hides the previous V(1)/V(2) messages.
- warn once at startup when a filer.toml points a local filer store
  (leveldb/leveldb2/leveldb3, including path-specific <store>.<id>
  sections) outside -dir and -master.dir: the filer skips the
  -dir-derived store dir whenever a filer.toml exists, and the metadata
  is lost on container recreate if that path is not persisted.
2026-10-15 18:36:46 -07:00
github-actions[bot] 4c81015bd3 docs: regenerate star history chart 2026-10-16 00:43:27 +00:00
Chris LuandGitHub a5ccefb661 filer: escape LIKE wildcards in SQL list prefixes (#11757)
* filer: escape LIKE wildcards in SQL list prefixes

ListDirectoryPrefixedEntries bound prefix+"%" straight into a LIKE
clause, so '_' and '%' in object names (e.g. Langfuse keys like
call_03_...) acted as wildcards: wrong results and a ~40x slower
range scan because (dirhash, name) can no longer be narrowed.

Escape '!'/'%'/'_' in the prefix and declare ESCAPE '!' in the
generated list SQL so every abstract_sql backend (mysql, mysql2,
postgres, postgres2, sqlite) matches literally with identical
semantics. '!' is used instead of '\' because SQLite shares
SqlGenMysql but would reject the MySQL-style '\\' literal, and
'\' would still need escaping itself.

* filer: escape LIKE wildcards in YDB list prefixes

Same wildcard-in-prefix bug as the abstract_sql backends: YQL LIKE
treated '_' and '%' in the prefix as wildcards. Apply the shared
LikePattern helper and declare ESCAPE '!' (YQL allows any escape
char except '%', '_' and '\').

* test: cover literal '_'/'%'/'!' prefixes in sqlite filer listing

End-to-end regression test through SqliteStore against a real
database; runs in CI under the sqlite build tag.
2026-10-15 16:51:15 -07:00
103328e70f filer: create parent directories for lazily fetched remote entries (#11727)
* filer: create parent directories for lazily fetched remote entries

A remote-mounted object first read through the filer was persisted without
its parent directories, so remote.uncache and the remote cache evictor, which
walk down from the mount directory, never reached it: its cached chunks could
not be evicted.

Persist the missing parents with the entry. With those directories in place,
the empty-folder cleaner leaves folders under a remote mount alone: deleting
one removes its whole prefix on the remote, which can still hold objects
that were never read locally.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep remote objects at parent keys and remote prefixes of empty folders

Look up a lazily fetched entry's missing parents through lazy fetch before
creating them, so a remote object stored at a parent's key is materialized
and promoted to a prefix object rather than hidden behind a new directory.

Deleting a folder that is empty in the filer is metadata-only under a remote
mount, both for a non-recursive delete and for the expiry cleanup of empty
parents: the mount is populated lazily, so the folder can still hold remote
objects never read, and removing its remote directory deletes every object
under that prefix. This replaces the empty-folder cleaner's own guard, which
covered only one of those paths.

A metadata-only delete records no remote-deletion tombstone, locally or from
its metadata event: nothing is owed to the remote, so lazy reads stay open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: stop the lazy parent lookup at a store error

Only a parent the store reports as not found is missing; a parent it cannot
read ends the walk, so the lookup does not stat ancestors above it on the
remote.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: stat each missing remote parent once

A lazy fetch that finds k missing parents asked FindEntry for each, and every nested fetch re-walked its own still-missing ancestors, re-stating remote keys already proven absent: 1+k(k+1)/2 remote stats for one first read. The flagged parent lookups now skip their own walk; their entries are created by the same CreateEntry that already persists the chain.

* filer: materialize parents for lazily listed remote entries

maybeLazyListFromRemote persisted children and the listing directory with skipCreateParentDir, so a first access that goes through the list path still left entries unreachable to a walk from the mount directory. Materialize the missing ancestors the same way the lazy fetch does, and let the synced_at directory create its parent chain.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-15 16:23:33 -07:00
1562621c5e filer: record the size of a multipart upload saved inside the entry (#11734)
* filer: record the size of a multipart upload saved inside the entry

* filer: guard save-inside multipart append and check Content-MD5

Addressing review feedback:

- Reject a save-inside multipart append when the target is chunk
  backed. The buffered content cannot be attached to existing chunks,
  so previously the request grew FileSize without storing the appended
  bytes, and reads returned zeros for the added range.
- Compare the request Content-MD5 against the buffered content before
  saving, matching the regular multipart path, so a bad digest no
  longer commits the file.
- Extend the size test to check the saved entry MD5, and cover the
  rejected append against chunk-backed and inline targets plus the
  bad-digest rejection.

* filer: return 400 for unsupported append modes

The save-inside append rejection and the existing inline append and offset write rejections are deterministic client errors, but the autoChunk default case mapped them to 500. Give them shared constants and return 400 instead, and assert the status in the tests.

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-15 14:20:42 -07:00
Jay HemnaniandGitHub 3659c369e0 webdav: read small files stored inline in the entry (#11733) 2026-10-15 06:38:53 -07:00
Robin TweedieandGitHub e12ec94747 fix(shell): compare fractional evacuation destination ratios (#11752) 2026-10-15 06:36:37 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
9e56651808 build(deps): bump github.com/go-redsync/redsync/v4 from 4.17.0 to 4.18.0 (#11743)
Bumps [github.com/go-redsync/redsync/v4](https://github.com/go-redsync/redsync) from 4.17.0 to 4.18.0.
- [Commits](https://github.com/go-redsync/redsync/compare/v4.17.0...v4.18.0)

---
updated-dependencies:
- dependency-name: github.com/go-redsync/redsync/v4
  dependency-version: 4.18.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-15 06:22:09 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
d167521270 build(deps): bump helm/chart-testing-action from 2.8.0 to 2.9.0 (#11745)
Bumps [helm/chart-testing-action](https://github.com/helm/chart-testing-action) from 2.8.0 to 2.9.0.
- [Release notes](https://github.com/helm/chart-testing-action/releases)
- [Commits](https://github.com/helm/chart-testing-action/compare/v2.8.0...v2.9.0)

---
updated-dependencies:
- dependency-name: helm/chart-testing-action
  dependency-version: 2.9.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-15 06:20:36 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
8608a4e75d build(deps): bump github.com/ydb-platform/ydb-go-sdk/v3 from 3.151.1 to 3.155.1 (#11739)
build(deps): bump github.com/ydb-platform/ydb-go-sdk/v3

Bumps [github.com/ydb-platform/ydb-go-sdk/v3](https://github.com/ydb-platform/ydb-go-sdk) from 3.151.1 to 3.155.1.
- [Release notes](https://github.com/ydb-platform/ydb-go-sdk/releases)
- [Changelog](https://github.com/ydb-platform/ydb-go-sdk/blob/master/CHANGELOG.md)
- [Commits](https://github.com/ydb-platform/ydb-go-sdk/compare/v3.151.1...v3.155.1)

---
updated-dependencies:
- dependency-name: github.com/ydb-platform/ydb-go-sdk/v3
  dependency-version: 3.155.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-15 06:19:29 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
896e4dd192 build(deps): bump github/codeql-action from 4.38.2 to 4.38.3 (#11746)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.38.2 to 4.38.3.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/v4.38.2...v4.38.3)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-version: 4.38.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-15 06:18:54 -07:00
github-actions[bot] 84512927ab docs: regenerate star history chart 2026-10-15 00:43:00 +00:00
github-actions[bot] 590a75b9d6 docs: regenerate star history chart 2026-10-14 00:42:11 +00:00
github-actions[bot] 8093ecf4c8 docs: regenerate star history chart 2026-10-13 00:52:42 +00:00
Jay HemnaniGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Chris LuDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
6840371774 sftp: keep a file's existing bytes when it is opened for write without truncate (#11723)
* sftp: keep a file's existing bytes when it is opened for write without truncate

* sftp: preserve sparse gaps and refuse unreadable entries when copying existing content

The copy used a single chunk-stream read, which ends at the first sparse
gap: the tail past a hole was dropped, and a remote-only entry with no
local chunks copied zero bytes, so closing the handle uploaded an empty
file. Read through PrepareStreamContent, which fills unstored regions
with zeros, and refuse entries that carry no local bytes at all.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: report a failed destination write from a completed chunk fetch

retriedStreamFetchChunkData broke out of the retry loop on a clean HTTP
end before checking writeErr, so a destination write that failed mid-
stream returned a short write and nil error. Check writeErr first.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* sftp: cover remote and all-holes sizes when copying existing content

Take the declared size from RemoteEntry when it exceeds FileSize, pad
inline content to the declared size, and treat a chunkless local file as
all holes rather than unreadable. A remote-only entry still refuses the
copy, now checked with a zero FileSize as well.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* sftp: size the existing-content copy the way filer.FileSize does

A locally shortened file still names its older, larger remote object;
taking RemoteSize unconditionally padded the copy back out to it.
filer.FileSize only honors the remote size when the remote is newer,
and also covers the chunk-extent size. A remote-only entry still
refuses the copy whenever it points at remote bytes.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-13 07:20:20 +08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
d67fc47a61 build(deps): bump rustls from 0.23.37 to 0.23.45 in /seaweed-volume (#11707)
Bumps [rustls](https://github.com/rustls/rustls) from 0.23.37 to 0.23.45.
- [Release notes](https://github.com/rustls/rustls/releases)
- [Changelog](https://github.com/rustls/rustls/blob/main/CHANGELOG.md)
- [Commits](https://github.com/rustls/rustls/compare/v/0.23.37...v/0.23.45)

---
updated-dependencies:
- dependency-name: rustls
  dependency-version: 0.23.45
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-13 05:57:00 +08:00
d0f3fad133 ec: avoid redundant needle.GetActualSize call in LocateEcShardNeedle (#11629)
LocateEcShardNeedle passed needle.GetActualSize(size, version) to
LocateEcShardNeedleInterval, which calls needle.GetActualSize internally.
This caused header and padding to be added twice to needle size.

Fixes #3864

Co-authored-by: Morteza <m.mohammadi@rahkar.team>
2026-10-13 05:50:09 +08:00
624a13ecc6 EC worker: pin jobs to replica-host workers, late-bind local source (#11718)
* feat: prefer same-host worker and local EC replica

Admin executor pick ranked idle workers only, so a job whose volume
lived on full-03 could still run on full-01 and pull the whole .dat
across the LAN. Rank workers whose Hello host matches any replica
first (fallback to others if the local worker is full).

Detection named Sources[0] / the lowest server id; execute then
SyncAndSelectBestReplica and could switch the copy source to another
rack. After sync, if a replica on this machine has a complete file
count, copy from that URL so affinity and pull stay on one host.

* feat: pin scheduler jobs to replica-host workers until load K

Reserve prefers the least-loaded replica-host worker and waits while
that load is below 5 instead of spilling to a free remote worker.
K=0 restores immediate spill. Waterfill uses reservation load so four
idle preferred workers each get a job.

* feat: park scheduled jobs on per-worker admin queues

Assign a detection batch to the least-loaded replica-host worker in
one pass, then let ExecutionConcurrency goroutines pop a home-runnable
job or steal when that home queue reaches K. Dispatch no longer holds
a job in reserve-backoff, so a free worker is not blocked by another
home's pin. Tracked jobs stay pending/queued and record the home
worker for the admin UI.

* fix: rehome queued jobs when the home worker dies

A live home still pins below K. A home missing from live executors
is stolen immediately onto a free worker so a short orphaned queue
does not sit until the dispatch window ends.

* feat: wait before stealing a dead home worker queue

Orphaned queues are held for 30s so a worker reconnect blink does
not dump jobs onto other machines. After the grace, steal without K
as before. Zero disables the wait. A home that comes back during
grace keeps its queue.

* fix: late-bind EC from to worker hostname

After replicas are consistent (equal FileCount), rebind encode source
to a volume server on the same hostname as worker -address. Any port
on that host is local. Diverged FileCount still uses the max replica.

* feat: assign homeless jobs to global FIFO

Jobs whose replica hostname has no registered worker skip pin and orphan grace.
Free workers take own home, then steal at K/orphan, then leftover FIFO.

* fix: take homeless FIFO before steal for no-home workers

Workers without a replica-host match take leftover first; steal (K/orphan)
only when FIFO is empty. Home workers keep own-queue, steal, then leftover.

---------

Co-authored-by: Chepurnov Simon <195125934+CHEPURNOV-S-S@users.noreply.github.com>
2026-10-13 05:31:34 +08:00
Bruce ZouGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Chris LuDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
492e700d1f Volume grow avg fill threshold (#11725)
* master: add conditional grow by average fill ratio to VolumeGrow gRPC

Callers that pre-grow volumes on fill level currently have to list all
volumes of a collection and sum sizes themselves, which is expensive on
clusters with millions of volumes. The master already tracks each
writable volume's effective size in memory, so it can answer the
question directly.

Add grow_avg_fill_threshold to VolumeGrowRequest: when set (e.g. 0.6),
growth only happens once the layout's writable volumes have reached
that average fill ratio. A layout with no writable volumes skips the
check, since that is the case growth exists to fix. VolumeGrowResponse
now reports the observed avg_fill and writable_count whether or not
growth was triggered.

* master: unit tests for WritableAvgFill and FindVolumeLayout

Covers the average-fill computation (mixed fills, no writables, a
writable volume without a size-tracking entry yet) and that
FindVolumeLayout resolves existing layouts without creating missing
collections.

* address review: report avg_fill and writable_count for every VolumeGrow

The stats were only read when grow_avg_fill_threshold was set, so a
request without a threshold got AvgFill=0 and WritableCount=0 even when
writable volumes existed. Read the layout stats unconditionally and use
the threshold only to decide whether growth happens.

* master: scope the grow fill check to the requested target

A VolumeGrow aimed at a data center, rack or node must not skip growth
because fill elsewhere keeps the layout average below the threshold.
Measure only writables serving the target; a target with none still
grows.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-13 05:30:03 +08:00
Jay HemnaniGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Chris LuDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
b5ee5b871e sftp: fail a directory listing on a stream error, and page past hidden entries (#11724)
* sftp: fail a directory listing on a stream error, and page past hidden entries

* sftp: test a fully hidden listing page

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 19:34:46 +08:00
Alex KGitHubChris LuDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
6fb21c9ec0 filer: copy a remote-only entry on the remote when renaming it (#11709)
* filer: copy a remote-only entry on the remote when renaming it

Renaming a file under a remote.mount whose content exists only on the
remote (never cached, or remote.uncache'd) lost it: the new entry kept
the old RemoteEntry, deleting the old entry deleted the old object, and
with no chunks nothing ever wrote the new key. filer.remote.sync could
not finish the rename either, since the old object was already gone.

moveSelfEntry now copies such an entry's object to the new key before
it touches anything, and the new entry gets the RemoteEntry of the
copy. The copy is a new optional RemoteStorageObjectCopier interface,
implemented for S3 with CopyObject up to 5 GB and UploadPartCopy above
that; the multipart copy carries the source's metadata and, where the
remote supports tagging, its tags. When an existing target is
overwritten, its delete keeps the remote object (the copy replaced it)
and its event is marked metadata-only so the sync leaves it alone too.

A remote-only rename that leaves its mount, or whose remote cannot
copy, is refused instead of deleting the only copy. Empty entries and
entries with local data rename as before.

UpdateFileMetadata builds its copy source with the same helper, which
escapes each key segment and keeps the slashes.

* filer: delete a copied rename's old object only after the commit

Review follow-ups:

- The old key of a copied remote-only entry, and the old directory of a
  rename that copied one, are now deleted once the rename has committed.
  A rename that fails after the copy leaves the source entry and its
  object in place, plus an extra object at the new key.
- Every part of a multipart copy is conditioned on the ETag the copy
  started from, so an object overwritten mid-copy fails the copy
  instead of mixing versions.
- Both copy paths keep the source's server-side encryption settings.
- filer.remote.sync no longer deletes a key that a remote-only entry
  holds by the time the delete event is processed: a file deleted and
  then replaced by a remote-only rename has its content only there.

* filer: do not sweep the old prefix of a directory rename that copied

Removing the old directory on the remote deletes everything under its
prefix. Run after the commit, that could also take an object another
rename copied in under the same path meanwhile. When a child was copied,
leave the prefix alone and delete only the copied children's old keys.

* filer: keep an old key a remote-only entry has retaken by cleanup time

A directory rename does not lock its children's paths, so between the
commit and the deferred delete of a copied child's old key another
rename can put a remote-only entry there, whose content is then only in
that object. Check the store before deleting.

* filer: keep a renamed entry's old object when the path lookup fails

The deferred delete must only run once the lookup shows no remote-only
entry at the old path. A failed lookup shows nothing, and an extra
object loses nothing.

* filer: copy over a remote-only rename target only after the commit

Copying the source object onto the target's key before the transaction
committed destroyed the target's only content, which a rolled-back
commit cannot restore. When the target is remote-only the copy now runs
after the commit: the new entry keeps the old key's stamp until the copy
lands and re-stamps it, the target's object is kept meanwhile, and a
failed copy keeps the old object so remote.sync can finish it.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* remote.sync: match the delete event's etag before deleting the object

A rename can copy another entry's content onto a key between the filer's
check and the remote delete, and the delete then removes the copy. The
delete event carries the object's etag; delete only when the live object
still matches it.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: finish a rename's remote work outside the committed transaction

The SQL stores keep the transaction in the context and refuse every
call through it once it has committed. The work that runs after
CommitTransaction used that context, so on postgres2 the deferred
old-key delete could not look up the old path and kept the object, and
finishRemoteCopies could not read or stamp the entry. Both failed with
"sql: transaction has already been committed or rolled back" on every
copied rename; remote.sync cleaned up after them.

Keep the context from before the transaction for the work after the
commit. The rename test store gets a strict mode that fails calls
through a committed transaction the way database/sql does.

* filer: carry a deferred copy's stamp in the rename event; delete orphaned uploads

- The deferred copy stamps the entry with a plain UpdateEntry, which
  publishes no event, so the queued rename event still carried the old
  key's stamp to peer filers and subscribers. Give it the copy's
  RemoteEntry before it is sent. The two notify loops share the code now.
- remote.sync kept any object whose ETag changed since the delete. With
  no entry at the path that is an upload which finished after the
  delete, and keeping it leaves it on the remote for good. Look up the
  path first and keep a changed object only when an entry holds it.

* remote.sync: keep a changed object even with no entry at its path

Deleting a changed object when no entry holds the path also deletes a
copy a directory rename has written but not yet made visible, while a
delete of that path waits in the sync; the rename then removes the old
key and the content is gone. Keep any object whose ETag changed since
the delete, as before. An upload that finishes after the delete leaks
its object instead, which is the cheaper failure.

* filer: read a pending remote rename copy from the source key

A remote-only rename over a remote-only target commits before its copy
lands, so reads on the new path served the target's old object until the
copy stamped the entry — and indefinitely when the copy failed without
remote.sync running. The committed entry now records the renamed-from
path, and remote-only reads resolve that object's key while the stamp
is pending; the copy's stamp and the sync's completion stamp both clear
the marker.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: stamp the rename event with the deferred copy's result

A remote-only rename over a remote-only target queued the new entry's
event with the source's stamp before the deferred copy ran. Peers
replaying it saved a stamp the copy had already superseded, and the
sync serving such a peer rejected the destination as a match and tried
to copy the deleted source. The event now carries the copy's result
and drops the pending-source marker once it lands.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* remote.sync: delete only the object the event's stamp describes

An object at a deleted key is not always the one the delete meant: a
rename copy can land before its filer entry is visible, and a rewrite
can upload over it. When the event recorded no remote stamp, or the
live stat cannot confirm the recorded etag, the delete now keeps the
object instead of treating it as a late upload.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: tighten the pending rename source marker

Follow-ups to reading a pending remote rename copy from its source key:

- Only the filer sets the marker. The HTTP upload and tagging handlers
  store Seaweed-* request headers as extended attributes, so they skip
  this one, and a marker that leads out of the entry's own mount is
  ignored, so it cannot point reads at another remote.
- A pending entry renamed again copies from the marker's source, not
  from its own key, which still holds the old target's bytes. A further
  deferred rename keeps naming the original source; an immediate copy
  clears the marker.
- When remote.sync finds the destination already holding the content,
  it stamps the entry (clearing the marker) before deleting the source,
  and deletes the source only once the entry no longer reads it.
- A read that resolved the source just before the copy finished and
  the source went away retries with the entry as it is now.

* filer, remote.sync: follow a pending rename's source across renames

When a remote-only entry is renamed over another remote-only file and
the deferred copy fails, it reads the source key through its marker.
Renamed again the same way, the marker keeps naming that source, but:

- remote.sync completed each rename from the event's old key. For the
  second one that is the first target's old object, which it copied
  over the destination before clearing the marker. It now copies from
  the source the entry's marker names.
- For the first rename it found no entry at the new path and deleted
  the old key, which the later entry still reads. When the event's new
  entry carried a marker and the path is gone, it now leaves the key to
  the entry that moved on.
- When the filer copies a pending entry immediately on a later rename,
  it now also releases the source the copy read, after the commit and
  with the same checks as the old key.

* remote.sync: release a pending rename's source, the filer leaves it

The filer deleted a pending entry's source after copying it on a later
rename, but that rename locks only its own paths: another rename can
copy onto the source key before its entry is visible, and the cleanup
then deleted that copy. And deleting a pending entry never released its
source at all.

The filer now leaves marked sources alone. remote.sync releases one
when the pending entry is deleted, or when a later rename was copied
right away, and only while the live object is still the one the
entry's stamp describes and no remote-only entry holds the path: a
copy that landed there since has a different ETag and is kept.

* remote.sync: release a pending source when the renamed entry is cached

A rename copied right away left its pending source to remote.sync,
which released it only while the destination was still remote-only.
Cached before the sync got there, the destination took the other path
and the source stayed on the remote for good. Release it there too,
with the same checks.

* remote.sync: release a pending source only after the destination is safe

e6d6438 released the source a renamed entry's old version read before
the destination was uploaded. A cached destination that still carried
the pending marker then had nothing behind it: had its upload failed,
remote.uncache could drop its chunks and leave it reading a deleted
source. Release the source only after the upload is stamped, and only
when the entry no longer reads through a marker.

* remote.sync: delete a rename's old key only if it is still the renamed content

completeRemoteOnlyRename deleted the old key once the destination held
the content. Another rename onto the vacated path can copy its object
there before the sync reaches the first rename, and that copy was then
deleted with it. Delete the old key only while the live object is still
the one the renamed entry's stamp describes.

Also keep the pending-source marker out of DELETE ?tagging, which
clears every Seaweed-* attribute: the marker is not a tag, and losing it
points reads back at the old target's bytes.

* remote.sync: compare a rename's old key against the source's own stamp

A retry can find the renamed entry already stamped with the copy, and a
copy of a multipart source has another ETag, so the unchanged source
was kept and stayed on the remote. Carry the stamp the event recorded
for the source (or the pending entry's, for a marked source) and compare
against that.

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 19:21:54 +08:00
92 changed files with 7009 additions and 1488 deletions
+3 -3
View File
@@ -27,7 +27,7 @@ jobs:
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v4.38.2
uses: github/codeql-action/init@v4.38.3
# Override language selection by uncommenting this and choosing your languages
with:
languages: go
@@ -35,7 +35,7 @@ jobs:
# Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
# If this step fails, then you should remove it and run the build manually (see below).
- name: Autobuild
uses: github/codeql-action/autobuild@v4.38.2
uses: github/codeql-action/autobuild@v4.38.3
# ℹ️ Command-line programs to run using the OS shell.
# 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun
@@ -49,4 +49,4 @@ jobs:
# make release
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v4.38.2
uses: github/codeql-action/analyze@v4.38.3
+1 -1
View File
@@ -405,7 +405,7 @@ jobs:
output: trivy-results.sarif
exit-code: '0'
- name: Upload Trivy scan results to GitHub Security
uses: github/codeql-action/upload-sarif@v4.38.2
uses: github/codeql-action/upload-sarif@v4.38.3
if: always()
with:
sarif_file: trivy-results.sarif
@@ -460,7 +460,7 @@ jobs:
- name: Upload Trivy scan results to GitHub Security
if: always()
uses: github/codeql-action/upload-sarif@v4.38.2
uses: github/codeql-action/upload-sarif@v4.38.3
with:
sarif_file: trivy-results.sarif
category: trivy-${{ matrix.variant }}
+2 -2
View File
@@ -39,7 +39,7 @@ jobs:
check-latest: true
- name: Set up chart-testing
uses: helm/chart-testing-action@v2.8.0
uses: helm/chart-testing-action@v2.9.0
- name: Run chart-testing (list-changed)
id: list-changed
@@ -1869,7 +1869,7 @@ jobs:
echo "IMAGE_TAG=$TAG" >> $GITHUB_ENV
- name: Create kind cluster
uses: helm/kind-action@v1.15.0
uses: helm/kind-action@v1.15.1
- name: Verify volume resize hook security contexts
run: |
+11 -10
View File
@@ -23,8 +23,8 @@ require (
github.com/facebookgo/stack v0.0.0-20160209184415-751773369052 // indirect
github.com/facebookgo/stats v0.0.0-20151006221625-1b76add642e4
github.com/facebookgo/subset v0.0.0-20200203212716-c811ad88dec4 // indirect
github.com/fsnotify/fsnotify v1.9.0 // indirect
github.com/go-redsync/redsync/v4 v4.17.0
github.com/fsnotify/fsnotify v1.10.1 // indirect
github.com/go-redsync/redsync/v4 v4.18.0
github.com/go-sql-driver/mysql v1.10.1
github.com/go-zookeeper/zk v1.0.4 // indirect
github.com/golang/protobuf v1.5.4
@@ -91,7 +91,7 @@ require (
gocloud.dev/pubsub/natspubsub v0.46.0
gocloud.dev/pubsub/rabbitpubsub v0.46.0
golang.org/x/crypto v0.58.0
golang.org/x/exp v0.0.0-20260709172345-9ea1abe57597
golang.org/x/exp v0.0.0-20260813180055-c1d0aacb2297
golang.org/x/image v0.46.0
golang.org/x/net v0.61.0
golang.org/x/oauth2 v0.37.0
@@ -116,13 +116,13 @@ require (
github.com/DATA-DOG/go-sqlmock v1.5.2
github.com/Jille/raft-grpc-transport v1.6.1
github.com/ThreeDotsLabs/watermill v1.5.2
github.com/a-h/templ v0.3.1020
github.com/a-h/templ v0.3.1070
github.com/apache/cassandra-gocql-driver/v2 v2.1.2
github.com/apache/iceberg-go v0.7.0
github.com/apple/foundationdb/bindings/go v0.0.0-20250911184653-27f7192f47c3
github.com/arangodb/go-driver v1.6.9
github.com/armon/go-metrics v0.4.1
github.com/aws/aws-sdk-go-v2 v1.47.1
github.com/aws/aws-sdk-go-v2 v1.47.3
github.com/aws/aws-sdk-go-v2/config v1.33.6
github.com/aws/aws-sdk-go-v2/credentials v1.20.6
github.com/aws/aws-sdk-go-v2/service/s3 v1.114.1
@@ -157,7 +157,7 @@ require (
github.com/twmb/avro v1.9.0
github.com/xeipuuv/gojsonschema v1.2.0
github.com/ydb-platform/ydb-go-sdk-auth-environ v0.5.3
github.com/ydb-platform/ydb-go-sdk/v3 v3.151.1
github.com/ydb-platform/ydb-go-sdk/v3 v3.155.1
go.etcd.io/etcd/client/pkg/v3 v3.7.1
go.uber.org/atomic v1.12.0
golang.org/x/sync v0.24.0
@@ -172,6 +172,7 @@ require (
cloud.google.com/go/pubsub/v2 v2.6.0 // indirect
dario.cat/mergo v1.0.2 // indirect
github.com/Azure/azure-sdk-for-go/sdk/keyvault/internal v0.7.1 // indirect
github.com/Azure/azure-sdk-for-go/sdk/storage/internal v0.1.0 // indirect
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c // indirect
github.com/FilenCloudDienste/filen-sdk-go v0.0.39 // indirect
github.com/ProtonMail/gopenpgp/v3 v3.4.1 // indirect
@@ -293,10 +294,10 @@ require (
cloud.google.com/go/iam v1.12.0 // indirect
cloud.google.com/go/monitoring v1.30.0 // indirect
filippo.io/edwards25519 v1.2.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.23.1
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.23.2
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.14.1
github.com/Azure/azure-sdk-for-go/sdk/internal v1.12.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.8.0
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.8.2
github.com/Azure/azure-sdk-for-go/sdk/storage/azfile v1.7.0 // indirect
github.com/Azure/go-ntlmssp v0.1.1 // indirect
github.com/AzureAD/microsoft-authentication-library-for-go v1.8.0 // indirect
@@ -313,7 +314,7 @@ require (
github.com/ProtonMail/go-srp v0.0.7 // indirect
github.com/PuerkitoBio/goquery v1.12.0 // indirect
github.com/abbot/go-http-auth v0.4.0 // indirect
github.com/andybalholm/brotli v1.2.3 // indirect
github.com/andybalholm/brotli v1.2.6 // indirect
github.com/andybalholm/cascadia v1.3.4 // indirect
github.com/appscode/go-querystring v0.0.0-20170504095604-0126cfb3f1dc // indirect
github.com/arangodb/go-velocypack v0.0.0-20200318135517-5af53c29c67e // indirect
@@ -332,7 +333,7 @@ require (
github.com/aws/aws-sdk-go-v2/service/sso v1.38.1 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.43.1 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.51.1
github.com/aws/smithy-go v1.28.1
github.com/aws/smithy-go v1.28.5
github.com/boltdb/bolt v1.3.1 // indirect
github.com/bradenaw/juniper v0.15.3 // indirect
github.com/buengese/sgzip v0.1.1 // indirect
+26 -24
View File
@@ -34,8 +34,8 @@ filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo=
filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc=
github.com/AdaLogics/go-fuzz-headers v0.0.0-20240806141605-e8a1dd7889d6 h1:He8afgbRMd7mFxO99hRNu+6tazq8nFF9lIwo9JFroBk=
github.com/AdaLogics/go-fuzz-headers v0.0.0-20240806141605-e8a1dd7889d6/go.mod h1:8o94RPi1/7XTJvwPpRSzSUedZrtlirdB3r9Z20bi2f8=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.23.1 h1:zvXfGJCWvywnCA814d8ZiVyt+fm9nnTE8xSb99zRyfo=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.23.1/go.mod h1:iptorS+VYKFL2N6PnebpS91dubG35eAOEERnT4PJbQU=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.23.2 h1:utpeoEeZjd+A8J41zvoLsOOrqXHhX1Kx/X/tCW9dEYQ=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.23.2/go.mod h1:iptorS+VYKFL2N6PnebpS91dubG35eAOEERnT4PJbQU=
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.14.1 h1:u93s+zU2JD62im61Bm5CZIc1ZrOJaIAWEg0WOrMVkEo=
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.14.1/go.mod h1:oXtinPO4OLj9d1DOTrqrL1oRwGhcqadvAmrl6wTeGlk=
github.com/Azure/azure-sdk-for-go/sdk/azidentity/cache v0.4.0 h1:xFaZZ+IubdftrDHnGGwZ6QvQ3KHTtWl2MCK+GMt2vxs=
@@ -48,10 +48,12 @@ github.com/Azure/azure-sdk-for-go/sdk/keyvault/internal v0.7.1 h1:FbH3BbSb4bvGlu
github.com/Azure/azure-sdk-for-go/sdk/keyvault/internal v0.7.1/go.mod h1:9V2j0jn9jDEkCkv8w/bKTNppX/d0FVA1ud77xCIP4KA=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage v1.8.1 h1:/Zt+cDPnpC3OVDm/JKLOs7M2DKmLRIIp3XIx9pHHiig=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage v1.8.1/go.mod h1:Ng3urmn6dYe8gnbCMoHHVl5APYz2txho3koEkV2o2HA=
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.8.0 h1:irsmOWwkp0KCTTNS5e2hdFeIvSQClQo2No3IaNmL3Vw=
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.8.0/go.mod h1:GWcBkQj3MqN7ozHKLaCCAuNLiXoIGv2RtanfAwSjY/Y=
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.8.2 h1:tQJdgQjbKKffEiMOKJmcQPkY01eFHRqWIZVefqP44nI=
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.8.2/go.mod h1:kTkpJchXAdO7LGkKZP97a+Ou8TgaF4NT6MHwOIy5uSQ=
github.com/Azure/azure-sdk-for-go/sdk/storage/azfile v1.7.0 h1:cuiKf1UVyWHu+XSQghPZR/qEF43JIcuk2CDqMlPiT6M=
github.com/Azure/azure-sdk-for-go/sdk/storage/azfile v1.7.0/go.mod h1:9JSyvgXLPAOC4jfhgZg58XeU2FJHsGmxuYSclCFQ4ZY=
github.com/Azure/azure-sdk-for-go/sdk/storage/internal v0.1.0 h1:X/bTwe0qpvmvPPutc+LWo7T4cKMqDP9JHP4k+Co9ZB4=
github.com/Azure/azure-sdk-for-go/sdk/storage/internal v0.1.0/go.mod h1:+kX7ahyFelQ42F3RScM2CFgHJljp0phc/FieKLYVpFI=
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c h1:udKWzYgxTojEKWjV8V+WSxDXJ4NFATAsZjh8iIbsQIg=
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E=
github.com/Azure/go-ntlmssp v0.1.1 h1:l+FM/EEMb0U9QZE7mKNEDw5Mu3mFiaa2GKOoTSsNDPw=
@@ -118,8 +120,8 @@ github.com/ThreeDotsLabs/watermill v1.5.2 h1:0ES33Eq1jEsP/pWvtE4n8bE0bs+9Jq7boT7
github.com/ThreeDotsLabs/watermill v1.5.2/go.mod h1:i9/968UriGphWfEbfMuYSD1qFbYRjb0mE0r+rV0FPp4=
github.com/TomiHiltunen/geohash-golang v0.0.0-20150112065804-b3e4e625abfb h1:wumPkzt4zaxO4rHPBrjDK8iZMR41C1qs7njNqlacwQg=
github.com/TomiHiltunen/geohash-golang v0.0.0-20150112065804-b3e4e625abfb/go.mod h1:QiYsIBRQEO+Z4Rz7GoI+dsHVneZNONvhczuA+llOZNM=
github.com/a-h/templ v0.3.1020 h1:ypAT/L5ySWEnZ6Zft/5yfoWXYYkhFNvEFOeeqecg4tw=
github.com/a-h/templ v0.3.1020/go.mod h1:A2DlK61v+K+NRoGnhmYbNYVmtYHcFO5/AisMvBdDxTM=
github.com/a-h/templ v0.3.1070 h1:3iCOXHXqFXa99OQzgkUOLJbNzv/QjFGLwKZkBAg+ADA=
github.com/a-h/templ v0.3.1070/go.mod h1:iIqB3g2RHBe/WlnPwa+Bvhh1TKXHWLRnBD9NWHOLm78=
github.com/a1ex3/zstd-seekable-format-go/pkg v0.10.0 h1:iLDOF0rdGTrol/q8OfPIIs5kLD8XvA2q75o6Uq/tgak=
github.com/a1ex3/zstd-seekable-format-go/pkg v0.10.0/go.mod h1:DrEWcQJjz7t5iF2duaiyhg4jyoF0kxOD6LtECNGkZ/Q=
github.com/aalpar/deheap v1.1.2 h1:MABHLcnjqsffb8GLkUFDigqpBBxOMz0DoKM9QfELeTw=
@@ -142,8 +144,8 @@ github.com/alexbrainman/sspi v0.0.0-20250919150558-7d374ff0d59e h1:4dAU9FXIyQktp
github.com/alexbrainman/sspi v0.0.0-20250919150558-7d374ff0d59e/go.mod h1:cEWa1LVoE5KvSD9ONXsZrj0z6KqySlCCNKHlLzbqAt4=
github.com/anchore/go-lzo v0.1.1 h1:IwL/fvkdtlIrYIXck6WxZ3nb8WjjHziYYmGxlooyOnM=
github.com/anchore/go-lzo v0.1.1/go.mod h1:3kLx0bve2oN1iDwgM1U5zGku1Tfbdb0No5qp1eL1fIk=
github.com/andybalholm/brotli v1.2.3 h1:8H1qwOkl2LPfjf3YezB90JnCliZb6SInJ/OJkEbA5NQ=
github.com/andybalholm/brotli v1.2.3/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
github.com/andybalholm/brotli v1.2.6 h1:ftYnfj6usCp+UGV5kSJ3+chpMQgU+gJf/AxsUQ52REI=
github.com/andybalholm/brotli v1.2.6/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
github.com/andybalholm/cascadia v1.3.4 h1:vM2lgh0Vru9Vwyfm4cQqWP2HHMW0u0+2PAW7Q38Qufg=
github.com/andybalholm/cascadia v1.3.4/go.mod h1:BLRmbRjpEtNKieZOCCvYj4RqN+KRA41GBe/5O+G93kM=
github.com/antihax/optional v1.0.0/go.mod h1:uupD/76wgC+ih3iEmQUL+0Ugr19nfwCT1kdvxnR2qWY=
@@ -171,8 +173,8 @@ github.com/armon/go-metrics v0.4.1 h1:hR91U9KYmb6bLBYLQjyM+3j+rcd/UhE+G78SFnF8gJ
github.com/armon/go-metrics v0.4.1/go.mod h1:E6amYzXo6aW1tqzoZGT755KkbgrJsSdpwZ+3JqfkOG4=
github.com/aws/aws-sdk-go v1.55.8 h1:JRmEUbU52aJQZ2AjX4q4Wu7t4uZjOu71uyNmaWlUkJQ=
github.com/aws/aws-sdk-go v1.55.8/go.mod h1:ZkViS9AqA6otK+JBBNH2++sx1sgxrPKcSzPPvQkUtXk=
github.com/aws/aws-sdk-go-v2 v1.47.1 h1:uOIZnp4PK3ZhKI0dNrJrhTEsLxbpXHTAJlwoS1pvAtw=
github.com/aws/aws-sdk-go-v2 v1.47.1/go.mod h1:bttEH6JqnUL8LepvDVfdrds/fZ5bCIxzpe3abyUrhDU=
github.com/aws/aws-sdk-go-v2 v1.47.3 h1:HRoT/MkNi3bjO5xF4dCzLzvVmkvRGHj3/cLINDnJejk=
github.com/aws/aws-sdk-go-v2 v1.47.3/go.mod h1:xSl+OSI5ZUw6bSMupWc218zE5Pl+65MCCu3ySWFY99A=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.20 h1:GPRlPwz40I2B2VrBEASOA3Bi77NyeqejNLkifosX0rs=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.20/go.mod h1:g7PNzKcsOKWb4fkSRBA7BZVAS6Y8IcxzN+nRohhQ1Q8=
github.com/aws/aws-sdk-go-v2/config v1.33.6 h1:MBjkSTLczek/UgiK+EYPIoRTqE7gP8vtW3OFbFo7Nug=
@@ -213,8 +215,8 @@ github.com/aws/aws-sdk-go-v2/service/ssooidc v1.43.1 h1:orIWdNiLgzrhu/11RcPPKO/S
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.43.1/go.mod h1:skwM/xsbR/1ReUTesv9BhpJp1VjajR7DWQnuVLwiXsQ=
github.com/aws/aws-sdk-go-v2/service/sts v1.51.1 h1:0HOqZXRvMytH6bFHVIc0oJX07sZjfhz0zXtjs6gdE8s=
github.com/aws/aws-sdk-go-v2/service/sts v1.51.1/go.mod h1:26zA0GhDrLo+yiLI2yXWxqB1PdsShfLikoI7GOEgugM=
github.com/aws/smithy-go v1.28.1 h1:R/nXH00c8qcfCzQVELtRw+eLQWtzv+VAIEFJ1/xxXlQ=
github.com/aws/smithy-go v1.28.1/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
github.com/aws/smithy-go v1.28.5 h1:OqbqKtzY9NUnr2yAe9+hiTdFb20tNaK/lI0PMOGXX9k=
github.com/aws/smithy-go v1.28.5/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
github.com/bahlo/generic-list-go v0.2.0 h1:5sz/EEAK+ls5wF+NeqDpk5+iNdMDXrh3z3nPnH1Wvgk=
github.com/bahlo/generic-list-go v0.2.0/go.mod h1:2KvAjgMlE5NNynlg/5iLrrCCZ2+5xWbdbCW3pNTGyYg=
github.com/bazelbuild/rules_go v0.46.0 h1:CTefzjN/D3Cdn3rkrM6qMWuQj59OBcuOjyIp3m4hZ7s=
@@ -462,8 +464,8 @@ github.com/fsnotify/fsevents v0.2.0 h1:BRlvlqjvNTfogHfeBOFvSC9N0Ddy+wzQCQukyoD7o
github.com/fsnotify/fsevents v0.2.0/go.mod h1:B3eEk39i4hz8y1zaWS/wPrAP4O6wkIl7HQwKBr1qH/w=
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/fsnotify/fsnotify v1.6.0/go.mod h1:sl3t1tCWJFWoRz9R8WJCbQihKKwmorjAbSClcnxKAGw=
github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
github.com/fsnotify/fsnotify v1.10.1 h1:b0/UzAf9yR5rhf3RPm9gf3ehBPpf0oZKIjtpKrx59Ho=
github.com/fsnotify/fsnotify v1.10.1/go.mod h1:TLheqan6HD6GBK6PrDWyDPBaEV8LspOxvPSjC+bVfgo=
github.com/fvbommel/sortorder v1.1.0 h1:fUmoe+HLsBTctBDoaBwpQo5N+nrCp8g/BjKb/6ZQmYw=
github.com/fvbommel/sortorder v1.1.0/go.mod h1:uk88iVf1ovNn1iLfgUVU2F9o5eO30ui720w+kxuqRs0=
github.com/gabriel-vasile/mimetype v1.4.13 h1:46nXokslUBsAJE/wMsp5gtO500a4F3Nkz9Ufpk2AcUM=
@@ -527,8 +529,8 @@ github.com/go-redis/redis/v7 v7.4.1 h1:PASvf36gyUpr2zdOUS/9Zqc80GbM+9BDyiJSJDDOr
github.com/go-redis/redis/v7 v7.4.1/go.mod h1:JDNMw23GTyLNC4GZu9njt15ctBQVn7xjRfnwdHj/Dcg=
github.com/go-redis/redis/v8 v8.11.5 h1:AcZZR7igkdvfVmQTPnu9WE37LRrO/YrBH5zWyjDC0oI=
github.com/go-redis/redis/v8 v8.11.5/go.mod h1:gREzHqY1hg6oD9ngVRbLStwAWKhA0FEgq8Jd4h5lpwo=
github.com/go-redsync/redsync/v4 v4.17.0 h1:FFJ+uxZs44y4Sq10//IFKic9T94AYl+u3Sog98AHzBo=
github.com/go-redsync/redsync/v4 v4.17.0/go.mod h1:CKVA6qwT07S/916i+Yd9h1/8YFQhCCpPYTQhvvYytJo=
github.com/go-redsync/redsync/v4 v4.18.0 h1:DDM09PEhmNTf/vziolf2x6391y1hHxF5vQ4ytHl9xQc=
github.com/go-redsync/redsync/v4 v4.18.0/go.mod h1:yuoqcQ55FS1VypSeVTMqFsJqTdWpEH5eA4MQGKJ9c4M=
github.com/go-resty/resty/v2 v2.17.2 h1:FQW5oHYcIlkCNrMD2lloGScxcHJ0gkjshV3qcQAyHQk=
github.com/go-resty/resty/v2 v2.17.2/go.mod h1:kCKZ3wWmwJaNc7S29BRtUhJwy7iqmn+2mLtQrOyQlVA=
github.com/go-sql-driver/mysql v1.10.1 h1:arlSnNLq6a5yxGxV7qg9lF4j0C+KwD6NbQyKr9QL6ME=
@@ -1063,10 +1065,10 @@ github.com/rdleal/intervalst v1.5.0 h1:SEB9bCFz5IqD1yhfH1Wv8IBnY/JQxDplwkxHjT6ha
github.com/rdleal/intervalst v1.5.0/go.mod h1:xO89Z6BC+LQDH+IPQQw/OESt5UADgFD41tYMUINGpxQ=
github.com/redis/go-redis/v9 v9.22.0 h1:laDvpYXTJtZLloinw1fA5Kqd6HAEH2XKxOkG/PDq2F0=
github.com/redis/go-redis/v9 v9.22.0/go.mod h1:y2g0Wj8rQvuK0ELM+oxSudcLtC09JScs98I/X9gRWY4=
github.com/redis/rueidis v1.0.76 h1:RdDWuvlYBSp+bTrBvaXqJnNEL3VVzsnjo+0psPFgLc4=
github.com/redis/rueidis v1.0.76/go.mod h1:UsfHPSbomB6QAVMk4iiFkzRy0nh9o7scDGa+SitvBY4=
github.com/redis/rueidis/rueidiscompat v1.0.76 h1:7LikbiqCQqCsZXeZ+akgZMnjIV/J0VHih9PIX4gGZC4=
github.com/redis/rueidis/rueidiscompat v1.0.76/go.mod h1:UatQQLVj4QMIsZtpvRWY28qm6r2d72idhcS+C/RM+Zg=
github.com/redis/rueidis v1.0.77 h1:ZR41bgJcm7oRFb3aSDPrRhC0eonDSrPzjvvZvHIlNjM=
github.com/redis/rueidis v1.0.77/go.mod h1:L8mnCQJJaSNL6I4pIR6Rz732HTGS9vmuXm0yT9dRvjo=
github.com/redis/rueidis/rueidiscompat v1.0.77 h1:S1xiLQsFv8XQ46e0LelC8ihGqbDzVRocXXZLzVAeh64=
github.com/redis/rueidis/rueidiscompat v1.0.77/go.mod h1:aXCvRVUTvTHZY0owHuz9JjveO1hyuzyDWrpJh8/E3Ds=
github.com/rekby/fixenv v0.6.1 h1:jUFiSPpajT4WY2cYuc++7Y1zWrnCxnovGCIX72PZniM=
github.com/rekby/fixenv v0.6.1/go.mod h1:/b5LRc06BYJtslRtHKxsPWFT/ySpHV+rWvzTg+XWk4c=
github.com/relvacode/iso8601 v1.7.0 h1:BXy+V60stMP6cpswc+a93Mq3e65PfXCgDFfhvNNGrdo=
@@ -1317,8 +1319,8 @@ github.com/ydb-platform/ydb-go-genproto v0.0.0-20260810122915-65bfd5c4b705 h1:7V
github.com/ydb-platform/ydb-go-genproto v0.0.0-20260810122915-65bfd5c4b705/go.mod h1:Er+FePu1dNUieD+XTMDduGpQuCPssK5Q4BjF+IIXJ3I=
github.com/ydb-platform/ydb-go-sdk-auth-environ v0.5.3 h1:A48PcmIETWwbEMIGwndv52uME6J1eJ28MBgF7HVR74o=
github.com/ydb-platform/ydb-go-sdk-auth-environ v0.5.3/go.mod h1:oRGF+67GWLbtNzqGygsPIUsjXmsRXD1KS5m3o5ppm2k=
github.com/ydb-platform/ydb-go-sdk/v3 v3.151.1 h1:T+fB2ZDHpYIGC7DWjK+rzZHLoexBF0zG/n3Q9DGNskY=
github.com/ydb-platform/ydb-go-sdk/v3 v3.151.1/go.mod h1:dJXJ1u00IqO8Vsph8fWmYj8b1E7jyphnvJPqA69emBY=
github.com/ydb-platform/ydb-go-sdk/v3 v3.155.1 h1:ZMByYe53zZ9R1o23fxJsbNhjzKFQkzpXm6uGj8gWHXQ=
github.com/ydb-platform/ydb-go-sdk/v3 v3.155.1/go.mod h1:dJXJ1u00IqO8Vsph8fWmYj8b1E7jyphnvJPqA69emBY=
github.com/ydb-platform/ydb-go-yc v0.12.5 h1:qYIeU1N6a92FaDA0+/vR55l5IzRF50MeLZwUWwvy2xQ=
github.com/ydb-platform/ydb-go-yc v0.12.5/go.mod h1:DlSltVQKEdXJotaAYwoHzMGDF9EJw5S3hg3LXe/5yYw=
github.com/ydb-platform/ydb-go-yc-metadata v0.6.2 h1:CZFawwHjLcjsxcl/g0dGIXbI2r41okPGLCydF4qTHcc=
@@ -1447,8 +1449,8 @@ golang.org/x/crypto v0.14.0/go.mod h1:MVFd36DqK4CsrnJYDkBA3VC4m2GkXAM0PvzMCn4JQf
golang.org/x/crypto v0.58.0 h1:COkYLr4k7nDI4r6QtMML2AfzeIswWHAJthLS6K/M/54=
golang.org/x/crypto v0.58.0/go.mod h1:Xh+kl5A+M0gMiWDytH2Hhr5Z7WbNCpv/eB/EFgfSk+w=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20260709172345-9ea1abe57597 h1:qLvzZeaANDgyVOA8pyHCOStGlXn0rseXma+GQjeuv2g=
golang.org/x/exp v0.0.0-20260709172345-9ea1abe57597/go.mod h1:EdfpwwqSu+0Li0mzskwHU6FWDV3t9Q+RZDo3QMUtL3Q=
golang.org/x/exp v0.0.0-20260813180055-c1d0aacb2297 h1:YXnL44eJ77R+ji4/ooy8UsXIhz+lbi2Qgdlc8iRN0gY=
golang.org/x/exp v0.0.0-20260813180055-c1d0aacb2297/go.mod h1:Mkmymgv+uMpSQ/XxJ/7GpdrdYoqm3u72jEbpCLiJmNk=
golang.org/x/image v0.46.0 h1:b1+oYj0Jbp6K5MDT4i4/eZpYlk3V8SJhhDKh6LBHAyQ=
golang.org/x/image v0.46.0/go.mod h1:3B3W05VGVQyuXucLINLjXKrqISASfi4Xj+iCVkLMwew=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
+1080 -1059
View File
File diff suppressed because it is too large Load Diff

Before

Width:  |  Height:  |  Size: 54 KiB

After

Width:  |  Height:  |  Size: 54 KiB

+15 -8
View File
@@ -234,9 +234,9 @@ dependencies = [
[[package]]
name = "aws-lc-rs"
version = "1.16.2"
version = "1.18.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a054912289d18629dc78375ba2c3726a3afe3ff71b4edba9dedfca0e3446d1fc"
checksum = "b281d307588d634de920874890732659e2e7672f72b5e10e81badc1a8a83621e"
dependencies = [
"aws-lc-sys",
"zeroize",
@@ -244,14 +244,15 @@ dependencies = [
[[package]]
name = "aws-lc-sys"
version = "0.39.1"
version = "0.45.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "83a25cf98105baa966497416dbd42565ce3a8cf8dbfd59803ec9ad46f3126399"
checksum = "9bff6c3b54fad79a2e60b8102caf565819711497c1f5f092f49508e2f5c31b27"
dependencies = [
"cc",
"cmake",
"dunce",
"fs_extra",
"pkg-config",
]
[[package]]
@@ -2677,6 +2678,12 @@ dependencies = [
"spki 0.7.3",
]
[[package]]
name = "pkg-config"
version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f6b464fbc74e149a392436b17d523f769e057cb6877f6a5c4618bc6f11800548"
[[package]]
name = "png"
version = "0.18.1"
@@ -3389,9 +3396,9 @@ dependencies = [
[[package]]
name = "rustls"
version = "0.23.37"
version = "0.23.45"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "758025cb5fccfd3bc2fd74708fd4682be41d99e5dff73c377c0646c6012c73a4"
checksum = "0d41d731c7d2f962d1ccc364cec258de3c0e93b38c2fb3ba97ac74513048d634"
dependencies = [
"aws-lc-rs",
"log",
@@ -3436,9 +3443,9 @@ dependencies = [
[[package]]
name = "rustls-webpki"
version = "0.103.13"
version = "0.103.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "61c429a8649f110dddef65e2a5ad240f747e85f7758a6bccc7e5777bd33f756e"
checksum = "f3c3cf1d8b1e7d4927e2d154c3fcb02979afb9939629c62cd9048d4f07b60ac2"
dependencies = [
"aws-lc-rs",
"ring",
+2 -2
View File
@@ -114,7 +114,7 @@ func TestS3Integration(t *testing.T) {
// findAvailablePort finds an available port by binding to port 0
// startMiniCluster starts a weed mini instance directly without exec.
// Extra flags (e.g. "-s3.allowDeleteBucketNotEmpty=false") can be appended via extraArgs.
// Extra flags can be appended via extraArgs.
func startMiniCluster(t *testing.T, extraArgs ...string) (*TestCluster, error) {
// Allocate non-colliding ports (including gRPC offsets) for weed mini.
ports := testutil.MustFreeMiniPorts(t, []string{"Master", "Volume", "Filer", "S3"})
@@ -872,7 +872,7 @@ func TestS3DeleteBucketNotEmpty(t *testing.T) {
t.Skip("Skipping integration test in short mode")
}
cluster, err := startMiniCluster(t, "-s3.allowDeleteBucketNotEmpty=false")
cluster, err := startMiniCluster(t)
require.NoError(t, err)
defer cluster.Stop()
+2 -2
View File
@@ -228,8 +228,8 @@ test-with-server: start-server
# Test with different configurations
test-versioning-with-configs: check-deps
@echo "Testing with different S3 configurations..."
@echo "Testing with empty folder allowed..."
@$(WEED_BINARY) mini -s3.port=$(S3_PORT) > weed-test-config1.log 2>&1 & echo $$! > weed-config1.pid
@echo "Testing with delete bucket not empty allowed..."
@$(WEED_BINARY) mini -s3.port=$(S3_PORT) -s3.allowDeleteBucketNotEmpty=true > weed-test-config1.log 2>&1 & echo $$! > weed-config1.pid
@sleep 5
@go test -v -timeout=5m -run "TestVersioningBasicWorkflow" . || true
@if [ -f weed-config1.pid ]; then kill -TERM $$(cat weed-config1.pid) 2>/dev/null || true; rm -f weed-config1.pid; fi
+27
View File
@@ -3,6 +3,7 @@ package sftp
import (
"bytes"
"io"
"os"
"path"
"testing"
@@ -602,6 +603,32 @@ func TestFileContent(t *testing.T) {
sftpClient.Remove("/binary.bin")
})
t.Run("WriteAtOffsetKeepsExistingBytes", func(t *testing.T) {
file, err := sftpClient.Create("/resume.txt")
require.NoError(t, err)
_, err = file.Write([]byte("hello world"))
require.NoError(t, err)
require.NoError(t, file.Close())
// Reopen without O_TRUNC and write past the start, as a resumed upload does
file, err = sftpClient.OpenFile("/resume.txt", os.O_WRONLY)
require.NoError(t, err)
_, err = file.WriteAt([]byte("WORLD!"), 6)
require.NoError(t, err)
require.NoError(t, file.Close())
readFile, err := sftpClient.Open("/resume.txt")
require.NoError(t, err)
content, err := io.ReadAll(readFile)
require.NoError(t, err)
readFile.Close()
require.Equal(t, "hello WORLD!", string(content))
// Clean up
sftpClient.Remove("/resume.txt")
})
t.Run("EmptyFile", func(t *testing.T) {
file, err := sftpClient.Create("/empty.txt")
require.NoError(t, err)
+66
View File
@@ -0,0 +1,66 @@
package plugin
import (
"strings"
"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
"github.com/seaweedfs/seaweedfs/weed/pb/worker_pb"
pluginworker "github.com/seaweedfs/seaweedfs/weed/plugin/worker"
"github.com/seaweedfs/seaweedfs/weed/util"
"google.golang.org/protobuf/proto"
)
// JobReplicaHosts are the volume-replica hosts a job will read from.
// Used so executor pick prefers a worker on one of those machines.
// Sources with ShardIds are EC-cleanup rows, not encode replicas.
func JobReplicaHosts(job *plugin_pb.JobSpec) []string {
if job == nil {
return nil
}
var addrs []string
addrs = append(addrs, pluginworker.ReadStringListConfig(job.Parameters, "replica_servers")...)
if src := strings.TrimSpace(pluginworker.ReadStringConfig(job.Parameters, "source_server", "")); src != "" {
addrs = append(addrs, src)
}
if payload := pluginworker.ReadBytesConfig(job.Parameters, "task_params_pb"); len(payload) > 0 {
params := &worker_pb.TaskParams{}
if err := proto.Unmarshal(payload, params); err == nil {
for _, s := range params.GetSources() {
if s == nil || s.Node == "" || len(s.ShardIds) > 0 {
continue
}
addrs = append(addrs, s.Node)
}
}
}
return util.HostKeysFromAddresses(addrs)
}
// partitionByPreferredHost splits candidates into (host matches, others).
// No preferred hosts, or no matches, returns (nil, all) so callers keep
// the existing slots+round-robin path.
func partitionByPreferredHost(candidates []*WorkerSession, preferredHosts []string) (pref, rest []*WorkerSession) {
want := util.HostKeysFromAddresses(preferredHosts)
if len(want) == 0 || len(candidates) == 0 {
return nil, candidates
}
set := make(map[string]struct{}, len(want))
for _, h := range want {
set[h] = struct{}{}
}
for _, c := range candidates {
if c == nil {
continue
}
if _, ok := set[util.HostKey(c.Address)]; ok {
pref = append(pref, c)
continue
}
rest = append(rest, c)
}
if len(pref) == 0 {
return nil, candidates
}
return pref, rest
}
+1 -1
View File
@@ -615,7 +615,7 @@ func (r *Plugin) ExecuteJob(
releaseGate := r.acquireDetectionExecutionGate(jobType, true)
defer releaseGate()
executor, err := r.registry.PickExecutor(jobType)
executor, err := r.registry.PickExecutor(jobType, JobReplicaHosts(job)...)
if err != nil {
return nil, err
}
+31
View File
@@ -887,6 +887,37 @@ func (r *Plugin) trackExecutionQueued(job *plugin_pb.JobSpec) {
})
}
// bindQueuedJobWorker records the home worker while the job stays pending/queued.
func (r *Plugin) bindQueuedJobWorker(job *plugin_pb.JobSpec, workerID string) {
if job == nil || strings.TrimSpace(job.JobId) == "" || strings.TrimSpace(workerID) == "" {
return
}
now := time.Now().UTC()
r.jobsMu.Lock()
tracked := r.jobs[job.JobId]
if tracked == nil {
r.jobsMu.Unlock()
return
}
if isTerminalTrackedJobState(tracked.State) {
r.jobsMu.Unlock()
return
}
tracked.WorkerID = workerID
tracked.Message = "queued on worker " + workerID
tracked.UpdatedAt = timeToPtr(now)
snap := cloneTrackedJob(*tracked)
r.dirtyJobs = true
r.jobsMu.Unlock()
r.persistJobDetailSnapshot(job.JobId, func(detail *TrackedJob) {
detail.WorkerID = snap.WorkerID
detail.Message = snap.Message
detail.UpdatedAt = snap.UpdatedAt
})
}
func (r *Plugin) cancelQueuedJob(job *plugin_pb.JobSpec, cause error) {
reason := "job canceled"
if cause != nil {
+206 -125
View File
@@ -4,6 +4,7 @@ import (
"context"
"errors"
"fmt"
"sort"
"strings"
"sync"
"sync/atomic"
@@ -28,12 +29,18 @@ const (
defaultScheduledMaxResults int32 = 1000
defaultScheduledExecutionConcurrency = 1
defaultScheduledPerWorkerConcurrency = 1
maxScheduledExecutionConcurrency = 128
defaultScheduledRetryBackoff = 5 * time.Second
defaultClusterContextTimeout = 10 * time.Second
defaultWaitingBacklogFloor = 8
defaultWaitingBacklogMultiplier = 4
maxEstimatedRuntimeCap = 8 * time.Hour
// Hold a job on replica-host workers until their reservation load
// reaches this; 0 disables the pin and spills immediately.
defaultLocalQueuePin = 5
// Wait this long after a home worker disappears before stealing
// its queue; 0 steals immediately. Covers a reconnect blink.
defaultOrphanRehomeGrace = 30 * time.Second
maxScheduledExecutionConcurrency = 128
defaultScheduledRetryBackoff = 5 * time.Second
defaultClusterContextTimeout = 10 * time.Second
defaultWaitingBacklogFloor = 8
defaultWaitingBacklogMultiplier = 4
maxEstimatedRuntimeCap = 8 * time.Hour
// How long started jobs may drain past the run window close.
scheduledExecutionDrainGrace = 30 * time.Minute
)
@@ -49,6 +56,12 @@ type schedulerPolicy struct {
PerWorkerConcurrency int
RetryLimit int
ExecutorReserveBackoff time.Duration
// LocalQueuePin is K: do not spill off replica-host workers while
// min(load(pref)) < K. Zero is the old immediate-spill path.
LocalQueuePin int
// OrphanRehomeGrace is how long a missing home's queue is held
// before steal-without-K. Zero steals as soon as the worker is gone.
OrphanRehomeGrace time.Duration
}
// laneSchedulerLoop is the main scheduling goroutine for a single lane.
@@ -466,6 +479,8 @@ func (r *Plugin) loadSchedulerPolicy(jobType string) (schedulerPolicy, bool, err
PerWorkerConcurrency: int(adminRuntime.PerWorkerExecutionConcurrency),
RetryLimit: int(adminRuntime.RetryLimit),
ExecutorReserveBackoff: 200 * time.Millisecond,
LocalQueuePin: defaultLocalQueuePin,
OrphanRehomeGrace: defaultOrphanRehomeGrace,
}
if policy.DetectionInterval < r.schedulerTick {
@@ -496,6 +511,9 @@ func (r *Plugin) loadSchedulerPolicy(jobType string) (schedulerPolicy, bool, err
if policy.ExecutionTimeout < defaultScheduledExecutionTimeout {
policy.ExecutionTimeout = defaultScheduledExecutionTimeout
}
if policy.LocalQueuePin < 0 {
policy.LocalQueuePin = 0
}
return policy, true, nil
}
@@ -813,19 +831,22 @@ func (r *Plugin) dispatchScheduledProposals(
// whole dispatch lets manual shell operations interleave between jobs.
jobsRequireLock := LaneRequiresLock(JobTypeLane(jobType))
jobQueue := make(chan *plugin_pb.JobSpec, len(proposals))
jobs := make([]*plugin_pb.JobSpec, 0, len(proposals))
for index, proposal := range proposals {
job := buildScheduledJobSpec(jobType, proposal, index)
r.trackExecutionQueued(job)
select {
case <-r.shutdownCh:
close(jobQueue)
return 0, 0, 0
for _, job := range jobs {
r.cancelQueuedJob(job, errSchedulerShutdown)
}
return 0, 0, len(jobs)
default:
jobQueue <- job
}
job := buildScheduledJobSpec(jobType, proposal, index)
r.trackExecutionQueued(job)
jobs = append(jobs, job)
}
close(jobQueue)
queues := r.assignJobsToWorkerQueues(jobType, jobs, policy)
var wg sync.WaitGroup
var statsMu sync.Mutex
@@ -842,130 +863,110 @@ func (r *Plugin) dispatchScheduledProposals(
wg.Add(1)
go func() {
defer wg.Done()
jobLoop:
for job := range jobQueue {
for {
select {
case <-r.shutdownCh:
return
default:
}
if ctx.Err() != nil {
r.cancelQueuedJob(job, ctx.Err())
statsMu.Lock()
canceledCount++
statsMu.Unlock()
return
}
if queues.isEmpty() {
return
}
item, targetID := r.takeQueuedJob(queues, jobType, policy)
if item == nil {
if queues.isEmpty() {
return
}
if !waitForShutdownOrTimerWithContext(r.shutdownCh, ctx, policy.ExecutorReserveBackoff) {
return
}
continue
}
if ctx.Err() != nil {
queues.pushFront(item)
return
}
executor, release, reserveErr := r.reserveExecutorByID(jobType, policy, targetID)
if reserveErr != nil {
queues.pushFront(item)
if ctx.Err() != nil {
return
}
if !waitForShutdownOrTimerWithContext(r.shutdownCh, ctx, policy.ExecutorReserveBackoff) {
return
}
continue
}
for {
select {
case <-r.shutdownCh:
return
default:
var releaseJobLock func()
if jobsRequireLock {
var lockErr error
releaseJobLock, lockErr = r.acquireAdminLock(fmt.Sprintf("plugin scheduler %s job %s", jobType, item.job.JobId))
if lockErr != nil {
release()
statsMu.Lock()
errorCount++
statsMu.Unlock()
r.appendActivity(JobActivity{
JobID: item.job.JobId,
JobType: item.job.JobType,
Source: "admin_scheduler",
Message: fmt.Sprintf("scheduled execution lock failed: %v", lockErr),
Stage: "failed",
OccurredAt: timeToPtr(time.Now().UTC()),
})
continue
}
if ctx.Err() != nil {
r.cancelQueuedJob(job, ctx.Err())
releaseJobLock()
release()
queues.pushFront(item)
return
}
}
err := r.executeScheduledJobWithExecutor(ctx, executor, item.job, clusterContext, policy)
if releaseJobLock != nil {
releaseJobLock()
}
release()
if errors.Is(err, errExecutorAtCapacity) {
queues.pushFront(item)
if !waitForShutdownOrTimerWithContext(r.shutdownCh, ctx, policy.ExecutorReserveBackoff) {
return
}
continue
}
if err != nil {
if ctx.Err() != nil || errors.Is(err, context.DeadlineExceeded) || errors.Is(err, context.Canceled) {
r.cancelQueuedJob(item.job, err)
statsMu.Lock()
canceledCount++
statsMu.Unlock()
continue jobLoop
}
executor, release, reserveErr := r.reserveScheduledExecutor(ctx, jobType, policy)
if reserveErr != nil {
if ctx.Err() != nil {
r.cancelQueuedJob(job, ctx.Err())
statsMu.Lock()
canceledCount++
statsMu.Unlock()
continue jobLoop
}
statsMu.Lock()
errorCount++
statsMu.Unlock()
r.appendActivity(JobActivity{
JobType: jobType,
Source: "admin_scheduler",
Message: fmt.Sprintf("scheduled execution reservation failed: %v", reserveErr),
Stage: "failed",
OccurredAt: timeToPtr(time.Now().UTC()),
})
break
}
var releaseJobLock func()
if jobsRequireLock {
var lockErr error
releaseJobLock, lockErr = r.acquireAdminLock(fmt.Sprintf("plugin scheduler %s job %s", jobType, job.JobId))
if lockErr != nil {
release()
statsMu.Lock()
errorCount++
statsMu.Unlock()
r.appendActivity(JobActivity{
JobID: job.JobId,
JobType: job.JobType,
Source: "admin_scheduler",
Message: fmt.Sprintf("scheduled execution lock failed: %v", lockErr),
Stage: "failed",
OccurredAt: timeToPtr(time.Now().UTC()),
})
break
}
// The lock may have been held by an operator for a
// while; re-check the execution window.
if ctx.Err() != nil {
releaseJobLock()
release()
r.cancelQueuedJob(job, ctx.Err())
statsMu.Lock()
canceledCount++
statsMu.Unlock()
continue jobLoop
}
}
err := r.executeScheduledJobWithExecutor(ctx, executor, job, clusterContext, policy)
if releaseJobLock != nil {
releaseJobLock()
}
release()
if errors.Is(err, errExecutorAtCapacity) {
r.trackExecutionQueued(job)
if !waitForShutdownOrTimerWithContext(r.shutdownCh, ctx, policy.ExecutorReserveBackoff) {
return
}
continue
}
if err != nil {
if ctx.Err() != nil || errors.Is(err, context.DeadlineExceeded) || errors.Is(err, context.Canceled) {
r.cancelQueuedJob(job, err)
statsMu.Lock()
canceledCount++
statsMu.Unlock()
continue jobLoop
}
statsMu.Lock()
errorCount++
statsMu.Unlock()
r.appendActivity(JobActivity{
JobID: job.JobId,
JobType: job.JobType,
Source: "admin_scheduler",
Message: fmt.Sprintf("scheduled execution failed: %v", err),
Stage: "failed",
OccurredAt: timeToPtr(time.Now().UTC()),
})
break
}
statsMu.Lock()
successCount++
errorCount++
statsMu.Unlock()
break
r.appendActivity(JobActivity{
JobID: item.job.JobId,
JobType: item.job.JobType,
Source: "admin_scheduler",
Message: fmt.Sprintf("scheduled execution failed: %v", err),
Stage: "failed",
OccurredAt: timeToPtr(time.Now().UTC()),
})
continue
}
statsMu.Lock()
successCount++
statsMu.Unlock()
}
}()
}
@@ -976,8 +977,8 @@ func (r *Plugin) dispatchScheduledProposals(
if drainErr == nil {
drainErr = errSchedulerShutdown
}
for job := range jobQueue {
r.cancelQueuedJob(job, drainErr)
for _, item := range queues.drain() {
r.cancelQueuedJob(item.job, drainErr)
canceledCount++
}
@@ -988,6 +989,7 @@ func (r *Plugin) reserveScheduledExecutor(
ctx context.Context,
jobType string,
policy schedulerPolicy,
preferredHosts []string,
) (*WorkerSession, func(), error) {
if ctx == nil {
ctx = context.Background()
@@ -1015,7 +1017,7 @@ func (r *Plugin) reserveScheduledExecutor(
return nil, nil, fmt.Errorf("timed out waiting for executor capacity for %s", jobType)
}
executors, err := r.registry.ListExecutors(jobType)
executors, err := r.registry.ListExecutors(jobType, preferredHosts...)
if err != nil {
if !waitForShutdownOrTimerWithContext(r.shutdownCh, ctx, policy.ExecutorReserveBackoff) {
if ctx.Err() != nil {
@@ -1026,11 +1028,19 @@ func (r *Plugin) reserveScheduledExecutor(
continue
}
for _, executor := range executors {
release, ok := r.tryReserveExecutorCapacity(executor, jobType, policy)
if !ok {
continue
pref, rest := partitionByPreferredHost(executors, preferredHosts)
lane := JobTypeLane(jobType)
loadOf := func(e *WorkerSession) int { return r.executorReservationLoad(e, lane) }
if len(pref) > 0 {
sortExecutorsByLoad(pref, loadOf)
if executor, release, ok := r.tryReserveFirst(pref, jobType, policy); ok {
return executor, release, nil
}
}
sortExecutorsByLoad(rest, loadOf)
if executor, release, ok := r.tryReserveFirst(rest, jobType, policy); ok {
return executor, release, nil
}
@@ -1051,6 +1061,75 @@ func (r *Plugin) tryReserveExecutorCapacity(
return r.tryReserveExecutorCapacityForLane(executor, jobType, policy, JobTypeLane(jobType))
}
func (r *Plugin) tryReserveFirst(executors []*WorkerSession, jobType string, policy schedulerPolicy) (*WorkerSession, func(), bool) {
for _, executor := range executors {
release, ok := r.tryReserveExecutorCapacity(executor, jobType, policy)
if ok {
return executor, release, true
}
}
return nil, nil, false
}
func (r *Plugin) reserveExecutorByID(jobType string, policy schedulerPolicy, workerID string) (*WorkerSession, func(), error) {
workerID = strings.TrimSpace(workerID)
if workerID == "" {
return nil, nil, fmt.Errorf("executor id is empty")
}
executors, err := r.registry.ListExecutors(jobType)
if err != nil {
return nil, nil, err
}
for _, executor := range executors {
if executor == nil || executor.WorkerID != workerID {
continue
}
release, ok := r.tryReserveExecutorCapacity(executor, jobType, policy)
if !ok {
return nil, nil, errExecutorAtCapacity
}
return executor, release, nil
}
return nil, nil, fmt.Errorf("executor %s is not available for %s", workerID, jobType)
}
func (r *Plugin) executorReservationLoad(executor *WorkerSession, lane SchedulerLane) int {
if executor == nil {
return 0
}
load := 0
if executor.Heartbeat != nil && executor.Heartbeat.ExecutionSlotsUsed > 0 {
load = int(executor.Heartbeat.ExecutionSlotsUsed)
}
workerID := strings.TrimSpace(executor.WorkerID)
if ls := r.lanes[lane]; ls != nil {
ls.execMu.Lock()
load += ls.execRes[workerID]
ls.execMu.Unlock()
return load
}
r.schedulerExecMu.Lock()
load += r.schedulerExecReservations[workerID]
r.schedulerExecMu.Unlock()
return load
}
func sortExecutorsByLoad(executors []*WorkerSession, load func(*WorkerSession) int) {
sort.SliceStable(executors, func(i, j int) bool {
if executors[i] == nil {
return false
}
if executors[j] == nil {
return true
}
li, lj := load(executors[i]), load(executors[j])
if li != lj {
return li < lj
}
return executors[i].WorkerID < executors[j].WorkerID
})
}
// tryReserveExecutorCapacityForLane reserves an execution slot on the
// per-lane reservation pool so that lanes cannot starve each other.
func (r *Plugin) tryReserveExecutorCapacityForLane(
@@ -1577,6 +1656,8 @@ func (r *Plugin) DispatchProposals(
ExecutionTimeout: defaultScheduledExecutionTimeout,
RetryBackoff: defaultScheduledRetryBackoff,
ExecutorReserveBackoff: 200 * time.Millisecond,
LocalQueuePin: defaultLocalQueuePin,
OrphanRehomeGrace: defaultOrphanRehomeGrace,
}
}
+58 -5
View File
@@ -61,6 +61,12 @@ func TestLoadSchedulerPolicyUsesAdminConfig(t *testing.T) {
if policy.JobTypeMaxRuntime != 30*time.Minute {
t.Fatalf("unexpected max runtime: got=%v", policy.JobTypeMaxRuntime)
}
if policy.LocalQueuePin != defaultLocalQueuePin {
t.Fatalf("unexpected local queue pin: got=%d", policy.LocalQueuePin)
}
if policy.OrphanRehomeGrace != defaultOrphanRehomeGrace {
t.Fatalf("unexpected orphan rehome grace: got=%v", policy.OrphanRehomeGrace)
}
}
func TestLoadSchedulerPolicyUsesDescriptorDefaultsWhenConfigMissing(t *testing.T) {
@@ -138,13 +144,13 @@ func TestReserveScheduledExecutorRespectsPerWorkerLimit(t *testing.T) {
ExecutorReserveBackoff: time.Millisecond,
}
executor1, release1, err := pluginSvc.reserveScheduledExecutor(context.Background(), "balance", policy)
executor1, release1, err := pluginSvc.reserveScheduledExecutor(context.Background(), "balance", policy, nil)
if err != nil {
t.Fatalf("reserve executor 1: %v", err)
}
defer release1()
executor2, release2, err := pluginSvc.reserveScheduledExecutor(context.Background(), "balance", policy)
executor2, release2, err := pluginSvc.reserveScheduledExecutor(context.Background(), "balance", policy, nil)
if err != nil {
t.Fatalf("reserve executor 2: %v", err)
}
@@ -266,7 +272,7 @@ func TestReserveScheduledExecutorTimesOutWhenNoExecutor(t *testing.T) {
start := time.Now()
pluginSvc.Shutdown()
_, _, err = pluginSvc.reserveScheduledExecutor(context.Background(), "missing-job-type", policy)
_, _, err = pluginSvc.reserveScheduledExecutor(context.Background(), "missing-job-type", policy, nil)
if err == nil {
t.Fatalf("expected reservation shutdown error")
}
@@ -297,7 +303,7 @@ func TestReserveScheduledExecutorWaitsForWorkerCapacity(t *testing.T) {
ExecutorReserveBackoff: 5 * time.Millisecond,
}
_, release1, err := pluginSvc.reserveScheduledExecutor(context.Background(), "balance", policy)
_, release1, err := pluginSvc.reserveScheduledExecutor(context.Background(), "balance", policy, nil)
if err != nil {
t.Fatalf("reserve executor 1: %v", err)
}
@@ -308,7 +314,7 @@ func TestReserveScheduledExecutorWaitsForWorkerCapacity(t *testing.T) {
}
secondReserveCh := make(chan reserveResult, 1)
go func() {
_, release2, reserveErr := pluginSvc.reserveScheduledExecutor(context.Background(), "balance", policy)
_, release2, reserveErr := pluginSvc.reserveScheduledExecutor(context.Background(), "balance", policy, nil)
if release2 != nil {
release2()
}
@@ -992,3 +998,50 @@ func TestLaneStatus_LifecycleNextDetectionSurvivesDefaultLanePrune(t *testing.T)
status.NextDetectionAt, expected)
}
}
func TestReserveWaterfillsPreferredHosts(t *testing.T) {
t.Parallel()
pluginSvc, err := New(Options{})
if err != nil {
t.Fatalf("New: %v", err)
}
defer pluginSvc.Shutdown()
ids := []string{"worker-a", "worker-b", "worker-c", "worker-d"}
for _, id := range ids {
pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{
WorkerId: id,
Address: "10.0.0.1:9334",
Capabilities: []*plugin_pb.JobTypeCapability{
{JobType: "erasure_coding", CanExecute: true, MaxExecutionConcurrency: 8},
},
})
}
policy := schedulerPolicy{
PerWorkerConcurrency: 8,
ExecutorReserveBackoff: time.Millisecond,
LocalQueuePin: 5,
}
preferred := []string{"10.0.0.1:8080"}
seen := map[string]int{}
var releases []func()
defer func() {
for _, release := range releases {
release()
}
}()
for i := 0; i < 4; i++ {
exec, release, err := pluginSvc.reserveScheduledExecutor(context.Background(), "erasure_coding", policy, preferred)
if err != nil {
t.Fatalf("reserve %d: %v", i, err)
}
releases = append(releases, release)
seen[exec.WorkerID]++
}
if len(seen) != 4 {
t.Fatalf("want one job per worker, got %v", seen)
}
}
+23 -5
View File
@@ -277,14 +277,24 @@ func (r *Registry) PickDetector(jobType string) (*WorkerSession, error) {
}
// PickExecutor picks one executor worker for a job type.
func (r *Registry) PickExecutor(jobType string) (*WorkerSession, error) {
return r.pickByKind(jobType, false)
// preferredHosts, when set, ranks workers whose Hello Address host matches a
// replica/source host ahead of others (same-machine EC pull). Empty means the
// previous slots+round-robin order.
func (r *Registry) PickExecutor(jobType string, preferredHosts ...string) (*WorkerSession, error) {
list, err := r.ListExecutors(jobType, preferredHosts...)
if err != nil {
return nil, err
}
return list[0], nil
}
// ListExecutors returns sorted executor candidates for one job type.
// Ordering is by most available execution slots, then lexical worker ID.
// The top tie group is rotated round-robin to prevent sticky assignment.
func (r *Registry) ListExecutors(jobType string) ([]*WorkerSession, error) {
// preferredHosts, when any worker matches, are listed first (then the rest);
// rotation applies only inside the preferred group so a local worker is not
// rotated away in favor of a remote one.
func (r *Registry) ListExecutors(jobType string, preferredHosts ...string) ([]*WorkerSession, error) {
r.mu.Lock()
defer r.mu.Unlock()
@@ -293,8 +303,16 @@ func (r *Registry) ListExecutors(jobType string) ([]*WorkerSession, error) {
return nil, fmt.Errorf("no executor worker available for job_type=%s", jobType)
}
sortByKind(candidates, jobType, false)
r.rotateTopCandidatesLocked(candidates, jobType, false)
pref, rest := partitionByPreferredHost(candidates, preferredHosts)
if len(pref) > 0 {
sortByKind(pref, jobType, false)
r.rotateTopCandidatesLocked(pref, jobType, false)
sortByKind(rest, jobType, false)
candidates = append(pref, rest...)
} else {
sortByKind(candidates, jobType, false)
r.rotateTopCandidatesLocked(candidates, jobType, false)
}
out := make([]*WorkerSession, 0, len(candidates))
for _, candidate := range candidates {
+67
View File
@@ -401,3 +401,70 @@ func TestRegistryHasCapableWorkerIgnoresNilCapability(t *testing.T) {
t.Fatalf("expected HasCapableWorker to return false when capability has no detect or execute")
}
}
func TestListExecutorsPrefersMatchingReplicaHost(t *testing.T) {
t.Parallel()
r := NewRegistry()
r.UpsertFromHello(&plugin_pb.WorkerHello{
WorkerId: "worker-remote",
Address: "10.87.0.4:33646",
Capabilities: []*plugin_pb.JobTypeCapability{
{JobType: "erasure_coding", CanExecute: true, MaxExecutionConcurrency: 8},
},
})
r.UpsertFromHello(&plugin_pb.WorkerHello{
WorkerId: "worker-local",
Address: "10.87.0.3:33646",
Capabilities: []*plugin_pb.JobTypeCapability{
{JobType: "erasure_coding", CanExecute: true, MaxExecutionConcurrency: 1},
},
})
r.UpdateHeartbeat("worker-remote", &plugin_pb.WorkerHeartbeat{
WorkerId: "worker-remote", ExecutionSlotsUsed: 0, ExecutionSlotsTotal: 8,
})
r.UpdateHeartbeat("worker-local", &plugin_pb.WorkerHeartbeat{
WorkerId: "worker-local", ExecutionSlotsUsed: 0, ExecutionSlotsTotal: 1,
})
// Without preference the emptier remote worker wins on slots.
plain, err := r.ListExecutors("erasure_coding")
if err != nil {
t.Fatalf("ListExecutors: %v", err)
}
if plain[0].WorkerID != "worker-remote" {
t.Fatalf("without preference want remote first, got %s", plain[0].WorkerID)
}
// Replica on :8080 of the same host as worker-local.
pref, err := r.ListExecutors("erasure_coding", "10.87.0.3:8080")
if err != nil {
t.Fatalf("ListExecutors preferred: %v", err)
}
if pref[0].WorkerID != "worker-local" {
t.Fatalf("preferred host want worker-local first, got %s", pref[0].WorkerID)
}
if pref[1].WorkerID != "worker-remote" {
t.Fatalf("remote should still be fallback, got %s", pref[1].WorkerID)
}
}
func TestJobReplicaHostsFromSourcesAndReplicaServers(t *testing.T) {
t.Parallel()
job := &plugin_pb.JobSpec{
Parameters: map[string]*plugin_pb.ConfigValue{
"source_server": {
Kind: &plugin_pb.ConfigValue_StringValue{StringValue: "10.87.0.1:8080"},
},
"replica_servers": {
Kind: &plugin_pb.ConfigValue_StringList{
StringList: &plugin_pb.StringList{Values: []string{"10.87.0.1:8080", "10.87.0.2:8081"}},
},
},
},
}
got := JobReplicaHosts(job)
if len(got) != 2 {
t.Fatalf("got %v", got)
}
}
+317
View File
@@ -0,0 +1,317 @@
package plugin
import (
"sync"
"time"
"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
)
// queuedJob is one scheduled spec parked on a home worker's admin-side queue.
type queuedJob struct {
job *plugin_pb.JobSpec
home string
}
// workerJobQueues holds per-worker FIFOs for one dispatch batch.
// Home assignment is done once up front; run goroutines only pop/steal.
// leftover is the homeless FIFO: replica hostnames with no currently
// registered worker. Orphan grace does not apply here.
type workerJobQueues struct {
mu sync.Mutex
q map[string][]*queuedJob
leftover []*queuedJob
remaining int
missingSince map[string]time.Time
homes map[string]struct{}
}
func newWorkerJobQueues() *workerJobQueues {
return &workerJobQueues{
q: make(map[string][]*queuedJob),
missingSince: make(map[string]time.Time),
homes: make(map[string]struct{}),
}
}
func (qs *workerJobQueues) hasHome(workerID string) bool {
if qs == nil || workerID == "" {
return false
}
_, ok := qs.homes[workerID]
return ok
}
func (qs *workerJobQueues) push(home string, job *plugin_pb.JobSpec) {
if qs == nil || job == nil {
return
}
item := &queuedJob{job: job, home: home}
qs.mu.Lock()
defer qs.mu.Unlock()
qs.pushLocked(item, false)
}
func (qs *workerJobQueues) pushFront(item *queuedJob) {
if qs == nil || item == nil {
return
}
qs.mu.Lock()
defer qs.mu.Unlock()
qs.pushLocked(item, true)
}
func (qs *workerJobQueues) pushLocked(item *queuedJob, front bool) {
qs.remaining++
if item.home == "" {
if front {
qs.leftover = append([]*queuedJob{item}, qs.leftover...)
return
}
qs.leftover = append(qs.leftover, item)
return
}
if front {
qs.q[item.home] = append([]*queuedJob{item}, qs.q[item.home]...)
return
}
qs.q[item.home] = append(qs.q[item.home], item)
}
func (qs *workerJobQueues) isEmpty() bool {
if qs == nil {
return true
}
qs.mu.Lock()
defer qs.mu.Unlock()
return qs.remaining == 0
}
func (qs *workerJobQueues) queueLen(workerID string) int {
if qs == nil {
return 0
}
qs.mu.Lock()
defer qs.mu.Unlock()
return len(qs.q[workerID])
}
func (qs *workerJobQueues) leftoverLen() int {
if qs == nil {
return 0
}
qs.mu.Lock()
defer qs.mu.Unlock()
return len(qs.leftover)
}
func (qs *workerJobQueues) drain() []*queuedJob {
if qs == nil {
return nil
}
qs.mu.Lock()
defer qs.mu.Unlock()
out := make([]*queuedJob, 0, qs.remaining)
for id, qq := range qs.q {
out = append(out, qq...)
qs.q[id] = nil
}
out = append(out, qs.leftover...)
qs.leftover = nil
qs.remaining = 0
return out
}
func popFront(qq []*queuedJob) (*queuedJob, []*queuedJob) {
item := qq[0]
if len(qq) == 1 {
return item, nil
}
return item, qq[1:]
}
// assignJobsToWorkerQueues parks each job on the least-loaded replica-host
// worker. A home exists only if a currently registered worker's -address
// hostname matches a replica host. Otherwise the job is homeless: global
// leftover FIFO, no fake home, no K, no orphan grace. Single-threaded.
func (r *Plugin) assignJobsToWorkerQueues(
jobType string,
jobs []*plugin_pb.JobSpec,
_ schedulerPolicy,
) *workerJobQueues {
qs := newWorkerJobQueues()
executors, err := r.registry.ListExecutors(jobType)
lane := JobTypeLane(jobType)
queued := make(map[string]int, len(executors))
for _, job := range jobs {
if job == nil {
continue
}
if err != nil || len(executors) == 0 {
qs.push("", job)
continue
}
pref, _ := partitionByPreferredHost(executors, JobReplicaHosts(job))
if len(pref) == 0 {
// Never registered for that hostname: homeless, not an orphan.
qs.push("", job)
continue
}
for _, e := range pref {
if e != nil && e.WorkerID != "" {
qs.homes[e.WorkerID] = struct{}{}
}
}
sortExecutorsByLoad(pref, func(e *WorkerSession) int {
return r.executorReservationLoad(e, lane) + queued[e.WorkerID]
})
home := pref[0].WorkerID
qs.push(home, job)
queued[home]++
r.bindQueuedJobWorker(job, home)
}
return qs
}
// takeQueuedJob pops a job for a free worker.
// Home worker (hostname matched a replica / has a home-queue identity):
// 1. own home queue
// 2. steal from another home if len>=K, or orphan past OrphanRehomeGrace
// 3. homeless leftover FIFO
// No-home worker (no matching replica, no home-queue identity):
// 1. homeless leftover FIFO
// 2. only if leftover empty: steal (K / orphan), same rules
func (r *Plugin) takeQueuedJob(
qs *workerJobQueues,
jobType string,
policy schedulerPolicy,
) (item *queuedJob, targetID string) {
if qs == nil {
return nil, ""
}
executors, err := r.registry.ListExecutors(jobType)
if err != nil || len(executors) == 0 {
return nil, ""
}
lane := JobTypeLane(jobType)
pin := policy.LocalQueuePin
defer func() {
if item != nil && targetID != "" {
r.bindQueuedJobWorker(item.job, targetID)
}
}()
qs.mu.Lock()
defer qs.mu.Unlock()
if qs.remaining == 0 {
return nil, ""
}
var bestHome *WorkerSession
bestLoad := 0
for _, exec := range executors {
if len(qs.q[exec.WorkerID]) == 0 {
continue
}
limit := schedulerWorkerExecutionLimit(exec, jobType, policy)
if limit <= 0 {
continue
}
load := r.executorReservationLoad(exec, lane)
if load >= limit {
continue
}
if bestHome == nil || load < bestLoad || (load == bestLoad && exec.WorkerID < bestHome.WorkerID) {
bestHome = exec
bestLoad = load
}
}
if bestHome != nil {
id := bestHome.WorkerID
item, qs.q[id] = popFront(qs.q[id])
qs.remaining--
return item, id
}
var thief *WorkerSession
for _, exec := range executors {
limit := schedulerWorkerExecutionLimit(exec, jobType, policy)
if limit <= 0 {
continue
}
if r.executorReservationLoad(exec, lane) >= limit {
continue
}
if thief == nil || exec.WorkerID < thief.WorkerID {
thief = exec
}
}
if thief == nil {
return nil, ""
}
takeLeftover := func() (*queuedJob, string) {
if len(qs.leftover) == 0 {
return nil, ""
}
item, qs.leftover = popFront(qs.leftover)
qs.remaining--
return item, thief.WorkerID
}
if !qs.hasHome(thief.WorkerID) {
if item, id := takeLeftover(); item != nil {
return item, id
}
}
live := make(map[string]struct{}, len(executors))
for _, exec := range executors {
live[exec.WorkerID] = struct{}{}
delete(qs.missingSince, exec.WorkerID)
}
now := time.Now()
grace := policy.OrphanRehomeGrace
victimID := ""
victimLen := 0
victimOrphan := false
for id, qq := range qs.q {
if len(qq) == 0 {
delete(qs.missingSince, id)
continue
}
_, isLive := live[id]
orphan := !isLive
if isLive {
if pin > 0 && len(qq) < pin {
continue
}
} else {
if _, ok := qs.missingSince[id]; !ok {
qs.missingSince[id] = now
}
if grace > 0 && now.Sub(qs.missingSince[id]) < grace {
continue
}
}
better := victimID == ""
if !better && orphan != victimOrphan {
better = orphan
}
if !better && orphan == victimOrphan {
better = len(qq) > victimLen || (len(qq) == victimLen && id < victimID)
}
if better {
victimID = id
victimLen = len(qq)
victimOrphan = orphan
}
}
if victimID != "" {
item, qs.q[victimID] = popFront(qs.q[victimID])
qs.remaining--
return item, thief.WorkerID
}
return takeLeftover()
}
+600
View File
@@ -0,0 +1,600 @@
package plugin
import (
"fmt"
"testing"
"time"
"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
)
func replicaJob(id, source string) *plugin_pb.JobSpec {
return &plugin_pb.JobSpec{
JobId: id,
JobType: "erasure_coding",
Summary: id,
DedupeKey: id,
Parameters: map[string]*plugin_pb.ConfigValue{
"source_server": {
Kind: &plugin_pb.ConfigValue_StringValue{StringValue: source},
},
},
}
}
func registerECWorkers(t *testing.T, pluginSvc *Plugin, workers map[string]string) {
t.Helper()
for id, addr := range workers {
pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{
WorkerId: id,
Address: addr,
Capabilities: []*plugin_pb.JobTypeCapability{
{JobType: "erasure_coding", CanExecute: true, MaxExecutionConcurrency: 8},
},
})
}
}
func TestAssignJobsWaterfillsPreferredWorkers(t *testing.T) {
t.Parallel()
pluginSvc, err := New(Options{})
if err != nil {
t.Fatalf("New: %v", err)
}
defer pluginSvc.Shutdown()
registerECWorkers(t, pluginSvc, map[string]string{
"worker-a": "10.0.0.1:9334",
"worker-b": "10.0.0.2:9334",
"worker-c": "10.0.0.3:9334",
"worker-d": "10.0.0.4:9334",
})
jobs := []*plugin_pb.JobSpec{
replicaJob("j1", "10.0.0.1:8080"),
replicaJob("j2", "10.0.0.1:8080"),
replicaJob("j3", "10.0.0.1:8080"),
replicaJob("j4", "10.0.0.1:8080"),
}
// All four replica IPs listed so every worker is preferred.
for _, job := range jobs {
job.Parameters["replica_servers"] = &plugin_pb.ConfigValue{
Kind: &plugin_pb.ConfigValue_StringList{StringList: &plugin_pb.StringList{
Values: []string{"10.0.0.1:8080", "10.0.0.2:8080", "10.0.0.3:8080", "10.0.0.4:8080"},
}},
}
}
qs := pluginSvc.assignJobsToWorkerQueues("erasure_coding", jobs, schedulerPolicy{LocalQueuePin: 5})
if qs.queueLen("worker-a")+qs.queueLen("worker-b")+qs.queueLen("worker-c")+qs.queueLen("worker-d") != 4 {
t.Fatalf("expected 4 assigned jobs")
}
for _, id := range []string{"worker-a", "worker-b", "worker-c", "worker-d"} {
if qs.queueLen(id) != 1 {
t.Fatalf("want 1 job on %s, got %d", id, qs.queueLen(id))
}
}
}
func TestAssignAndTakeDoesNotStarveOtherHomes(t *testing.T) {
t.Parallel()
pluginSvc, err := New(Options{})
if err != nil {
t.Fatalf("New: %v", err)
}
defer pluginSvc.Shutdown()
registerECWorkers(t, pluginSvc, map[string]string{
"worker-local": "10.0.0.1:9334",
"worker-remote": "10.0.0.2:9334",
})
jobs := []*plugin_pb.JobSpec{
replicaJob("local-1", "10.0.0.1:8080"),
replicaJob("local-2", "10.0.0.1:8080"),
replicaJob("remote-1", "10.0.0.2:8080"),
}
policy := schedulerPolicy{PerWorkerConcurrency: 1, LocalQueuePin: 5}
qs := pluginSvc.assignJobsToWorkerQueues("erasure_coding", jobs, policy)
if qs.queueLen("worker-local") != 2 || qs.queueLen("worker-remote") != 1 {
t.Fatalf("homes local=%d remote=%d", qs.queueLen("worker-local"), qs.queueLen("worker-remote"))
}
_, target1 := pluginSvc.takeQueuedJob(qs, "erasure_coding", policy)
_, release1, err := pluginSvc.reserveExecutorByID("erasure_coding", policy, target1)
if err != nil {
t.Fatalf("reserve 1: %v", err)
}
defer release1()
item2, target2 := pluginSvc.takeQueuedJob(qs, "erasure_coding", policy)
if item2 == nil || target2 != "worker-remote" {
t.Fatalf("second take should start remote home, got job=%v target=%s", item2, target2)
}
}
func TestTakeStealsOnlyWhenHomeQueueReachesK(t *testing.T) {
t.Parallel()
pluginSvc, err := New(Options{})
if err != nil {
t.Fatalf("New: %v", err)
}
defer pluginSvc.Shutdown()
registerECWorkers(t, pluginSvc, map[string]string{
"worker-local": "10.0.0.1:9334",
"worker-remote": "10.0.0.2:9334",
})
jobs := make([]*plugin_pb.JobSpec, 0, 6)
for i := 0; i < 6; i++ {
jobs = append(jobs, replicaJob(fmt.Sprintf("j%d", i), "10.0.0.1:8080"))
}
policy := schedulerPolicy{PerWorkerConcurrency: 1, LocalQueuePin: 5}
qs := pluginSvc.assignJobsToWorkerQueues("erasure_coding", jobs, policy)
if qs.queueLen("worker-local") != 6 {
t.Fatalf("all jobs should home on local, got %d", qs.queueLen("worker-local"))
}
_, target := pluginSvc.takeQueuedJob(qs, "erasure_coding", policy)
if target != "worker-local" {
t.Fatalf("first take want local, got %s", target)
}
_, release, err := pluginSvc.reserveExecutorByID("erasure_coding", policy, "worker-local")
if err != nil {
t.Fatalf("reserve local: %v", err)
}
defer release()
// 5 still queued on local (>=K) and local has no slot → steal to remote.
stolen, stealTarget := pluginSvc.takeQueuedJob(qs, "erasure_coding", policy)
if stolen == nil || stealTarget != "worker-remote" {
t.Fatalf("expected steal to remote, got job=%v target=%s", stolen, stealTarget)
}
if qs.queueLen("worker-local") != 4 {
t.Fatalf("after steal want 4 left on local, got %d", qs.queueLen("worker-local"))
}
// 4 < K: do not steal again.
if item, tgt := pluginSvc.takeQueuedJob(qs, "erasure_coding", policy); item != nil {
t.Fatalf("pin should hold remaining jobs, got target=%s", tgt)
}
}
func TestTakeStealsImmediatelyWhenPinDisabled(t *testing.T) {
t.Parallel()
pluginSvc, err := New(Options{})
if err != nil {
t.Fatalf("New: %v", err)
}
defer pluginSvc.Shutdown()
registerECWorkers(t, pluginSvc, map[string]string{
"worker-local": "10.0.0.1:9334",
"worker-remote": "10.0.0.2:9334",
})
jobs := []*plugin_pb.JobSpec{
replicaJob("a", "10.0.0.1:8080"),
replicaJob("b", "10.0.0.1:8080"),
}
policy := schedulerPolicy{PerWorkerConcurrency: 1, LocalQueuePin: 0}
qs := pluginSvc.assignJobsToWorkerQueues("erasure_coding", jobs, policy)
_, target := pluginSvc.takeQueuedJob(qs, "erasure_coding", policy)
if target != "worker-local" {
t.Fatalf("first take want local, got %s", target)
}
_, release, err := pluginSvc.reserveExecutorByID("erasure_coding", policy, "worker-local")
if err != nil {
t.Fatalf("reserve local: %v", err)
}
defer release()
stolen, stealTarget := pluginSvc.takeQueuedJob(qs, "erasure_coding", policy)
if stolen == nil || stealTarget != "worker-remote" {
t.Fatalf("K=0 should steal immediately, got job=%v target=%s", stolen, stealTarget)
}
}
func TestTakeStealsOrphanedHomeWithoutK(t *testing.T) {
t.Parallel()
pluginSvc, err := New(Options{})
if err != nil {
t.Fatalf("New: %v", err)
}
defer pluginSvc.Shutdown()
registerECWorkers(t, pluginSvc, map[string]string{
"worker-local": "10.0.0.1:9334",
"worker-remote": "10.0.0.2:9334",
})
jobs := []*plugin_pb.JobSpec{
replicaJob("o1", "10.0.0.1:8080"),
replicaJob("o2", "10.0.0.1:8080"),
}
policy := schedulerPolicy{
PerWorkerConcurrency: 1,
LocalQueuePin: 5,
OrphanRehomeGrace: 30 * time.Second,
}
qs := pluginSvc.assignJobsToWorkerQueues("erasure_coding", jobs, policy)
if qs.queueLen("worker-local") != 2 {
t.Fatalf("want 2 on local, got %d", qs.queueLen("worker-local"))
}
pluginSvc.registry.Remove("worker-local")
if item, tgt := pluginSvc.takeQueuedJob(qs, "erasure_coding", policy); item != nil {
t.Fatalf("within grace should hold orphan queue, got target=%s", tgt)
}
if qs.queueLen("worker-local") != 2 {
t.Fatalf("held queue changed: %d", qs.queueLen("worker-local"))
}
qs.mu.Lock()
qs.missingSince["worker-local"] = time.Now().Add(-time.Minute)
qs.mu.Unlock()
stolen, stealTarget := pluginSvc.takeQueuedJob(qs, "erasure_coding", policy)
if stolen == nil || stealTarget != "worker-remote" {
t.Fatalf("after grace should steal without K, got job=%v target=%s", stolen, stealTarget)
}
if qs.queueLen("worker-local") != 1 {
t.Fatalf("want 1 orphan left, got %d", qs.queueLen("worker-local"))
}
stolen2, stealTarget2 := pluginSvc.takeQueuedJob(qs, "erasure_coding", policy)
if stolen2 == nil || stealTarget2 != "worker-remote" {
t.Fatalf("second orphan should also move, got job=%v target=%s", stolen2, stealTarget2)
}
}
func TestTakeKeepsOrphanWhenHomeReconnectsDuringGrace(t *testing.T) {
t.Parallel()
pluginSvc, err := New(Options{})
if err != nil {
t.Fatalf("New: %v", err)
}
defer pluginSvc.Shutdown()
registerECWorkers(t, pluginSvc, map[string]string{
"worker-local": "10.0.0.1:9334",
"worker-remote": "10.0.0.2:9334",
})
jobs := []*plugin_pb.JobSpec{replicaJob("blink", "10.0.0.1:8080")}
policy := schedulerPolicy{
PerWorkerConcurrency: 1,
LocalQueuePin: 5,
OrphanRehomeGrace: 30 * time.Second,
}
qs := pluginSvc.assignJobsToWorkerQueues("erasure_coding", jobs, policy)
pluginSvc.registry.Remove("worker-local")
if item, _ := pluginSvc.takeQueuedJob(qs, "erasure_coding", policy); item != nil {
t.Fatal("expected hold during grace")
}
registerECWorkers(t, pluginSvc, map[string]string{"worker-local": "10.0.0.1:9334"})
item, target := pluginSvc.takeQueuedJob(qs, "erasure_coding", policy)
if item == nil || target != "worker-local" {
t.Fatalf("reconnected home should run its queue, got job=%v target=%s", item, target)
}
}
func TestTakeStealsOrphanImmediatelyWhenGraceDisabled(t *testing.T) {
t.Parallel()
pluginSvc, err := New(Options{})
if err != nil {
t.Fatalf("New: %v", err)
}
defer pluginSvc.Shutdown()
registerECWorkers(t, pluginSvc, map[string]string{
"worker-local": "10.0.0.1:9334",
"worker-remote": "10.0.0.2:9334",
})
jobs := []*plugin_pb.JobSpec{replicaJob("o1", "10.0.0.1:8080")}
policy := schedulerPolicy{PerWorkerConcurrency: 1, LocalQueuePin: 5, OrphanRehomeGrace: 0}
qs := pluginSvc.assignJobsToWorkerQueues("erasure_coding", jobs, policy)
pluginSvc.registry.Remove("worker-local")
stolen, stealTarget := pluginSvc.takeQueuedJob(qs, "erasure_coding", policy)
if stolen == nil || stealTarget != "worker-remote" {
t.Fatalf("grace=0 should steal immediately, got job=%v target=%s", stolen, stealTarget)
}
}
func TestBindQueuedJobWorkerKeepsPendingState(t *testing.T) {
t.Parallel()
pluginSvc, err := New(Options{})
if err != nil {
t.Fatalf("New: %v", err)
}
defer pluginSvc.Shutdown()
job := replicaJob("bind-1", "10.0.0.1:8080")
pluginSvc.trackExecutionQueued(job)
pluginSvc.bindQueuedJobWorker(job, "worker-local")
pluginSvc.jobsMu.Lock()
tracked := pluginSvc.jobs[job.JobId]
pluginSvc.jobsMu.Unlock()
if tracked == nil {
t.Fatal("missing tracked job")
}
if tracked.State != "job_state_pending" {
t.Fatalf("state=%s", tracked.State)
}
if tracked.Stage != "queued" {
t.Fatalf("stage=%s", tracked.Stage)
}
if tracked.WorkerID != "worker-local" {
t.Fatalf("worker=%s", tracked.WorkerID)
}
}
func TestHomelessAllRemoteWorkersUseFIFOWithoutPinOrGrace(t *testing.T) {
t.Parallel()
pluginSvc, err := New(Options{})
if err != nil {
t.Fatalf("New: %v", err)
}
defer pluginSvc.Shutdown()
// Workers never registered on the replica hostname — homeless, not orphan.
registerECWorkers(t, pluginSvc, map[string]string{
"worker-a": "10.0.0.5:9334",
"worker-b": "10.0.0.6:9334",
})
jobs := []*plugin_pb.JobSpec{
replicaJob("h1", "10.0.0.1:8080"),
replicaJob("h2", "10.0.0.1:8080"),
replicaJob("h3", "10.0.0.1:8080"),
}
policy := schedulerPolicy{
PerWorkerConcurrency: 1,
LocalQueuePin: 5,
OrphanRehomeGrace: 30 * time.Second,
}
qs := pluginSvc.assignJobsToWorkerQueues("erasure_coding", jobs, policy)
if qs.queueLen("worker-a") != 0 || qs.queueLen("worker-b") != 0 {
t.Fatalf("homeless jobs must not get a fake home, a=%d b=%d", qs.queueLen("worker-a"), qs.queueLen("worker-b"))
}
if qs.leftoverLen() != 3 {
t.Fatalf("want 3 homeless FIFO, got %d", qs.leftoverLen())
}
item1, target1 := pluginSvc.takeQueuedJob(qs, "erasure_coding", policy)
if item1 == nil || item1.job.JobId != "h1" {
t.Fatalf("FIFO first want h1, got %v", item1)
}
_, release1, err := pluginSvc.reserveExecutorByID("erasure_coding", policy, target1)
if err != nil {
t.Fatalf("reserve 1: %v", err)
}
defer release1()
item2, target2 := pluginSvc.takeQueuedJob(qs, "erasure_coding", policy)
if item2 == nil || item2.job.JobId != "h2" {
t.Fatalf("FIFO second want h2 immediately, got %v", item2)
}
if target2 == "" || target2 == target1 {
t.Fatalf("second take should hit the other idle worker, t1=%s t2=%s", target1, target2)
}
if qs.leftoverLen() != 1 {
t.Fatalf("want 1 homeless left, got %d", qs.leftoverLen())
}
}
func TestMixedReplicaLocalJobsStayHomeBeforeHomeless(t *testing.T) {
t.Parallel()
pluginSvc, err := New(Options{})
if err != nil {
t.Fatalf("New: %v", err)
}
defer pluginSvc.Shutdown()
registerECWorkers(t, pluginSvc, map[string]string{
"worker-local": "10.0.0.1:9334",
"worker-remote": "10.0.0.2:9334",
})
jobs := []*plugin_pb.JobSpec{
replicaJob("local-1", "10.0.0.1:8080"),
replicaJob("local-2", "10.0.0.1:8080"),
replicaJob("homeless-1", "10.0.0.9:8080"),
replicaJob("homeless-2", "10.0.0.9:8080"),
}
policy := schedulerPolicy{
PerWorkerConcurrency: 1,
LocalQueuePin: 5,
OrphanRehomeGrace: 30 * time.Second,
}
qs := pluginSvc.assignJobsToWorkerQueues("erasure_coding", jobs, policy)
if qs.queueLen("worker-local") != 2 || qs.queueLen("worker-remote") != 0 || qs.leftoverLen() != 2 {
t.Fatalf("homes local=%d remote=%d leftover=%d", qs.queueLen("worker-local"), qs.queueLen("worker-remote"), qs.leftoverLen())
}
item1, target1 := pluginSvc.takeQueuedJob(qs, "erasure_coding", policy)
if item1 == nil || item1.job.JobId != "local-1" || target1 != "worker-local" {
t.Fatalf("first take want local-1 on home, got job=%v target=%s", item1, target1)
}
_, release, err := pluginSvc.reserveExecutorByID("erasure_coding", policy, "worker-local")
if err != nil {
t.Fatalf("reserve local: %v", err)
}
defer release()
item2, target2 := pluginSvc.takeQueuedJob(qs, "erasure_coding", policy)
if item2 == nil || item2.job.JobId != "homeless-1" || target2 != "worker-remote" {
t.Fatalf("remote should take homeless, not steal pinned home, got job=%v target=%s", item2, target2)
}
if qs.queueLen("worker-local") != 1 {
t.Fatalf("remaining replica-local job must stay home, got %d", qs.queueLen("worker-local"))
}
if qs.leftoverLen() != 1 {
t.Fatalf("one homeless should remain, leftover=%d", qs.leftoverLen())
}
}
func TestIdleHomeWorkerTakesHomelessAfterEmptyHomeQueue(t *testing.T) {
t.Parallel()
pluginSvc, err := New(Options{})
if err != nil {
t.Fatalf("New: %v", err)
}
defer pluginSvc.Shutdown()
registerECWorkers(t, pluginSvc, map[string]string{
"worker-local": "10.0.0.1:9334",
})
jobs := []*plugin_pb.JobSpec{
replicaJob("home-1", "10.0.0.1:8080"),
replicaJob("homeless-1", "10.0.0.9:8080"),
}
policy := schedulerPolicy{
PerWorkerConcurrency: 1,
LocalQueuePin: 5,
OrphanRehomeGrace: 30 * time.Second,
}
qs := pluginSvc.assignJobsToWorkerQueues("erasure_coding", jobs, policy)
if qs.queueLen("worker-local") != 1 || qs.leftoverLen() != 1 {
t.Fatalf("home=%d leftover=%d", qs.queueLen("worker-local"), qs.leftoverLen())
}
item1, target1 := pluginSvc.takeQueuedJob(qs, "erasure_coding", policy)
if item1 == nil || item1.job.JobId != "home-1" || target1 != "worker-local" {
t.Fatalf("first take want home-1, got job=%v target=%s", item1, target1)
}
item2, target2 := pluginSvc.takeQueuedJob(qs, "erasure_coding", policy)
if item2 == nil || item2.job.JobId != "homeless-1" || target2 != "worker-local" {
t.Fatalf("idle home with empty queue may take homeless, got job=%v target=%s", item2, target2)
}
if qs.leftoverLen() != 0 {
t.Fatalf("homeless should be gone, leftover=%d", qs.leftoverLen())
}
}
func TestHomelessWorkerTakesFIFOBeforeStealableHome(t *testing.T) {
t.Parallel()
pluginSvc, err := New(Options{})
if err != nil {
t.Fatalf("New: %v", err)
}
defer pluginSvc.Shutdown()
registerECWorkers(t, pluginSvc, map[string]string{
"worker-local": "10.0.0.1:9334",
"worker-remote": "10.0.0.2:9334",
})
jobs := make([]*plugin_pb.JobSpec, 0, 7)
for i := 0; i < 6; i++ {
jobs = append(jobs, replicaJob(fmt.Sprintf("j%d", i), "10.0.0.1:8080"))
}
jobs = append(jobs, replicaJob("homeless-1", "10.0.0.9:8080"))
policy := schedulerPolicy{PerWorkerConcurrency: 1, LocalQueuePin: 5}
qs := pluginSvc.assignJobsToWorkerQueues("erasure_coding", jobs, policy)
if qs.queueLen("worker-local") != 6 || qs.leftoverLen() != 1 {
t.Fatalf("home=%d leftover=%d", qs.queueLen("worker-local"), qs.leftoverLen())
}
if qs.hasHome("worker-remote") {
t.Fatal("remote must have no home identity")
}
_, target := pluginSvc.takeQueuedJob(qs, "erasure_coding", policy)
if target != "worker-local" {
t.Fatalf("first take want local, got %s", target)
}
_, release, err := pluginSvc.reserveExecutorByID("erasure_coding", policy, "worker-local")
if err != nil {
t.Fatalf("reserve local: %v", err)
}
defer release()
item, takeTarget := pluginSvc.takeQueuedJob(qs, "erasure_coding", policy)
if item == nil || takeTarget != "worker-remote" || item.job.JobId != "homeless-1" {
t.Fatalf("remote must take homeless FIFO before K-steal, got job=%v target=%s", item, takeTarget)
}
if qs.leftoverLen() != 0 {
t.Fatalf("homeless should be gone, leftover=%d", qs.leftoverLen())
}
if qs.queueLen("worker-local") != 5 {
t.Fatalf("stealable home must stay until FIFO empty, got %d", qs.queueLen("worker-local"))
}
stolen, stealTarget := pluginSvc.takeQueuedJob(qs, "erasure_coding", policy)
if stolen == nil || stealTarget != "worker-remote" || stolen.job.JobId == "homeless-1" {
t.Fatalf("after FIFO empty remote may steal >=K, got job=%v target=%s", stolen, stealTarget)
}
}
func TestHomeWorkerStealsBeforeHomelessAfterOwnQueueEmpty(t *testing.T) {
t.Parallel()
pluginSvc, err := New(Options{})
if err != nil {
t.Fatalf("New: %v", err)
}
defer pluginSvc.Shutdown()
registerECWorkers(t, pluginSvc, map[string]string{
"worker-local": "10.0.0.1:9334",
"worker-other": "10.0.0.2:9334",
})
jobs := make([]*plugin_pb.JobSpec, 0, 8)
for i := 0; i < 6; i++ {
jobs = append(jobs, replicaJob(fmt.Sprintf("j%d", i), "10.0.0.1:8080"))
}
jobs = append(jobs, replicaJob("other-1", "10.0.0.2:8080"))
jobs = append(jobs, replicaJob("homeless-1", "10.0.0.9:8080"))
policy := schedulerPolicy{PerWorkerConcurrency: 1, LocalQueuePin: 5}
qs := pluginSvc.assignJobsToWorkerQueues("erasure_coding", jobs, policy)
if qs.queueLen("worker-local") != 6 || qs.queueLen("worker-other") != 1 || qs.leftoverLen() != 1 {
t.Fatalf("local=%d other=%d leftover=%d", qs.queueLen("worker-local"), qs.queueLen("worker-other"), qs.leftoverLen())
}
if !qs.hasHome("worker-other") {
t.Fatal("other must keep home identity")
}
item1, target1 := pluginSvc.takeQueuedJob(qs, "erasure_coding", policy)
if item1 == nil || target1 != "worker-local" {
t.Fatalf("first take want local home, got job=%v target=%s", item1, target1)
}
_, release, err := pluginSvc.reserveExecutorByID("erasure_coding", policy, "worker-local")
if err != nil {
t.Fatalf("reserve local: %v", err)
}
defer release()
item2, target2 := pluginSvc.takeQueuedJob(qs, "erasure_coding", policy)
if item2 == nil || item2.job.JobId != "other-1" || target2 != "worker-other" {
t.Fatalf("other should drain own home first, got job=%v target=%s", item2, target2)
}
stolen, stealTarget := pluginSvc.takeQueuedJob(qs, "erasure_coding", policy)
if stolen == nil || stealTarget != "worker-other" || stolen.job.JobId == "homeless-1" {
t.Fatalf("home worker with empty own queue still steals before homeless, got job=%v target=%s", stolen, stealTarget)
}
if qs.leftoverLen() != 1 {
t.Fatalf("homeless should still wait, leftover=%d", qs.leftoverLen())
}
}
+1 -1
View File
@@ -154,7 +154,7 @@ func init() {
filerS3Options.metricsHttpPort = cmdFiler.Flag.Int("s3.metricsPort", 0, "Prometheus metrics listen port")
filerS3Options.metricsHttpIp = cmdFiler.Flag.String("s3.metricsIp", "", "metrics listen ip. If empty, default to same as -s3.ip.bind option.")
cmdFiler.Flag.Bool("s3.allowEmptyFolder", true, "deprecated, ignored. Empty folder cleanup is now automatic.")
filerS3Options.allowDeleteBucketNotEmpty = cmdFiler.Flag.Bool("s3.allowDeleteBucketNotEmpty", true, "allow recursive deleting all entries along with bucket")
filerS3Options.allowDeleteBucketNotEmpty = cmdFiler.Flag.Bool("s3.allowDeleteBucketNotEmpty", false, "allow recursive deleting all entries along with bucket")
filerS3Options.autoCreateBucket = cmdFiler.Flag.Bool("s3.autoCreateBucket", true, "create the bucket on upload if it does not exist, for admin identities only")
filerS3Options.localSocket = cmdFiler.Flag.String("s3.localSocket", "", "default to /tmp/seaweedfs-s3-<port>.sock")
filerS3Options.tlsCACertificate = cmdFiler.Flag.String("s3.cacert.file", "", "path to the TLS CA certificate file")
+191 -18
View File
@@ -195,7 +195,7 @@ func (option *RemoteSyncOptions) makeEventProcessor(remoteStorage *remote_pb.Rem
return updateLocalEntry(option, message.NewParentPath, message.NewEntry, remoteEntry)
}
if filer_pb.IsDelete(resp) {
return processDeleteEvent(client, mountedDir, remoteStorageMountLocation, resp)
return processDeleteEvent(filerSource, client, mountedDir, remoteStorageMountLocation, resp)
}
if message.OldEntry != nil && message.NewEntry != nil {
return processUpdateEvent(option, filerSource, *option.storageClass, client, mountedDir, remoteStorageMountLocation, resp)
@@ -212,8 +212,11 @@ func (option *RemoteSyncOptions) makeEventProcessor(remoteStorage *remote_pb.Rem
// event the inline delete already handled), and GCS reports that case as
// ErrRemoteObjectNotFound where S3 and Azure answer an idempotent success.
// Returning the error would pin the sync offset on an event that has nothing
// left to do.
// left to do. A file whose path a remote-only entry has taken since keeps its
// object: a rename copied that entry's content to this key, and it exists
// nowhere else.
func processDeleteEvent(
filerSource filer_pb.FilerClient,
client remote_storage.RemoteStorageClient,
mountedDir string,
remoteStorageMountLocation *remote_pb.RemoteStorageLocation,
@@ -232,6 +235,36 @@ func processDeleteEvent(
}
glog.V(2).Infof("delete: %+v", resp)
dest := toRemoteStorageLocation(util.FullPath(mountedDir), util.NewFullPath(resp.Directory, message.OldEntry.Name), remoteStorageMountLocation)
if !message.OldEntry.IsDirectory {
// a deleted entry whose rename copy was still pending read its
// content from the source its marker names, which nothing else
// releases
if err := releasePendingSource(filerSource, client, mountedDir, remoteStorageMountLocation, message.OldEntry); err != nil {
return err
}
// The object at the key may have been replaced since the delete: a
// rename copies another entry's content onto it, a rewrite uploads
// over it, and a copy can land before its entry is visible. Delete
// only an object the event's stamp still describes.
if live, statErr := client.StatFile(dest); statErr == nil {
if oldRemote := message.OldEntry.GetRemoteEntry(); oldRemote == nil ||
!describes(oldRemote, live) ||
oldRemote.RemoteETag != "" && live.RemoteETag == "" {
glog.V(0).Infof("keep %s: the object changed since the delete", remote_storage.FormatLocation(dest))
return nil
}
} else if !errors.Is(statErr, remote_storage.ErrRemoteObjectNotFound) {
return statErr
}
current, err := currentEntry(filerSource, resp.Directory, message.OldEntry.Name)
if err != nil {
return err
}
if isRemoteOnly(current) {
glog.V(0).Infof("keep %s: a remote-only entry has taken its path since", remote_storage.FormatLocation(dest))
return nil
}
}
if message.OldEntry.IsDirectory {
glog.V(0).Infof("rmdir %s", remote_storage.FormatLocation(dest))
return client.RemoveDirectory(dest)
@@ -240,6 +273,41 @@ func processDeleteEvent(
return deleteRemoteFile(client, dest)
}
// releasePendingSource deletes the source a pending-rename marker on entry
// names, once nothing needs it. The filer never deletes a marked source: a
// concurrent rename can copy onto that key before its entry is visible. So
// the object goes only while it is still the one entry's stamp describes, and
// no remote-only entry holds its path.
func releasePendingSource(filerSource filer_pb.FilerClient, client remote_storage.RemoteStorageClient, mountedDir string, remoteStorageMountLocation *remote_pb.RemoteStorageLocation, entry *filer_pb.Entry) error {
src := util.FullPath(entry.GetExtended()[filer.ExtRemoteRenameSourceKey])
if src == "" || !strings.HasPrefix(string(src), strings.TrimSuffix(mountedDir, "/")+"/") {
return nil
}
srcDest := toRemoteStorageLocation(util.FullPath(mountedDir), src, remoteStorageMountLocation)
srcDir, srcName := src.DirAndName()
current, err := currentEntry(filerSource, srcDir, srcName)
if err != nil {
return err
}
if isRemoteOnly(current) {
glog.V(0).Infof("keep %s: a remote-only entry holds the pending rename's source", remote_storage.FormatLocation(srcDest))
return nil
}
live, err := client.StatFile(srcDest)
if errors.Is(err, remote_storage.ErrRemoteObjectNotFound) {
return nil
}
if err != nil {
return err
}
if stamp := entry.GetRemoteEntry(); !describes(stamp, live) || stamp.RemoteETag != "" && live.RemoteETag == "" {
glog.V(0).Infof("keep %s: the pending rename's source changed since", remote_storage.FormatLocation(srcDest))
return nil
}
glog.V(0).Infof("delete %s, the source of a pending rename", remote_storage.FormatLocation(srcDest))
return deleteRemoteFile(client, srcDest)
}
// deleteRemoteFile deletes the object and treats an already-absent object as
// deleted.
func deleteRemoteFile(client remote_storage.RemoteStorageClient, dest *remote_pb.RemoteStorageLocation) error {
@@ -268,6 +336,17 @@ func processUpdateEvent(
}
oldDest := toRemoteStorageLocation(util.FullPath(mountedDir), util.NewFullPath(resp.Directory, message.OldEntry.Name), remoteStorageMountLocation)
dest := toRemoteStorageLocation(util.FullPath(mountedDir), util.NewFullPath(message.NewParentPath, message.NewEntry.Name), remoteStorageMountLocation)
eventSource := renameSource{path: util.NewFullPath(resp.Directory, message.OldEntry.Name), loc: oldDest, stamp: message.OldEntry.GetRemoteEntry()}
// sourceOf is where a remote-only entry's content is: the source its
// pending-rename marker names, which may be an earlier rename's old key,
// or else the event's old key
sourceOf := func(entry *filer_pb.Entry, fallback renameSource) renameSource {
if src := string(entry.Extended[filer.ExtRemoteRenameSourceKey]); strings.HasPrefix(src, strings.TrimSuffix(mountedDir, "/")+"/") {
// a pending entry carries the stamp of the source it reads
return renameSource{path: util.FullPath(src), loc: toRemoteStorageLocation(util.FullPath(mountedDir), util.FullPath(src), remoteStorageMountLocation), stamp: entry.GetRemoteEntry()}
}
return fallback
}
if proto.Equal(oldDest, dest) && !shouldSendToRemote(message.NewEntry) {
glog.V(2).Infof("skipping updating: %+v", resp)
return nil
@@ -307,7 +386,23 @@ func processUpdateEvent(
return err
}
if isRemoteOnly(current) {
return completeRemoteOnlyRename(filerClient, client, message.NewParentPath, current, oldDest, dest, storageClass)
source := sourceOf(current, eventSource)
if err := completeRemoteOnlyRename(filerClient, client, message.NewParentPath, current, source, dest, storageClass); err != nil {
return err
}
// a pending entry the filer copied on this rename leaves the
// source its marker named
if pending := sourceOf(message.OldEntry, renameSource{}); pending.loc != nil && !proto.Equal(pending.loc, source.loc) {
return releasePendingSource(filerSource, client, mountedDir, remoteStorageMountLocation, message.OldEntry)
}
return nil
}
if current == nil && len(message.NewEntry.Extended[filer.ExtRemoteRenameSourceKey]) > 0 {
// the rename's copy was pending and the entry has moved on: a
// later rename carries the source in its own marker and finishes
// or releases it, so this one leaves it
glog.V(0).Infof("keep %s: the pending rename moved on", remote_storage.FormatLocation(oldDest))
return nil
}
glog.V(0).Infof("delete %s", remote_storage.FormatLocation(oldDest))
if err := client.DeleteFile(oldDest); err != nil {
@@ -323,14 +418,43 @@ func processUpdateEvent(
if errors.Is(writeErr, errSuperseded) {
glog.Errorf("skipping %s: %v", remote_storage.FormatLocation(dest), writeErr)
if !proto.Equal(oldDest, dest) {
return uploadCurrentEntry(filerClient, filerSource, client, message.NewParentPath, message.NewEntry.Name, oldDest, dest, storageClass)
if err := uploadCurrentEntry(filerClient, filerSource, client, message.NewParentPath, message.NewEntry.Name, eventSource, dest, storageClass, sourceOf); err != nil {
return err
}
return releaseRenamedSource(filerSource, client, mountedDir, remoteStorageMountLocation, message)
}
return nil
}
if writeErr != nil {
return writeErr
}
return updateLocalEntry(filerClient, message.NewParentPath, message.NewEntry, remoteEntry)
if err := updateLocalEntry(filerClient, message.NewParentPath, message.NewEntry, remoteEntry); err != nil {
return err
}
if !proto.Equal(oldDest, dest) {
return releaseRenamedSource(filerSource, client, mountedDir, remoteStorageMountLocation, message)
}
return nil
}
// releaseRenamedSource releases a pending source the renamed entry's old
// version read, once the destination holds the content on the remote under
// its own key: only after the upload and its stamp, which clears the marker,
// and only if the entry does not read through a marker any more. Released
// earlier, an upload that fails followed by remote.uncache would leave the
// entry reading a deleted source.
func releaseRenamedSource(filerSource filer_pb.FilerClient, client remote_storage.RemoteStorageClient, mountedDir string, remoteStorageMountLocation *remote_pb.RemoteStorageLocation, message *filer_pb.EventNotification) error {
if len(message.OldEntry.GetExtended()[filer.ExtRemoteRenameSourceKey]) == 0 {
return nil
}
current, err := currentEntry(filerSource, message.NewParentPath, message.NewEntry.Name)
if err != nil {
return err
}
if current != nil && len(current.Extended[filer.ExtRemoteRenameSourceKey]) > 0 {
return nil
}
return releasePendingSource(filerSource, client, mountedDir, remoteStorageMountLocation, message.OldEntry)
}
// currentEntry returns what the filer holds at dir/name now, nil when the
@@ -359,11 +483,24 @@ func isRemoteOnly(entry *filer_pb.Entry) bool {
// copied over the destination, the entry is stamped with the copy, and the
// old key goes. With neither object present the content is lost: the event
// fails and holds the offset for recovery.
func completeRemoteOnlyRename(filerClient filer_pb.FilerClient, client remote_storage.RemoteStorageClient, dir string, current *filer_pb.Entry, oldDest, dest *remote_pb.RemoteStorageLocation, storageClass string) error {
func completeRemoteOnlyRename(filerClient filer_pb.FilerClient, client remote_storage.RemoteStorageClient, dir string, current *filer_pb.Entry, source renameSource, dest *remote_pb.RemoteStorageLocation, storageClass string) error {
oldDest := source.loc
// the source's recorded stamp; a retry can find the entry already
// stamped with the copy, whose ETag a multipart source does not share
sourceStamp := source.stamp
if sourceStamp == nil {
sourceStamp = proto.Clone(current.RemoteEntry).(*filer_pb.RemoteEntry)
}
if existing, err := client.StatFile(dest); err == nil {
if describes(current.RemoteEntry, existing) {
glog.V(0).Infof("%s already holds the renamed content", remote_storage.FormatLocation(dest))
return deleteRemoteFile(client, oldDest)
if len(current.Extended[filer.ExtRemoteRenameSourceKey]) > 0 {
// reads still go to the source until the stamp clears the marker
if err := updateLocalEntry(filerClient, dir, current, existing); err != nil {
return err
}
}
return deleteRenameSource(filerClient, client, dir, current.Name, source, sourceStamp)
}
glog.V(0).Infof("%s holds an object the entry does not describe (size %d, want %d); replacing it from %s", remote_storage.FormatLocation(dest), existing.RemoteSize, current.RemoteEntry.RemoteSize, remote_storage.FormatLocation(oldDest))
} else if !errors.Is(err, remote_storage.ErrRemoteObjectNotFound) {
@@ -389,7 +526,42 @@ func completeRemoteOnlyRename(filerClient filer_pb.FilerClient, client remote_st
if err := updateLocalEntry(filerClient, dir, current, remoteEntry); err != nil {
return err
}
return deleteRemoteFile(client, oldDest)
return deleteRenameSource(filerClient, client, dir, current.Name, source, sourceStamp)
}
// renameSource is where a renamed remote-only entry's content was read
// from: the path, the object it maps to, and the stamp describing it.
type renameSource struct {
path util.FullPath
loc *remote_pb.RemoteStorageLocation
stamp *filer_pb.RemoteEntry
}
// deleteRenameSource deletes a rename's old key once the renamed entry no
// longer reads it: an entry still carrying the pending-source marker (its
// stamp lost a race) keeps the source, and the event is retried. The key is
// left alone when the object there is no longer the one stamp describes, as
// when another rename copied onto it, visible yet or not.
func deleteRenameSource(filerClient filer_pb.FilerClient, client remote_storage.RemoteStorageClient, dir, name string, source renameSource, stamp *filer_pb.RemoteEntry) error {
current, err := currentEntry(filerClient, dir, name)
if err != nil {
return err
}
if current != nil && len(current.Extended[filer.ExtRemoteRenameSourceKey]) > 0 {
return fmt.Errorf("%s still reads %s", util.NewFullPath(dir, name), remote_storage.FormatLocation(source.loc))
}
live, err := client.StatFile(source.loc)
if errors.Is(err, remote_storage.ErrRemoteObjectNotFound) {
return nil
}
if err != nil {
return err
}
if !describes(stamp, live) || stamp.GetRemoteETag() != "" && live.RemoteETag == "" {
glog.V(0).Infof("keep %s: it is no longer the renamed content", remote_storage.FormatLocation(source.loc))
return nil
}
return deleteRemoteFile(client, source.loc)
}
// describes reports whether the object a stat returned is the one the entry's
@@ -417,15 +589,15 @@ func openRemoteObject(client remote_storage.RemoteStorageClient, loc *remote_pb.
}
// uploadCurrentEntry uploads what the filer holds at dir/name now, when the
// event that superseded this rename will not. A superseded rename has
// already deleted the old key; the rewrite behind it in the log uploads the
// destination itself whenever shouldSendToRemote sees it as unsynced, so
// only a current entry that reads synced is uploaded here and the content
// goes up once. A remote-only entry is finished the way
// completeRemoteOnlyRename finishes a rename: its stamp can already describe
// the destination (a sync plus remote.uncache in the meantime), and only
// then is it complete.
func uploadCurrentEntry(filerClient filer_pb.FilerClient, filerSource filer_pb.FilerClient, client remote_storage.RemoteStorageClient, dir, name string, oldDest, dest *remote_pb.RemoteStorageLocation, storageClass string) error {
// event that superseded this rename will not. A rename whose snapshot is
// superseded has already deleted the old key; the rewrite behind it in the
// log uploads the destination itself unless shouldSendToRemote skips it on
// the inherited RemoteEntry, whose RemoteMtime can equal the rewrite's mtime
// within the same second. Only that case uploads here, so the content goes
// up once. A remote-only entry is finished the way completeRemoteOnlyRename
// finishes a rename: its stamp can already describe the destination (a sync
// plus remote.uncache in the meantime), and only then is it complete.
func uploadCurrentEntry(filerClient filer_pb.FilerClient, filerSource filer_pb.FilerClient, client remote_storage.RemoteStorageClient, dir, name string, oldSource renameSource, dest *remote_pb.RemoteStorageLocation, storageClass string, sourceOf func(*filer_pb.Entry, renameSource) renameSource) error {
current, _, _, err := filer_pb.GetEntry(context.Background(), filerSource, util.NewFullPath(dir, name))
if errors.Is(err, filer_pb.ErrNotFound) {
return nil
@@ -437,7 +609,7 @@ func uploadCurrentEntry(filerClient filer_pb.FilerClient, filerSource filer_pb.F
return nil
}
if isRemoteOnly(current) {
return completeRemoteOnlyRename(filerClient, client, dir, current, oldDest, dest, storageClass)
return completeRemoteOnlyRename(filerClient, client, dir, current, sourceOf(current, oldSource), dest, storageClass)
}
if shouldSendToRemote(current) {
glog.V(0).Infof("leaving %s to the rewrite that superseded the rename", remote_storage.FormatLocation(dest))
@@ -634,6 +806,7 @@ func updateLocalEntry(filerClient filer_pb.FilerClient, dir string, entry *filer
remoteEntry.LastLocalSyncTsNs = time.Now().UnixNano()
expected := proto.Clone(entry).(*filer_pb.Entry)
entry.RemoteEntry = remoteEntry
delete(entry.Extended, filer.ExtRemoteRenameSourceKey)
err := filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
_, err := client.UpdateEntry(context.Background(), &filer_pb.UpdateEntryRequest{
Directory: dir,
+375 -7
View File
@@ -5,6 +5,7 @@ import (
"errors"
"fmt"
"io"
"slices"
"strings"
"testing"
"time"
@@ -686,12 +687,20 @@ type recordingRemote struct {
// chunk ids of each written entry: which version of the file went up
writtenChunks [][]string
deleteErr error
writeErr error
// objects present on the remote, by path: StatFile and ReadFile answer
// from it, everything else is ErrRemoteObjectNotFound.
objects map[string][]byte
// statETag is the ETag StatFile reports for a present object, unless
// statETags names one for its path.
statETag string
statETags map[string]string
}
func (r *recordingRemote) WriteFile(loc *remote_pb.RemoteStorageLocation, entry *filer_pb.Entry, reader io.Reader) (*filer_pb.RemoteEntry, error) {
if r.writeErr != nil {
return nil, r.writeErr
}
r.writes = append(r.writes, loc)
var ids []string
for _, c := range entry.GetChunks() {
@@ -714,7 +723,11 @@ func (r *recordingRemote) StatFile(loc *remote_pb.RemoteStorageLocation) (*filer
if !ok {
return nil, remote_storage.ErrRemoteObjectNotFound
}
return &filer_pb.RemoteEntry{StorageName: loc.Name, RemoteSize: int64(len(body))}, nil
etag := r.statETag
if e, ok := r.statETags[loc.Path]; ok {
etag = e
}
return &filer_pb.RemoteEntry{StorageName: loc.Name, RemoteSize: int64(len(body)), RemoteETag: etag}, nil
}
func (r *recordingRemote) ReadFile(loc *remote_pb.RemoteStorageLocation, offset int64, size int64) ([]byte, error) {
@@ -1008,7 +1021,7 @@ func TestDeleteEventAbsentRemoteObjectIsSuccess(t *testing.T) {
t.Run("absent object", func(t *testing.T) {
remote := &recordingRemote{deleteErr: remote_storage.ErrRemoteObjectNotFound}
if err := processDeleteEvent(remote, mountedDir, mountLoc, resp); err != nil {
if err := processDeleteEvent(&stubFilerClient{}, remote, mountedDir, mountLoc, resp); err != nil {
t.Fatalf("err = %v, want nil: deleting an absent object is complete", err)
}
if len(remote.deletes) != 1 || !proto.Equal(remote.deletes[0], wantDelete) {
@@ -1020,7 +1033,7 @@ func TestDeleteEventAbsentRemoteObjectIsSuccess(t *testing.T) {
marked := proto.Clone(resp).(*filer_pb.SubscribeMetadataResponse)
marked.EventNotification.OldEntry.Extended = map[string][]byte{filer.ExtKeepRemoteObjectKey: []byte("true")}
remote := &recordingRemote{}
if err := processDeleteEvent(remote, mountedDir, mountLoc, marked); err != nil {
if err := processDeleteEvent(&stubFilerClient{}, remote, mountedDir, mountLoc, marked); err != nil {
t.Fatalf("err = %v", err)
}
if len(remote.deletes) != 0 {
@@ -1028,9 +1041,108 @@ func TestDeleteEventAbsentRemoteObjectIsSuccess(t *testing.T) {
}
})
t.Run("a remote-only entry that took the path keeps the object", func(t *testing.T) {
// deleted, then a remote-only file renamed onto the path: the rename
// copied its content to this key before the sync got to the delete
remote := &recordingRemote{}
filerClient := &stubFilerClient{entry: &filer_pb.Entry{
Name: "state.db.tmp",
Attributes: &filer_pb.FuseAttributes{FileSize: 10},
RemoteEntry: &filer_pb.RemoteEntry{RemoteSize: 10},
}}
if err := processDeleteEvent(filerClient, remote, mountedDir, mountLoc, resp); err != nil {
t.Fatalf("err = %v", err)
}
if len(remote.deletes) != 0 {
t.Errorf("deletes = %+v, want none", remote.deletes)
}
})
t.Run("a replaced object is kept", func(t *testing.T) {
// the delete recorded the old object's etag; a rename copied another
// entry's content onto the key since, and it must not go with it
deleted := proto.Clone(resp).(*filer_pb.SubscribeMetadataResponse)
deleted.EventNotification.OldEntry.RemoteEntry = &filer_pb.RemoteEntry{RemoteETag: `"orig"`, RemoteSize: 10}
remote := &recordingRemote{objects: map[string][]byte{"/b/vol/state.db.tmp": []byte("copied")}, statETag: `"copy"`}
filerClient := &stubFilerClient{entry: &filer_pb.Entry{Name: "state.db.tmp", Content: []byte("copied")}}
if err := processDeleteEvent(filerClient, remote, mountedDir, mountLoc, deleted); err != nil {
t.Fatalf("err = %v", err)
}
if len(remote.deletes) != 0 {
t.Errorf("deletes = %+v, want none", remote.deletes)
}
})
t.Run("a changed object with no entry at the path yet is kept", func(t *testing.T) {
// a directory rename copies a child onto the key before its new entry
// becomes visible; the copy may be the content's only remaining home
deleted := proto.Clone(resp).(*filer_pb.SubscribeMetadataResponse)
deleted.EventNotification.OldEntry.RemoteEntry = &filer_pb.RemoteEntry{RemoteETag: `"orig"`, RemoteSize: 10}
remote := &recordingRemote{objects: map[string][]byte{"/b/vol/state.db.tmp": []byte("copied")}, statETag: `"copy"`}
if err := processDeleteEvent(&stubFilerClient{}, remote, mountedDir, mountLoc, deleted); err != nil {
t.Fatalf("err = %v", err)
}
if len(remote.deletes) != 0 {
t.Errorf("deletes = %+v, want none", remote.deletes)
}
})
t.Run("an unchanged object still goes", func(t *testing.T) {
deleted := proto.Clone(resp).(*filer_pb.SubscribeMetadataResponse)
deleted.EventNotification.OldEntry.RemoteEntry = &filer_pb.RemoteEntry{RemoteETag: `"orig"`, RemoteSize: 3}
remote := &recordingRemote{objects: map[string][]byte{"/b/vol/state.db.tmp": []byte("old")}, statETag: `"orig"`}
if err := processDeleteEvent(&stubFilerClient{}, remote, mountedDir, mountLoc, deleted); err != nil {
t.Fatalf("err = %v", err)
}
if len(remote.deletes) != 1 {
t.Errorf("deletes = %+v, want one", remote.deletes)
}
})
t.Run("a copy that landed before its entry is kept", func(t *testing.T) {
// the deleted entry was never replicated, so its event records no
// stamp to match the fresh rename copy waiting for its filer entry
remote := &recordingRemote{objects: map[string][]byte{"/b/vol/state.db.tmp": []byte("copied")}, statETag: `"copy"`}
if err := processDeleteEvent(&stubFilerClient{}, remote, mountedDir, mountLoc, resp); err != nil {
t.Fatalf("err = %v", err)
}
if len(remote.deletes) != 0 {
t.Errorf("deletes = %+v, want none", remote.deletes)
}
})
t.Run("an unverifiable object is kept", func(t *testing.T) {
// the delete recorded an etag the live stat does not report; the
// object cannot be confirmed as the one the delete meant
deleted := proto.Clone(resp).(*filer_pb.SubscribeMetadataResponse)
deleted.EventNotification.OldEntry.RemoteEntry = &filer_pb.RemoteEntry{RemoteETag: `"orig"`, RemoteSize: 3}
remote := &recordingRemote{objects: map[string][]byte{"/b/vol/state.db.tmp": []byte("old")}}
if err := processDeleteEvent(&stubFilerClient{}, remote, mountedDir, mountLoc, deleted); err != nil {
t.Fatalf("err = %v", err)
}
if len(remote.deletes) != 0 {
t.Errorf("deletes = %+v, want none", remote.deletes)
}
})
t.Run("an entry with local data at the path does not stop the delete", func(t *testing.T) {
remote := &recordingRemote{}
filerClient := &stubFilerClient{entry: &filer_pb.Entry{
Name: "state.db.tmp",
Content: []byte("local"),
RemoteEntry: &filer_pb.RemoteEntry{RemoteSize: 10},
}}
if err := processDeleteEvent(filerClient, remote, mountedDir, mountLoc, resp); err != nil {
t.Fatalf("err = %v", err)
}
if len(remote.deletes) != 1 {
t.Errorf("deletes = %+v, want one", remote.deletes)
}
})
t.Run("other failure still fails", func(t *testing.T) {
remote := &recordingRemote{deleteErr: errors.New("AccessDenied: Access Denied")}
if err := processDeleteEvent(remote, mountedDir, mountLoc, resp); err == nil {
if err := processDeleteEvent(&stubFilerClient{}, remote, mountedDir, mountLoc, resp); err == nil {
t.Fatal("err = nil, want the delete failure")
}
})
@@ -1281,7 +1393,7 @@ func TestSupersededRenameUploadsCurrentEntry(t *testing.T) {
current := entryWith("b.txt", &filer_pb.RemoteEntry{StorageName: "gcs", RemoteMtime: 1786096669, RemoteSize: 1024})
remote := &recordingRemote{}
filerClient := &stubFilerClient{entry: current}
err := uploadCurrentEntry(filerClient, filerClient, remote, "/buckets/b/dir", "b.txt", wantDelete, wantWrite, "")
err := uploadCurrentEntry(filerClient, filerClient, remote, "/buckets/b/dir", "b.txt", renameSource{loc: wantDelete}, wantWrite, "", eventSource)
if err == nil || !strings.Contains(err.Error(), "neither") {
t.Errorf("err = %v, want the lost-content failure", err)
}
@@ -1292,9 +1404,9 @@ func TestSupersededRenameUploadsCurrentEntry(t *testing.T) {
t.Run("uncached with the destination already synced: complete without a write", func(t *testing.T) {
current := entryWith("b.txt", &filer_pb.RemoteEntry{StorageName: "gcs", RemoteMtime: 1786096669, RemoteSize: 7})
remote := &recordingRemote{objects: map[string][]byte{"/b/dir/b.txt": []byte("payload")}}
remote := &recordingRemote{objects: map[string][]byte{"/b/dir/a.txt": []byte("payload"), "/b/dir/b.txt": []byte("payload")}}
filerClient := &stubFilerClient{entry: current}
if err := uploadCurrentEntry(filerClient, filerClient, remote, "/buckets/b/dir", "b.txt", wantDelete, wantWrite, ""); err != nil {
if err := uploadCurrentEntry(filerClient, filerClient, remote, "/buckets/b/dir", "b.txt", renameSource{loc: wantDelete}, wantWrite, "", eventSource); err != nil {
t.Fatalf("err = %v, want nil: the destination object matches the entry's stamp", err)
}
if len(remote.writes) != 0 {
@@ -1306,6 +1418,228 @@ func TestSupersededRenameUploadsCurrentEntry(t *testing.T) {
})
}
func TestCompleteRemoteOnlyRenameClearsThePendingSourceFirst(t *testing.T) {
oldDest := &remote_pb.RemoteStorageLocation{Name: "b2", Bucket: "bucket", Path: "/b/src/a.jpg"}
dest := &remote_pb.RemoteStorageLocation{Name: "b2", Bucket: "bucket", Path: "/b/dst/a.jpg"}
pending := func() *filer_pb.Entry {
return &filer_pb.Entry{
Name: "a.jpg",
Attributes: &filer_pb.FuseAttributes{FileSize: 4},
RemoteEntry: &filer_pb.RemoteEntry{RemoteETag: `"copy"`, RemoteSize: 4},
Extended: map[string][]byte{filer.ExtRemoteRenameSourceKey: []byte("/buckets/b/src/a.jpg")},
}
}
objects := func() map[string][]byte {
return map[string][]byte{"/b/src/a.jpg": []byte("data"), "/b/dst/a.jpg": []byte("data")}
}
t.Run("stamped, then the source goes", func(t *testing.T) {
// the destination already holds the copy; the entry still reads the
// source until its stamp clears the marker
current := pending()
filerClient := &stubFilerClient{entry: current}
remote := &recordingRemote{objects: objects(), statETag: `"copy"`}
if err := completeRemoteOnlyRename(filerClient, remote, "/buckets/b/dst", current, renameSource{path: "/buckets/b/src/a.jpg", loc: oldDest}, dest, ""); err != nil {
t.Fatalf("err = %v", err)
}
if filerClient.updates != 1 {
t.Errorf("updates = %d, want the stamp that clears the marker", filerClient.updates)
}
if len(remote.deletes) != 1 || remote.deletes[0].Path != "/b/src/a.jpg" {
t.Errorf("deletes = %+v, want the source", remote.deletes)
}
})
t.Run("a marker still in place keeps the source", func(t *testing.T) {
// the stamp did not land: the filer still has the marker
filerClient := &stubFilerClient{entry: pending()}
remote := &recordingRemote{objects: objects(), statETag: `"copy"`}
if err := completeRemoteOnlyRename(filerClient, remote, "/buckets/b/dst", pending(), renameSource{path: "/buckets/b/src/a.jpg", loc: oldDest}, dest, ""); err == nil {
t.Fatal("err = nil, want the event retried")
}
if len(remote.deletes) != 0 {
t.Errorf("deletes = %+v, want none while the entry reads the source", remote.deletes)
}
})
}
// eventSource resolves a rename's source to the event's old key.
func eventSource(_ *filer_pb.Entry, source renameSource) renameSource {
return source
}
func TestRenameOfAPendingEntryFollowsItsSource(t *testing.T) {
// A was renamed over B and B over C, both copies failing: C reads A's
// object through the marker, and B's key holds the old target's bytes
const mountedDir = "/buckets"
mountLoc := &remote_pb.RemoteStorageLocation{Name: "b2", Bucket: "bucket", Path: "/"}
remoteOnly := &filer_pb.RemoteEntry{StorageName: "b2", RemoteETag: `"a"`, RemoteSize: 6}
pendingC := func() *filer_pb.Entry {
return &filer_pb.Entry{
Name: "c.jpg",
Attributes: &filer_pb.FuseAttributes{FileSize: 6},
RemoteEntry: remoteOnly,
Extended: map[string][]byte{filer.ExtRemoteRenameSourceKey: []byte("/buckets/b/a.jpg")},
}
}
event := func(oldName, newName string) *filer_pb.SubscribeMetadataResponse {
newEntry := pendingC()
newEntry.Name = newName
return &filer_pb.SubscribeMetadataResponse{
Directory: "/buckets/b",
EventNotification: &filer_pb.EventNotification{
OldEntry: &filer_pb.Entry{Name: oldName, RemoteEntry: remoteOnly},
NewParentPath: "/buckets/b",
NewEntry: newEntry,
},
}
}
objects := func() map[string][]byte {
return map[string][]byte{"/b/a.jpg": []byte("data-a"), "/b/b.jpg": []byte("old-b"), "/b/c.jpg": []byte("old-c")}
}
t.Run("the first rename leaves the source to the entry that moved on", func(t *testing.T) {
remote := &recordingRemote{objects: objects()}
filerClient := &stubFilerClient{} // nothing at b.jpg any more
if err := processUpdateEvent(filerClient, filerClient, "", remote, mountedDir, mountLoc, event("a.jpg", "b.jpg")); err != nil {
t.Fatal(err)
}
if len(remote.deletes) != 0 || len(remote.writes) != 0 {
t.Errorf("deletes = %+v, writes = %+v, want neither: c.jpg still reads a.jpg", remote.deletes, remote.writes)
}
})
t.Run("the second rename copies from the marker's source", func(t *testing.T) {
current := pendingC()
remote := &recordingRemote{objects: objects(), statETag: `"a"`}
filerClient := &stubFilerClient{entry: current}
if err := processUpdateEvent(filerClient, filerClient, "", remote, mountedDir, mountLoc, event("b.jpg", "c.jpg")); err != nil {
t.Fatal(err)
}
if len(remote.written) != 1 || string(remote.written[0]) != "data-a" {
t.Errorf("written = %q, want a.jpg's content", remote.written)
}
if len(remote.deletes) != 1 || remote.deletes[0].Path != "/b/a.jpg" {
t.Errorf("deletes = %+v, want the source once the stamp cleared the marker", remote.deletes)
}
})
}
func TestPendingRenameSourceIsReleasedOnlyWhileUnchanged(t *testing.T) {
const mountedDir = "/buckets"
mountLoc := &remote_pb.RemoteStorageLocation{Name: "b2", Bucket: "bucket", Path: "/"}
// b.jpg's copy failed after it replaced a target: it read a.jpg's object
pendingB := func() *filer_pb.Entry {
return &filer_pb.Entry{
Name: "b.jpg",
RemoteEntry: &filer_pb.RemoteEntry{StorageName: "b2", RemoteETag: `"a"`, RemoteSize: 6},
Extended: map[string][]byte{filer.ExtRemoteRenameSourceKey: []byte("/buckets/b/a.jpg")},
}
}
deleted := func() *filer_pb.SubscribeMetadataResponse {
return &filer_pb.SubscribeMetadataResponse{Directory: "/buckets/b", EventNotification: &filer_pb.EventNotification{OldEntry: pendingB()}}
}
deletedPaths := func(r *recordingRemote) (paths []string) {
for _, d := range r.deletes {
paths = append(paths, d.Path)
}
return
}
t.Run("deleting the pending entry releases its source", func(t *testing.T) {
remote := &recordingRemote{objects: map[string][]byte{"/b/a.jpg": []byte("data-a")}, statETag: `"a"`}
if err := processDeleteEvent(&stubFilerClient{}, remote, mountedDir, mountLoc, deleted()); err != nil {
t.Fatal(err)
}
if got := deletedPaths(remote); !slices.Contains(got, "/b/a.jpg") {
t.Errorf("deletes = %v, want the source", got)
}
})
t.Run("a source replaced since is kept", func(t *testing.T) {
// another rename copied onto a.jpg before its entry became visible
remote := &recordingRemote{objects: map[string][]byte{"/b/a.jpg": []byte("data-d")}, statETag: `"d"`}
if err := processDeleteEvent(&stubFilerClient{}, remote, mountedDir, mountLoc, deleted()); err != nil {
t.Fatal(err)
}
if got := deletedPaths(remote); slices.Contains(got, "/b/a.jpg") {
t.Errorf("deletes = %v, want the replaced source kept", got)
}
})
t.Run("a destination cached since releases the source too", func(t *testing.T) {
// c.jpg was cached before the sync reached the rename: it is no
// longer remote-only, and a.jpg is still left over
cached := &filer_pb.Entry{Name: "c.jpg", Content: []byte("data-a"), Attributes: &filer_pb.FuseAttributes{FileSize: 6}, RemoteEntry: &filer_pb.RemoteEntry{StorageName: "b2", RemoteETag: `"a"`, RemoteSize: 6}}
remote := &recordingRemote{objects: map[string][]byte{"/b/a.jpg": []byte("data-a"), "/b/b.jpg": []byte("old-b"), "/b/c.jpg": []byte("data-a")}, statETag: `"a"`}
event := &filer_pb.SubscribeMetadataResponse{Directory: "/buckets/b", EventNotification: &filer_pb.EventNotification{
OldEntry: pendingB(), NewParentPath: "/buckets/b", NewEntry: cached,
}}
filerClient := &pathFilerClient{entries: map[string]*filer_pb.Entry{"/buckets/b/c.jpg": cached}}
if err := processUpdateEvent(filerClient, filerClient, "", remote, mountedDir, mountLoc, event); err != nil {
t.Fatal(err)
}
if got := deletedPaths(remote); !slices.Contains(got, "/b/a.jpg") {
t.Errorf("deletes = %v, want the pending source released", got)
}
})
t.Run("a cached entry still reading the source keeps it until its upload lands", func(t *testing.T) {
// c.jpg was cached through its marker; until the upload is stamped
// and the marker cleared, a.jpg is what remote.uncache would fall
// back to
marked := func() *filer_pb.Entry {
return &filer_pb.Entry{Name: "c.jpg", Content: []byte("data-a"), Attributes: &filer_pb.FuseAttributes{FileSize: 6},
RemoteEntry: &filer_pb.RemoteEntry{StorageName: "b2", RemoteETag: `"a"`, RemoteSize: 6},
Extended: map[string][]byte{filer.ExtRemoteRenameSourceKey: []byte("/buckets/b/a.jpg")}}
}
for name, writeErr := range map[string]error{"upload fails": errors.New("upload failed"), "stamp does not land": nil} {
t.Run(name, func(t *testing.T) {
remote := &recordingRemote{objects: map[string][]byte{"/b/a.jpg": []byte("data-a"), "/b/b.jpg": []byte("old-b")}, statETag: `"a"`, writeErr: writeErr}
event := &filer_pb.SubscribeMetadataResponse{Directory: "/buckets/b", EventNotification: &filer_pb.EventNotification{
OldEntry: pendingB(), NewParentPath: "/buckets/b", NewEntry: marked(),
}}
// the filer keeps the marker: the stub does not store updates
filerClient := &pathFilerClient{entries: map[string]*filer_pb.Entry{"/buckets/b/c.jpg": marked()}}
_ = processUpdateEvent(filerClient, filerClient, "", remote, mountedDir, mountLoc, event)
if got := deletedPaths(remote); slices.Contains(got, "/b/a.jpg") {
t.Errorf("deletes = %v, want the source kept while the entry reads it", got)
}
})
}
})
t.Run("a later rename the filer copied releases the source", func(t *testing.T) {
// b.jpg was renamed to c.jpg and the filer copied a.jpg there at once
copied := &filer_pb.Entry{Name: "c.jpg", Attributes: &filer_pb.FuseAttributes{FileSize: 6}, RemoteEntry: &filer_pb.RemoteEntry{StorageName: "b2", RemoteETag: `"a"`, RemoteSize: 6}}
remote := &recordingRemote{objects: map[string][]byte{"/b/a.jpg": []byte("data-a"), "/b/b.jpg": []byte("old-b"), "/b/c.jpg": []byte("data-a")}, statETag: `"a"`}
event := &filer_pb.SubscribeMetadataResponse{Directory: "/buckets/b", EventNotification: &filer_pb.EventNotification{
OldEntry: pendingB(), NewParentPath: "/buckets/b", NewEntry: copied,
}}
filerClient := &pathFilerClient{entries: map[string]*filer_pb.Entry{"/buckets/b/c.jpg": copied}}
if err := processUpdateEvent(filerClient, filerClient, "", remote, mountedDir, mountLoc, event); err != nil {
t.Fatal(err)
}
if got := deletedPaths(remote); !slices.Contains(got, "/b/a.jpg") {
t.Errorf("deletes = %v, want the pending source released", got)
}
})
}
// pathFilerClient serves entries by path; a path it lacks is not found.
type pathFilerClient struct {
stubFilerClient
entries map[string]*filer_pb.Entry
}
func (c *pathFilerClient) LookupDirectoryEntry(_ context.Context, req *filer_pb.LookupDirectoryEntryRequest, _ ...grpc.CallOption) (*filer_pb.LookupDirectoryEntryResponse, error) {
return &filer_pb.LookupDirectoryEntryResponse{Entry: c.entries[req.Directory+"/"+req.Name]}, nil
}
func (c *pathFilerClient) WithFilerClient(_ bool, fn func(filer_pb.SeaweedFilerClient) error) error {
return fn(c)
}
func TestShouldSendToRemote(t *testing.T) {
const mtime = 1786096669
@@ -1383,3 +1717,37 @@ func TestMetadataOnlyUpdateRestampsSyncTs(t *testing.T) {
t.Error("the restamped entry should read synced")
}
}
func TestCompleteRemoteOnlyRenameKeepsAReusedOldKey(t *testing.T) {
// A was renamed to B, then D to A before the sync got to the first
// rename: the object at A is now D's, and the only copy of it
oldDest := &remote_pb.RemoteStorageLocation{Name: "b2", Bucket: "bucket", Path: "/b/a.jpg"}
dest := &remote_pb.RemoteStorageLocation{Name: "b2", Bucket: "bucket", Path: "/b/b.jpg"}
current := &filer_pb.Entry{Name: "b.jpg", RemoteEntry: &filer_pb.RemoteEntry{RemoteETag: `"a"`, RemoteSize: 6}}
remote := &recordingRemote{objects: map[string][]byte{"/b/a.jpg": []byte("data-d"), "/b/b.jpg": []byte("data-a")}, statETag: `"a"`}
remote.statETags = map[string]string{"/b/a.jpg": `"d"`}
filerClient := &stubFilerClient{entry: current}
if err := completeRemoteOnlyRename(filerClient, remote, "/buckets/b", current, renameSource{path: "/buckets/b/a.jpg", loc: oldDest}, dest, ""); err != nil {
t.Fatal(err)
}
if len(remote.deletes) != 0 {
t.Errorf("deletes = %+v, want the reused key kept", remote.deletes)
}
}
func TestCompleteRemoteOnlyRenameRetryUsesTheSourceStamp(t *testing.T) {
// an earlier run copied a multipart source and stamped the entry with
// the copy's ETag, then failed to delete the source
oldDest := &remote_pb.RemoteStorageLocation{Name: "b2", Bucket: "bucket", Path: "/b/a.jpg"}
dest := &remote_pb.RemoteStorageLocation{Name: "b2", Bucket: "bucket", Path: "/b/b.jpg"}
current := &filer_pb.Entry{Name: "b.jpg", RemoteEntry: &filer_pb.RemoteEntry{RemoteETag: `"copy"`, RemoteSize: 6}}
remote := &recordingRemote{objects: map[string][]byte{"/b/a.jpg": []byte("data-a"), "/b/b.jpg": []byte("data-a")},
statETags: map[string]string{"/b/a.jpg": `"a-2"`, "/b/b.jpg": `"copy"`}}
source := renameSource{path: "/buckets/b/a.jpg", loc: oldDest, stamp: &filer_pb.RemoteEntry{RemoteETag: `"a-2"`, RemoteSize: 6}}
if err := completeRemoteOnlyRename(&stubFilerClient{entry: current}, remote, "/buckets/b", current, source, dest, ""); err != nil {
t.Fatal(err)
}
if len(remote.deletes) != 1 || remote.deletes[0].Path != "/b/a.jpg" {
t.Errorf("deletes = %+v, want the unchanged source deleted on the retry", remote.deletes)
}
}
+158 -55
View File
@@ -12,6 +12,7 @@ import (
"net/http"
"os"
"path/filepath"
"sort"
"strings"
"sync"
"time"
@@ -64,8 +65,9 @@ var (
miniS3Options S3Options
miniWebDavOptions WebDavOption
miniAdminOptions AdminOptions
// Track which port flags were explicitly passed on CLI before config file is applied
explicitPortFlags map[string]bool
// Flags explicitly provided on the command line or via WEED_* env vars,
// captured before the saved options file is applied
explicitFlags map[string]bool
miniEnableWebDAV *bool
miniEnableS3 *bool
miniEnableAdminUI *bool
@@ -376,6 +378,19 @@ The S3 endpoint is available at http://localhost:8333. For client
configuration and IAM setup, see the project documentation or use the
Admin UI (http://localhost:23646) to manage users and policies.
Notes:
- Values mini computes itself (currently master.volumeSizeLimitMB) are
pinned in <dir>/mini.options and applied again on the next start to
keep them stable; applied values are printed at startup. Command-line
flags are never saved and always win over saved values. Edit or
delete mini.options to reset a computed value.
- The official Docker image ships /etc/seaweedfs/filer.toml, which keeps
the filer metadata in /data/filerldb2 regardless of -dir; mini warns
at startup when a configured store dir is outside -dir. If you set
-dir to another path, keep /data on a persistent volume as well.
Otherwise the metadata is lost when the container is recreated, and
buckets list no objects although the volume files are still in -dir.
`,
}
@@ -392,7 +407,7 @@ var (
miniMetricsHttpIp = cmdMini.Flag.String("metricsIp", "", "metrics listen ip. If empty, default to same as -ip.bind option.")
miniS3Config = cmdMini.Flag.String("s3.config", "", "path to the S3 config file")
miniIamConfig = cmdMini.Flag.String("s3.iam.config", "", "path to the advanced IAM config file for S3")
miniS3AllowDeleteBucketNotEmpty = cmdMini.Flag.Bool("s3.allowDeleteBucketNotEmpty", true, "allow recursive deleting all entries along with bucket")
miniS3AllowDeleteBucketNotEmpty = cmdMini.Flag.Bool("s3.allowDeleteBucketNotEmpty", false, "allow recursive deleting all entries along with bucket")
miniS3AutoCreateBucket = cmdMini.Flag.Bool("s3.autoCreateBucket", true, "create the bucket on upload if it does not exist, for admin identities only")
miniBucket = cmdMini.Flag.String("bucket", "", "comma-separated S3 bucket names to create on startup if they do not already exist; leave empty to skip. Falls back to S3_BUCKET env var.")
miniTableBucket = cmdMini.Flag.String("tableBucket", "", "comma-separated S3 Tables buckets to create on startup if they do not already exist, each name[:FORMAT] with FORMAT one of ICEBERG (default) or LANCE, e.g. warehouse,vectors:LANCE; leave empty to skip. Falls back to S3_TABLE_BUCKET env var.")
@@ -823,8 +838,8 @@ func findAvailablePortOnIP(ip string, startPort int, maxAttempts int, reservedPo
// If the port is not available, it finds the next available port and updates the pointer
// The reservedPorts map contains ports that should not be allocated (for gRPC collision avoidance)
func ensurePortAvailableOnIP(portPtr *int, serviceName string, ip string, reservedPorts map[int]bool, flagName string) error {
// Check if this port was explicitly specified by the user (from CLI, before config file was applied)
isExplicitPort := explicitPortFlags[flagName]
// Check if this port was explicitly specified by the user (from CLI or env, before the config file was applied)
isExplicitPort := explicitFlags[flagName]
if *portPtr == 0 {
return nil
@@ -1072,10 +1087,16 @@ func initializeGrpcPortsOnIP(bindIp string) {
}
}
// miniOptionsFilePath returns the path of the mini.options file under the
// first -dir folder.
func miniOptionsFilePath(dataFolder string) string {
return filepath.Join(util.ResolvePath(util.StringSplit(dataFolder, ",")[0]), "mini.options")
}
// loadMiniConfigurationFile reads the mini.options file and returns parsed options
// File format: one option per line, without leading dash (e.g., "ip=127.0.0.1")
func loadMiniConfigurationFile(dataFolder string) (map[string]string, error) {
configFile := filepath.Join(util.ResolvePath(util.StringSplit(dataFolder, ",")[0]), "mini.options")
configFile := miniOptionsFilePath(dataFolder)
options := make(map[string]string)
@@ -1123,31 +1144,112 @@ func loadMiniConfigurationFile(dataFolder string) (map[string]string, error) {
return options, nil
}
// applyConfigFileOptions sets command-line flags from loaded configuration file
func applyConfigFileOptions(options map[string]string) {
for key, value := range options {
// Skip port flags that were explicitly passed on CLI
if explicitPortFlags[key] {
glog.V(2).Infof("Skipping config file option %s=%s (explicitly specified on command line)", key, value)
// applyConfigFileOptions sets command-line flags from loaded configuration
// file. Options for flags the user provided explicitly (command line or
// WEED_* env var) are skipped, so passing a flag always wins over the saved
// value — including passing it at its default value. Returns a map of the
// applied keys to their display values (secrets redacted).
func applyConfigFileOptions(options map[string]string) map[string]string {
keys := make([]string, 0, len(options))
for key := range options {
keys = append(keys, key)
}
sort.Strings(keys)
applied := make(map[string]string)
for _, key := range keys {
value := options[key]
// Redact secrets (admin.password, admin.readOnlyPassword, ...) in
// logs and the startup message; only flag.Value.Set sees the value.
display := value
if isSensitiveOption(key) {
display = "*****"
}
if explicitFlags[key] {
glog.V(2).Infof("Skipping config file option %s=%s (explicitly specified)", key, display)
continue
}
// Set the flag value if it hasn't been explicitly set on command line
flag := cmdMini.Flag.Lookup(key)
if flag != nil {
// Only set if not already set (by command line)
if flag.Value.String() == flag.DefValue {
if err := flag.Value.Set(value); err != nil {
glog.Warningf("Failed to apply config file option: %s=%s: %v", key, value, err)
} else {
glog.V(2).Infof("Applied config file option: %s=%s", key, value)
if flag == nil {
continue
}
if err := flag.Value.Set(value); err != nil {
glog.Warningf("Failed to apply config file option: %s=%s: %v", key, display, err)
continue
}
glog.V(2).Infof("Applied config file option: %s=%s", key, display)
applied[key] = display
}
return applied
}
// isSensitiveOption reports whether a saved option holds a secret that must
// not appear in logs or the startup message.
func isSensitiveOption(key string) bool {
k := strings.ToLower(key)
return strings.Contains(k, "password") || strings.Contains(k, "secret") || strings.Contains(k, "token")
}
// warnIfFilerStoreOutsideDir warns when a filer.toml configures a local
// filer store outside the -dir folders and the master meta folder. When a
// filer.toml is found the filer skips defaultLevelDbDirectory, so metadata
// can silently land on a path the volume data doesn't use — in a container
// that path is lost on recreate unless it is persisted separately.
func warnIfFilerStoreOutsideDir(dataFolders string, metaFolder string) {
if !util.LoadConfiguration("filer", false) {
return
}
v := util.GetViper()
folders := append(util.StringSplit(dataFolders, ","), metaFolder)
inside := func(storeDir string) bool {
// The filer stores open the configured dir verbatim, so absolutize it
// the same way (relative to the working directory) rather than
// expanding "~", which the store never expands.
resolved, err := filepath.Abs(storeDir)
if err != nil {
return false
}
for _, folder := range folders {
base, err := filepath.Abs(util.ResolvePath(folder))
if err != nil {
continue
}
// Resolve symlinks only when both sides exist, so a not-yet-
// created store dir under a symlinked -dir still compares
// lexically against the same spelling.
r, b := resolved, base
if er, err := filepath.EvalSymlinks(resolved); err == nil {
if eb, err := filepath.EvalSymlinks(base); err == nil {
r, b = er, eb
}
}
if rel, err := filepath.Rel(b, r); err == nil && rel != ".." && !strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
return true
}
}
return false
}
// Local filer stores keep metadata under a "dir" key; check both the
// default store (leveldb2.enabled) and path-specific stores
// (leveldb2.<id>.enabled, keyed under <section>.location).
localStores := map[string]bool{"leveldb": true, "leveldb2": true, "leveldb3": true}
for _, key := range v.AllKeys() {
if !strings.HasSuffix(key, ".enabled") || !v.GetBool(key) {
continue
}
section := strings.TrimSuffix(key, ".enabled")
if !localStores[strings.Split(section, ".")[0]] {
continue
}
if storeDir := v.GetString(section + ".dir"); storeDir != "" && !inside(storeDir) {
glog.Warningf("filer store %s uses dir %s from filer.toml, outside -dir %s; persist that path or filer metadata will be lost when the container is recreated", section, storeDir, dataFolders)
}
}
}
// saveMiniConfiguration saves the current mini configuration to a file
// The file format uses option=value format without leading dashes
// saveMiniConfiguration writes values mini computed itself to mini.options.
// Command-line flags are never saved, so a dropped flag cannot resurrect on
// the next start; the file exists to pin environment-derived values.
func saveMiniConfiguration(dataFolder string) error {
configDir := util.ResolvePath(util.StringSplit(dataFolder, ",")[0])
if err := os.MkdirAll(configDir, 0755); err != nil {
@@ -1159,29 +1261,15 @@ func saveMiniConfiguration(dataFolder string) error {
var sb strings.Builder
sb.WriteString("#!/bin/bash\n")
sb.WriteString("# Mini server configuration\n")
sb.WriteString("# Mini server auto-computed options\n")
sb.WriteString("# Format: option=value (no leading dash)\n")
sb.WriteString("# This file is loaded on startup if it exists\n\n")
sb.WriteString("# This file is loaded on startup if it exists\n")
sb.WriteString("# Command-line flags are not saved; only values mini computed itself.\n\n")
// Collect all flags that were explicitly passed (except "dir")
cmdMini.Flag.Visit(func(f *flag.Flag) {
// Skip the "dir" option - it's environment-specific
if f.Name == "dir" {
return
}
value := f.Value.String()
// Quote the value if it contains spaces
if strings.Contains(value, " ") {
sb.WriteString(fmt.Sprintf("%s=\"%s\"\n", f.Name, value))
} else {
sb.WriteString(fmt.Sprintf("%s=%s\n", f.Name, value))
}
})
// Add auto-calculated volume size if it was computed
// Pin the auto-calculated volume size so it stays stable across restarts
// even as free disk space changes. Delete this line to force
// recalculation on the next startup.
if !isFlagPassed("master.volumeSizeLimitMB") && miniMasterOptions.volumeSizeLimitMB != nil {
sb.WriteString(fmt.Sprintf("\n# Auto-calculated volume size based on total disk capacity\n"))
sb.WriteString(fmt.Sprintf("# Delete this line to force recalculation on next startup\n"))
sb.WriteString(fmt.Sprintf("master.volumeSizeLimitMB=%d\n", *miniMasterOptions.volumeSizeLimitMB))
}
@@ -1203,21 +1291,32 @@ func runMini(cmd *Command, args []string) bool {
// can opt back into verbose output.
quietMiniLogs()
// Capture which port flags were explicitly passed on CLI BEFORE config file is applied
// This is necessary to distinguish user-specified ports from defaults or config file options
explicitPortFlags = make(map[string]bool)
portFlagNames := []string{"master.port", "filer.port", "volume.port", "s3.port", "s3.port.iceberg", "webdav.port", "admin.port", "s3.iam.readOnly"}
for _, flagName := range portFlagNames {
explicitPortFlags[flagName] = isFlagPassed(flagName)
}
// Capture which flags were explicitly provided on the command line or via
// WEED_* env vars BEFORE the saved options file is applied. Saved options
// only fill in flags the user did not provide this run, so passing a flag
// at its default value also resets a saved value.
explicitFlags = make(map[string]bool)
cmdMini.Flag.Visit(func(f *flag.Flag) {
explicitFlags[f.Name] = true
})
// Load configuration from file if it exists
// Load configuration from file if it exists. Capture the path before
// applying: a saved "dir" option can change miniDataFolders.
optionsFile := miniOptionsFilePath(*miniDataFolders)
configOptions, err := loadMiniConfigurationFile(*miniDataFolders)
if err != nil {
glog.Warningf("Error loading configuration file: %v", err)
}
// Apply loaded options to flags (CLI flags will override these)
applyConfigFileOptions(configOptions)
// Apply loaded options to flags (explicitly provided flags win)
appliedOptions := applyConfigFileOptions(configOptions)
if len(appliedOptions) > 0 {
pairs := make([]string, 0, len(appliedOptions))
for k, v := range appliedOptions {
pairs = append(pairs, k+"="+v)
}
sort.Strings(pairs)
fmt.Printf(" Applying %d saved option(s) from %s: %s\n", len(appliedOptions), optionsFile, strings.Join(pairs, " "))
}
if *miniOptions.debug {
grace.StartDebugServer(*miniOptions.debugPort)
@@ -1346,10 +1445,14 @@ func runMini(cmd *Command, args []string) bool {
glog.Fatalf("Check Meta Folder (-dir=\"%s\") Writable: %s", *miniMasterOptions.metaFolder, err)
}
miniFilerOptions.defaultLevelDbDirectory = miniMasterOptions.metaFolder
warnIfFilerStoreOutsideDir(*miniDataFolders, *miniMasterOptions.metaFolder)
// Calculate and set optimal volume size limit based on available disk space
// Only auto-calculate if user didn't explicitly specify a value via -master.volumeSizeLimitMB
if !isFlagPassed("master.volumeSizeLimitMB") {
// Calculate and set optimal volume size limit based on available disk space.
// Skip when the user passed -master.volumeSizeLimitMB explicitly, and when
// the value was loaded from mini.options — file-applied flags are not in
// Visit, and the saved pin must survive restarts.
_, sizeLimitFromFile := appliedOptions["master.volumeSizeLimitMB"]
if !isFlagPassed("master.volumeSizeLimitMB") && !sizeLimitFromFile {
// User didn't override, use auto-calculated value
// The -dir flag can accept comma-separated directories; use the first one for disk space calculation.
// miniDataFolders was already tilde-resolved at the top of runMini.
+1 -1
View File
@@ -114,7 +114,7 @@ func init() {
s3StandaloneOptions.metricsHttpPort = cmdS3.Flag.Int("metricsPort", 0, "Prometheus metrics listen port")
s3StandaloneOptions.metricsHttpIp = cmdS3.Flag.String("metricsIp", "", "metrics listen ip. If empty, default to same as -ip.bind option.")
cmdS3.Flag.Bool("allowEmptyFolder", true, "deprecated, ignored. Empty folder cleanup is now automatic.")
s3StandaloneOptions.allowDeleteBucketNotEmpty = cmdS3.Flag.Bool("allowDeleteBucketNotEmpty", true, "allow recursive deleting all entries along with bucket")
s3StandaloneOptions.allowDeleteBucketNotEmpty = cmdS3.Flag.Bool("allowDeleteBucketNotEmpty", false, "allow recursive deleting all entries along with bucket")
s3StandaloneOptions.autoCreateBucket = cmdS3.Flag.Bool("autoCreateBucket", true, "create the bucket on upload if it does not exist, for admin identities only")
s3StandaloneOptions.localFilerSocket = cmdS3.Flag.String("localFilerSocket", "", "local filer socket path")
s3StandaloneOptions.localSocket = cmdS3.Flag.String("localSocket", "", "default to /tmp/seaweedfs-s3-<port>.sock")
+1 -1
View File
@@ -182,7 +182,7 @@ func init() {
s3Options.iamConfig = cmdServer.Flag.String("s3.iam.config", "", "path to the advanced IAM config file for S3. Overrides -iam.config if both are provided.")
s3Options.auditLogConfig = cmdServer.Flag.String("s3.auditLogConfig", "", "path to the audit log config file")
cmdServer.Flag.Bool("s3.allowEmptyFolder", true, "deprecated, ignored. Empty folder cleanup is now automatic.")
s3Options.allowDeleteBucketNotEmpty = cmdServer.Flag.Bool("s3.allowDeleteBucketNotEmpty", true, "allow recursive deleting all entries along with bucket")
s3Options.allowDeleteBucketNotEmpty = cmdServer.Flag.Bool("s3.allowDeleteBucketNotEmpty", false, "allow recursive deleting all entries along with bucket")
s3Options.autoCreateBucket = cmdServer.Flag.Bool("s3.autoCreateBucket", true, "create the bucket on upload if it does not exist, for admin identities only")
s3Options.localSocket = cmdServer.Flag.String("s3.localSocket", "", "default to /tmp/seaweedfs-s3-<port>.sock")
s3Options.bindIp = cmdServer.Flag.String("s3.ip.bind", "", "ip address to bind to. If empty, default to same as -ip.bind option.")
+8 -2
View File
@@ -82,7 +82,7 @@ func runPluginWorkerWithOptions(options pluginWorkerRunOptions) bool {
return false
}
handlers, err := buildPluginWorkerHandlers(options.JobTypes, dialOption, options.MaxExecute, options.WorkingDir)
handlers, err := buildPluginWorkerHandlers(options.JobTypes, dialOption, options.MaxExecute, options.WorkingDir, options.Address)
if err != nil {
glog.Errorf("Failed to build plugin worker handlers: %v", err)
return false
@@ -141,7 +141,7 @@ func resolvePluginWorkerID(explicitID string, workingDir string) (string, error)
// buildPluginWorkerHandlers resolves the comma-separated jobTypes string
// (which may contain category names like "all", "default", "heavy" and/or
// explicit job type names/aliases) into a deduplicated slice of JobHandlers.
func buildPluginWorkerHandlers(jobTypes string, dialOption grpc.DialOption, maxExecute int, workingDir string) ([]pluginworker.JobHandler, error) {
func buildPluginWorkerHandlers(jobTypes string, dialOption grpc.DialOption, maxExecute int, workingDir string, workerAddress ...string) ([]pluginworker.JobHandler, error) {
jobTypes = strings.TrimSpace(jobTypes)
if jobTypes == "" {
jobTypes = defaultPluginWorkerJobTypes
@@ -152,10 +152,16 @@ func buildPluginWorkerHandlers(jobTypes string, dialOption grpc.DialOption, maxE
return nil, err
}
addr := ""
if len(workerAddress) > 0 {
addr = strings.TrimSpace(workerAddress[0])
}
opts := pluginworker.HandlerBuildOptions{
GrpcDialOption: dialOption,
MaxExecute: maxExecute,
WorkingDir: workingDir,
WorkerAddress: addr,
}
handlers := make([]pluginworker.JobHandler, 0, len(factories))
+359
View File
@@ -0,0 +1,359 @@
package ec
import (
"context"
"fmt"
"net"
"strings"
"sync"
"testing"
"time"
"github.com/seaweedfs/seaweedfs/weed/pb/master_pb"
"github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
"github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding"
"github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding/ecbalancer"
"github.com/seaweedfs/seaweedfs/weed/storage/needle"
"github.com/seaweedfs/seaweedfs/weed/storage/types"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
)
// ecBalanceFakeServer answers the EC-shard RPC subset a balance run drives:
// copy (rejected when full), mount, inventory, unmount, delete. The disk state
// a real server tracks lives in shards; what the planner saw is rebuilt from it
// through FetchTopology, so a landed move is visible to the next plan.
type ecBalanceFakeServer struct {
volume_server_pb.UnimplementedVolumeServerServer
mu sync.Mutex
shards map[uint32]map[uint32]bool // vid -> shardId -> present
full bool // VolumeEcShardsCopy answers "no space left"
copyCalls int
grpcAddr string
stop func()
}
func (f *ecBalanceFakeServer) VolumeEcShardsCopy(ctx context.Context, req *volume_server_pb.VolumeEcShardsCopyRequest) (*volume_server_pb.VolumeEcShardsCopyResponse, error) {
f.mu.Lock()
defer f.mu.Unlock()
f.copyCalls++
if f.full {
return nil, fmt.Errorf("no space left")
}
return &volume_server_pb.VolumeEcShardsCopyResponse{}, nil
}
func (f *ecBalanceFakeServer) VolumeEcShardsMount(ctx context.Context, req *volume_server_pb.VolumeEcShardsMountRequest) (*volume_server_pb.VolumeEcShardsMountResponse, error) {
f.mu.Lock()
defer f.mu.Unlock()
for _, sid := range req.ShardIds {
f.shards[req.VolumeId][sid] = true
}
return &volume_server_pb.VolumeEcShardsMountResponse{}, nil
}
func (f *ecBalanceFakeServer) VolumeEcShardsInfo(ctx context.Context, req *volume_server_pb.VolumeEcShardsInfoRequest) (*volume_server_pb.VolumeEcShardsInfoResponse, error) {
f.mu.Lock()
defer f.mu.Unlock()
resp := &volume_server_pb.VolumeEcShardsInfoResponse{}
for sid := range f.shards[req.VolumeId] {
resp.EcShardInfos = append(resp.EcShardInfos, &volume_server_pb.EcShardInfo{
VolumeId: req.VolumeId,
ShardId: sid,
Size: 1,
})
}
return resp, nil
}
func (f *ecBalanceFakeServer) VolumeEcShardsUnmount(ctx context.Context, req *volume_server_pb.VolumeEcShardsUnmountRequest) (*volume_server_pb.VolumeEcShardsUnmountResponse, error) {
return &volume_server_pb.VolumeEcShardsUnmountResponse{}, nil
}
func (f *ecBalanceFakeServer) VolumeEcShardsDelete(ctx context.Context, req *volume_server_pb.VolumeEcShardsDeleteRequest) (*volume_server_pb.VolumeEcShardsDeleteResponse, error) {
f.mu.Lock()
defer f.mu.Unlock()
for _, sid := range req.ShardIds {
delete(f.shards[req.VolumeId], sid)
}
return &volume_server_pb.VolumeEcShardsDeleteResponse{}, nil
}
// nodeId returns the "host:httpPort.grpcPort" node id form the cluster uses, so
// ToGrpcAddress resolves to this server's listener.
func (f *ecBalanceFakeServer) nodeId() string {
host, port, err := net.SplitHostPort(f.grpcAddr)
if err != nil {
return f.grpcAddr
}
return fmt.Sprintf("%s:1.%s", host, port)
}
func (f *ecBalanceFakeServer) shardCount(vid uint32) int {
f.mu.Lock()
defer f.mu.Unlock()
return len(f.shards[vid])
}
func startEcBalanceFakeServer(t *testing.T, vid uint32, present ...uint32) *ecBalanceFakeServer {
t.Helper()
listener, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatalf("listen: %v", err)
}
f := &ecBalanceFakeServer{
shards: map[uint32]map[uint32]bool{vid: {}},
grpcAddr: listener.Addr().String(),
}
for _, sid := range present {
f.shards[vid][sid] = true
}
server := grpc.NewServer()
volume_server_pb.RegisterVolumeServerServer(server, f)
go func() { _ = server.Serve(listener) }()
f.stop = server.Stop
t.Cleanup(server.Stop)
return f
}
// fetchEcBalanceTopology builds a FetchTopology that reports the fake servers'
// live shard inventory.
func fetchEcBalanceTopology(servers []*ecBalanceFakeServer) func(delay time.Duration) (*master_pb.TopologyInfo, uint64, error) {
return func(delay time.Duration) (*master_pb.TopologyInfo, uint64, error) {
var nodes []*master_pb.DataNodeInfo
for _, f := range servers {
di := &master_pb.DiskInfo{Type: string(types.HardDriveType), MaxVolumeCount: 100}
f.mu.Lock()
for v, set := range f.shards {
if len(set) == 0 {
continue
}
si := erasure_coding.NewShardsInfo()
for sid := range set {
si.Set(erasure_coding.NewShardInfo(erasure_coding.ShardId(sid), 1))
}
di.EcShardInfos = append(di.EcShardInfos, &master_pb.VolumeEcShardInformationMessage{
Id: v,
Collection: "c1",
EcIndexBits: si.Bitmap(),
ShardSizes: si.SizesInt64(),
DiskType: string(types.HardDriveType),
})
}
f.mu.Unlock()
nodes = append(nodes, &master_pb.DataNodeInfo{
Id: f.nodeId(),
DiskInfos: map[string]*master_pb.DiskInfo{string(types.HardDriveType): di},
})
}
return &master_pb.TopologyInfo{
DataCenterInfos: []*master_pb.DataCenterInfo{{
Id: "dc1",
RackInfos: []*master_pb.RackInfo{{
Id: "r1",
DataNodeInfos: nodes,
}},
}},
}, 0, nil
}
}
// TestEcBalanceReplanAfterNoSpaceLeft reproduces seaweedfs/seaweedfs#11753:
// the planner works from heartbeat free-slot counts that can disagree with the
// volume server's own check, so a planned target can reject the copy with
// "no space left" mid-run. The balance must re-plan without that node as a
// target rather than fail the whole run.
func TestEcBalanceReplanAfterNoSpaceLeft(t *testing.T) {
const vid = uint32(1)
var allShards []uint32
for sid := 0; sid < erasure_coding.DataShardsCount+erasure_coding.ParityShardsCount; sid++ {
allShards = append(allShards, uint32(sid))
}
full := startEcBalanceFakeServer(t, vid)
accept := startEcBalanceFakeServer(t, vid)
source := startEcBalanceFakeServer(t, vid, allShards...)
full.full = true
env := &Env{
GrpcDialOption: grpc.WithTransportCredentials(insecure.NewCredentials()),
FetchTopology: fetchEcBalanceTopology([]*ecBalanceFakeServer{source, full, accept}),
}
err := EcBalance(env, []string{"c1"}, "", nil, types.HardDriveType, 3, 0, true, nil, nil, nil)
if err != nil {
t.Fatalf("balance should re-plan around the full target, got: %v", err)
}
if full.copyCalls == 0 {
t.Fatalf("expected the full server to have rejected at least one copy")
}
if got := full.shardCount(vid); got != 0 {
t.Fatalf("full server should receive no shards, holds %d", got)
}
total := source.shardCount(vid) + full.shardCount(vid) + accept.shardCount(vid)
if total != len(allShards) {
t.Fatalf("shards lost or duplicated: total %d, want %d", total, len(allShards))
}
if source.shardCount(vid) == len(allShards) {
t.Fatalf("no shards were moved off the source")
}
}
// TestEcBalanceReplanAfterNoSpaceLeftStaleSnapshot serves the pre-move
// topology on every collection after the first, simulating a master heartbeat
// that lags the landed moves. The re-plan must replay its completed moves onto
// the stale snapshot instead of re-moving shards off the deleted source.
func TestEcBalanceReplanAfterNoSpaceLeftStaleSnapshot(t *testing.T) {
const vid = uint32(1)
var allShards []uint32
for sid := 0; sid < erasure_coding.DataShardsCount+erasure_coding.ParityShardsCount; sid++ {
allShards = append(allShards, uint32(sid))
}
full := startEcBalanceFakeServer(t, vid)
accept := startEcBalanceFakeServer(t, vid)
source := startEcBalanceFakeServer(t, vid, allShards...)
full.full = true
live := fetchEcBalanceTopology([]*ecBalanceFakeServer{source, full, accept})
var stale *master_pb.TopologyInfo
calls := 0
fetchTopology := func(delay time.Duration) (*master_pb.TopologyInfo, uint64, error) {
topo, size, err := live(delay)
if err != nil {
return nil, 0, err
}
calls++
if stale == nil {
stale = topo
}
if calls > 1 {
return stale, size, nil
}
return topo, size, nil
}
env := &Env{
GrpcDialOption: grpc.WithTransportCredentials(insecure.NewCredentials()),
FetchTopology: fetchTopology,
}
err := EcBalance(env, []string{"c1"}, "", nil, types.HardDriveType, 3, 0, true, nil, nil, nil)
if err != nil {
t.Fatalf("balance should re-plan around the full target, got: %v", err)
}
if calls < 2 {
t.Fatalf("expected a re-plan, topology collected %d time(s)", calls)
}
// Each shard lands on the accepting server exactly once; a stale re-plan
// would copy already-moved shards again.
if got, want := accept.copyCalls, accept.shardCount(vid); got != want {
t.Fatalf("accepting server saw %d copies for %d held shards: completed moves were re-planned", got, want)
}
total := source.shardCount(vid) + full.shardCount(vid) + accept.shardCount(vid)
if total != len(allShards) {
t.Fatalf("shards lost or duplicated: total %d, want %d", total, len(allShards))
}
}
// TestEcBalanceNoReceivingTargetFails surfaces the refusal when every
// destination that could take a shard has rejected a copy, rather than
// reporting success for a plan that moved nothing.
func TestEcBalanceNoReceivingTargetFails(t *testing.T) {
const vid = uint32(1)
var allShards []uint32
for sid := 0; sid < erasure_coding.DataShardsCount+erasure_coding.ParityShardsCount; sid++ {
allShards = append(allShards, uint32(sid))
}
full := startEcBalanceFakeServer(t, vid)
source := startEcBalanceFakeServer(t, vid, allShards...)
full.full = true
env := &Env{
GrpcDialOption: grpc.WithTransportCredentials(insecure.NewCredentials()),
FetchTopology: fetchEcBalanceTopology([]*ecBalanceFakeServer{source, full}),
}
err := EcBalance(env, []string{"c1"}, "", nil, types.HardDriveType, 3, 0, true, nil, nil, nil)
if err == nil {
t.Fatalf("balance with no receiving target should fail")
}
if !strings.Contains(err.Error(), "no space left") {
t.Fatalf("expected the refusal to surface, got: %v", err)
}
}
// TestApplyCompletedMovesReplaysDiskState checks that replaying a landed move
// onto a stale snapshot carries the shard's disk attribution and the per-disk
// slot accounting, so a re-plan does not treat the just-filled disk as empty.
func TestApplyCompletedMovesReplaysDiskState(t *testing.T) {
const vid = uint32(1)
source := newEcNode("dc1", "r1", "dn1", 99)
source.Info.DiskInfos["hdd"] = &master_pb.DiskInfo{
Type: "hdd",
EcShardInfos: []*master_pb.VolumeEcShardInformationMessage{{
Id: vid,
Collection: "c1",
DiskId: 1,
DiskType: "hdd",
EcIndexBits: uint32(erasure_coding.ShardBits(0).Set(5)),
}},
}
si := erasure_coding.NewShardsInfo()
si.Set(erasure_coding.NewShardInfo(5, 1))
source.Disks = map[uint32]*EcDisk{
1: {DiskId: 1, DiskType: "hdd", FreeEcSlots: 99, EcShardCount: 1, EcShards: map[needle.VolumeId]*erasure_coding.ShardsInfo{1: si}},
}
target := newEcNode("dc1", "r1", "dn2", 100)
target.Disks = map[uint32]*EcDisk{
1: {DiskId: 1, DiskType: "hdd", FreeEcSlots: 10, EcShardCount: 0, EcShards: map[needle.VolumeId]*erasure_coding.ShardsInfo{}},
2: {DiskId: 2, DiskType: "hdd", FreeEcSlots: 10, EcShardCount: 0, EcShards: map[needle.VolumeId]*erasure_coding.ShardsInfo{}},
}
nodes := []*EcNode{source, target}
move := ecbalancer.Move{
VolumeID: vid,
ShardID: 5,
Collection: "c1",
SourceNode: "dn1",
TargetNode: "dn2",
TargetDisk: 2,
Phase: "within_rack",
}
applyCompletedMoves(nodes, []ecbalancer.Move{move}, types.HardDriveType)
if ecNodeHasShard(source, needle.VolumeId(vid), 5) {
t.Fatalf("source still holds shard 5")
}
if d := source.Disks[1]; d.EcShardCount != 0 || d.FreeEcSlots != 100 {
t.Fatalf("source disk counts not restored: %+v", d)
}
if !ecNodeHasShard(target, needle.VolumeId(vid), 5) {
t.Fatalf("target missing shard 5")
}
if d := target.Disks[2]; d.EcShardCount != 1 || d.FreeEcSlots != 9 {
t.Fatalf("target disk 2 not charged: %+v", d)
}
if d := target.Disks[1]; d.EcShardCount != 0 {
t.Fatalf("shard attributed to wrong disk: %+v", d)
}
if target.FreeEcSlot != 99 {
t.Fatalf("target free slots = %d, want 99", target.FreeEcSlot)
}
// Replaying again (or onto a fresher snapshot) must not double-count.
applyCompletedMoves(nodes, []ecbalancer.Move{move}, types.HardDriveType)
if d := target.Disks[2]; d.EcShardCount != 1 || d.FreeEcSlots != 9 {
t.Fatalf("double-applied move: %+v", d)
}
if target.FreeEcSlot != 99 {
t.Fatalf("target free slots = %d, want 99", target.FreeEcSlot)
}
}
+310 -49
View File
@@ -8,6 +8,7 @@ import (
"sort"
"strconv"
"strings"
"sync"
"time"
"github.com/seaweedfs/seaweedfs/weed/glog"
@@ -762,6 +763,26 @@ type ecBalancer struct {
// migratingVolumeIds are mid-encode volumes whose shards are ingested from
// every disk-type bucket; see EcBalance.
migratingVolumeIds map[uint32]bool
// noReceiveTargets are node ids the planner may not pick as destinations;
// noReceiveDisks narrows that to individual disks on a node: a server-side
// auto-pick refusal means no disk qualifies, but an explicit-disk refusal
// only convicts that disk. See EcBalance.
noReceiveTargets map[string]bool
noReceiveDisks map[string]map[uint32]bool
// mu guards the execution journal written by concurrent move tasks.
mu sync.Mutex
// rejectedTargets and rejectedDisks collect the destinations that refused a
// copy this attempt; completedMoves records applied moves so a retry can
// replay them onto a stale-heartbeat snapshot; sawOtherErr marks any
// failure that is not a "no space left" refusal and makes the run
// non-retryable.
rejectedTargets map[string]bool
rejectedDisks map[string]map[uint32]bool
completedMoves []ecbalancer.Move
sawOtherErr bool
// plannedMoves is the move count of the last plan, for the empty-replan
// check in EcBalance.
plannedMoves int
}
// excludeNodes is a set of server addresses kept out of the balance as copy/move
@@ -782,32 +803,6 @@ type ecBalancer struct {
// so a plain ec.balance -diskType=X never drags deliberately tiered shards of
// other types onto X disks.
func EcBalance(env *Env, collections []string, dc string, ecReplicaPlacement *super_block.ReplicaPlacement, diskType types.DiskType, maxParallelization int, ioBytePerSecond int64, applyBalancing bool, excludeNodes map[pb.ServerAddress]struct{}, volumeIds []needle.VolumeId, migratingVolumeIds []needle.VolumeId) (err error) {
// collect all ec nodes
allEcNodes, totalFreeEcSlots, err := CollectEcNodesForDC(env, dc, diskType)
if err != nil {
return err
}
// Drop excluded nodes (and the slots they contribute) before planning so they
// can be neither a target nor a source for any move this balance plans.
if len(excludeNodes) > 0 {
kept := allEcNodes[:0]
var excludedFreeSlots int
for _, en := range allEcNodes {
if _, skip := excludeNodes[pb.NewServerAddressFromDataNode(en.Info)]; skip {
excludedFreeSlots += en.FreeEcSlot
glog.V(0).Infof("EC balance excluding node %s: skipped as unreachable by the encode orphan sweep", en.Info.Id)
continue
}
kept = append(kept, en)
}
allEcNodes = kept
totalFreeEcSlots -= excludedFreeSlots
}
if totalFreeEcSlots < 1 {
return fmt.Errorf("no free ec shard slots. only %d left", totalFreeEcSlots)
}
var volumeIdFilter map[uint32]bool
if len(volumeIds) > 0 {
@@ -824,22 +819,277 @@ func EcBalance(env *Env, collections []string, dc string, ecReplicaPlacement *su
}
}
ecb := &ecBalancer{
env: env,
ecNodes: allEcNodes,
replicaPlacement: ecReplicaPlacement,
applyBalancing: applyBalancing,
maxParallelization: maxParallelization,
ioBytePerSecond: ioBytePerSecond,
diskType: diskType,
volumeIds: volumeIdFilter,
migratingVolumeIds: migrating,
}
// The planner's free-slot numbers come from heartbeat snapshots that can lag
// the volume server's own accounting, so a planned target can refuse a copy
// with "no space left" while shards move concurrently (issue #11753). The
// server's refusal is authoritative: re-collect the topology, mark the
// refusing targets no-receive, and plan again. They stay usable as sources.
noReceiveTargets := make(map[string]bool)
noReceiveDisks := make(map[string]map[uint32]bool)
var completed []ecbalancer.Move
var lastErr error
for {
// collect all ec nodes
allEcNodes, totalFreeEcSlots, err := CollectEcNodesForDC(env, dc, diskType)
if err != nil {
return err
}
if len(collections) == 0 {
glog.V(1).Infof("WARNING: No collections to balance EC volumes across.\n")
// Drop excluded nodes (and the slots they contribute) before planning so they
// can be neither a target nor a source for any move this balance plans.
if len(excludeNodes) > 0 {
kept := allEcNodes[:0]
var excludedFreeSlots int
for _, en := range allEcNodes {
if _, skip := excludeNodes[pb.NewServerAddressFromDataNode(en.Info)]; skip {
excludedFreeSlots += en.FreeEcSlot
glog.V(0).Infof("EC balance excluding node %s: skipped as unreachable by the encode orphan sweep", en.Info.Id)
continue
}
kept = append(kept, en)
}
allEcNodes = kept
totalFreeEcSlots -= excludedFreeSlots
}
// A refused destination's slots cannot take a copy, so they do not
// count toward the free-space precheck.
for _, en := range allEcNodes {
if noReceiveTargets[en.Info.Id] {
totalFreeEcSlots -= en.FreeEcSlot
continue
}
for diskId := range noReceiveDisks[en.Info.Id] {
if d := en.Disks[diskId]; d != nil {
totalFreeEcSlots -= d.FreeEcSlots
}
}
}
if totalFreeEcSlots < 1 {
if lastErr != nil {
return fmt.Errorf("no receiving targets remain: %w", lastErr)
}
return fmt.Errorf("no free ec shard slots. only %d left", totalFreeEcSlots)
}
// A fresh collection can still describe an applied move through a
// heartbeat that predates its source delete; replay the journal.
applyCompletedMoves(allEcNodes, completed, diskType)
ecb := &ecBalancer{
env: env,
ecNodes: allEcNodes,
replicaPlacement: ecReplicaPlacement,
applyBalancing: applyBalancing,
maxParallelization: maxParallelization,
ioBytePerSecond: ioBytePerSecond,
diskType: diskType,
volumeIds: volumeIdFilter,
migratingVolumeIds: migrating,
noReceiveTargets: noReceiveTargets,
noReceiveDisks: noReceiveDisks,
}
if len(collections) == 0 {
glog.V(1).Infof("WARNING: No collections to balance EC volumes across.\n")
}
err = ecb.balance(collections)
if !applyBalancing {
return err
}
if err == nil {
if lastErr != nil && ecb.plannedMoves == 0 {
return fmt.Errorf("no receiving targets remain: %w", lastErr)
}
return nil
}
// Replan while the failed attempt surfaced newly-refusing targets; a node
// once marked can no longer be picked, so the loop terminates. Any other
// failure aborts instead of being hidden by the retry.
newRejects := ecb.mergeRejectedTargets(noReceiveTargets, noReceiveDisks)
if len(newRejects) == 0 || ecb.hasOtherError() {
return err
}
lastErr = err
completed = append(completed, ecb.completedMoves...)
fmt.Printf("ec.balance: %v reported no space left; re-collecting topology and re-planning without them as targets\n", newRejects)
}
}
// applyCompletedMoves replays already-applied moves onto a freshly collected
// node list: heartbeats can lag, so the snapshot may still show a moved shard
// on its deleted source or lack it on its mounted target.
func applyCompletedMoves(ecNodes []*EcNode, moves []ecbalancer.Move, diskType types.DiskType) {
if len(moves) == 0 {
return
}
byID := make(map[string]*EcNode, len(ecNodes))
for _, en := range ecNodes {
byID[en.Info.Id] = en
}
for _, m := range moves {
vid := needle.VolumeId(m.VolumeID)
shardId := erasure_coding.ShardId(m.ShardID)
if src := byID[m.SourceNode]; src != nil {
removeEcShard(src, vid, shardId)
}
if m.Phase == "dedup" {
continue
}
if dst := byID[m.TargetNode]; dst != nil && !ecNodeHasShard(dst, vid, shardId) {
addEcShard(dst, m.Collection, vid, shardId, m.TargetDisk, diskType)
}
}
}
func ecNodeHasShard(ecNode *EcNode, vid needle.VolumeId, shardId erasure_coding.ShardId) bool {
for _, diskInfo := range ecNode.Info.DiskInfos {
for _, eci := range diskInfo.EcShardInfos {
if needle.VolumeId(eci.Id) == vid && erasure_coding.ShardBits(eci.EcIndexBits).Has(shardId) {
return true
}
}
}
return false
}
// removeEcShard drops the shard from the node model, including the per-disk
// bookkeeping of the disk that reported it.
func removeEcShard(en *EcNode, vid needle.VolumeId, shardId erasure_coding.ShardId) {
for _, diskInfo := range en.Info.DiskInfos {
for _, eci := range diskInfo.EcShardInfos {
if needle.VolumeId(eci.Id) != vid || !erasure_coding.ShardBits(eci.EcIndexBits).Has(shardId) {
continue
}
if d := en.Disks[eci.DiskId]; d != nil {
if si := d.EcShards[vid]; si != nil {
si.Delete(shardId)
}
d.EcShardCount--
d.FreeEcSlots++
}
}
}
en.DeleteEcVolumeShards(vid, []erasure_coding.ShardId{shardId})
}
// addEcShard places the shard on the disk the move used when that disk is
// known; otherwise (TargetDisk 0, server-picked) it falls back to the node's
// disk-type bucket.
func addEcShard(en *EcNode, collection string, vid needle.VolumeId, shardId erasure_coding.ShardId, diskId uint32, diskType types.DiskType) {
d := en.Disks[diskId]
if diskId == 0 || d == nil {
en.AddEcVolumeShards(vid, collection, []erasure_coding.ShardId{shardId}, diskType)
return
}
diskInfo := en.Info.DiskInfos[string(diskType)]
if diskInfo == nil {
diskInfo = &master_pb.DiskInfo{Type: string(diskType)}
en.Info.DiskInfos[string(diskType)] = diskInfo
}
var eci *master_pb.VolumeEcShardInformationMessage
for _, e := range diskInfo.EcShardInfos {
if needle.VolumeId(e.Id) == vid && e.DiskId == diskId {
eci = e
break
}
}
if eci == nil {
eci = &master_pb.VolumeEcShardInformationMessage{
Id: uint32(vid),
Collection: collection,
DiskId: diskId,
DiskType: string(diskType),
}
diskInfo.EcShardInfos = append(diskInfo.EcShardInfos, eci)
}
si := erasure_coding.ShardsInfoFromVolumeEcShardInformationMessage(eci)
before := si.Count()
si.Set(erasure_coding.NewShardInfo(shardId, 0))
eci.EcIndexBits = si.Bitmap()
eci.ShardSizes = si.SizesInt64()
added := si.Count() - before
en.FreeEcSlot -= added
d.EcShardCount += added
d.FreeEcSlots -= added
if d.EcShards == nil {
d.EcShards = make(map[needle.VolumeId]*erasure_coding.ShardsInfo)
}
if d.EcShards[vid] == nil {
d.EcShards[vid] = erasure_coding.NewShardsInfo()
}
d.EcShards[vid].Set(erasure_coding.NewShardInfo(shardId, 0))
}
// mergeRejectedTargets folds the destinations this run saw refuse into the
// no-receive sets and returns the newly added ids; an empty result means no
// progress is possible.
func (ecb *ecBalancer) mergeRejectedTargets(noReceiveNodes map[string]bool, noReceiveDisks map[string]map[uint32]bool) (added []string) {
ecb.mu.Lock()
defer ecb.mu.Unlock()
for id := range ecb.rejectedTargets {
if !noReceiveNodes[id] {
noReceiveNodes[id] = true
added = append(added, id)
}
}
for id, disks := range ecb.rejectedDisks {
if noReceiveNodes[id] {
continue
}
if noReceiveDisks[id] == nil {
noReceiveDisks[id] = make(map[uint32]bool)
}
for diskId := range disks {
if !noReceiveDisks[id][diskId] {
noReceiveDisks[id][diskId] = true
added = append(added, fmt.Sprintf("%s:disk%d", id, diskId))
}
}
}
sort.Strings(added)
return
}
func (ecb *ecBalancer) hasOtherError() bool {
ecb.mu.Lock()
defer ecb.mu.Unlock()
return ecb.sawOtherErr
}
// recordMoveResult journals an apply-mode move outcome: a success is replayed
// onto the next attempt's snapshot, a "no space left" marks the target
// no-receive, and any other failure makes the run non-retryable.
func (ecb *ecBalancer) recordMoveResult(m ecbalancer.Move, err error) {
ecb.mu.Lock()
defer ecb.mu.Unlock()
switch {
case err == nil:
ecb.completedMoves = append(ecb.completedMoves, m)
case m.TargetNode != "" && volume_move.IsNoSpaceLeftError(err):
if m.TargetDisk > 0 {
// The refusal convicts only the named disk; the node's other
// disks may still have room.
if ecb.rejectedDisks == nil {
ecb.rejectedDisks = make(map[string]map[uint32]bool)
}
if ecb.rejectedDisks[m.TargetNode] == nil {
ecb.rejectedDisks[m.TargetNode] = make(map[uint32]bool)
}
ecb.rejectedDisks[m.TargetNode][m.TargetDisk] = true
} else {
// The server auto-picked and found no usable disk: the whole
// node is out.
if ecb.rejectedTargets == nil {
ecb.rejectedTargets = make(map[string]bool)
}
ecb.rejectedTargets[m.TargetNode] = true
}
default:
ecb.sawOtherErr = true
}
return ecb.balance(collections)
}
// defaultECRatio resolves a collection's EC data/parity counts, defaulting to
@@ -852,7 +1102,7 @@ func defaultECRatio(_ string) (int, int) {
// balance plans EC shard moves with the shared planner and executes them. When
// collections is empty all collections present are balanced.
func (ecb *ecBalancer) balance(collections []string) error {
topo, volumeRatio, selected := toBalancerTopology(ecb.ecNodes, collections, ecb.diskType, ecb.volumeIds, ecb.migratingVolumeIds)
topo, volumeRatio, selected := toBalancerTopology(ecb.ecNodes, collections, ecb.diskType, ecb.volumeIds, ecb.migratingVolumeIds, ecb.noReceiveTargets, ecb.noReceiveDisks)
if len(ecb.volumeIds) > 0 {
requested := make([]uint32, 0, len(ecb.volumeIds))
for vid := range ecb.volumeIds {
@@ -896,6 +1146,7 @@ func (ecb *ecBalancer) balance(collections []string) error {
}
fmt.Printf("planned %d ec shard move(s) and %d ec shard deletion(s)\n", len(moves)-deletions, deletions)
}
ecb.plannedMoves = len(moves)
return ecb.executeMoves(moves)
}
@@ -907,7 +1158,7 @@ func (ecb *ecBalancer) balance(collections []string) error {
// (0,0 when unreported, e.g. always in OSS), which Plan prefers over the
// collection ratio for mixed-ratio clusters, and the set of volume ids that made
// it into the topology.
func toBalancerTopology(ecNodes []*EcNode, collections []string, diskType types.DiskType, volumeIds map[uint32]bool, migratingVolumeIds map[uint32]bool) (*ecbalancer.Topology, func(collection string, vid uint32) (int, int), map[uint32]bool) {
func toBalancerTopology(ecNodes []*EcNode, collections []string, diskType types.DiskType, volumeIds map[uint32]bool, migratingVolumeIds map[uint32]bool, noReceiveTargets map[string]bool, noReceiveDisks map[string]map[uint32]bool) (*ecbalancer.Topology, func(collection string, vid uint32) (int, int), map[uint32]bool) {
allowed := make(map[string]bool, len(collections))
for _, c := range collections {
allowed[c] = true
@@ -924,11 +1175,17 @@ func toBalancerTopology(ecNodes []*EcNode, collections []string, diskType types.
for _, en := range ecNodes {
rackKey := string(en.DC) + ":" + string(en.Rack)
node := topo.AddNode(en.Info.Id, string(en.DC), rackKey, en.FreeEcSlot)
if noReceiveTargets[en.Info.Id] {
node.SetNoReceive()
}
// Group by physical machine (host) so shards spread across machines, not just
// nodes; the id stays the node identity used for moves.
node.SetHost(pb.NewServerAddressFromDataNode(en.Info).ToHost())
for diskId, d := range en.Disks {
node.AddDisk(diskId, d.DiskType, d.FreeEcSlots, d.EcShardCount)
if noReceiveDisks[en.Info.Id][diskId] {
node.SetDiskNoReceive(diskId)
}
}
for diskTypeKey, diskInfo := range en.Info.DiskInfos {
if diskInfo == nil {
@@ -1069,14 +1326,16 @@ func (ecb *ecBalancer) executeMove(byID map[string]*EcNode, m ecbalancer.Move) e
// thing making this safe -- and the plan saying so is not evidence. A
// topology entry can name a location holding nothing, and deleting on
// that basis removes the last copy. Confirm the keep node has it.
if err := verifyEcShardOnKeepNode(grpcDialOption, m.Collection, vid, m.KeepNode, shardId); err != nil {
return err
}
err := verifyEcShardOnKeepNode(grpcDialOption, m.Collection, vid, m.KeepNode, shardId)
addr := pb.NewServerAddressFromDataNode(src.Info)
if err := UnmountEcShards(grpcDialOption, vid, addr, shardIds); err != nil {
return err
if err == nil {
err = UnmountEcShards(grpcDialOption, vid, addr, shardIds)
}
return SourceServerDeleteEcShards(grpcDialOption, m.Collection, vid, addr, shardIds)
if err == nil {
err = SourceServerDeleteEcShards(grpcDialOption, m.Collection, vid, addr, shardIds)
}
ecb.recordMoveResult(m, err)
return err
}
dst := byID[m.TargetNode]
@@ -1092,7 +1351,9 @@ func (ecb *ecBalancer) executeMove(byID map[string]*EcNode, m ecbalancer.Move) e
// Dry-run: update the in-memory model only.
return MoveMountedShardToEcNode(ecb.env, src, m.Collection, vid, shardId, dst, m.TargetDisk, false, ecb.diskType)
}
return ecb.applyShardMoveRPC(src, dst, m.Collection, vid, shardId, m.TargetDisk)
err := ecb.applyShardMoveRPC(src, dst, m.Collection, vid, shardId, m.TargetDisk)
ecb.recordMoveResult(m, err)
return err
}
// applyShardMoveRPC copies a shard to the destination disk, verifies the
@@ -390,6 +390,14 @@ func (store *AbstractSqlStore) DeleteFolderChildren(ctx context.Context, fullpat
return doDeleteFolderChildren()
}
var likeEscaper = strings.NewReplacer("!", "!!", "%", "!%", "_", "!_")
// LikePattern escapes LIKE wildcards so the prefix matches literally;
// '!' pairs with the ESCAPE '!' clause emitted by the SqlGenerators.
func LikePattern(prefix string) string {
return likeEscaper.Replace(prefix) + "%"
}
func (store *AbstractSqlStore) ListDirectoryPrefixedEntries(ctx context.Context, dirPath util.FullPath, startFileName string, includeStartFile bool, limit int64, prefix string, eachEntryFunc filer.ListEachEntryFunc) (lastFileName string, err error) {
db, bucket, shortPath, err := store.getTxOrDB(ctx, dirPath, true)
@@ -402,7 +410,7 @@ func (store *AbstractSqlStore) ListDirectoryPrefixedEntries(ctx context.Context,
sqlText = store.GetSqlListInclusive(bucket)
}
rows, err := db.QueryContext(ctx, sqlText, util.HashStringToLong(string(shortPath)), startFileName, string(shortPath), prefix+"%", limit+1)
rows, err := db.QueryContext(ctx, sqlText, util.HashStringToLong(string(shortPath)), startFileName, string(shortPath), LikePattern(prefix), limit+1)
if err != nil {
return lastFileName, fmt.Errorf("list %s : %v", dirPath, err)
}
@@ -52,6 +52,24 @@ func TestGetTxOrDBRealBucket(t *testing.T) {
}
}
func TestLikePattern(t *testing.T) {
cases := []struct {
prefix, want string
}{
{"", "%"},
{"plain", "plain%"},
{"call_03_KNl", "call!_03!_KNl%"},
{"100%", "100!%%"},
{"a!b", "a!!b%"},
{`a\b`, `a\b%`},
}
for _, c := range cases {
if got := LikePattern(c.prefix); got != c.want {
t.Errorf("LikePattern(%q) = %q, want %q", c.prefix, got, c.want)
}
}
}
func TestSplitPoolForKv(t *testing.T) {
cases := []struct {
maxOpen, mainOpen, kvOpen int
+3 -3
View File
@@ -458,13 +458,13 @@ func retriedStreamFetchChunkData(ctx context.Context, writer io.Writer, urlStrin
localProcessed += writtenCount
totalWritten += writtenCount
})
if !shouldRetry {
break
}
if writeErr != nil {
err = writeErr
break
}
if !shouldRetry {
break
}
if err != nil {
failed = true
glog.V(0).InfofCtx(ctx, "read %s failed, err: %v", urlString, err)
+20
View File
@@ -787,3 +787,23 @@ func TestFetchWholeChunkUsesProvidedFilerJwt(t *testing.T) {
return ""
}))
}
// A completed fetch whose destination write failed is a failed copy, not a
// short success: the caller must see the write error, or a temp disk filling
// mid-copy would still upload the truncated file.
func TestRetriedStreamFetchChunkDataReportsWriteError(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("0123456789"))
}))
t.Cleanup(srv.Close)
written, err := retriedStreamFetchChunkData(context.Background(), &failOnWrite{}, []string{srv.URL + "/5,abc"}, "", nil, false, true, 0, 0, nil)
assert.Error(t, err)
assert.Zero(t, written)
}
type failOnWrite struct{}
func (w *failOnWrite) Write(p []byte) (int, error) {
return 0, fmt.Errorf("no space left")
}
+1 -1
View File
@@ -890,7 +890,7 @@ func (f *Filer) DeleteEmptyParentDirectories(ctx context.Context, dirPath util.F
glog.V(2).InfofCtx(ctx, "DeleteEmptyParentDirectories: deleting empty directory %s", dirPath)
parentDir, _ := dirPath.DirAndName()
if dirEntry, findErr := f.FindEntry(ctx, dirPath); findErr == nil {
if delErr := f.doDeleteEntryMetaAndData(ctx, dirEntry, false, false, nil); delErr == nil {
if delErr := f.doDeleteEntryMetaAndData(f.emptyFolderDeleteContext(ctx, dirPath), dirEntry, false, false, nil); delErr == nil {
// Successfully deleted, continue checking upwards
f.DeleteEmptyParentDirectories(ctx, util.FullPath(parentDir), stopAtPath)
} else {
+32 -3
View File
@@ -80,6 +80,27 @@ func keepsRemoteObject(ctx context.Context) bool {
return ctx.Value(keepRemoteObjectKey{}) != nil
}
// emptyFolderDeleteContext makes deleting a folder that is empty in the filer
// metadata-only when it is under a remote mount. A mount is populated lazily,
// so the folder can still hold remote objects never read, and removing its
// remote directory deletes every object under that prefix.
func (f *Filer) emptyFolderDeleteContext(ctx context.Context, dirPath util.FullPath) context.Context {
if f.IsUnderRemoteMount(dirPath) {
return WithKeepRemoteObject(ctx)
}
return ctx
}
// IsUnderRemoteMount reports whether p is a remote mount directory or lies under one.
func (f *Filer) IsUnderRemoteMount(p util.FullPath) bool {
if f.RemoteStorage == nil {
return false
}
// Mount rules are keyed with a trailing slash, so this also matches the mount directory itself.
_, remoteLoc := f.RemoteStorage.FindMountDirectory(util.FullPath(string(p) + "/"))
return remoteLoc != nil
}
func deleteEventEntry(ctx context.Context, entry *Entry) *Entry {
if !keepsRemoteObject(ctx) {
return entry
@@ -108,6 +129,9 @@ func (f *Filer) DeleteEntryMetaAndData(ctx context.Context, p util.FullPath, isR
if ifNotModifiedAfter > 0 && entry.Attr.Mtime.Unix() > ifNotModifiedAfter {
return nil
}
if !isRecursive && entry.IsDirectory() {
ctx = f.emptyFolderDeleteContext(ctx, p)
}
isDeleteCollection := f.IsBucket(entry)
collectionName := ""
if isDeleteCollection {
@@ -170,7 +194,7 @@ func (f *Filer) DeleteEntryMetaAndData(ctx context.Context, p util.FullPath, isR
func (f *Filer) doBatchDeleteFolderMetaAndData(ctx context.Context, entry *Entry, isRecursive, ignoreRecursiveError, shouldDeleteChunks, isDeletingBucket, isFromOtherCluster bool, signatures []int32, onHardLinkIdsFn OnHardLinkIdsFunc) (err error) {
var dirTombstoneTs int64
if isRecursive {
if isRecursive && !keepsRemoteObject(ctx) {
// Tombstone the directory before its children: when the store drops
// the subtree without listing it, or a child error aborts the walk,
// the ancestor tombstone still covers every descendant.
@@ -201,7 +225,9 @@ func (f *Filer) doBatchDeleteFolderMetaAndData(ctx context.Context, entry *Entry
for _, sub := range entries {
lastFileName = sub.Name()
f.noteRemoteDeletion(sub.FullPath, sub.IsDirectory(), time.Now().UnixNano())
if !keepsRemoteObject(ctx) {
f.noteRemoteDeletion(sub.FullPath, sub.IsDirectory(), time.Now().UnixNano())
}
if sub.IsDirectory() {
subIsDeletingBucket := f.IsBucket(sub)
err = f.doBatchDeleteFolderMetaAndData(ctx, sub, isRecursive, ignoreRecursiveError, shouldDeleteChunks, subIsDeletingBucket, isFromOtherCluster, nil, onHardLinkIdsFn)
@@ -264,7 +290,10 @@ func (f *Filer) doDeleteEntryMetaAndData(ctx context.Context, entry *Entry, shou
}
}
f.noteRemoteDeletion(entry.FullPath, entry.IsDirectory(), time.Now().UnixNano())
// A metadata-only delete owes the remote nothing, so lazy reads stay open.
if !keepsRemoteObject(ctx) {
f.noteRemoteDeletion(entry.FullPath, entry.IsDirectory(), time.Now().UnixNano())
}
if storeDeletionErr := f.Store.DeleteOneEntry(ctx, entry); storeDeletionErr != nil {
return fmt.Errorf("filer store delete: %w", storeDeletionErr)
+123 -3
View File
@@ -18,11 +18,15 @@ import (
type lazyFetchContextKey struct{}
// lazyParentsContextKey marks a FindEntry issued while materializing missing
// parents: that fetch's own parent walk would only re-stat the same gaps.
type lazyParentsContextKey struct{}
// maybeLazyFetchFromRemote is called by FindEntry when the store returns no
// entry for p. If p is under a remote-storage mount, it stats the remote
// object, builds a filer Entry from the result, and persists it via
// CreateEntry with SkipCheckParentDirectory so phantom parent directories
// under the mount are not required.
// CreateEntry together with its missing parent directories: remote.uncache
// and cache eviction find entries by walking down from the mount directory.
//
// On a CreateEntry failure after a successful StatFile the in-memory entry is
// still returned (availability over consistency); the singleflight key is
@@ -127,7 +131,10 @@ func (f *Filer) maybeLazyFetchFromRemote(ctx context.Context, p util.FullPath) (
glog.V(2).InfofCtx(ctx, "maybeLazyFetchFromRemote: %s deleted during remote stat", p)
return lazyFetchResult{nil}, nil
}
saveErr := f.CreateEntry(persistCtx, entry, nil, false, false, nil, true, f.MaxFilenameLength)
if ctx.Value(lazyParentsContextKey{}) == nil {
f.fetchMissingRemoteParents(persistBaseCtx, mountDir, p)
}
saveErr := f.CreateEntry(persistCtx, entry, nil, false, false, nil, false, f.MaxFilenameLength)
if saveErr != nil {
glog.Warningf("maybeLazyFetchFromRemote: failed to persist filer entry for %s: %v", p, saveErr)
f.lazyFetchGroup.Forget(key)
@@ -157,6 +164,25 @@ func (f *Filer) maybeLazyFetchFromRemote(ctx context.Context, p util.FullPath) (
return result.entry, nil
}
// fetchMissingRemoteParents looks up p's missing parents under mountDir, top
// down, through lazy fetch. A remote object stored at a parent's key is then
// materialized first, so creating p promotes it to a prefix object instead of
// hiding it behind a plain directory. Each missing parent costs one remote
// stat, once per directory.
func (f *Filer) fetchMissingRemoteParents(ctx context.Context, mountDir, p util.FullPath) {
var missing []util.FullPath
for dir, _ := p.DirAndName(); len(dir) > len(mountDir); dir, _ = util.FullPath(dir).DirAndName() {
if _, err := f.Store.FindEntry(ctx, util.FullPath(dir)); !errors.Is(err, filer_pb.ErrNotFound) {
break
}
missing = append(missing, util.FullPath(dir))
}
parentCtx := context.WithValue(ctx, lazyParentsContextKey{}, true)
for i := len(missing) - 1; i >= 0; i-- {
f.FindEntry(parentCtx, missing[i])
}
}
// retractLazyRemoteEntry deletes the entry at entry.FullPath only when it is
// still the entry a lazy remote read just materialized — a concurrent write
// may have replaced it, and deleting by path alone would take that write down.
@@ -225,3 +251,97 @@ func (f *Filer) maybeDeleteFromRemote(ctx context.Context, entry *Entry) (bool,
glog.V(3).InfofCtx(ctx, "maybeDeleteFromRemote: deleted %s from remote", entry.FullPath)
return true, nil
}
// DeleteFromRemote deletes the object, or for a directory every object, the
// entry maps to on its mounted remote storage.
func (f *Filer) DeleteFromRemote(ctx context.Context, entry *Entry) error {
_, err := f.maybeDeleteFromRemote(ctx, entry)
return err
}
// ExtRemoteRenameSourceKey marks a remote-only entry whose rename committed
// while its copy to the new key was still pending: it holds the path whose
// object remote-only reads must serve until the copy stamps the entry.
const ExtRemoteRenameSourceKey = "Seaweed-X-Remote-Rename-Source"
// RemoteSourcePath returns the path whose remote object holds the content of
// the remote-only entry at p: the renamed-from path a pending deferred copy
// still reads, or p itself. A marker leading out of p's mount is ignored, so
// whoever can set extended attributes cannot point reads at another mount.
func (f *Filer) RemoteSourcePath(entry *Entry, p util.FullPath) util.FullPath {
src := util.FullPath(entry.Extended[ExtRemoteRenameSourceKey])
if src == "" || f.RemoteStorage == nil {
return p
}
mountDir, remoteLoc := f.RemoteStorage.FindMountDirectory(p)
if remoteLoc == nil {
return p
}
if srcMountDir, _ := f.RemoteStorage.FindMountDirectory(src); srcMountDir != mountDir {
return p
}
return src
}
// RemoteReadPath returns the dir and name whose object a remote-only read of
// the entry at dir/name serves.
func (f *Filer) RemoteReadPath(entry *Entry, dir, name string) (string, string) {
return f.RemoteSourcePath(entry, util.NewFullPath(dir, name)).DirAndName()
}
// CopyRemoteOnlyEntry copies the remote object of a renamed entry whose
// content exists only on the remote to the key of newPath, and returns the
// RemoteEntry of the copy. The rename deletes the old key along with the old
// entry, and with no local data nothing would write the new key. It returns
// nil when no copy is needed: the entry has local data, which the remote sync
// uploads to the new key, it is empty, which the sync writes as an empty
// object, or it is not under a remote mount. A rename that
// moves such an entry out of its mount, or onto a remote that cannot copy, is
// refused, since it would delete the only copy of the content.
func (f *Filer) CopyRemoteOnlyEntry(ctx context.Context, entry *Entry, oldPath, newPath util.FullPath) (*filer_pb.RemoteEntry, error) {
src, dst, copier, err := f.RemoteCopyLocations(ctx, entry, oldPath, newPath)
if err != nil || copier == nil {
return nil, err
}
remoteEntry, err := copier.CopyFile(src, dst)
if err != nil {
return nil, fmt.Errorf("copy remote %s to %s: %w", remote_storage.FormatLocation(src), remote_storage.FormatLocation(dst), err)
}
glog.V(3).InfofCtx(ctx, "CopyRemoteOnlyEntry: copied %s to %s", remote_storage.FormatLocation(src), remote_storage.FormatLocation(dst))
return remoteEntry, nil
}
// RemoteCopyLocations resolves the object locations a remote-only rename copy
// of entry between oldPath and newPath would use, and the client able to copy
// it. A nil copier means the entry needs no copy; an error means it needs one
// that cannot be served, which a rename must refuse before committing.
func (f *Filer) RemoteCopyLocations(ctx context.Context, entry *Entry, oldPath, newPath util.FullPath) (src, dst *remote_pb.RemoteStorageLocation, copier remote_storage.RemoteStorageObjectCopier, err error) {
if f.RemoteStorage == nil || entry.IsDirectory() || !entry.IsInRemoteOnly() || len(entry.Content) > 0 {
return nil, nil, nil, nil
}
mountDir, remoteLoc := f.RemoteStorage.FindMountDirectory(oldPath)
if remoteLoc == nil {
return nil, nil, nil, nil
}
if newMountDir, _ := f.RemoteStorage.FindMountDirectory(newPath); newMountDir != mountDir {
return nil, nil, nil, fmt.Errorf("%s exists only on the remote of mount %s and cannot be renamed out of it; cache it first", oldPath, mountDir)
}
remoteConf, found := f.RemoteStorage.GetRemoteStorageConf(remoteLoc.Name)
if !found {
return nil, nil, nil, fmt.Errorf("resolve remote storage client for %s: not found", oldPath)
}
client, clientErr := f.buildRemoteStorageClient(ctx, remoteConf)
if clientErr != nil {
return nil, nil, nil, fmt.Errorf("resolve remote storage client for %s: %w", oldPath, clientErr)
}
copier, ok := client.(remote_storage.RemoteStorageObjectCopier)
if !ok {
return nil, nil, nil, fmt.Errorf("%s exists only on the remote and remote storage type %s cannot copy it to the new name; cache it first", oldPath, remoteConf.Type)
}
src = MapFullPathToRemoteStorageLocation(mountDir, remoteLoc, f.RemoteSourcePath(entry, oldPath))
dst = MapFullPathToRemoteStorageLocation(mountDir, remoteLoc, newPath)
return src, dst, copier, nil
}
+5 -1
View File
@@ -103,6 +103,10 @@ func (f *Filer) maybeLazyListFromRemote(ctx context.Context, p util.FullPath) {
persistCtx := context.WithValue(context.WithoutCancel(ctx), lazyListContextKey{}, true)
persistCtx = context.WithValue(persistCtx, lazyFetchContextKey{}, true)
// Entries listed under p must stay reachable by a walk from the mount
// directory, the same reason the lazy fetch fills in missing parents.
f.fetchMissingRemoteParents(context.WithoutCancel(ctx), mountDir, p)
listErr := client.ListDirectory(persistCtx, objectLoc, func(dir string, name string, isDirectory bool, remoteEntry *filer_pb.RemoteEntry) error {
childPath := p.Child(name)
@@ -215,7 +219,7 @@ func (f *Filer) updateDirectoryListingSyncedAt(ctx context.Context, p util.FullP
dirEntry.Extended = make(map[string][]byte)
}
dirEntry.Extended[xattrRemoteListingSyncedAt] = []byte(fmt.Sprintf("%d", syncTime.Unix()))
if saveErr := f.CreateEntry(ctx, dirEntry, nil, false, false, nil, true, f.MaxFilenameLength); saveErr != nil {
if saveErr := f.CreateEntry(ctx, dirEntry, nil, false, false, nil, false, f.MaxFilenameLength); saveErr != nil {
glog.Warningf("maybeLazyListFromRemote: create dir synced_at for %s: %v", p, saveErr)
}
return
+203
View File
@@ -17,6 +17,7 @@ import (
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
"github.com/seaweedfs/seaweedfs/weed/pb/remote_pb"
"github.com/seaweedfs/seaweedfs/weed/remote_storage"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
"github.com/seaweedfs/seaweedfs/weed/util"
"github.com/seaweedfs/seaweedfs/weed/util/log_buffer"
"github.com/seaweedfs/seaweedfs/weed/wdclient"
@@ -35,6 +36,7 @@ type stubFilerStore struct {
kv map[string][]byte
insertErr error
findErr error
findErrByPath map[string]error
kvPutErr error
kvGetErr error
kvDeleteErr error
@@ -200,6 +202,9 @@ func (s *stubFilerStore) FindEntry(_ context.Context, p util.FullPath) (*Entry,
if s.findErr != nil {
return nil, s.findErr
}
if err := s.findErrByPath[string(p)]; err != nil {
return nil, err
}
if e, ok := s.entries[string(p)]; ok {
return e, nil
}
@@ -340,6 +345,11 @@ func putConfigEntry(store *stubFilerStore, path string, content []byte) {
}
}
// putDirEntry stores a plain directory entry, so a lazy fetch below it finds its parent in place.
func putDirEntry(store *stubFilerStore, path string) {
store.entries[path] = &Entry{FullPath: util.FullPath(path), Attr: Attr{Mode: os.ModeDir | 0755}}
}
// --- tests ---
func TestMaybeLazyFetchFromRemote_HitsRemoteAndPersists(t *testing.T) {
@@ -375,6 +385,199 @@ func TestMaybeLazyFetchFromRemote_HitsRemoteAndPersists(t *testing.T) {
assert.Equal(t, []byte("zstd"), stored.Extended["Content-Encoding"])
}
// pathStatRemoteClient stats only the objects it holds, keyed by remote path.
type pathStatRemoteClient struct {
stubRemoteClient
objects map[string]*filer_pb.RemoteEntry
statCalls []string
}
func (c *pathStatRemoteClient) StatFile(loc *remote_pb.RemoteStorageLocation) (*filer_pb.RemoteEntry, error) {
c.statCalls = append(c.statCalls, loc.Path)
if remoteEntry, found := c.objects[loc.Path]; found {
return remoteEntry, nil
}
return nil, remote_storage.ErrRemoteObjectNotFound
}
func newPathStatFiler(t *testing.T, storageType string, client remote_storage.RemoteStorageClient) (*Filer, *stubFilerStore) {
t.Helper()
t.Cleanup(registerStubMaker(t, storageType, client))
conf := &remote_pb.RemoteConf{Name: "mystore", Type: storageType}
rs := NewFilerRemoteStorage()
rs.storageNameToConf[conf.Name] = conf
rs.mapDirectoryToRemoteStorage("/buckets/mybucket", &remote_pb.RemoteStorageLocation{
Name: "mystore",
Bucket: "mybucket",
Path: "/",
})
store := newStubFilerStore()
return newTestFiler(t, store, rs), store
}
func TestMaybeLazyFetchFromRemote_PersistsParentDirectories(t *testing.T) {
stub := &pathStatRemoteClient{objects: map[string]*filer_pb.RemoteEntry{
"/a/b/file.bin": {RemoteMtime: 1700000000, RemoteSize: 1234},
"/a/b/other.bin": {RemoteMtime: 1700000000, RemoteSize: 5678},
}}
f, store := newPathStatFiler(t, "stub_lazy_parents", stub)
const filePath = "/buckets/mybucket/a/b/file.bin"
entry, err := f.maybeLazyFetchFromRemote(context.Background(), filePath)
require.NoError(t, err)
require.NotNil(t, entry)
assert.Equal(t, []string{"/a/b/file.bin", "/a", "/a/b"}, stub.statCalls, "each missing parent is stat'ed once")
for _, dir := range []string{"/buckets/mybucket/a", "/buckets/mybucket/a/b"} {
dirEntry := store.getEntry(dir)
require.NotNil(t, dirEntry, "parent %s not persisted", dir)
assert.True(t, dirEntry.IsDirectory(), "%s should be a directory", dir)
assert.Nil(t, dirEntry.Remote, "%s holds no remote object", dir)
}
// A sibling finds its parents in place and stats only itself.
stub.statCalls = nil
_, err = f.maybeLazyFetchFromRemote(context.Background(), "/buckets/mybucket/a/b/other.bin")
require.NoError(t, err)
assert.Equal(t, []string{"/a/b/other.bin"}, stub.statCalls)
// Once the content is cached, eviction walks down from the mount and must reach it.
stored := store.getEntry(filePath)
require.NotNil(t, stored)
stored.Chunks = []*filer_pb.FileChunk{{FileId: "1,01637037d6", Size: 1234}}
stored.Remote.LastLocalSyncTsNs = time.Now().UnixNano()
require.NoError(t, store.UpdateEntry(context.Background(), stored))
evictable := f.ListEvictableRemoteEntries(context.Background(), []util.FullPath{"/buckets/mybucket"}, 0)
require.Len(t, evictable, 1)
assert.Equal(t, util.FullPath(filePath), evictable[0].FullPath)
}
func TestMaybeLazyFetchFromRemote_ParentLookupStopsAtStoreError(t *testing.T) {
stub := &pathStatRemoteClient{objects: map[string]*filer_pb.RemoteEntry{
"/a/b/file.bin": {RemoteMtime: 1700000000, RemoteSize: 1234},
}}
f, store := newPathStatFiler(t, "stub_lazy_parent_store_error", stub)
store.findErrByPath = map[string]error{"/buckets/mybucket/a/b": errors.New("store unavailable")}
_, err := f.maybeLazyFetchFromRemote(context.Background(), "/buckets/mybucket/a/b/file.bin")
require.NoError(t, err)
assert.Equal(t, []string{"/a/b/file.bin"}, stub.statCalls, "a parent the store cannot read is not treated as missing")
}
func TestMaybeLazyFetchFromRemote_PromotesRemoteObjectAtParentKey(t *testing.T) {
stub := &pathStatRemoteClient{objects: map[string]*filer_pb.RemoteEntry{
"/reports": {RemoteMtime: 1700000000, RemoteSize: 10},
"/reports/2026.csv": {RemoteMtime: 1700000000, RemoteSize: 20},
}}
f, store := newPathStatFiler(t, "stub_lazy_prefix_object", stub)
_, err := f.maybeLazyFetchFromRemote(context.Background(), "/buckets/mybucket/reports/2026.csv")
require.NoError(t, err)
parent := store.getEntry("/buckets/mybucket/reports")
require.NotNil(t, parent)
assert.True(t, parent.IsDirectory())
assert.Equal(t, []byte("true"), parent.Extended[s3_constants.SeaweedFSPrefixObject], "the remote object at the parent key must stay readable")
require.NotNil(t, parent.Remote)
assert.Equal(t, int64(10), parent.Remote.RemoteSize)
}
func TestMaybeLazyFetchFromRemote_StatsEachParentOnce(t *testing.T) {
stub := &pathStatRemoteClient{objects: map[string]*filer_pb.RemoteEntry{
"/a/b/c/file.bin": {RemoteMtime: 1700000000, RemoteSize: 1234},
}}
f, _ := newPathStatFiler(t, "stub_lazy_parent_stats", stub)
_, err := f.maybeLazyFetchFromRemote(context.Background(), "/buckets/mybucket/a/b/c/file.bin")
require.NoError(t, err)
assert.Equal(t, []string{"/a/b/c/file.bin", "/a", "/a/b", "/a/b/c"}, stub.statCalls,
"a parent still missing after its own stat must not be re-stat'ed by the next lookup")
}
func TestMaybeLazyListFromRemote_PersistsParentDirectories(t *testing.T) {
stub := &pathStatRemoteClient{objects: map[string]*filer_pb.RemoteEntry{}}
stub.listDirFn = func(loc *remote_pb.RemoteStorageLocation, visitFn remote_storage.VisitFunc) error {
return visitFn("/", "file.bin", false, &filer_pb.RemoteEntry{RemoteMtime: 1700000000, RemoteSize: 1234})
}
f, store := newPathStatFiler(t, "stub_lazy_list_parents", stub)
f.RemoteStorage.mapDirectoryToRemoteStorage("/buckets/listmount", &remote_pb.RemoteStorageLocation{
Name: "mystore",
Bucket: "listmount",
Path: "/",
ListingCacheTtlSeconds: 300,
})
f.maybeLazyListFromRemote(context.Background(), util.FullPath("/buckets/listmount/a/b"))
assert.Equal(t, []string{"/a"}, stub.statCalls, "each missing parent is stat'ed once")
for _, dir := range []string{"/buckets/listmount", "/buckets/listmount/a", "/buckets/listmount/a/b"} {
dirEntry := store.getEntry(dir)
require.NotNil(t, dirEntry, "%s not persisted", dir)
assert.True(t, dirEntry.IsDirectory(), "%s should be a directory", dir)
}
child := store.getEntry("/buckets/listmount/a/b/file.bin")
require.NotNil(t, child)
child.Chunks = []*filer_pb.FileChunk{{FileId: "1,01637037d6", Size: 1234}}
child.Remote.LastLocalSyncTsNs = time.Now().UnixNano()
require.NoError(t, store.UpdateEntry(context.Background(), child))
evictable := f.ListEvictableRemoteEntries(context.Background(), []util.FullPath{"/buckets/listmount"}, 0)
require.Len(t, evictable, 1)
assert.Equal(t, util.FullPath("/buckets/listmount/a/b/file.bin"), evictable[0].FullPath)
}
func TestDeleteEntryMetaAndData_EmptyFolderUnderMountKeepsRemotePrefix(t *testing.T) {
stub := &pathStatRemoteClient{objects: map[string]*filer_pb.RemoteEntry{
"/dir/unread.bin": {RemoteMtime: 1700000000, RemoteSize: 1},
}}
f, store := newPathStatFiler(t, "stub_lazy_empty_folder", stub)
dirPath := util.FullPath("/buckets/mybucket/dir")
putDirEntry(store, string(dirPath))
require.NoError(t, f.DeleteEntryMetaAndData(context.Background(), dirPath, false, false, false, false, nil, 0))
assert.Empty(t, stub.removeCalls, "an empty local folder must not remove its remote prefix")
assert.Nil(t, store.getEntry(string(dirPath)))
entry, err := f.maybeLazyFetchFromRemote(context.Background(), "/buckets/mybucket/dir/unread.bin")
require.NoError(t, err)
assert.NotNil(t, entry, "a metadata-only delete must not block lazy reads under the folder")
}
func TestDeleteEmptyParentDirectories_UnderMountKeepsRemotePrefix(t *testing.T) {
stub := &pathStatRemoteClient{}
f, store := newPathStatFiler(t, "stub_lazy_expired_parents", stub)
f.DirBucketsPath = "/buckets"
putDirEntry(store, "/buckets/mybucket/a")
putDirEntry(store, "/buckets/mybucket/a/b")
f.DeleteEmptyParentDirectories(context.Background(), "/buckets/mybucket/a/b", "/buckets")
assert.Empty(t, stub.removeCalls, "an empty local folder must not remove its remote prefix")
assert.Nil(t, store.getEntry("/buckets/mybucket/a/b"))
assert.Nil(t, store.getEntry("/buckets/mybucket/a"))
}
func TestIsUnderRemoteMount(t *testing.T) {
rs := NewFilerRemoteStorage()
rs.mapDirectoryToRemoteStorage("/buckets/mirror", &remote_pb.RemoteStorageLocation{
Name: "mystore",
Bucket: "mirror",
Path: "/",
})
f := newTestFiler(t, newStubFilerStore(), rs)
for path, want := range map[util.FullPath]bool{
"/buckets/mirror": true,
"/buckets/mirror/a/b": true,
"/buckets/mirror2/a": false,
"/buckets/local/a": false,
} {
assert.Equal(t, want, f.IsUnderRemoteMount(path), path)
}
}
func TestMaybeLazyFetchFromRemote_NotUnderMount(t *testing.T) {
rs := NewFilerRemoteStorage()
store := newStubFilerStore()
+1 -1
View File
@@ -407,7 +407,7 @@ func (f *Filer) onRemoteDeletionEvents(event *filer_pb.SubscribeMetadataResponse
if message == nil {
return
}
if message.OldEntry != nil {
if message.OldEntry != nil && !IsMetadataOnlyDelete(message.OldEntry) {
sourcePath := filer_pb.MetadataEventSourceFullPath(event)
if message.NewEntry == nil || sourcePath != filer_pb.MetadataEventTargetFullPath(event) {
if f.RemoteStorage != nil && f.remoteTombstones != nil {
+31
View File
@@ -187,6 +187,7 @@ func TestMaybeLazyFetchFromRemote_SyncOffsetReleasesTombstone(t *testing.T) {
},
}
f, store := newMountedTestFiler(t, storageType, stub, 0)
putDirEntry(store, "/buckets/mybucket/dir")
filePath := util.FullPath("/buckets/mybucket/dir/a.txt")
deleteTsNs := time.Now().UnixNano()
@@ -222,6 +223,7 @@ func TestDeleteEntryMetaAndData_TombstonesPath(t *testing.T) {
},
}
f, store := newMountedTestFiler(t, storageType, stub, 0)
putDirEntry(store, "/buckets/mybucket/dir")
filePath := util.FullPath("/buckets/mybucket/dir/a.txt")
store.entries[string(filePath)] = &Entry{
@@ -257,6 +259,35 @@ func TestDeleteEntryMetaAndData_TombstonesPath(t *testing.T) {
assert.Equal(t, 1, stub.statCalls)
}
func TestOnMetadataChangeEvent_MetadataOnlyDeleteSkipsTombstone(t *testing.T) {
const storageType = "stub_tomb_metadata_only"
stub := &countingRemoteClient{
stubRemoteClient: stubRemoteClient{
statResult: &filer_pb.RemoteEntry{RemoteMtime: 1700000000, RemoteSize: 11},
},
}
f, store := newMountedTestFiler(t, storageType, stub, 0)
putDirEntry(store, "/buckets/mybucket/dir")
f.onMetadataChangeEvent(&filer_pb.SubscribeMetadataResponse{
Directory: "/buckets/mybucket",
TsNs: time.Now().UnixNano(),
EventNotification: &filer_pb.EventNotification{
OldEntry: &filer_pb.Entry{
Name: "dir",
IsDirectory: true,
Extended: map[string][]byte{ExtKeepRemoteObjectKey: []byte("true")},
},
},
})
// nothing is owed to the remote, so objects under the folder stay readable
entry, err := f.maybeLazyFetchFromRemote(context.Background(), "/buckets/mybucket/dir/a.txt")
require.NoError(t, err)
require.NotNil(t, entry)
assert.Equal(t, 1, stub.statCalls)
}
func TestOnMetadataChangeEvent_PeerDeleteTombstones(t *testing.T) {
const storageType = "stub_tomb_peer"
stub := &countingRemoteClient{
+2 -2
View File
@@ -64,12 +64,12 @@ func (gen *SqlGenMysql) nameExpr() string {
func (gen *SqlGenMysql) GetSqlListExclusive(tableName string) string {
name := gen.nameExpr()
return fmt.Sprintf("SELECT `name`, `meta` FROM `%s` WHERE `dirhash` = ? AND %s > ? AND `directory` = ? AND %s LIKE ? ORDER BY %s ASC LIMIT ?", tableName, name, name, name)
return fmt.Sprintf("SELECT `name`, `meta` FROM `%s` WHERE `dirhash` = ? AND %s > ? AND `directory` = ? AND %s LIKE ? ESCAPE '!' ORDER BY %s ASC LIMIT ?", tableName, name, name, name)
}
func (gen *SqlGenMysql) GetSqlListInclusive(tableName string) string {
name := gen.nameExpr()
return fmt.Sprintf("SELECT `name`, `meta` FROM `%s` WHERE `dirhash` = ? AND %s >= ? AND `directory` = ? AND %s LIKE ? ORDER BY %s ASC LIMIT ?", tableName, name, name, name)
return fmt.Sprintf("SELECT `name`, `meta` FROM `%s` WHERE `dirhash` = ? AND %s >= ? AND `directory` = ? AND %s LIKE ? ESCAPE '!' ORDER BY %s ASC LIMIT ?", tableName, name, name, name)
}
func (gen *SqlGenMysql) GetSqlCreateTable(tableName string) string {
+9
View File
@@ -65,6 +65,15 @@ func TestListSqlBinaryOrderingOnNonBinaryColumn(t *testing.T) {
}
}
func TestListSqlEscapesLikeWildcards(t *testing.T) {
gen := &SqlGenMysql{}
for _, got := range []string{gen.GetSqlListExclusive("filemeta"), gen.GetSqlListInclusive("filemeta")} {
if !strings.Contains(got, "LIKE ? ESCAPE '!'") {
t.Fatalf("list query must declare the escape char matching LikePattern, got: %s", got)
}
}
}
func TestIsBinaryCollation(t *testing.T) {
binary := []string{"", "binary", "utf8mb4_bin", "utf8mb3_bin", "latin1_bin", "UTF8MB4_BIN"}
for _, c := range binary {
+2 -2
View File
@@ -77,12 +77,12 @@ func (gen *SqlGenPostgres) nameExpr() string {
func (gen *SqlGenPostgres) GetSqlListExclusive(tableName string) string {
name := gen.nameExpr()
return fmt.Sprintf(`SELECT NAME, meta FROM "%s" WHERE dirhash=$1 AND %s>$2 AND directory=$3 AND %s like $4 ORDER BY %s ASC LIMIT $5`, tableName, name, name, name)
return fmt.Sprintf(`SELECT NAME, meta FROM "%s" WHERE dirhash=$1 AND %s>$2 AND directory=$3 AND %s like $4 ESCAPE '!' ORDER BY %s ASC LIMIT $5`, tableName, name, name, name)
}
func (gen *SqlGenPostgres) GetSqlListInclusive(tableName string) string {
name := gen.nameExpr()
return fmt.Sprintf(`SELECT NAME, meta FROM "%s" WHERE dirhash=$1 AND %s>=$2 AND directory=$3 AND %s like $4 ORDER BY %s ASC LIMIT $5`, tableName, name, name, name)
return fmt.Sprintf(`SELECT NAME, meta FROM "%s" WHERE dirhash=$1 AND %s>=$2 AND directory=$3 AND %s like $4 ESCAPE '!' ORDER BY %s ASC LIMIT $5`, tableName, name, name, name)
}
func (gen *SqlGenPostgres) GetSqlCreateTable(tableName string) string {
@@ -62,6 +62,15 @@ func TestListSqlBinaryOrderingOnNonBinaryColumn(t *testing.T) {
}
}
func TestListSqlEscapesLikeWildcards(t *testing.T) {
gen := &SqlGenPostgres{}
for _, got := range []string{gen.GetSqlListExclusive("filemeta"), gen.GetSqlListInclusive("filemeta")} {
if !strings.Contains(got, "like $4 ESCAPE '!'") {
t.Fatalf("list query must declare the escape char matching LikePattern, got: %s", got)
}
}
}
func TestIsByteOrderedCollation(t *testing.T) {
byteOrdered := []string{"C", "POSIX", "c", "C.UTF-8", "C.utf8"}
for _, c := range byteOrdered {
@@ -0,0 +1,75 @@
//go:build sqlite
package sqlite
import (
"context"
"database/sql"
"testing"
"github.com/seaweedfs/seaweedfs/weed/filer"
"github.com/seaweedfs/seaweedfs/weed/filer/mysql"
"github.com/seaweedfs/seaweedfs/weed/util"
_ "modernc.org/sqlite"
)
func TestListDirectoryPrefixedEntriesEscapesLikeWildcards(t *testing.T) {
db, err := sql.Open("sqlite", "file::memory:?cache=shared")
if err != nil {
t.Fatal(err)
}
defer db.Close()
_, err = db.Exec(`CREATE TABLE filemeta (dirhash BIGINT, name VARCHAR(1000), directory TEXT, meta BLOB, PRIMARY KEY (dirhash, name))`)
if err != nil {
t.Fatal(err)
}
store := &SqliteStore{}
store.SqlGenerator = &mysql.SqlGenMysql{}
store.DB = db
ctx := context.Background()
names := []string{
"call_03_KNlHPt6grkZMe4WZVwQi9710", // target: literal _ prefix
"callX03XKNlZZZ", // would match if _ were a wildcard
"call_99_other",
"100%_done", // % and _ in the name
"100aXdone", // would match prefix "100%_" if unescaped
"bang!file",
}
for _, n := range names {
entry := &filer.Entry{FullPath: util.FullPath("/dir/" + n)}
if err := store.InsertEntry(ctx, entry); err != nil {
t.Fatalf("insert %s: %v", n, err)
}
}
list := func(prefix string) []string {
var got []string
_, err := store.ListDirectoryPrefixedEntries(ctx, "/dir", "", false, 100, prefix, func(e *filer.Entry) (bool, error) {
got = append(got, e.Name())
return true, nil
})
if err != nil {
t.Fatalf("list %q: %v", prefix, err)
}
return got
}
if got := list("call_03_"); len(got) != 1 || got[0] != "call_03_KNlHPt6grkZMe4WZVwQi9710" {
t.Fatalf("prefix with _: got %v", got)
}
if got := list("call_"); len(got) != 2 {
t.Fatalf("prefix call_: got %v", got)
}
if got := list("100%_"); len(got) != 1 || got[0] != "100%_done" {
t.Fatalf("prefix with %%: got %v", got)
}
if got := list("bang!"); len(got) != 1 || got[0] != "bang!file" {
t.Fatalf("prefix with !: got %v", got)
}
if got := list(""); len(got) != len(names) {
t.Fatalf("empty prefix: got %v", got)
}
}
+2 -2
View File
@@ -55,7 +55,7 @@ const (
SELECT name, meta
FROM ` + asql.DEFAULT_TABLE + `
WHERE dir_hash = $dir_hash AND directory = $directory and name > $start_name and name LIKE $prefix
WHERE dir_hash = $dir_hash AND directory = $directory and name > $start_name and name LIKE $prefix ESCAPE '!'
ORDER BY name ASC LIMIT $limit;`
listInclusiveDirectoryQuery = `
@@ -68,6 +68,6 @@ const (
SELECT name, meta
FROM ` + asql.DEFAULT_TABLE + `
WHERE dir_hash = $dir_hash AND directory = $directory and name >= $start_name and name LIKE $prefix
WHERE dir_hash = $dir_hash AND directory = $directory and name >= $start_name and name LIKE $prefix ESCAPE '!'
ORDER BY name ASC LIMIT $limit;`
)
+1 -1
View File
@@ -288,7 +288,7 @@ func (store *YdbStore) ListDirectoryPrefixedEntries(ctx context.Context, dirPath
table.ValueParam("$dir_hash", types.Int64Value(util.HashStringToLong(*shortDir))),
table.ValueParam("$directory", types.UTF8Value(*shortDir)),
table.ValueParam("$start_name", types.UTF8Value(startFileName)),
table.ValueParam("$prefix", types.UTF8Value(prefix+"%")),
table.ValueParam("$prefix", types.UTF8Value(abstract_sql.LikePattern(prefix))),
table.ValueParam("$limit", types.Uint64Value(uint64(chunkLimit))),
)
+9
View File
@@ -4,6 +4,7 @@ import (
"context"
"fmt"
"io"
"strings"
"github.com/seaweedfs/seaweedfs/weed/pb"
"github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
@@ -202,6 +203,14 @@ func (m *Mover) DeleteEcShards(ctx context.Context, volumeId needle.VolumeId, co
})
}
// IsNoSpaceLeftError matches the volume server's refusal when no local disk
// can take the copied shard ("no space left" from FindEcShardTargetLocation,
// an ENOSPC writing to a caller-pinned disk, and the Rust server's same
// reply), however the move chain wraps it.
func IsNoSpaceLeftError(err error) bool {
return err != nil && strings.Contains(err.Error(), "no space left")
}
// RemoveEcShards unmounts then deletes shards in place — the dedup path for a
// shard that already has a copy elsewhere.
func (m *Mover) RemoveEcShards(ctx context.Context, volumeId needle.VolumeId, collection string, server pb.ServerAddress, shardIds []erasure_coding.ShardId) error {
+10
View File
@@ -305,6 +305,11 @@ message VolumeGrowRequest {
string data_node = 7;
uint32 memory_map_max_size_mb = 8;
string disk_type = 9;
// If > 0, grow only when the average fill ratio of the layout's writable
// volumes has reached this threshold (e.g. 0.6). Lets callers pre-grow on
// fill level with one call, instead of listing volumes to compute the
// ratio themselves. Ignored when there are no writable volumes.
float grow_avg_fill_threshold = 10;
}
message AssignResponse {
@@ -611,4 +616,9 @@ message RaftLeadershipTransferResponse {
}
message VolumeGrowResponse {
// Average fill ratio of the layout's writable volumes at request time
// (0 when there are none). Reported whether or not growth was triggered.
float avg_fill = 1;
// Number of writable volumes at request time.
uint32 writable_count = 2;
}
+42 -7
View File
@@ -1739,8 +1739,13 @@ type VolumeGrowRequest struct {
DataNode string `protobuf:"bytes,7,opt,name=data_node,json=dataNode,proto3" json:"data_node,omitempty"`
MemoryMapMaxSizeMb uint32 `protobuf:"varint,8,opt,name=memory_map_max_size_mb,json=memoryMapMaxSizeMb,proto3" json:"memory_map_max_size_mb,omitempty"`
DiskType string `protobuf:"bytes,9,opt,name=disk_type,json=diskType,proto3" json:"disk_type,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
// If > 0, grow only when the average fill ratio of the layout's writable
// volumes has reached this threshold (e.g. 0.6). Lets callers pre-grow on
// fill level with one call, instead of listing volumes to compute the
// ratio themselves. Ignored when there are no writable volumes.
GrowAvgFillThreshold float32 `protobuf:"fixed32,10,opt,name=grow_avg_fill_threshold,json=growAvgFillThreshold,proto3" json:"grow_avg_fill_threshold,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *VolumeGrowRequest) Reset() {
@@ -1836,6 +1841,13 @@ func (x *VolumeGrowRequest) GetDiskType() string {
return ""
}
func (x *VolumeGrowRequest) GetGrowAvgFillThreshold() float32 {
if x != nil {
return x.GrowAvgFillThreshold
}
return 0
}
type AssignResponse struct {
state protoimpl.MessageState `protogen:"open.v1"`
Fid string `protobuf:"bytes,1,opt,name=fid,proto3" json:"fid,omitempty"`
@@ -4659,7 +4671,12 @@ func (x *RaftLeadershipTransferResponse) GetNewLeader() string {
}
type VolumeGrowResponse struct {
state protoimpl.MessageState `protogen:"open.v1"`
state protoimpl.MessageState `protogen:"open.v1"`
// Average fill ratio of the layout's writable volumes at request time
// (0 when there are none). Reported whether or not growth was triggered.
AvgFill float32 `protobuf:"fixed32,1,opt,name=avg_fill,json=avgFill,proto3" json:"avg_fill,omitempty"`
// Number of writable volumes at request time.
WritableCount uint32 `protobuf:"varint,2,opt,name=writable_count,json=writableCount,proto3" json:"writable_count,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
@@ -4694,6 +4711,20 @@ func (*VolumeGrowResponse) Descriptor() ([]byte, []int) {
return file_master_proto_rawDescGZIP(), []int{68}
}
func (x *VolumeGrowResponse) GetAvgFill() float32 {
if x != nil {
return x.AvgFill
}
return 0
}
func (x *VolumeGrowResponse) GetWritableCount() uint32 {
if x != nil {
return x.WritableCount
}
return 0
}
type SuperBlockExtra_ErasureCoding struct {
state protoimpl.MessageState `protogen:"open.v1"`
Data uint32 `protobuf:"varint,1,opt,name=data,proto3" json:"data,omitempty"`
@@ -5229,7 +5260,7 @@ const file_master_proto_rawDesc = "" +
"\x15writable_volume_count\x18\t \x01(\rR\x13writableVolumeCount\x12\x1b\n" +
"\tdisk_type\x18\n" +
" \x01(\tR\bdiskType\x12,\n" +
"\x12expected_data_size\x18\v \x01(\x04R\x10expectedDataSize\"\xbe\x02\n" +
"\x12expected_data_size\x18\v \x01(\x04R\x10expectedDataSize\"\xf5\x02\n" +
"\x11VolumeGrowRequest\x122\n" +
"\x15writable_volume_count\x18\x01 \x01(\rR\x13writableVolumeCount\x12 \n" +
"\vreplication\x18\x02 \x01(\tR\vreplication\x12\x1e\n" +
@@ -5242,7 +5273,9 @@ const file_master_proto_rawDesc = "" +
"\x04rack\x18\x06 \x01(\tR\x04rack\x12\x1b\n" +
"\tdata_node\x18\a \x01(\tR\bdataNode\x122\n" +
"\x16memory_map_max_size_mb\x18\b \x01(\rR\x12memoryMapMaxSizeMb\x12\x1b\n" +
"\tdisk_type\x18\t \x01(\tR\bdiskType\"\xc4\x01\n" +
"\tdisk_type\x18\t \x01(\tR\bdiskType\x125\n" +
"\x17grow_avg_fill_threshold\x18\n" +
" \x01(\x02R\x14growAvgFillThreshold\"\xc4\x01\n" +
"\x0eAssignResponse\x12\x10\n" +
"\x03fid\x18\x01 \x01(\tR\x03fid\x12\x14\n" +
"\x05count\x18\x04 \x01(\x04R\x05count\x12\x14\n" +
@@ -5477,8 +5510,10 @@ const file_master_proto_rawDesc = "" +
"\x1eRaftLeadershipTransferResponse\x12'\n" +
"\x0fprevious_leader\x18\x01 \x01(\tR\x0epreviousLeader\x12\x1d\n" +
"\n" +
"new_leader\x18\x02 \x01(\tR\tnewLeader\"\x14\n" +
"\x12VolumeGrowResponse2\xf1\x12\n" +
"new_leader\x18\x02 \x01(\tR\tnewLeader\"V\n" +
"\x12VolumeGrowResponse\x12\x19\n" +
"\bavg_fill\x18\x01 \x01(\x02R\aavgFill\x12%\n" +
"\x0ewritable_count\x18\x02 \x01(\rR\rwritableCount2\xf1\x12\n" +
"\aSeaweed\x12I\n" +
"\rSendHeartbeat\x12\x14.master_pb.Heartbeat\x1a\x1c.master_pb.HeartbeatResponse\"\x00(\x010\x01\x12X\n" +
"\rKeepConnected\x12\x1f.master_pb.KeepConnectedRequest\x1a .master_pb.KeepConnectedResponse\"\x00(\x010\x01\x12Q\n" +
+3
View File
@@ -37,6 +37,9 @@ type HandlerBuildOptions struct {
GrpcDialOption grpc.DialOption
MaxExecute int
WorkingDir string
// WorkerAddress is the advertised weed worker -address (host:port).
// It is not a volume-server URL; handlers match replicas by hostname.
WorkerAddress string
}
var (
+8
View File
@@ -110,6 +110,14 @@ type RemoteStorageStreamReader interface {
ReadFileAsStream(ctx context.Context, loc *remote_pb.RemoteStorageLocation, offset int64, size int64) (reader io.ReadCloser, err error)
}
// RemoteStorageObjectCopier is an optional interface for remote storage clients
// that copy an object to another key within the storage, without the content
// passing through the client. A rename of an entry whose content exists only
// on the remote needs it.
type RemoteStorageObjectCopier interface {
CopyFile(src *remote_pb.RemoteStorageLocation, dst *remote_pb.RemoteStorageLocation) (remoteEntry *filer_pb.RemoteEntry, err error)
}
// CacheWaitTimeout is how long a read of an uncached remote-only object waits
// for the local cache before serving another way: small files wait longer since
// their cache completes quickly, large files fail fast for better TTFB. A mount
+163 -1
View File
@@ -610,7 +610,7 @@ func (s *s3RemoteStorageClient) UpdateFileMetadata(loc *remote_pb.RemoteStorageL
copyInput := &s3.CopyObjectInput{
Bucket: aws.String(loc.Bucket),
Key: aws.String(key),
CopySource: aws.String(url.PathEscape(loc.Bucket + "/" + key)),
CopySource: aws.String(s3CopySource(loc.Bucket, key)),
MetadataDirective: aws.String(s3.MetadataDirectiveReplace),
Metadata: headOut.Metadata,
ContentType: headOut.ContentType,
@@ -663,6 +663,168 @@ func (s *s3RemoteStorageClient) UpdateFileMetadata(loc *remote_pb.RemoteStorageL
}
return
}
// s3CopyPartSize is the smallest part a multipart copy uses; parts grow so
// the largest object stays within the 10000-part limit
const s3CopyPartSize = 512 * 1024 * 1024
// CopyFile copies the object at src to dst on the server side, with a single
// CopyObject up to the 5 GB limit and a multipart copy above it. The copy
// keeps the source's metadata.
func (s *s3RemoteStorageClient) CopyFile(src *remote_pb.RemoteStorageLocation, dst *remote_pb.RemoteStorageLocation) (remoteEntry *filer_pb.RemoteEntry, err error) {
srcKey, dstKey := src.Path[1:], dst.Path[1:]
headOut, err := s.conn.HeadObject(&s3.HeadObjectInput{
Bucket: aws.String(src.Bucket),
Key: aws.String(srcKey),
})
if err != nil {
if reqErr, ok := err.(awserr.RequestFailure); ok && reqErr.StatusCode() == http.StatusNotFound {
return nil, remote_storage.ErrRemoteObjectNotFound
}
return nil, fmt.Errorf("stat s3 %s/%s before copy: %w", src.Bucket, srcKey, err)
}
copySource := aws.String(s3CopySource(src.Bucket, srcKey))
var storageClass *string
if s.conf.S3StorageClass != "" {
storageClass = aws.String(s.conf.S3StorageClass)
}
if size := aws.Int64Value(headOut.ContentLength); size <= s3CopyObjectSizeLimit {
// a copy takes the bucket's default encryption unless told otherwise
if _, err = s.conn.CopyObject(&s3.CopyObjectInput{
Bucket: aws.String(dst.Bucket),
Key: aws.String(dstKey),
CopySource: copySource,
StorageClass: storageClass,
ServerSideEncryption: headOut.ServerSideEncryption,
SSEKMSKeyId: headOut.SSEKMSKeyId,
}); err != nil {
return nil, fmt.Errorf("copy s3 %s/%s to %s/%s: %w", src.Bucket, srcKey, dst.Bucket, dstKey, err)
}
} else if err = s.multipartCopy(headOut, src, copySource, size, dst.Bucket, dstKey, storageClass); err != nil {
return nil, fmt.Errorf("copy s3 %s/%s to %s/%s: %w", src.Bucket, srcKey, dst.Bucket, dstKey, err)
}
return s.readFileRemoteEntry(dst)
}
// s3CopySource escapes each segment of the copy source and keeps the slashes,
// which every S3 implementation reads back
func s3CopySource(bucket, key string) string {
segments := strings.Split(bucket+"/"+key, "/")
for i, segment := range segments {
segments[i] = url.PathEscape(segment)
}
return strings.Join(segments, "/")
}
// s3TaggingHeader encodes tags as the query string x-amz-tagging takes, with
// spaces as %20: a + is read back as a space by some implementations and
// literally by others
func s3TaggingHeader(tagSet []*s3.Tag) string {
escape := func(s string) string {
return strings.ReplaceAll(url.QueryEscape(s), "+", "%20")
}
pairs := make([]string, 0, len(tagSet))
for _, tag := range tagSet {
pairs = append(pairs, escape(aws.StringValue(tag.Key))+"="+escape(aws.StringValue(tag.Value)))
}
return strings.Join(pairs, "&")
}
// multipartCopy copies an object above the CopyObject limit part by part. A
// multipart upload does not inherit the source's metadata or tags, so they are
// carried over like CopyObject would: metadata from the source's HeadObject,
// tags when the remote supports tagging.
func (s *s3RemoteStorageClient) multipartCopy(headOut *s3.HeadObjectOutput, src *remote_pb.RemoteStorageLocation, copySource *string, size int64, bucket, key string, storageClass *string) error {
createInput := &s3.CreateMultipartUploadInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
Metadata: headOut.Metadata,
ContentType: headOut.ContentType,
ContentEncoding: headOut.ContentEncoding,
CacheControl: headOut.CacheControl,
ContentDisposition: headOut.ContentDisposition,
ContentLanguage: headOut.ContentLanguage,
WebsiteRedirectLocation: headOut.WebsiteRedirectLocation,
ServerSideEncryption: headOut.ServerSideEncryption,
SSEKMSKeyId: headOut.SSEKMSKeyId,
StorageClass: storageClass,
}
if headOut.Expires != nil {
if expires, parseErr := http.ParseTime(*headOut.Expires); parseErr == nil {
createInput.Expires = aws.Time(expires)
}
}
if s.conf.S3SupportTagging {
tagOut, tagErr := s.conn.GetObjectTagging(&s3.GetObjectTaggingInput{
Bucket: aws.String(src.Bucket),
Key: aws.String(src.Path[1:]),
})
if tagErr != nil {
return fmt.Errorf("read tags: %w", tagErr)
}
if len(tagOut.TagSet) > 0 {
createInput.Tagging = aws.String(s3TaggingHeader(tagOut.TagSet))
}
}
createOut, err := s.conn.CreateMultipartUpload(createInput)
if err != nil {
return err
}
partSize := int64(s3CopyPartSize)
for partSize*10000 < size {
partSize *= 2
}
var parts []*s3.CompletedPart
for offset, partNumber := int64(0), int64(1); offset < size; offset, partNumber = offset+partSize, partNumber+1 {
partOut, partErr := s.conn.UploadPartCopy(&s3.UploadPartCopyInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
UploadId: createOut.UploadId,
PartNumber: aws.Int64(partNumber),
CopySource: copySource,
CopySourceRange: aws.String(fmt.Sprintf("bytes=%d-%d", offset, min(offset+partSize, size)-1)),
// every part from the object the HeadObject saw, not a mix of
// versions if it is overwritten mid-copy
CopySourceIfMatch: headOut.ETag,
})
if partErr == nil && partOut.CopyPartResult == nil {
partErr = errors.New("no part result")
}
if partErr != nil {
s.abortMultipartUpload(bucket, key, createOut.UploadId)
return fmt.Errorf("copy part %d: %w", partNumber, partErr)
}
parts = append(parts, &s3.CompletedPart{
ETag: partOut.CopyPartResult.ETag,
PartNumber: aws.Int64(partNumber),
})
}
if _, err = s.conn.CompleteMultipartUpload(&s3.CompleteMultipartUploadInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
UploadId: createOut.UploadId,
MultipartUpload: &s3.CompletedMultipartUpload{Parts: parts},
}); err != nil {
s.abortMultipartUpload(bucket, key, createOut.UploadId)
return err
}
return nil
}
func (s *s3RemoteStorageClient) abortMultipartUpload(bucket, key string, uploadId *string) {
if _, err := s.conn.AbortMultipartUpload(&s3.AbortMultipartUploadInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
UploadId: uploadId,
}); err != nil {
glog.Warningf("abort multipart copy to s3 %s/%s: %v", bucket, key, err)
}
}
func (s *s3RemoteStorageClient) DeleteFile(loc *remote_pb.RemoteStorageLocation) (err error) {
_, err = s.conn.DeleteObject(&s3.DeleteObjectInput{
Bucket: aws.String(loc.Bucket),
@@ -0,0 +1,169 @@
package s3
import (
"errors"
"net/http"
"testing"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/awserr"
awss3 "github.com/aws/aws-sdk-go/service/s3"
"github.com/aws/aws-sdk-go/service/s3/s3iface"
"github.com/seaweedfs/seaweedfs/weed/pb/remote_pb"
"github.com/seaweedfs/seaweedfs/weed/remote_storage"
"github.com/stretchr/testify/require"
)
// copyObjectMock serves HeadObject for a source of size bytes and records the
// copy calls.
type copyObjectMock struct {
s3iface.S3API
size int64
missing bool
partErr error
copyInputs []*awss3.CopyObjectInput
partInputs []*awss3.UploadPartCopyInput
createInput *awss3.CreateMultipartUploadInput
tagInputs []*awss3.GetObjectTaggingInput
completed *awss3.CompleteMultipartUploadInput
aborted bool
}
func (m *copyObjectMock) HeadObject(input *awss3.HeadObjectInput) (*awss3.HeadObjectOutput, error) {
if m.missing {
return nil, awserr.NewRequestFailure(awserr.New("NotFound", "not found", nil), http.StatusNotFound, "")
}
return &awss3.HeadObjectOutput{
ContentLength: aws.Int64(m.size),
ContentType: aws.String("image/jpeg"),
Expires: aws.String("Wed, 21 Oct 2026 07:28:00 GMT"),
WebsiteRedirectLocation: aws.String("/elsewhere"),
ServerSideEncryption: aws.String("aws:kms"),
SSEKMSKeyId: aws.String("key-1"),
ETag: aws.String(`"etag-` + aws.StringValue(input.Key) + `"`),
}, nil
}
func (m *copyObjectMock) GetObjectTagging(input *awss3.GetObjectTaggingInput) (*awss3.GetObjectTaggingOutput, error) {
m.tagInputs = append(m.tagInputs, input)
return &awss3.GetObjectTaggingOutput{TagSet: []*awss3.Tag{
{Key: aws.String("album"), Value: aws.String("2026 & co")},
{Key: aws.String("a+b"), Value: aws.String("x=y")},
}}, nil
}
func (m *copyObjectMock) CopyObject(input *awss3.CopyObjectInput) (*awss3.CopyObjectOutput, error) {
m.copyInputs = append(m.copyInputs, input)
return &awss3.CopyObjectOutput{}, nil
}
func (m *copyObjectMock) CreateMultipartUpload(input *awss3.CreateMultipartUploadInput) (*awss3.CreateMultipartUploadOutput, error) {
m.createInput = input
return &awss3.CreateMultipartUploadOutput{UploadId: aws.String("upload-1")}, nil
}
func (m *copyObjectMock) UploadPartCopy(input *awss3.UploadPartCopyInput) (*awss3.UploadPartCopyOutput, error) {
m.partInputs = append(m.partInputs, input)
if m.partErr != nil && len(m.partInputs) == 2 {
return nil, m.partErr
}
return &awss3.UploadPartCopyOutput{CopyPartResult: &awss3.CopyPartResult{ETag: aws.String(`"part"`)}}, nil
}
func (m *copyObjectMock) CompleteMultipartUpload(input *awss3.CompleteMultipartUploadInput) (*awss3.CompleteMultipartUploadOutput, error) {
m.completed = input
return &awss3.CompleteMultipartUploadOutput{}, nil
}
func (m *copyObjectMock) AbortMultipartUpload(*awss3.AbortMultipartUploadInput) (*awss3.AbortMultipartUploadOutput, error) {
m.aborted = true
return &awss3.AbortMultipartUploadOutput{}, nil
}
func newCopyTestClient(mock *copyObjectMock) *s3RemoteStorageClient {
return &s3RemoteStorageClient{conf: &remote_pb.RemoteConf{Name: "test"}, conn: mock}
}
var (
copySrc = &remote_pb.RemoteStorageLocation{Name: "test", Bucket: "bucket", Path: "/src/a b.jpg"}
copyDst = &remote_pb.RemoteStorageLocation{Name: "test", Bucket: "bucket", Path: "/dst/a b.jpg"}
)
func TestS3ClientImplementsObjectCopier(t *testing.T) {
var _ remote_storage.RemoteStorageObjectCopier = &s3RemoteStorageClient{}
}
func TestS3CopyFileUsesCopyObjectUpToTheLimit(t *testing.T) {
mock := &copyObjectMock{size: s3CopyObjectSizeLimit}
remoteEntry, err := newCopyTestClient(mock).CopyFile(copySrc, copyDst)
require.NoError(t, err)
require.Len(t, mock.copyInputs, 1)
require.Equal(t, "bucket/src/a%20b.jpg", aws.StringValue(mock.copyInputs[0].CopySource))
require.Equal(t, "dst/a b.jpg", aws.StringValue(mock.copyInputs[0].Key))
require.Equal(t, "aws:kms", aws.StringValue(mock.copyInputs[0].ServerSideEncryption))
require.Equal(t, "key-1", aws.StringValue(mock.copyInputs[0].SSEKMSKeyId))
require.Nil(t, mock.createInput)
// the RemoteEntry describes the destination
require.Equal(t, `"etag-dst/a b.jpg"`, remoteEntry.RemoteETag)
require.Equal(t, int64(s3CopyObjectSizeLimit), remoteEntry.RemoteSize)
}
func TestS3CopyFileCopiesLargeObjectsInParts(t *testing.T) {
size := int64(s3CopyObjectSizeLimit) + 1
mock := &copyObjectMock{size: size}
_, err := newCopyTestClient(mock).CopyFile(copySrc, copyDst)
require.NoError(t, err)
require.Empty(t, mock.copyInputs)
require.NotNil(t, mock.createInput)
require.Equal(t, "image/jpeg", aws.StringValue(mock.createInput.ContentType))
require.Equal(t, "/elsewhere", aws.StringValue(mock.createInput.WebsiteRedirectLocation))
require.Equal(t, "aws:kms", aws.StringValue(mock.createInput.ServerSideEncryption))
require.Equal(t, "key-1", aws.StringValue(mock.createInput.SSEKMSKeyId))
require.Equal(t, int64(1792567680), aws.TimeValue(mock.createInput.Expires).Unix())
// tags only where the remote supports tagging
require.Empty(t, mock.tagInputs)
require.Nil(t, mock.createInput.Tagging)
require.Len(t, mock.partInputs, 11)
require.Equal(t, "bytes=0-536870911", aws.StringValue(mock.partInputs[0].CopySourceRange))
require.Equal(t, "bytes=5368709120-5368709120", aws.StringValue(mock.partInputs[10].CopySourceRange))
for _, part := range mock.partInputs {
require.Equal(t, `"etag-src/a b.jpg"`, aws.StringValue(part.CopySourceIfMatch))
}
require.Len(t, mock.completed.MultipartUpload.Parts, 11)
require.Equal(t, int64(11), aws.Int64Value(mock.completed.MultipartUpload.Parts[10].PartNumber))
require.False(t, mock.aborted)
}
func TestS3CopyFileCarriesTagsIntoAMultipartCopy(t *testing.T) {
mock := &copyObjectMock{size: s3CopyObjectSizeLimit + 1}
client := newCopyTestClient(mock)
client.conf.S3SupportTagging = true
_, err := client.CopyFile(copySrc, copyDst)
require.NoError(t, err)
require.Len(t, mock.tagInputs, 1)
require.Equal(t, "src/a b.jpg", aws.StringValue(mock.tagInputs[0].Key))
require.Equal(t, "album=2026%20%26%20co&a%2Bb=x%3Dy", aws.StringValue(mock.createInput.Tagging))
}
func TestS3CopyFileAbortsAFailedMultipartCopy(t *testing.T) {
mock := &copyObjectMock{size: 2 * s3CopyObjectSizeLimit, partErr: errors.New("part failed")}
_, err := newCopyTestClient(mock).CopyFile(copySrc, copyDst)
require.ErrorContains(t, err, "part failed")
require.True(t, mock.aborted)
require.Nil(t, mock.completed)
}
func TestS3CopyFileMissingSource(t *testing.T) {
mock := &copyObjectMock{missing: true}
_, err := newCopyTestClient(mock).CopyFile(copySrc, copyDst)
require.ErrorIs(t, err, remote_storage.ErrRemoteObjectNotFound)
require.Empty(t, mock.copyInputs)
}
+2
View File
@@ -176,6 +176,8 @@ func TestBuildBucketMetadata(t *testing.T) {
}
func TestGetBucketMetadata(t *testing.T) {
original := loadBucketMetadataFromFiler
t.Cleanup(func() { loadBucketMetadataFromFiler = original })
loadBucketMetadataFromFiler = func(r *BucketRegistry, bucketName string) (*BucketMetaData, error) {
time.Sleep(time.Second)
loadFilerBucketLock.Lock()
+14 -1
View File
@@ -162,7 +162,7 @@ func TestBucketHasUserObjects_EmptyDirectories(t *testing.T) {
wantHasUser: false,
},
{
name: "nested versions-named directories are internal",
name: "nested versions directory with stored versions counts",
entriesByDir: map[string][]*filer_pb.Entry{
"/buckets/b": {
{Name: "logs", IsDirectory: true},
@@ -174,6 +174,19 @@ func TestBucketHasUserObjects_EmptyDirectories(t *testing.T) {
{Name: "v1", IsDirectory: false},
},
},
wantHasUser: true,
},
{
name: "nested versions directory left empty does not count",
entriesByDir: map[string][]*filer_pb.Entry{
"/buckets/b": {
{Name: "logs", IsDirectory: true},
},
"/buckets/b/logs": {
{Name: "foo" + s3_constants.VersionsFolder, IsDirectory: true},
},
"/buckets/b/logs/foo" + s3_constants.VersionsFolder: {},
},
wantHasUser: false,
},
{
+7 -5
View File
@@ -521,8 +521,9 @@ func (s3a *S3ApiServer) DeleteBucketHandler(w http.ResponseWriter, r *http.Reque
}
// bucketHasUserObjects checks whether a bucket contains any user objects.
// Empty directories left behind by deleted objects and internal folders
// (.uploads, *.versions) do not count as user objects.
// Empty directories left behind by deleted objects and the .uploads staging
// folder do not count as user objects; stored versions inside a <key>.versions
// directory do.
func (s3a *S3ApiServer) bucketHasUserObjects(bucket string) (bool, error) {
return s3a.dirHasUserObjects(s3a.option.BucketsPath + "/" + bucket)
}
@@ -556,9 +557,10 @@ func (s3a *S3ApiServer) dirHasUserObjects(root string) (bool, error) {
if entry.IsDirectoryKeyObject() {
return true, nil
}
// Internal folders are skipped by object listing at every level,
// so a directory with a reserved name never counts.
if isReservedDirectoryName(entry.Name) {
// Multipart staging folders are invisible to object listing at
// every level, so they never count. A <key>.versions directory
// is descended into since it may still hold stored versions.
if entry.Name == s3_constants.MultipartUploadsFolder {
continue
}
dirs = append(dirs, dir+"/"+entry.Name)
+2 -1
View File
@@ -96,7 +96,8 @@ func (fs *FilerServer) doCacheRemoteObjectToLocalCluster(ctx context.Context, re
glog.V(1).Infof("CacheRemoteObjectToLocalCluster: caching %s/%s (remote size: %d)", req.Directory, req.Name, entry.Remote.RemoteSize)
storageConf, remoteLocation, err := fs.resolveMountedRemote(ctx, req.Directory, req.Name)
readDir, readName := fs.filer.RemoteReadPath(entry, req.Directory, req.Name)
storageConf, remoteLocation, err := fs.resolveMountedRemote(ctx, readDir, readName)
if err != nil {
return nil, err
}
+188 -15
View File
@@ -2,7 +2,9 @@ package weed_server
import (
"context"
"errors"
"fmt"
"maps"
"path/filepath"
"time"
@@ -45,6 +47,9 @@ func (fs *FilerServer) AtomicRenameEntry(ctx context.Context, req *filer_pb.Atom
defer fs.acquireRenamePathLocks("AtomicRenameEntry", oldParent.Child(req.OldName), newParent.Child(req.NewName))()
// a SQL store keeps the transaction in the context and refuses calls
// through it once committed; the work after the commit uses this one
afterCommitCtx := ctx
ctx, err := fs.filer.BeginTransaction(ctx)
if err != nil {
return nil, err
@@ -58,7 +63,8 @@ func (fs *FilerServer) AtomicRenameEntry(ctx context.Context, req *filer_pb.Atom
var metadataEvents []metadataEvent
var pendingChunkDeletes []*filer_pb.FileChunk
moveErr := fs.moveEntry(ctx, nil, oldParent, oldEntry, newParent, req.NewName, req.Signatures, false, &metadataEvents, &pendingChunkDeletes)
var pendingRemoteCopies []*pendingRemoteCopy
moveErr := fs.moveEntry(ctx, nil, oldParent, oldEntry, newParent, req.NewName, req.Signatures, false, &metadataEvents, &pendingChunkDeletes, &pendingRemoteCopies)
if moveErr != nil {
fs.filer.RollbackTransaction(ctx)
return nil, fmt.Errorf("%s/%s move error: %v", req.OldDirectory, req.OldName, moveErr)
@@ -75,8 +81,10 @@ func (fs *FilerServer) AtomicRenameEntry(ctx context.Context, req *filer_pb.Atom
if len(pendingChunkDeletes) > 0 {
fs.filer.DeleteChunksNotRecursive(pendingChunkDeletes)
}
fs.finishRemoteCopies(afterCommitCtx, pendingRemoteCopies)
applyRemoteCopies(metadataEvents, pendingRemoteCopies)
for _, event := range metadataEvents {
event.notify(fs.filer, ctx, req.Signatures)
event.notify(fs.filer, afterCommitCtx, req.Signatures)
}
return &filer_pb.AtomicRenameEntryResponse{}, nil
@@ -97,6 +105,7 @@ func (fs *FilerServer) StreamRenameEntry(req *filer_pb.StreamRenameEntryRequest,
ctx := context.Background()
afterCommitCtx := ctx
ctx, err = fs.filer.BeginTransaction(ctx)
if err != nil {
return err
@@ -125,7 +134,8 @@ func (fs *FilerServer) StreamRenameEntry(req *filer_pb.StreamRenameEntryRequest,
var metadataEvents []metadataEvent
var pendingChunkDeletes []*filer_pb.FileChunk
moveErr := fs.moveEntry(ctx, stream, oldParent, oldEntry, newParent, req.NewName, req.Signatures, false, &metadataEvents, &pendingChunkDeletes)
var pendingRemoteCopies []*pendingRemoteCopy
moveErr := fs.moveEntry(ctx, stream, oldParent, oldEntry, newParent, req.NewName, req.Signatures, false, &metadataEvents, &pendingChunkDeletes, &pendingRemoteCopies)
if moveErr != nil {
fs.filer.RollbackTransaction(ctx)
return fmt.Errorf("%s/%s move error: %v", req.OldDirectory, req.OldName, moveErr)
@@ -138,8 +148,10 @@ func (fs *FilerServer) StreamRenameEntry(req *filer_pb.StreamRenameEntryRequest,
if len(pendingChunkDeletes) > 0 {
fs.filer.DeleteChunksNotRecursive(pendingChunkDeletes)
}
fs.finishRemoteCopies(afterCommitCtx, pendingRemoteCopies)
applyRemoteCopies(metadataEvents, pendingRemoteCopies)
for _, event := range metadataEvents {
event.notify(fs.filer, ctx, req.Signatures)
event.notify(fs.filer, afterCommitCtx, req.Signatures)
}
return nil
@@ -149,29 +161,119 @@ type metadataEvent struct {
oldEntry *filer.Entry
newEntry *filer.Entry
deleteChunks bool
// the old entry's remote object is deleted once the rename has committed:
// it was copied to the new key, and a rename that fails must not lose it
deleteOldRemote bool
}
// pendingRemoteCopy is a remote-only rename over a remote-only target whose
// copy runs after the commit: the store transaction cannot roll a remote
// write back, so the new entry keeps the old key's stamp until the copy lands.
type pendingRemoteCopy struct {
source *filer.Entry
oldPath util.FullPath
newPath util.FullPath
failed bool
copied *filer_pb.RemoteEntry
}
// finishRemoteCopies runs the deferred remote copies after the commit. A
// failure leaves the new entry pointing at the old key, whose object the
// caller then keeps: remote.sync completes the copy the same way it does a
// rename whose event it finds unfinished.
func (fs *FilerServer) finishRemoteCopies(ctx context.Context, pendings []*pendingRemoteCopy) {
for _, pending := range pendings {
remoteEntry, err := fs.filer.CopyRemoteOnlyEntry(ctx, pending.source, pending.oldPath, pending.newPath)
if err != nil {
glog.WarningfCtx(ctx, "remote rename copy of %s: %v", pending.newPath, err)
pending.failed = true
continue
}
entry, findErr := fs.filer.FindEntry(ctx, pending.newPath)
if findErr != nil {
glog.WarningfCtx(ctx, "stamp remote rename copy of %s: %v", pending.newPath, findErr)
pending.failed = true
continue
}
entry.Remote = remoteEntry
delete(entry.Extended, filer.ExtRemoteRenameSourceKey)
if updateErr := fs.filer.UpdateEntry(ctx, entry, entry, false); updateErr != nil {
glog.WarningfCtx(ctx, "stamp remote rename copy of %s: %v", pending.newPath, updateErr)
pending.failed = true
continue
}
pending.copied = remoteEntry
}
}
// applyRemoteCopies brings the queued rename events in line with the deferred
// copies. The stamp is written with a plain UpdateEntry, which publishes no
// event, so the rename event carries it, without the pending-source marker,
// to peer filers and subscribers. A failed copy keeps the old object, which
// the entry still reads through the marker.
func applyRemoteCopies(events []metadataEvent, pendings []*pendingRemoteCopy) {
for i := range events {
event := &events[i]
for _, pending := range pendings {
if event.oldEntry.FullPath != pending.oldPath {
continue
}
if pending.failed {
event.deleteOldRemote = false
} else if pending.copied != nil && event.newEntry != nil {
newEntry := event.newEntry.ShallowClone()
newEntry.Remote = pending.copied
newEntry.Extended = maps.Clone(newEntry.Extended)
delete(newEntry.Extended, filer.ExtRemoteRenameSourceKey)
event.newEntry = newEntry
}
}
}
}
func (event metadataEvent) notify(f *filer.Filer, ctx context.Context, signatures []int32) {
if event.deleteOldRemote {
deleteRenamedRemoteObject(f, ctx, event.oldEntry)
}
f.NotifyUpdateEvent(ctx, event.oldEntry, event.newEntry, event.deleteChunks, false, signatures)
}
func (fs *FilerServer) moveEntry(ctx context.Context, stream filer_pb.SeaweedFiler_StreamRenameEntryServer, oldParent util.FullPath, entry *filer.Entry, newParent util.FullPath, newName string, signatures []int32, skipTargetLookup bool, metadataEvents *[]metadataEvent, pendingChunkDeletes *[]*filer_pb.FileChunk) error {
// deleteRenamedRemoteObject deletes the remote object a committed rename no
// longer needs. A child path is not locked by its directory's rename, so
// another rename may have put a remote-only entry at the key meanwhile; the
// store is asked directly, a lazy fetch would revive the key. When in doubt
// the object stays: an extra object loses nothing.
func deleteRenamedRemoteObject(f *filer.Filer, ctx context.Context, entry *filer.Entry) {
current, findErr := f.Store.FindEntry(ctx, entry.FullPath)
switch {
case findErr != nil && !errors.Is(findErr, filer_pb.ErrNotFound):
glog.WarningfCtx(ctx, "keep remote object of renamed %s: look up its old path: %v", entry.FullPath, findErr)
case findErr == nil && current.IsInRemoteOnly():
glog.V(0).InfofCtx(ctx, "keep remote object of renamed %s: a remote-only entry has taken its path", entry.FullPath)
default:
if err := f.DeleteFromRemote(ctx, entry); err != nil {
glog.WarningfCtx(ctx, "delete remote object of renamed %s: %v", entry.FullPath, err)
}
}
}
func (fs *FilerServer) moveEntry(ctx context.Context, stream filer_pb.SeaweedFiler_StreamRenameEntryServer, oldParent util.FullPath, entry *filer.Entry, newParent util.FullPath, newName string, signatures []int32, skipTargetLookup bool, metadataEvents *[]metadataEvent, pendingChunkDeletes *[]*filer_pb.FileChunk, pendingRemoteCopies *[]*pendingRemoteCopy) error {
if err := fs.moveSelfEntry(ctx, stream, oldParent, entry, newParent, newName, func() error {
if entry.IsDirectory() {
if err := fs.moveFolderSubEntries(ctx, stream, oldParent, entry, newParent, newName, signatures, metadataEvents, pendingChunkDeletes); err != nil {
if err := fs.moveFolderSubEntries(ctx, stream, oldParent, entry, newParent, newName, signatures, metadataEvents, pendingChunkDeletes, pendingRemoteCopies); err != nil {
return err
}
}
return nil
}, signatures, skipTargetLookup, metadataEvents, pendingChunkDeletes); err != nil {
}, signatures, skipTargetLookup, metadataEvents, pendingChunkDeletes, pendingRemoteCopies); err != nil {
return fmt.Errorf("fail to move %s => %s: %v", oldParent.Child(entry.Name()), newParent.Child(newName), err)
}
return nil
}
func (fs *FilerServer) moveFolderSubEntries(ctx context.Context, stream filer_pb.SeaweedFiler_StreamRenameEntryServer, oldParent util.FullPath, entry *filer.Entry, newParent util.FullPath, newName string, signatures []int32, metadataEvents *[]metadataEvent, pendingChunkDeletes *[]*filer_pb.FileChunk) error {
func (fs *FilerServer) moveFolderSubEntries(ctx context.Context, stream filer_pb.SeaweedFiler_StreamRenameEntryServer, oldParent util.FullPath, entry *filer.Entry, newParent util.FullPath, newName string, signatures []int32, metadataEvents *[]metadataEvent, pendingChunkDeletes *[]*filer_pb.FileChunk, pendingRemoteCopies *[]*pendingRemoteCopy) error {
currentDirPath := oldParent.Child(entry.Name())
newDirPath := newParent.Child(newName)
@@ -194,7 +296,7 @@ func (fs *FilerServer) moveFolderSubEntries(ctx context.Context, stream filer_pb
// println("processing", lastFileName)
newChildPath := newDirPath.Child(item.Name())
skipTarget := fs.filer.Store.SameActualStore(newDirPath, newChildPath)
err := fs.moveEntry(ctx, stream, currentDirPath, item, newDirPath, item.Name(), signatures, skipTarget, metadataEvents, pendingChunkDeletes)
err := fs.moveEntry(ctx, stream, currentDirPath, item, newDirPath, item.Name(), signatures, skipTarget, metadataEvents, pendingChunkDeletes, pendingRemoteCopies)
if err != nil {
return err
}
@@ -206,7 +308,7 @@ func (fs *FilerServer) moveFolderSubEntries(ctx context.Context, stream filer_pb
return nil
}
func (fs *FilerServer) moveSelfEntry(ctx context.Context, stream filer_pb.SeaweedFiler_StreamRenameEntryServer, oldParent util.FullPath, entry *filer.Entry, newParent util.FullPath, newName string, moveFolderSubEntries func() error, signatures []int32, skipTargetLookup bool, metadataEvents *[]metadataEvent, pendingChunkDeletes *[]*filer_pb.FileChunk) error {
func (fs *FilerServer) moveSelfEntry(ctx context.Context, stream filer_pb.SeaweedFiler_StreamRenameEntryServer, oldParent util.FullPath, entry *filer.Entry, newParent util.FullPath, newName string, moveFolderSubEntries func() error, signatures []int32, skipTargetLookup bool, metadataEvents *[]metadataEvent, pendingChunkDeletes *[]*filer_pb.FileChunk, pendingRemoteCopies *[]*pendingRemoteCopy) error {
oldPath, newPath := oldParent.Child(entry.Name()), newParent.Child(newName)
@@ -235,8 +337,48 @@ func (fs *FilerServer) moveSelfEntry(ctx context.Context, stream filer_pb.Seawee
case !existingTarget.IsDirectory() && entry.IsDirectory():
return fmt.Errorf("%s: %w", existingTarget.FullPath, filer_pb.ErrExistingIsFile)
}
}
// a remote-only entry is copied on the remote before anything changes:
// deleting the old entry deletes the old object. Over a remote-only
// target the copy runs after the commit instead — it overwrites the
// target's only content, which a rolled-back commit cannot restore —
// and the new entry reads the old key until the copy stamps it.
deferredCopy := false
var copiedRemote *filer_pb.RemoteEntry
if existingTarget != nil && existingTarget.IsInRemoteOnly() && entry.IsInRemoteOnly() {
if _, _, copier, copyErr := fs.filer.RemoteCopyLocations(ctx, entry, oldPath, newPath); copyErr != nil {
return copyErr
} else if copier != nil {
deferredCopy = true
*pendingRemoteCopies = append(*pendingRemoteCopies, &pendingRemoteCopy{source: sourceEntry, oldPath: oldPath, newPath: newPath})
}
}
if !deferredCopy {
var copyErr error
copiedRemote, copyErr = fs.filer.CopyRemoteOnlyEntry(ctx, entry, oldPath, newPath)
if copyErr != nil {
return copyErr
}
}
remoteEntry := entry.Remote
if copiedRemote != nil {
remoteEntry = copiedRemote
}
if existingTarget != nil {
targetCtx := filer.WithSuppressedMetadataEvents(ctx)
if copiedRemote != nil || deferredCopy {
// the copy has replaced or still needs the target's object; keep it
targetCtx = filer.WithKeepRemoteObject(targetCtx)
existingTarget.Extended = maps.Clone(existingTarget.Extended)
if existingTarget.Extended == nil {
existingTarget.Extended = map[string][]byte{}
}
existingTarget.Extended[filer.ExtKeepRemoteObjectKey] = []byte("true")
}
if deleteErr := fs.filer.DeleteEntryMetaAndData(
filer.WithSuppressedMetadataEvents(ctx),
targetCtx,
newPath,
false,
false,
@@ -258,9 +400,23 @@ func (fs *FilerServer) moveSelfEntry(ctx context.Context, stream filer_pb.Seawee
Content: entry.Content,
HardLinkCounter: entry.HardLinkCounter,
HardLinkId: entry.HardLinkId,
Remote: entry.Remote,
Remote: remoteEntry,
Quota: entry.Quota,
}
if deferredCopy {
// a pending entry renamed again still reads its original source
newEntry.Extended = maps.Clone(newEntry.Extended)
if newEntry.Extended == nil {
newEntry.Extended = map[string][]byte{}
}
newEntry.Extended[filer.ExtRemoteRenameSourceKey] = []byte(fs.filer.RemoteSourcePath(entry, oldPath))
} else if copiedRemote != nil && len(newEntry.Extended[filer.ExtRemoteRenameSourceKey]) > 0 {
// the copy came from the pending source; the new key holds it now.
// The source itself is left to remote.sync, which deletes it only
// while it is still the object the rename event recorded
newEntry.Extended = maps.Clone(newEntry.Extended)
delete(newEntry.Extended, filer.ExtRemoteRenameSourceKey)
}
if skipTargetLookup {
if newEntry.FullPath.IsLongerFileName(fs.filer.MaxFilenameLength) {
return filer_pb.ErrEntryNameTooLong
@@ -311,19 +467,36 @@ func (fs *FilerServer) moveSelfEntry(ctx context.Context, stream filer_pb.Seawee
})
}
*metadataEvents = append(*metadataEvents, metadataEvent{
oldEntry: sourceEntry,
newEntry: newEntry,
oldEntry: sourceEntry,
newEntry: newEntry,
deleteOldRemote: copiedRemote != nil || deferredCopy,
})
selfEvent := len(*metadataEvents) - 1
keepOldRemote := copiedRemote != nil || deferredCopy
if moveFolderSubEntries != nil {
if moveChildrenErr := moveFolderSubEntries(); moveChildrenErr != nil {
return moveChildrenErr
}
// removing the old directory on the remote removes every object under
// its prefix, the copied children's old keys included, which must wait
// for the commit. Those are deleted one by one after it instead, and
// nothing else under the prefix is touched.
for _, event := range (*metadataEvents)[selfEvent+1:] {
if event.deleteOldRemote {
keepOldRemote = true
break
}
}
}
// delete old entry
ctx = context.WithValue(ctx, "OP", "MV")
deleteErr := fs.filer.DeleteEntryMetaAndData(filer.WithSuppressedMetadataEvents(ctx), oldPath, false, false, false, false, signatures, 0)
deleteCtx := filer.WithSuppressedMetadataEvents(ctx)
if keepOldRemote {
deleteCtx = filer.WithKeepRemoteObject(deleteCtx)
}
deleteErr := fs.filer.DeleteEntryMetaAndData(deleteCtx, oldPath, false, false, false, false, signatures, 0)
if deleteErr != nil {
return deleteErr
}
@@ -0,0 +1,430 @@
package weed_server
import (
"context"
"errors"
"net/http"
"net/http/httptest"
"sync"
"testing"
"github.com/seaweedfs/seaweedfs/weed/filer"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
"github.com/seaweedfs/seaweedfs/weed/pb/remote_pb"
"github.com/seaweedfs/seaweedfs/weed/remote_storage"
"github.com/seaweedfs/seaweedfs/weed/util"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"google.golang.org/protobuf/proto"
)
// renameRemoteClient records the remote calls a rename makes. It cannot copy;
// copyingRenameRemoteClient adds CopyFile.
type renameRemoteClient struct {
remote_storage.RemoteStorageClient
mu sync.Mutex
calls []string
copyErr error
}
func (c *renameRemoteClient) record(call string) {
c.mu.Lock()
defer c.mu.Unlock()
c.calls = append(c.calls, call)
}
func (c *renameRemoteClient) recorded() []string {
c.mu.Lock()
defer c.mu.Unlock()
return append([]string(nil), c.calls...)
}
func (c *renameRemoteClient) StatFile(*remote_pb.RemoteStorageLocation) (*filer_pb.RemoteEntry, error) {
return nil, remote_storage.ErrRemoteObjectNotFound
}
func (c *renameRemoteClient) ListDirectory(context.Context, *remote_pb.RemoteStorageLocation, remote_storage.VisitFunc) error {
return nil
}
func (c *renameRemoteClient) DeleteFile(loc *remote_pb.RemoteStorageLocation) error {
c.record("delete " + loc.Bucket + loc.Path)
return nil
}
func (c *renameRemoteClient) RemoveDirectory(loc *remote_pb.RemoteStorageLocation) error {
c.record("rmdir " + loc.Bucket + loc.Path)
return nil
}
type copyingRenameRemoteClient struct {
*renameRemoteClient
}
func (c copyingRenameRemoteClient) CopyFile(src, dst *remote_pb.RemoteStorageLocation) (*filer_pb.RemoteEntry, error) {
if c.copyErr != nil {
return nil, c.copyErr
}
c.record("copy " + src.Bucket + src.Path + " " + dst.Bucket + dst.Path)
return &filer_pb.RemoteEntry{StorageName: "r1", RemoteSize: 10, RemoteMtime: 1800000000, RemoteETag: `"copy"`}, nil
}
// newRenameRemoteTestServer mounts /buckets/b, /buckets/c and /data/m on r1,
// through client.
func newRenameRemoteTestServer(t *testing.T, client remote_storage.RemoteStorageClient) (*FilerServer, *renameTestStore) {
t.Helper()
store := newRenameTestStore()
f := newRenameTestFiler(t, store)
f.BuildGuardedRemoteClient = func(context.Context, *remote_pb.RemoteConf, bool) (remote_storage.RemoteStorageClient, error) {
return client, nil
}
mapping, err := proto.Marshal(&remote_pb.RemoteStorageMapping{Mappings: map[string]*remote_pb.RemoteStorageLocation{
"/buckets/b": {Name: "r1", Bucket: "origin", Path: "/"},
"/buckets/c": {Name: "r1", Bucket: "other", Path: "/"},
"/data/m": {Name: "r1", Bucket: "origin", Path: "/m"},
}})
require.NoError(t, err)
conf, err := proto.Marshal(&remote_pb.RemoteConf{Name: "r1", Type: "s3"})
require.NoError(t, err)
for name, content := range map[string][]byte{filer.REMOTE_STORAGE_MOUNT_FILE: mapping, "r1" + filer.REMOTE_STORAGE_CONF_SUFFIX: conf} {
entry := newFileEntry(filer.DirectoryEtcRemote+"/"+name, 1)
entry.Content = content
store.entries[string(entry.FullPath)] = entry
}
require.NoError(t, f.RemoteStorage.LoadRemoteStorageConfigurationsAndMapping(f))
for _, dir := range []string{"/buckets", "/buckets/b", "/buckets/b/src", "/buckets/b/dst", "/buckets/c", "/data", "/data/m"} {
store.entries[dir] = newDirectoryEntry(dir, 2)
}
return &FilerServer{filer: f, option: &FilerOption{}, entryLockTable: util.NewLockTable[util.FullPath]()}, store
}
func remoteOnlyRenameEntry(path string, inode uint64) *filer.Entry {
entry := newFileEntry(path, inode)
entry.FileSize = 10
entry.Remote = &filer_pb.RemoteEntry{StorageName: "r1", RemoteSize: 10, RemoteMtime: 1700000000, RemoteETag: `"orig"`}
return entry
}
func renameFile(server *FilerServer, oldPath, newPath util.FullPath) error {
oldDir, oldName := oldPath.DirAndName()
newDir, newName := newPath.DirAndName()
_, err := server.AtomicRenameEntry(context.Background(), &filer_pb.AtomicRenameEntryRequest{
OldDirectory: oldDir,
OldName: oldName,
NewDirectory: newDir,
NewName: newName,
})
return err
}
func TestRenameRemoteOnlyEntryCopiesBeforeDeletingTheOldObject(t *testing.T) {
client := copyingRenameRemoteClient{&renameRemoteClient{}}
server, store := newRenameRemoteTestServer(t, client)
store.entries["/buckets/b/src/a.jpg"] = remoteOnlyRenameEntry("/buckets/b/src/a.jpg", 101)
require.NoError(t, renameFile(server, "/buckets/b/src/a.jpg", "/buckets/b/dst/a.jpg"))
assert.Equal(t, []string{"copy origin/src/a.jpg origin/dst/a.jpg", "delete origin/src/a.jpg"}, client.recorded())
_, err := store.FindEntry(context.Background(), "/buckets/b/src/a.jpg")
assert.ErrorIs(t, err, filer_pb.ErrNotFound)
dst, err := store.FindEntry(context.Background(), "/buckets/b/dst/a.jpg")
require.NoError(t, err)
assert.Equal(t, `"copy"`, dst.Remote.RemoteETag)
assert.Equal(t, uint64(101), dst.Attr.Inode)
}
func TestRenameRemoteOnlyEntryOverTargetKeepsTheCopy(t *testing.T) {
client := copyingRenameRemoteClient{&renameRemoteClient{}}
server, store := newRenameRemoteTestServer(t, client)
store.entries["/buckets/b/src/a.jpg"] = remoteOnlyRenameEntry("/buckets/b/src/a.jpg", 101)
store.entries["/buckets/b/dst/a.jpg"] = remoteOnlyRenameEntry("/buckets/b/dst/a.jpg", 202)
queue := &captureQueue{}
swapNotificationQueue(t, queue)
require.NoError(t, renameFile(server, "/buckets/b/src/a.jpg", "/buckets/b/dst/a.jpg"))
// the target's delete must not take the copy that replaced its object
assert.Equal(t, []string{"copy origin/src/a.jpg origin/dst/a.jpg", "delete origin/src/a.jpg"}, client.recorded())
events := queue.snapshot()
require.Len(t, events, 2)
assert.True(t, filer.IsMetadataOnlyDelete(events[0].notification.OldEntry), "the target's delete event must leave the remote object alone")
assert.False(t, filer.IsMetadataOnlyDelete(events[1].notification.OldEntry))
// the copy lands after the commit; the rename event still carries its stamp
assert.Equal(t, `"copy"`, events[1].notification.NewEntry.RemoteEntry.GetRemoteETag())
dst, err := store.FindEntry(context.Background(), "/buckets/b/dst/a.jpg")
require.NoError(t, err)
assert.Equal(t, uint64(101), dst.Attr.Inode)
assert.Equal(t, `"copy"`, dst.Remote.RemoteETag)
assert.Empty(t, dst.Extended[filer.ExtRemoteRenameSourceKey])
// peers replay the post-copy state, not the source's stamp the entry
// committed with
newEvent := events[1].notification.NewEntry
assert.Equal(t, `"copy"`, newEvent.RemoteEntry.RemoteETag)
assert.Empty(t, newEvent.Extended[filer.ExtRemoteRenameSourceKey])
}
func TestRenameRemoteOnlyEntryCopyFailureKeepsTheOldObject(t *testing.T) {
client := copyingRenameRemoteClient{&renameRemoteClient{copyErr: errors.New("copy refused")}}
server, store := newRenameRemoteTestServer(t, client)
store.entries["/buckets/b/src/a.jpg"] = remoteOnlyRenameEntry("/buckets/b/src/a.jpg", 101)
store.entries["/buckets/b/dst/a.jpg"] = remoteOnlyRenameEntry("/buckets/b/dst/a.jpg", 202)
require.NoError(t, renameFile(server, "/buckets/b/src/a.jpg", "/buckets/b/dst/a.jpg"))
// the copy runs after the commit over a remote-only target, so its
// failure cannot roll anything back: the entry still stamps the old
// object, which is kept for remote.sync to finish the copy
assert.Empty(t, client.recorded())
_, err := store.FindEntry(context.Background(), "/buckets/b/src/a.jpg")
assert.ErrorIs(t, err, filer_pb.ErrNotFound)
dst, err := store.FindEntry(context.Background(), "/buckets/b/dst/a.jpg")
require.NoError(t, err)
assert.Equal(t, `"orig"`, dst.Remote.RemoteETag)
// reads keep resolving the source's key until the copy stamps the entry:
// its own key still holds the target's old object
assert.Equal(t, "/buckets/b/src/a.jpg", string(dst.Extended[filer.ExtRemoteRenameSourceKey]))
readDir, readName := server.filer.RemoteReadPath(dst, "/buckets/b/dst", "a.jpg")
assert.Equal(t, "/buckets/b/src", readDir)
assert.Equal(t, "a.jpg", readName)
}
func TestRenameRemoteOnlyEntryWithoutCopierIsRefused(t *testing.T) {
client := &renameRemoteClient{}
server, store := newRenameRemoteTestServer(t, client)
store.entries["/buckets/b/src/a.jpg"] = remoteOnlyRenameEntry("/buckets/b/src/a.jpg", 101)
err := renameFile(server, "/buckets/b/src/a.jpg", "/buckets/b/dst/a.jpg")
require.ErrorContains(t, err, "cannot copy")
assert.Empty(t, client.recorded())
_, err = store.FindEntry(context.Background(), "/buckets/b/src/a.jpg")
require.NoError(t, err)
_, err = store.FindEntry(context.Background(), "/buckets/b/dst/a.jpg")
assert.ErrorIs(t, err, filer_pb.ErrNotFound)
}
func TestRenameRemoteOnlyEntryOutOfItsMountIsRefused(t *testing.T) {
for _, paths := range [][2]util.FullPath{
{"/buckets/b/src/a.jpg", "/buckets/c/a.jpg"},
{"/data/m/a.jpg", "/data/a.jpg"},
} {
t.Run(string(paths[1]), func(t *testing.T) {
client := copyingRenameRemoteClient{&renameRemoteClient{}}
server, store := newRenameRemoteTestServer(t, client)
store.entries[string(paths[0])] = remoteOnlyRenameEntry(string(paths[0]), 101)
err := renameFile(server, paths[0], paths[1])
require.ErrorContains(t, err, "cannot be renamed out of it")
assert.Empty(t, client.recorded())
_, err = store.FindEntry(context.Background(), paths[0])
require.NoError(t, err)
_, err = store.FindEntry(context.Background(), paths[1])
assert.ErrorIs(t, err, filer_pb.ErrNotFound)
})
}
}
func TestRenameEmptyRemoteEntryDoesNotCopy(t *testing.T) {
// nothing to lose: no copy, and no refusal even out of the mount or
// without a copier; the sync writes an empty object at a new key
client := &renameRemoteClient{}
server, store := newRenameRemoteTestServer(t, client)
for _, path := range []string{"/buckets/b/src/empty", "/data/m/empty"} {
entry := remoteOnlyRenameEntry(path, 101)
entry.FileSize = 0
entry.Remote.RemoteSize = 0
store.entries[path] = entry
}
require.NoError(t, renameFile(server, "/buckets/b/src/empty", "/buckets/b/dst/empty"))
require.NoError(t, renameFile(server, "/data/m/empty", "/data/empty"))
assert.Equal(t, []string{"delete origin/src/empty", "delete origin/m/empty"}, client.recorded())
}
func TestRenameRemoteOnlyEntryFailingAfterTheCopyKeepsTheSource(t *testing.T) {
for _, src := range []util.FullPath{"/buckets/b/src/a.jpg", "/buckets/b/src"} {
t.Run(string(src), func(t *testing.T) {
client := copyingRenameRemoteClient{&renameRemoteClient{}}
server, store := newRenameRemoteTestServer(t, client)
store.entries["/buckets/b/src/a.jpg"] = remoteOnlyRenameEntry("/buckets/b/src/a.jpg", 101)
store.commitErr = errors.New("commit failed")
err := renameFile(server, src, "/buckets/b/moved")
require.ErrorContains(t, err, "commit failed")
// the old object, and the old directory holding it, are deleted
// only once the rename has committed
dst := "origin/moved"
if src == "/buckets/b/src" {
dst = "origin/moved/a.jpg"
}
assert.Equal(t, []string{"copy origin/src/a.jpg " + dst}, client.recorded())
})
}
}
func TestRenameEntryWithLocalDataDoesNotCopy(t *testing.T) {
client := copyingRenameRemoteClient{&renameRemoteClient{}}
server, store := newRenameRemoteTestServer(t, client)
entry := remoteOnlyRenameEntry("/data/m/a.jpg", 101)
entry.Chunks = []*filer_pb.FileChunk{{FileId: "1,0123456789", Size: 10}}
store.entries["/data/m/a.jpg"] = entry
// even out of the mount: the old object goes, the content stays local
require.NoError(t, renameFile(server, "/data/m/a.jpg", "/data/a.jpg"))
assert.Equal(t, []string{"delete origin/m/a.jpg"}, client.recorded())
dst, err := store.FindEntry(context.Background(), "/data/a.jpg")
require.NoError(t, err)
assert.Equal(t, `"orig"`, dst.Remote.RemoteETag)
assert.Len(t, dst.GetChunks(), 1)
}
func TestRenameDirectoryCopiesRemoteOnlyChildren(t *testing.T) {
client := copyingRenameRemoteClient{&renameRemoteClient{}}
server, store := newRenameRemoteTestServer(t, client)
store.entries["/buckets/b/src/a.jpg"] = remoteOnlyRenameEntry("/buckets/b/src/a.jpg", 101)
local := remoteOnlyRenameEntry("/buckets/b/src/b.jpg", 102)
local.Content = []byte("0123456789")
store.entries["/buckets/b/src/b.jpg"] = local
require.NoError(t, renameFile(server, "/buckets/b/src", "/buckets/b/moved"))
// b.jpg's old object goes with its entry, a.jpg's waits for the commit,
// and the old prefix is not swept: it could hold a newer object by then
assert.Equal(t, []string{
"copy origin/src/a.jpg origin/moved/a.jpg",
"delete origin/src/b.jpg",
"delete origin/src/a.jpg",
}, client.recorded())
moved, err := store.FindEntry(context.Background(), "/buckets/b/moved/a.jpg")
require.NoError(t, err)
assert.Equal(t, `"copy"`, moved.Remote.RemoteETag)
moved, err = store.FindEntry(context.Background(), "/buckets/b/moved/b.jpg")
require.NoError(t, err)
assert.Equal(t, `"orig"`, moved.Remote.RemoteETag)
}
func TestDeferredOldKeyDeleteKeepsAKeyRetakenSince(t *testing.T) {
client := copyingRenameRemoteClient{&renameRemoteClient{}}
server, store := newRenameRemoteTestServer(t, client)
event := metadataEvent{
oldEntry: remoteOnlyRenameEntry("/buckets/b/src/a.jpg", 101),
newEntry: remoteOnlyRenameEntry("/buckets/b/moved/a.jpg", 101),
deleteOldRemote: true,
}
// another rename put a remote-only entry at the old path after the commit
store.entries["/buckets/b/src/a.jpg"] = remoteOnlyRenameEntry("/buckets/b/src/a.jpg", 303)
event.notify(server.filer, context.Background(), nil)
assert.Empty(t, client.recorded())
// a lookup that fails says nothing about the path
delete(store.entries, "/buckets/b/src/a.jpg")
store.findErr = errors.New("store unavailable")
event.notify(server.filer, context.Background(), nil)
assert.Empty(t, client.recorded())
store.findErr = nil
event.notify(server.filer, context.Background(), nil)
assert.Equal(t, []string{"delete origin/src/a.jpg"}, client.recorded())
}
func TestRenameRemoteOnlyEntryFinishesAfterTheCommitOnASQLStore(t *testing.T) {
// a SQL store keeps the transaction in the context and refuses calls
// through it once committed, so the work after the commit must not use it
for _, overTarget := range []bool{false, true} {
t.Run(map[bool]string{false: "new path", true: "over a remote-only target"}[overTarget], func(t *testing.T) {
client := copyingRenameRemoteClient{&renameRemoteClient{}}
server, store := newRenameRemoteTestServer(t, client)
store.strictTx = true
store.entries["/buckets/b/src/a.jpg"] = remoteOnlyRenameEntry("/buckets/b/src/a.jpg", 101)
if overTarget {
store.entries["/buckets/b/dst/a.jpg"] = remoteOnlyRenameEntry("/buckets/b/dst/a.jpg", 202)
}
require.NoError(t, renameFile(server, "/buckets/b/src/a.jpg", "/buckets/b/dst/a.jpg"))
// the copy is stamped and the old key deleted by the filer itself
assert.Equal(t, []string{"copy origin/src/a.jpg origin/dst/a.jpg", "delete origin/src/a.jpg"}, client.recorded())
dst, err := store.FindEntry(context.Background(), "/buckets/b/dst/a.jpg")
require.NoError(t, err)
assert.Equal(t, `"copy"`, dst.Remote.RemoteETag)
})
}
}
func TestRenamePendingEntryAgainCopiesFromItsSource(t *testing.T) {
client := copyingRenameRemoteClient{&renameRemoteClient{copyErr: errors.New("copy refused")}}
server, store := newRenameRemoteTestServer(t, client)
store.entries["/buckets/b/src/a.jpg"] = remoteOnlyRenameEntry("/buckets/b/src/a.jpg", 101)
store.entries["/buckets/b/dst/a.jpg"] = remoteOnlyRenameEntry("/buckets/b/dst/a.jpg", 202)
store.entries["/buckets/b/dst/d.jpg"] = remoteOnlyRenameEntry("/buckets/b/dst/d.jpg", 303)
// the deferred copy fails: dst/a.jpg reads src/a.jpg's object
require.NoError(t, renameFile(server, "/buckets/b/src/a.jpg", "/buckets/b/dst/a.jpg"))
// renamed again over another remote-only file, failing again: the marker
// keeps naming the original source, not dst/a.jpg's old object
require.NoError(t, renameFile(server, "/buckets/b/dst/a.jpg", "/buckets/b/dst/d.jpg"))
d, err := store.FindEntry(context.Background(), "/buckets/b/dst/d.jpg")
require.NoError(t, err)
assert.Equal(t, "/buckets/b/src/a.jpg", string(d.Extended[filer.ExtRemoteRenameSourceKey]))
// renamed to a new path, the copy comes from the source and clears the marker
client.copyErr = nil
require.NoError(t, renameFile(server, "/buckets/b/dst/d.jpg", "/buckets/b/dst/c.jpg"))
assert.Contains(t, client.recorded(), "copy origin/src/a.jpg origin/dst/c.jpg")
// the source is left to remote.sync: a concurrent rename may have copied
// onto that key before its entry is visible
assert.NotContains(t, client.recorded(), "delete origin/src/a.jpg")
c, err := store.FindEntry(context.Background(), "/buckets/b/dst/c.jpg")
require.NoError(t, err)
assert.Empty(t, c.Extended[filer.ExtRemoteRenameSourceKey])
assert.Equal(t, `"copy"`, c.Remote.RemoteETag)
}
func TestRemoteSourcePathStaysInTheMount(t *testing.T) {
server, _ := newRenameRemoteTestServer(t, &renameRemoteClient{})
entry := remoteOnlyRenameEntry("/buckets/b/x.jpg", 101)
for marker, want := range map[string]util.FullPath{
"/buckets/b/src/y.jpg": "/buckets/b/src/y.jpg", // same mount: the pending source
"/buckets/c/y.jpg": "/buckets/b/x.jpg", // another mount: ignored
"/tmp/y.jpg": "/buckets/b/x.jpg", // no mount: ignored
} {
entry.Extended = map[string][]byte{filer.ExtRemoteRenameSourceKey: []byte(marker)}
assert.Equal(t, want, server.filer.RemoteSourcePath(entry, "/buckets/b/x.jpg"), marker)
}
}
func TestTaggingCannotSetThePendingSourceMarker(t *testing.T) {
server, store := newRenameRemoteTestServer(t, &renameRemoteClient{})
store.entries["/buckets/b/x.jpg"] = remoteOnlyRenameEntry("/buckets/b/x.jpg", 101)
r := httptest.NewRequest(http.MethodPut, "/buckets/b/x.jpg?tagging", nil)
r.Header.Set("Seaweed-X-Remote-Rename-Source", "/buckets/b/other.jpg")
r.Header.Set("Seaweed-Album", "2026")
server.PutTaggingHandler(httptest.NewRecorder(), r)
x, err := store.FindEntry(context.Background(), "/buckets/b/x.jpg")
require.NoError(t, err)
assert.Equal(t, "2026", string(x.Extended["Seaweed-Album"]))
assert.Empty(t, x.Extended[filer.ExtRemoteRenameSourceKey], "only the filer sets the pending source")
}
func TestDeletingTagsKeepsThePendingSourceMarker(t *testing.T) {
server, store := newRenameRemoteTestServer(t, &renameRemoteClient{})
entry := remoteOnlyRenameEntry("/buckets/b/x.jpg", 101)
entry.Extended = map[string][]byte{"Seaweed-Album": []byte("2026"), filer.ExtRemoteRenameSourceKey: []byte("/buckets/b/src/x.jpg")}
store.entries["/buckets/b/x.jpg"] = entry
// no tag names: delete every tag
server.DeleteTaggingHandler(httptest.NewRecorder(), httptest.NewRequest(http.MethodDelete, "/buckets/b/x.jpg?tagging", nil))
x, err := store.FindEntry(context.Background(), "/buckets/b/x.jpg")
require.NoError(t, err)
assert.Empty(t, x.Extended["Seaweed-Album"])
assert.Equal(t, "/buckets/b/src/x.jpg", string(x.Extended[filer.ExtRemoteRenameSourceKey]), "the marker is not a tag")
}
+49 -6
View File
@@ -31,6 +31,21 @@ type renameTestStore struct {
deleteErr error
listErr error // simulates a transient store/RPC failure from a directory listing
findDelay time.Duration // optional: widen check-then-act windows in tests
findErr error // simulates a failing entry lookup
strictTx bool // like a SQL store: calls through a committed transaction fail
}
type renameTestTxKey struct{}
type renameTestTx struct{ done bool }
// txDone reports a call through a transaction that has already committed or
// rolled back, which a SQL store refuses.
func (s *renameTestStore) txDone(ctx context.Context) error {
if tx, ok := ctx.Value(renameTestTxKey{}).(*renameTestTx); ok && tx.done {
return errors.New("sql: transaction has already been committed or rolled back")
}
return nil
}
func newRenameTestStore() *renameTestStore {
@@ -44,10 +59,23 @@ func (s *renameTestStore) GetName() string { return
func (s *renameTestStore) Initialize(util.Configuration, string) error { return nil }
func (s *renameTestStore) Shutdown() {}
func (s *renameTestStore) BeginTransaction(ctx context.Context) (context.Context, error) {
if s.strictTx {
return context.WithValue(ctx, renameTestTxKey{}, &renameTestTx{}), nil
}
return ctx, nil
}
func (s *renameTestStore) CommitTransaction(context.Context) error { return s.commitErr }
func (s *renameTestStore) RollbackTransaction(context.Context) error { return nil }
func (s *renameTestStore) CommitTransaction(ctx context.Context) error {
if tx, ok := ctx.Value(renameTestTxKey{}).(*renameTestTx); ok {
tx.done = true
}
return s.commitErr
}
func (s *renameTestStore) RollbackTransaction(ctx context.Context) error {
if tx, ok := ctx.Value(renameTestTxKey{}).(*renameTestTx); ok {
tx.done = true
}
return nil
}
func (s *renameTestStore) KvPut(context.Context, []byte, []byte) error {
return nil
}
@@ -56,27 +84,39 @@ func (s *renameTestStore) KvGet(context.Context, []byte) ([]byte, error) {
}
func (s *renameTestStore) KvDelete(context.Context, []byte) error { return nil }
func (s *renameTestStore) InsertEntry(_ context.Context, entry *filer.Entry) error {
func (s *renameTestStore) InsertEntry(ctx context.Context, entry *filer.Entry) error {
if err := s.txDone(ctx); err != nil {
return err
}
s.mu.Lock()
defer s.mu.Unlock()
s.entries[string(entry.FullPath)] = entry.ShallowClone()
return nil
}
func (s *renameTestStore) UpdateEntry(_ context.Context, entry *filer.Entry) error {
func (s *renameTestStore) UpdateEntry(ctx context.Context, entry *filer.Entry) error {
if err := s.txDone(ctx); err != nil {
return err
}
s.mu.Lock()
defer s.mu.Unlock()
s.entries[string(entry.FullPath)] = entry.ShallowClone()
return nil
}
func (s *renameTestStore) FindEntry(_ context.Context, p util.FullPath) (*filer.Entry, error) {
func (s *renameTestStore) FindEntry(ctx context.Context, p util.FullPath) (*filer.Entry, error) {
if err := s.txDone(ctx); err != nil {
return nil, err
}
if s.findDelay > 0 {
time.Sleep(s.findDelay)
}
s.mu.Lock()
defer s.mu.Unlock()
s.findCalls[string(p)]++
if s.findErr != nil {
return nil, s.findErr
}
entry, found := s.entries[string(p)]
if !found {
return nil, filer_pb.ErrNotFound
@@ -84,7 +124,10 @@ func (s *renameTestStore) FindEntry(_ context.Context, p util.FullPath) (*filer.
return entry.ShallowClone(), nil
}
func (s *renameTestStore) DeleteEntry(_ context.Context, p util.FullPath) error {
func (s *renameTestStore) DeleteEntry(ctx context.Context, p util.FullPath) error {
if err := s.txDone(ctx); err != nil {
return err
}
if s.deleteErr != nil {
return s.deleteErr
}
+23 -10
View File
@@ -206,15 +206,18 @@ func (fs *FilerServer) GetOrHeadHandler(w http.ResponseWriter, r *http.Request)
// Every part of a multipart Range is prepared on its own, so the origin
// preflight below is memoized to one stat per request.
statOrigin := sync.OnceValue(func() error {
dir, name := entry.FullPath.DirAndName()
client, remoteLocation, err := fs.mountedRemoteClient(ctx, dir, name)
if err != nil {
statOriginAt := func(readDir, readName string) func() error {
return sync.OnceValue(func() error {
client, remoteLocation, err := fs.mountedRemoteClient(ctx, readDir, readName)
if err != nil {
return err
}
_, err = client.StatFile(remoteLocation)
return err
}
_, err = client.StatFile(remoteLocation)
return err
})
})
}
entryDir, entryName := entry.FullPath.DirAndName()
statOrigin := statOriginAt(fs.filer.RemoteReadPath(entry, entryDir, entryName))
ProcessRangeRequest(r, w, totalSize, mimeType, func(offset int64, size int64) (filer.DoStreamContent, error) {
if offset+size <= int64(len(entry.Content)) {
@@ -230,13 +233,23 @@ func (fs *FilerServer) GetOrHeadHandler(w http.ResponseWriter, r *http.Request)
chunks := entry.GetChunks()
if entry.IsInRemoteOnly() {
dir, name := entry.FullPath.DirAndName()
readDir, readName := fs.filer.RemoteReadPath(entry, dir, name)
var mountedLocation *remote_pb.RemoteStorageLocation
if fs.filer.RemoteStorage != nil {
_, mountedLocation = fs.filer.RemoteStorage.FindMountDirectory(entry.FullPath)
}
cacheWait := remote_storage.CacheWaitTimeout(entry.Remote.RemoteSize, mountedLocation)
if cacheWait <= 0 {
return fs.streamFromRemoteOnly(ctx, r, dir, name, offset, size, statOrigin)
streamFn, err := fs.streamFromRemoteOnly(ctx, r, readDir, readName, offset, size, statOrigin)
if errors.Is(err, filer_pb.ErrNotFound) && (readDir != dir || readName != name) {
// the pending copy finished and its source went away since
// the entry was read; serve what the entry names now
if fresh, findErr := fs.filer.FindEntry(ctx, entry.FullPath); findErr == nil {
freshDir, freshName := fs.filer.RemoteReadPath(fresh, dir, name)
return fs.streamFromRemoteOnly(ctx, r, freshDir, freshName, offset, size, statOriginAt(freshDir, freshName))
}
}
return streamFn, err
}
// Bounded wait: a large download outlasts any client timeout, so
// serve straight from the origin once the wait expires while the
@@ -264,7 +277,7 @@ func (fs *FilerServer) GetOrHeadHandler(w http.ResponseWriter, r *http.Request)
// when a later prepare fails; keep those on the 503 retry path.
if !strings.Contains(r.Header.Get("Range"), ",") {
glog.V(1).InfofCtx(ctx, "stream %s from remote while caching: %v", entry.FullPath, err)
if streamFn, remoteErr := fs.streamFromRemote(ctx, dir, name, offset, size); remoteErr == nil {
if streamFn, remoteErr := fs.streamFromRemote(ctx, readDir, readName, offset, size); remoteErr == nil {
return streamFn, nil
} else {
glog.WarningfCtx(ctx, "stream %s from remote: %v", entry.FullPath, remoteErr)
+3 -2
View File
@@ -4,6 +4,7 @@ import (
"net/http"
"strings"
"github.com/seaweedfs/seaweedfs/weed/filer"
"github.com/seaweedfs/seaweedfs/weed/glog"
"github.com/seaweedfs/seaweedfs/weed/storage/needle"
"github.com/seaweedfs/seaweedfs/weed/util"
@@ -35,7 +36,7 @@ func (fs *FilerServer) PutTaggingHandler(w http.ResponseWriter, r *http.Request)
}
for header, values := range r.Header {
if strings.HasPrefix(header, needle.PairNamePrefix) {
if strings.HasPrefix(header, needle.PairNamePrefix) && header != filer.ExtRemoteRenameSourceKey {
for _, value := range values {
existingEntry.Extended[header] = []byte(value)
}
@@ -89,7 +90,7 @@ func (fs *FilerServer) DeleteTaggingHandler(w http.ResponseWriter, r *http.Reque
// delete all tags or specific tags
hasDeletion := false
for header, _ := range existingEntry.Extended {
if strings.HasPrefix(header, needle.PairNamePrefix) {
if strings.HasPrefix(header, needle.PairNamePrefix) && header != filer.ExtRemoteRenameSourceKey {
if len(deletions) == 0 {
delete(existingEntry.Extended, header)
hasDeletion = true
@@ -58,7 +58,10 @@ func (fs *FilerServer) autoChunk(ctx context.Context, w http.ResponseWriter, r *
writeJsonError(w, r, util.HttpStatusCancelled, err)
case strings.HasSuffix(errStr, "is a file") || strings.HasSuffix(errStr, "already exists"):
writeJsonError(w, r, http.StatusConflict, err)
case errStr == constants.ErrMsgBadDigest:
case errStr == constants.ErrMsgBadDigest,
errStr == constants.ErrMsgAppendSaveInsideChunked,
errStr == constants.ErrMsgAppendInlineMode,
errStr == constants.ErrMsgOffsetWriteInline:
writeJsonError(w, r, http.StatusBadRequest, err)
default:
writeJsonError(w, r, http.StatusInternalServerError, err)
@@ -99,8 +102,17 @@ func (fs *FilerServer) doPostAutoChunk(ctx context.Context, w http.ResponseWrite
if so.SaveInside {
buf := bufPool.Get().(*bytes.Buffer)
buf.Reset()
buf.ReadFrom(part1)
filerResult, replyerr, _ = fs.saveMetaData(ctx, r, fileName, contentType, so, chunkSize, nil, nil, 0, buf.Bytes())
if _, err := buf.ReadFrom(part1); err != nil {
bufPool.Put(buf)
return nil, nil, fmt.Errorf("read input: %w", err)
}
md5bytes = util.Md5(buf.Bytes())
headerMd5 := r.Header.Get("Content-Md5")
if headerMd5 != "" && !(util.Base64Encode(md5bytes) == headerMd5 || fmt.Sprintf("%x", md5bytes) == headerMd5) {
bufPool.Put(buf)
return nil, nil, errors.New(constants.ErrMsgBadDigest)
}
filerResult, replyerr, _ = fs.saveMetaData(ctx, r, fileName, contentType, so, chunkSize, md5bytes, nil, int64(buf.Len()), buf.Bytes())
bufPool.Put(buf)
return
}
@@ -293,7 +305,11 @@ func (fs *FilerServer) saveMetaData(ctx context.Context, r *http.Request, fileNa
if entry != nil {
inlineCandidate := len(entry.Content) > 0 ||
(isAppend && entry.FileSize == 0 && entry.Remote == nil && len(entry.HardLinkId) == 0 && len(entry.GetChunks()) == 0)
if isAppend && !so.SaveInside && content != nil && !inlineCandidate {
if isAppend && content != nil && !inlineCandidate {
if so.SaveInside {
replyerr = errors.New(constants.ErrMsgAppendSaveInsideChunked)
return
}
replyerr = fmt.Errorf("inline file changed while preparing append; retry the request")
return
}
@@ -301,11 +317,11 @@ func (fs *FilerServer) saveMetaData(ctx context.Context, r *http.Request, fileNa
if inlineCandidate {
if isOffsetWrite {
// TODO: support inline offset writes separately from append semantics.
replyerr = fmt.Errorf("offset write to inline small file is not supported yet")
replyerr = errors.New(constants.ErrMsgOffsetWriteInline)
return
}
if !isAppend || so.SaveInside || entry.Remote != nil || len(entry.HardLinkId) != 0 || len(entry.GetChunks()) != 0 {
replyerr = fmt.Errorf("append to inline content with this storage mode is not supported")
replyerr = errors.New(constants.ErrMsgAppendInlineMode)
return
}
if entry.FileSize > uint64(len(entry.Content)) {
@@ -468,6 +484,9 @@ func (fs *FilerServer) saveMetaData(ctx context.Context, r *http.Request, fileNa
}
for k, v := range r.Header {
if len(v) > 0 && len(v[0]) > 0 {
if k == filer.ExtRemoteRenameSourceKey {
continue // set by the filer only: it picks the remote key reads serve
}
if strings.HasPrefix(k, needle.PairNamePrefix) || k == "Cache-Control" || k == "Expires" || k == "Content-Disposition" {
entry.Extended[k] = []byte(v[0])
}
@@ -270,6 +270,164 @@ func TestFilerInlineAppendMultipartPostStaysInline(t *testing.T) {
}
}
func TestFilerSaveInsideMultipartPostRecordsSize(t *testing.T) {
ctx := context.Background()
content := []byte(`{"identities":[]}`)
store := newRenameTestStore()
f := newRenameTestFiler(t, store)
fs := &FilerServer{filer: f, option: &FilerOption{MaxMB: 1}}
var body bytes.Buffer
multipartWriter := multipart.NewWriter(&body)
part, err := multipartWriter.CreateFormFile("file", "identity.json")
if err != nil {
t.Fatalf("create multipart file part: %v", err)
}
if _, err = part.Write(content); err != nil {
t.Fatalf("write multipart file part: %v", err)
}
if err = multipartWriter.Close(); err != nil {
t.Fatalf("close multipart body: %v", err)
}
r := httptest.NewRequest(http.MethodPost, "/", &body)
r.Header.Set("Content-Type", multipartWriter.FormDataContentType())
result, requestMD5, err := fs.doPostAutoChunk(ctx, httptest.NewRecorder(), r, 4, int64(body.Len()), &operation.StorageOption{SaveInside: true})
if err != nil {
t.Fatalf("POST saveInside: %v", err)
}
if result == nil || result.Size != int64(len(content)) {
t.Fatalf("result = %#v, want size %d", result, len(content))
}
wantMD5 := md5.Sum(content)
if !bytes.Equal(requestMD5, wantMD5[:]) {
t.Fatalf("request MD5 = %x, want %x", requestMD5, wantMD5)
}
saved, err := f.FindEntry(ctx, "/identity.json")
if err != nil {
t.Fatalf("find saved entry: %v", err)
}
if !bytes.Equal(saved.Content, content) || saved.FileSize != uint64(len(content)) {
t.Fatalf("saved entry: content=%q size=%d, want %q size %d", saved.Content, saved.FileSize, content, len(content))
}
if !bytes.Equal(saved.Md5, wantMD5[:]) {
t.Fatalf("saved MD5 = %x, want %x", saved.Md5, wantMD5)
}
}
func saveInsideMultipartRequest(t *testing.T, target, fileName string, content []byte) *http.Request {
t.Helper()
var body bytes.Buffer
multipartWriter := multipart.NewWriter(&body)
part, err := multipartWriter.CreateFormFile("file", fileName)
if err != nil {
t.Fatalf("create multipart file part: %v", err)
}
if _, err = part.Write(content); err != nil {
t.Fatalf("write multipart file part: %v", err)
}
if err = multipartWriter.Close(); err != nil {
t.Fatalf("close multipart body: %v", err)
}
r := httptest.NewRequest(http.MethodPost, target, &body)
r.Header.Set("Content-Type", multipartWriter.FormDataContentType())
return r
}
func TestFilerSaveInsideMultipartPostRejectsBadDigest(t *testing.T) {
ctx := context.Background()
content := []byte(`{"identities":[]}`)
store := newRenameTestStore()
f := newRenameTestFiler(t, store)
fs := &FilerServer{filer: f, option: &FilerOption{MaxMB: 1}}
r := saveInsideMultipartRequest(t, "/", "identity.json", content)
r.Header.Set("Content-Md5", util.Base64Encode([]byte("not the digest")))
if _, _, err := fs.doPostAutoChunk(ctx, httptest.NewRecorder(), r, 4, r.ContentLength, &operation.StorageOption{SaveInside: true}); err == nil || !strings.Contains(err.Error(), "Content-Md5") {
t.Fatalf("POST saveInside error = %v, want bad Content-MD5 error", err)
}
if _, err := f.FindEntry(ctx, "/identity.json"); !errors.Is(err, filer_pb.ErrNotFound) {
t.Fatalf("bad digest still saved the entry: %v", err)
}
}
func TestFilerSaveInsideMultipartAppendRejected(t *testing.T) {
ctx := context.Background()
const filePath = "/append-target.txt"
appendContent := []byte("new")
t.Run("chunk backed target", func(t *testing.T) {
oldChunk := &filer_pb.FileChunk{FileId: "1,00000001", Offset: 0, Size: 4}
store := newRenameTestStore()
f := newRenameTestFiler(t, store)
if err := store.InsertEntry(ctx, &filer.Entry{
FullPath: util.FullPath(filePath),
Attr: filer.Attr{
Mtime: time.Unix(1, 0),
Crtime: time.Unix(1, 0),
FileSize: 4,
},
Chunks: []*filer_pb.FileChunk{oldChunk},
}); err != nil {
t.Fatalf("insert chunk-backed entry: %v", err)
}
fs := &FilerServer{filer: f, option: &FilerOption{MaxMB: 1}}
r := saveInsideMultipartRequest(t, filePath+"?op=append", "append-target.txt", appendContent)
if _, _, err := fs.doPostAutoChunk(ctx, httptest.NewRecorder(), r, 4, r.ContentLength, &operation.StorageOption{SaveInside: true}); err == nil {
t.Fatal("saveInside append to chunk-backed entry succeeded")
}
recorder := httptest.NewRecorder()
r = saveInsideMultipartRequest(t, filePath+"?op=append", "append-target.txt", appendContent)
fs.autoChunk(ctx, recorder, r, r.ContentLength, &operation.StorageOption{SaveInside: true})
if recorder.Code != http.StatusBadRequest {
t.Fatalf("rejected saveInside append status = %d, want %d", recorder.Code, http.StatusBadRequest)
}
unchanged, err := f.FindEntry(ctx, util.FullPath(filePath))
if err != nil {
t.Fatalf("find entry after rejected append: %v", err)
}
if unchanged.FileSize != 4 || len(unchanged.Chunks) != 1 || len(unchanged.Content) != 0 {
t.Fatalf("rejected append changed entry: size=%d chunks=%d content=%q", unchanged.FileSize, len(unchanged.Chunks), unchanged.Content)
}
})
t.Run("inline target", func(t *testing.T) {
oldContent := []byte("old-")
store := newRenameTestStore()
f := newRenameTestFiler(t, store)
if err := store.InsertEntry(ctx, &filer.Entry{
FullPath: util.FullPath(filePath),
Attr: filer.Attr{
Mtime: time.Unix(1, 0),
Crtime: time.Unix(1, 0),
FileSize: uint64(len(oldContent)),
},
Content: append([]byte(nil), oldContent...),
}); err != nil {
t.Fatalf("insert inline entry: %v", err)
}
fs := &FilerServer{filer: f, option: &FilerOption{MaxMB: 1}}
r := saveInsideMultipartRequest(t, filePath+"?op=append", "append-target.txt", appendContent)
if _, _, err := fs.doPostAutoChunk(ctx, httptest.NewRecorder(), r, 4, r.ContentLength, &operation.StorageOption{SaveInside: true}); err == nil {
t.Fatal("saveInside append to inline entry succeeded")
}
recorder := httptest.NewRecorder()
r = saveInsideMultipartRequest(t, filePath+"?op=append", "append-target.txt", appendContent)
fs.autoChunk(ctx, recorder, r, r.ContentLength, &operation.StorageOption{SaveInside: true})
if recorder.Code != http.StatusBadRequest {
t.Fatalf("rejected saveInside append status = %d, want %d", recorder.Code, http.StatusBadRequest)
}
unchanged, err := f.FindEntry(ctx, util.FullPath(filePath))
if err != nil {
t.Fatalf("find entry after rejected append: %v", err)
}
if !bytes.Equal(unchanged.Content, oldContent) || unchanged.FileSize != uint64(len(oldContent)) {
t.Fatalf("rejected append changed entry: content=%q size=%d", unchanged.Content, unchanged.FileSize)
}
})
}
func TestFilerInlineOffsetWriteRemainsUnsupported(t *testing.T) {
ctx := context.Background()
const filePath = "/offset.txt"
+25 -1
View File
@@ -509,6 +509,27 @@ func (ms *MasterServer) VolumeGrow(ctx context.Context, req *master_pb.VolumeGro
Force: true,
Reason: "grpc volume grow",
}
// Report the layout's fill observed at request time whether or not
// growth was triggered, so callers can tell an empty layout from a
// skipped one. A missing collection/layout reports zeros.
var avgFill float64
var writableCount int
if vl := ms.Topo.FindVolumeLayout(req.Collection, replicaPlacement, ttl, types.ToDiskType(req.DiskType)); vl != nil {
avgFill, writableCount = vl.WritableAvgFill(&volumeGrowOption)
}
// Conditional grow: with a threshold set, grow only when the layout's
// writable volumes have reached this average fill. A layout with no
// writable volumes (none yet, or all just filled) skips the check — that
// is the case growth exists to fix.
if req.GrowAvgFillThreshold > 0 && writableCount > 0 && avgFill < float64(req.GrowAvgFillThreshold) {
return &master_pb.VolumeGrowResponse{
AvgFill: float32(avgFill),
WritableCount: uint32(writableCount),
}, nil
}
replicaCount := int64(req.WritableVolumeCount * uint32(replicaPlacement.GetCopyCount()))
if ms.Topo.AvailableSpaceFor(&volumeGrowOption) < replicaCount {
@@ -517,5 +538,8 @@ func (ms *MasterServer) VolumeGrow(ctx context.Context, req *master_pb.VolumeGro
ms.DoAutomaticVolumeGrow(&volumeGrowRequest)
return &master_pb.VolumeGrowResponse{}, nil
return &master_pb.VolumeGrowResponse{
AvgFill: float32(avgFill),
WritableCount: uint32(writableCount),
}, nil
}
+1 -1
View File
@@ -394,7 +394,7 @@ func (vs *VolumeServer) VolumeEcShardsCopy(ctx context.Context, req *volume_serv
}
location = vs.store.FindEcShardTargetLocation(req.Collection, needle.VolumeId(req.VolumeId), erasure_coding.DataShardsCount, shardIds...)
if location == nil {
return nil, fmt.Errorf("no space left")
return nil, status.Error(codes.ResourceExhausted, "no space left")
}
}
+8
View File
@@ -570,6 +570,14 @@ func (f *WebDavFile) Read(p []byte) (readSize int, err error) {
if fileSize == 0 {
return 0, io.EOF
}
if len(f.entry.Content) > 0 {
if f.off >= int64(len(f.entry.Content)) {
return 0, io.EOF
}
readSize = copy(p, f.entry.Content[f.off:])
f.off += int64(readSize)
return readSize, nil
}
if f.visibleIntervals == nil {
f.visibleIntervals, err = filer.NonOverlappingVisibleIntervals(f.ctx, f.fs.filerClient.GetLookupFileIdFunction(), f.entry.GetChunks(), 0, fileSize)
if err != nil {
+26
View File
@@ -3,6 +3,7 @@ package weed_server
import (
"context"
"fmt"
"io"
"net"
"os"
"testing"
@@ -104,3 +105,28 @@ func TestWebDavFile_Read_ManifestResolveFailure(t *testing.T) {
assert.Error(t, err)
assert.Equal(t, 0, n)
}
// Small files stored inline in the entry have no chunks to read from.
func TestWebDavFile_Read_InlineContent(t *testing.T) {
content := []byte("hello inline")
entry := &filer_pb.Entry{
Name: "small.txt",
Attributes: &filer_pb.FuseAttributes{FileSize: uint64(len(content))},
Content: content,
}
filerAddr := startFakeWebDavFiler(t, &noVolumeFiler{})
fc := wdclient.NewFilerClient([]pb.ServerAddress{filerAddr}, grpc.WithTransportCredentials(insecure.NewCredentials()), "")
t.Cleanup(fc.Close)
f := &WebDavFile{fs: &WebDavFileSystem{filerClient: fc}, name: "/small.txt", entry: entry, ctx: context.Background()}
got, err := io.ReadAll(f)
assert.NoError(t, err)
assert.Equal(t, content, got)
_, err = f.Seek(6, io.SeekStart)
assert.NoError(t, err)
buf := make([]byte, 3)
n, err := f.Read(buf)
assert.NoError(t, err)
assert.Equal(t, "inl", string(buf[:n]))
}
+210
View File
@@ -0,0 +1,210 @@
package sftpd
import (
"bytes"
"context"
"net"
"net/http"
"net/http/httptest"
"testing"
"github.com/seaweedfs/seaweedfs/weed/pb"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
"github.com/seaweedfs/seaweedfs/weed/sftpd/user"
"google.golang.org/grpc"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/credentials/insecure"
"google.golang.org/grpc/status"
)
type copyTestFiler struct {
filer_pb.UnimplementedSeaweedFilerServer
entries map[string]*filer_pb.Entry
volumeUrl string
}
func (f *copyTestFiler) LookupDirectoryEntry(_ context.Context, req *filer_pb.LookupDirectoryEntryRequest) (*filer_pb.LookupDirectoryEntryResponse, error) {
if entry, ok := f.entries[req.Directory+"/"+req.Name]; ok {
return &filer_pb.LookupDirectoryEntryResponse{Entry: entry}, nil
}
return nil, status.Error(codes.NotFound, "no entry is found")
}
func (f *copyTestFiler) LookupVolume(_ context.Context, req *filer_pb.LookupVolumeRequest) (*filer_pb.LookupVolumeResponse, error) {
resp := &filer_pb.LookupVolumeResponse{LocationsMap: make(map[string]*filer_pb.Locations)}
for _, vid := range req.VolumeIds {
resp.LocationsMap[vid] = &filer_pb.Locations{Locations: []*filer_pb.Location{{Url: f.volumeUrl}}}
}
return resp, nil
}
func newCopyTestServer(t *testing.T, volumeUrl string, entries map[string]*filer_pb.Entry) *SftpServer {
lis, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatalf("listen: %v", err)
}
grpcServer := grpc.NewServer()
filer_pb.RegisterSeaweedFilerServer(grpcServer, &copyTestFiler{entries: entries, volumeUrl: volumeUrl})
go func() { _ = grpcServer.Serve(lis) }()
t.Cleanup(grpcServer.Stop)
_, port, _ := net.SplitHostPort(lis.Addr().String())
return &SftpServer{
filerAddr: pb.ServerAddress("127.0.0.1:0." + port),
grpcDialOption: grpc.WithTransportCredentials(insecure.NewCredentials()),
user: &user.User{Username: "alice", Uid: 1000, HomeDir: "/"},
}
}
func chunkedEntry(size int64, chunks ...*filer_pb.FileChunk) *filer_pb.Entry {
return &filer_pb.Entry{
Attributes: &filer_pb.FuseAttributes{FileSize: uint64(size)},
Chunks: chunks,
IsDirectory: false,
}
}
func fileChunk(fileId string, offset, size int64) *filer_pb.FileChunk {
return &filer_pb.FileChunk{FileId: fileId, Offset: offset, Size: uint64(size), ModifiedTsNs: 1}
}
// A non-truncating write must keep untouched bytes, including a sparse file's
// zeros between chunks. Stopping at the first gap would drop the tail.
func TestCopyExistingContentFillsSparseGaps(t *testing.T) {
chunkA := []byte("AAAAAAAAAA")
chunkB := []byte("BBBBBBBBBB")
objects := map[string][]byte{"3,01": chunkA, "3,02": chunkB}
volume := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
data, ok := objects[r.URL.Path[1:]]
if !ok {
http.Error(w, "not found", http.StatusNotFound)
return
}
_, _ = w.Write(data)
}))
t.Cleanup(volume.Close)
entries := map[string]*filer_pb.Entry{
"/home/sparse.bin": chunkedEntry(100, fileChunk("3,01", 0, 10), fileChunk("3,02", 90, 10)),
}
fs := newCopyTestServer(t, volume.Listener.Addr().String(), entries)
var dst bytes.Buffer
if err := fs.copyExistingContent("/home/sparse.bin", &dst); err != nil {
t.Fatalf("copyExistingContent: %v", err)
}
want := append(append(append([]byte{}, chunkA...), make([]byte, 80)...), chunkB...)
if !bytes.Equal(dst.Bytes(), want) {
t.Fatalf("copied %d bytes, want %d with a zero-filled gap; tail present: %v", dst.Len(), len(want), bytes.Contains(dst.Bytes(), chunkB))
}
}
// A remote-only file has no local chunks: copying nothing would upload an
// empty replacement on close. Refusing the read keeps the remote object.
func TestCopyExistingContentRejectsUnreadableEntry(t *testing.T) {
remoteOnly := chunkedEntry(0)
remoteOnly.RemoteEntry = &filer_pb.RemoteEntry{RemoteSize: 100}
entries := map[string]*filer_pb.Entry{
"/home/remote.bin": remoteOnly,
}
fs := newCopyTestServer(t, "", entries)
var dst bytes.Buffer
if err := fs.copyExistingContent("/home/remote.bin", &dst); err == nil {
t.Fatal("expected refusal when no local bytes exist")
}
}
// A locally shortened file still names its older, larger remote object. Once
// the local entry is newer the remote size is stale: copying must not pad the
// file back out to it.
func TestCopyExistingContentIgnoresStaleRemoteSize(t *testing.T) {
chunkA := []byte("AAAAAAAAAA")
volume := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/3,01" {
http.Error(w, "not found", http.StatusNotFound)
return
}
_, _ = w.Write(chunkA)
}))
t.Cleanup(volume.Close)
entries := map[string]*filer_pb.Entry{
"/home/short.bin": {
Attributes: &filer_pb.FuseAttributes{FileSize: 10, Mtime: 200},
Chunks: []*filer_pb.FileChunk{fileChunk("3,01", 0, 10)},
RemoteEntry: &filer_pb.RemoteEntry{RemoteSize: 100, RemoteMtime: 100},
},
}
fs := newCopyTestServer(t, volume.Listener.Addr().String(), entries)
var dst bytes.Buffer
if err := fs.copyExistingContent("/home/short.bin", &dst); err != nil {
t.Fatalf("copyExistingContent: %v", err)
}
if !bytes.Equal(dst.Bytes(), chunkA) {
t.Fatalf("copied %d bytes, want the %d local bytes only", dst.Len(), len(chunkA))
}
}
// A file whose size is all holes still keeps its zeros.
func TestCopyExistingContentAllHoles(t *testing.T) {
entries := map[string]*filer_pb.Entry{
"/home/holes.bin": chunkedEntry(10),
}
fs := newCopyTestServer(t, "", entries)
var dst bytes.Buffer
if err := fs.copyExistingContent("/home/holes.bin", &dst); err != nil {
t.Fatalf("all-holes file: %v", err)
}
if dst.Len() != 10 || !bytes.Equal(dst.Bytes(), make([]byte, 10)) {
t.Fatalf("wrote %d bytes, want 10 zeros", dst.Len())
}
}
func TestCopyExistingContentInlineAndMissing(t *testing.T) {
entries := map[string]*filer_pb.Entry{
"/home/inline.txt": {Attributes: &filer_pb.FuseAttributes{FileSize: 5}, Content: []byte("hello")},
}
fs := newCopyTestServer(t, "", entries)
var dst bytes.Buffer
if err := fs.copyExistingContent("/home/inline.txt", &dst); err != nil {
t.Fatalf("inline content: %v", err)
}
if got := dst.String(); got != "hello" {
t.Fatalf("inline content = %q, want %q", got, "hello")
}
dst.Reset()
if err := fs.copyExistingContent("/home/missing.txt", &dst); err != nil || dst.Len() != 0 {
t.Fatalf("missing file: err=%v written=%d", err, dst.Len())
}
}
func TestCopyExistingContentTrailingGap(t *testing.T) {
chunkA := []byte("AAAAAAAAAA")
volume := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/3,01" {
http.Error(w, "not found", http.StatusNotFound)
return
}
_, _ = w.Write(chunkA)
}))
t.Cleanup(volume.Close)
entries := map[string]*filer_pb.Entry{
"/home/trailing.bin": chunkedEntry(50, fileChunk("3,01", 0, 10)),
}
fs := newCopyTestServer(t, volume.Listener.Addr().String(), entries)
var dst bytes.Buffer
if err := fs.copyExistingContent("/home/trailing.bin", &dst); err != nil {
t.Fatalf("copyExistingContent: %v", err)
}
want := append(append([]byte{}, chunkA...), make([]byte, 40)...)
if !bytes.Equal(dst.Bytes(), want) {
t.Fatalf("copied %d bytes, want %d with a zero-filled tail", dst.Len(), len(want))
}
}
+93 -6
View File
@@ -15,6 +15,7 @@ import (
"time"
"github.com/pkg/sftp"
"github.com/seaweedfs/seaweedfs/weed/filer"
"github.com/seaweedfs/seaweedfs/weed/glog"
"github.com/seaweedfs/seaweedfs/weed/pb"
filer_pb "github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
@@ -23,6 +24,7 @@ import (
"github.com/seaweedfs/seaweedfs/weed/sftpd/user"
"github.com/seaweedfs/seaweedfs/weed/util"
util_http "github.com/seaweedfs/seaweedfs/weed/util/http"
"github.com/seaweedfs/seaweedfs/weed/wdclient"
"google.golang.org/grpc"
)
@@ -159,6 +161,14 @@ func (fs *SftpServer) newFileWriter(r *sftp.Request) (io.WriterAt, error) {
if err != nil {
return nil, fmt.Errorf("failed to create temp file: %w", err)
}
// Without O_TRUNC the client writes into the existing file, so start from its current bytes.
if !r.Pflags().Trunc {
if err := fs.copyExistingContent(absPath, tmpFile); err != nil {
tmpFile.Close()
os.Remove(tmpFile.Name())
return nil, err
}
}
return &SeaweedSftpFileWriter{
fs: *fs,
@@ -172,6 +182,75 @@ func (fs *SftpServer) newFileWriter(r *sftp.Request) (io.WriterAt, error) {
}, nil
}
func (fs *SftpServer) copyExistingContent(absPath string, dst io.Writer) error {
entry, err := fs.getEntry(absPath)
if err == os.ErrNotExist {
return nil
}
if err != nil {
return err
}
if entry.IsDirectory {
return nil
}
size := int64(filer.FileSize(entry))
remote := entry.GetRemoteEntry()
if len(entry.Content) == 0 && len(entry.GetChunks()) == 0 {
if remote != nil && remote.RemoteSize > 0 {
return fmt.Errorf("existing content of %s is not readable", absPath)
}
if size == 0 {
return nil
}
return writeZeros(dst, size)
}
if size == 0 {
return nil
}
if len(entry.Content) > 0 {
if _, err := dst.Write(entry.Content); err != nil {
return err
}
if remaining := size - int64(len(entry.Content)); remaining > 0 {
return writeZeros(dst, remaining)
}
return nil
}
streamFn, err := filer.PrepareStreamContent(lookupFileIdFn(filer.LookupFn(fs)), filer.JwtForVolumeServer, entry.GetChunks(), 0, size)
if err != nil {
return fmt.Errorf("read existing content of %s: %w", absPath, err)
}
if err := streamFn(dst); err != nil {
return fmt.Errorf("read existing content of %s: %w", absPath, err)
}
return nil
}
// lookupFileIdFn adapts a filer LookupFn to the chunk streamer's provider
// interface. PrepareStreamContent fills unstored gaps with zeros, so a sparse
// file's tail survives the copy.
type lookupFileIdFn wdclient.LookupFileIdFunctionType
func (fn lookupFileIdFn) GetLookupFileIdFunction() wdclient.LookupFileIdFunctionType {
return wdclient.LookupFileIdFunctionType(fn)
}
func writeZeros(w io.Writer, size int64) error {
zeros := make([]byte, 8192)
for size > 0 {
n := int64(len(zeros))
if size < n {
n = size
}
written, err := w.Write(zeros[:n])
size -= int64(written)
if err != nil {
return err
}
}
return nil
}
func (fs *SftpServer) removeEntry(absPath string) error {
return fs.deleteEntry(absPath, false)
}
@@ -246,21 +325,24 @@ func (fs *SftpServer) listAllPages(dirPath string) (sftp.ListerAt, error) {
var all []os.FileInfo
last := ""
for {
page, err := fs.fetchDirectoryPage(dirPath, last)
page, lastName, received, err := fs.fetchDirectoryPage(dirPath, last)
if err != nil {
return nil, err
}
all = append(all, page...)
if len(page) < defaultListLimit {
if received < defaultListLimit {
break
}
last = page[len(page)-1].Name()
last = lastName
}
return listerat(all), nil
}
func (fs *SftpServer) fetchDirectoryPage(dirPath, start string) ([]os.FileInfo, error) {
// fetchDirectoryPage returns the entries the user may list, plus the last name and count the filer sent.
func (fs *SftpServer) fetchDirectoryPage(dirPath, start string) ([]os.FileInfo, string, int, error) {
var list []os.FileInfo
lastName := ""
received := 0
err := fs.callWithClient(true, func(ctx context.Context, client filer_pb.SeaweedFilerClient) error {
stream, err := client.ListEntries(ctx, &filer_pb.ListEntriesRequest{Directory: dirPath, StartFromFileName: start, Limit: defaultListLimit})
if err != nil {
@@ -271,9 +353,14 @@ func (fs *SftpServer) fetchDirectoryPage(dirPath, start string) ([]os.FileInfo,
if err == io.EOF {
break
}
if err != nil || r.Entry == nil {
if err != nil {
return err
}
if r.Entry == nil {
continue
}
received++
lastName = r.Entry.Name
p := path.Join(dirPath, r.Entry.Name)
if err := fs.checkFilePermission(p, "list"); err != nil {
continue
@@ -282,7 +369,7 @@ func (fs *SftpServer) fetchDirectoryPage(dirPath, start string) ([]os.FileInfo,
}
return nil
})
return list, err
return list, lastName, received, err
}
// makeDir creates a new directory with proper permissions.
+141
View File
@@ -1,12 +1,24 @@
package sftpd
import (
"context"
"fmt"
"net"
"net/http"
"net/http/httptest"
"os"
"strings"
"testing"
"time"
"github.com/pkg/sftp"
"github.com/seaweedfs/seaweedfs/weed/pb"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
"github.com/seaweedfs/seaweedfs/weed/sftpd/user"
"google.golang.org/grpc"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/credentials/insecure"
"google.golang.org/grpc/status"
)
// TestPutFileEscapesQueryInjection ensures a filename containing "?" cannot be
@@ -37,3 +49,132 @@ func TestPutFileEscapesQueryInjection(t *testing.T) {
t.Errorf("path not delivered literally: got %q, want %q", gotPath, malicious)
}
}
type fakeListFiler struct {
filer_pb.UnimplementedSeaweedFilerServer
names []string
failFrom int // ListEntries fails after sending this many entries, when > 0
}
func (f *fakeListFiler) ListEntries(req *filer_pb.ListEntriesRequest, stream grpc.ServerStreamingServer[filer_pb.ListEntriesResponse]) error {
sent := 0
for _, name := range f.names {
if name <= req.StartFromFileName || sent >= int(req.Limit) {
continue
}
if f.failFrom > 0 && sent == f.failFrom {
return status.Error(codes.Unavailable, "filer went away")
}
if err := stream.Send(&filer_pb.ListEntriesResponse{Entry: &filer_pb.Entry{Name: name, Attributes: &filer_pb.FuseAttributes{}}}); err != nil {
return err
}
sent++
}
return nil
}
func (f *fakeListFiler) LookupDirectoryEntry(ctx context.Context, req *filer_pb.LookupDirectoryEntryRequest) (*filer_pb.LookupDirectoryEntryResponse, error) {
return &filer_pb.LookupDirectoryEntryResponse{Entry: &filer_pb.Entry{Name: req.Name, Attributes: &filer_pb.FuseAttributes{}}}, nil
}
func newListTestServer(t *testing.T, filer *fakeListFiler) *SftpServer {
t.Helper()
lis, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
srv := grpc.NewServer()
filer_pb.RegisterSeaweedFilerServer(srv, filer)
go srv.Serve(lis)
t.Cleanup(srv.Stop)
return &SftpServer{
filerAddr: pb.ServerAddress(fmt.Sprintf("127.0.0.1:1.%d", lis.Addr().(*net.TCPAddr).Port)),
grpcDialOption: grpc.WithTransportCredentials(insecure.NewCredentials()),
user: &user.User{
Username: "alice",
HomeDir: "/",
Uid: 1001,
Gid: 1001,
// every entry under /d is listable except f00000
Permissions: map[string][]string{"/d": {"list"}, "/d/f00000": {}},
},
}
}
func listNames(t *testing.T, fs *SftpServer) ([]string, error) {
t.Helper()
type result struct {
lister sftp.ListerAt
err error
}
done := make(chan result, 1)
go func() {
lister, err := fs.listAllPages("/d")
done <- result{lister, err}
}()
select {
case r := <-done:
if r.err != nil {
return nil, r.err
}
infos := make([]os.FileInfo, 10000)
n, _ := r.lister.ListAt(infos, 0)
var names []string
for _, fi := range infos[:n] {
names = append(names, fi.Name())
}
return names, nil
case <-time.After(10 * time.Second):
t.Fatal("listing did not return")
return nil, nil
}
}
func fileNames(n int) []string {
names := make([]string, n)
for i := range names {
names[i] = fmt.Sprintf("f%05d", i)
}
return names
}
func TestListAllPagesReturnsStreamError(t *testing.T) {
fs := newListTestServer(t, &fakeListFiler{names: fileNames(10), failFrom: 3})
_, err := listNames(t, fs)
if status.Code(err) != codes.Unavailable {
t.Fatalf("expected the stream error, got %v", err)
}
}
func TestListAllPagesContinuesPastHiddenEntries(t *testing.T) {
fs := newListTestServer(t, &fakeListFiler{names: fileNames(defaultListLimit + 500)})
names, err := listNames(t, fs)
if err != nil {
t.Fatal(err)
}
if len(names) != defaultListLimit+499 {
t.Fatalf("expected %d entries, got %d", defaultListLimit+499, len(names))
}
if names[0] != "f00001" || names[len(names)-1] != fmt.Sprintf("f%05d", defaultListLimit+499) {
t.Fatalf("unexpected first/last entries %q, %q", names[0], names[len(names)-1])
}
}
func TestListAllPagesContinuesPastFullyHiddenPage(t *testing.T) {
const tail = 10
fs := newListTestServer(t, &fakeListFiler{names: fileNames(defaultListLimit + tail)})
// The whole first page is hidden from the user; only the tail is listable.
for i := 0; i < defaultListLimit; i++ {
fs.user.Permissions[fmt.Sprintf("/d/f%05d", i)] = []string{}
}
names, err := listNames(t, fs)
if err != nil {
t.Fatal(err)
}
if len(names) != tail {
t.Fatalf("expected the %d entries after the hidden page, got %d", tail, len(names))
}
if names[0] != fmt.Sprintf("f%05d", defaultListLimit) {
t.Fatalf("unexpected first entry %q", names[0])
}
}
+2 -1
View File
@@ -1,6 +1,7 @@
package shell
import (
"cmp"
"flag"
"fmt"
"io"
@@ -273,7 +274,7 @@ func moveAwayOneNormalVolume(commandEnv *CommandEnv, volumeReplicas map[uint32][
}
// most empty one is in the front
slices.SortFunc(otherNodes, func(a, b *Node) int {
return int(a.localVolumeRatio(maxVolumeCountFn) - b.localVolumeRatio(maxVolumeCountFn))
return cmp.Compare(a.localVolumeRatio(maxVolumeCountFn), b.localVolumeRatio(maxVolumeCountFn))
})
for i := 0; i < len(otherNodes); i++ {
emptyNode := otherNodes[i]
@@ -5,6 +5,22 @@ import (
"testing"
)
func TestEvacuatePrefersLowerVolumeRatio(t *testing.T) {
volumes := mkByteVolumes(1, 1, 1)
source := makeByteNode("source", 10, 0, 0, volumes)
fuller := makeByteNode("fuller", 10, 0, 0, mkByteVolumes(10, 4, 1))
emptier := makeByteNode("emptier", 10, 0, 0, mkByteVolumes(20, 2, 1))
// Put the 40%-full server first: truncating 0.2 - 0.4 to int makes the ratios tie.
moved, err := moveAwayOneNormalVolume(nil, nil, volumes[0], source, []*Node{fuller, emptier}, false)
if err != nil || !moved {
t.Fatalf("evacuate: moved=%v, err=%v", moved, err)
}
if _, found := emptier.selectedVolumes[volumes[0].Id]; !found {
t.Fatal("expected evacuation to choose the server with lower occupancy")
}
}
func TestVolumeServerEvacuate(t *testing.T) {
targetServer := ""
volumeRack := "rack1"
+1 -1
View File
@@ -982,7 +982,7 @@ func (ev *EcVolume) LocateEcShardNeedle(needleId types.NeedleId, version needle.
return types.Offset{}, 0, nil, fmt.Errorf("FindNeedleFromEcx: %w", err)
}
intervals = ev.LocateEcShardNeedleInterval(version, offset.ToActualOffset(), types.Size(needle.GetActualSize(size, version)))
intervals = ev.LocateEcShardNeedleInterval(version, offset.ToActualOffset(), size)
return
}
@@ -256,3 +256,30 @@ func TestEcVolumeDestroySweepsBothDirs(t *testing.T) {
}
}
}
func TestLocateEcShardNeedleMatchesInterval(t *testing.T) {
ecxFile, err := os.OpenFile("./test_files/389.ecx", os.O_RDONLY, 0)
assert.NoError(t, err)
defer ecxFile.Close()
stat, err := ecxFile.Stat()
assert.NoError(t, err)
fileSize := stat.Size()
ev := &EcVolume{
VolumeId: needle.VolumeId(389),
ecxFile: ecxFile,
ecxFileSize: fileSize,
Shards: []*EcVolumeShard{
{ecdFileSize: 1118830592},
},
ECContext: NewDefaultECContext("", 389),
}
needleId, _ := types.ParseNeedleId("0f0edb92")
offset, size, intervals, err := ev.LocateEcShardNeedle(needleId, needle.GetCurrentVersion())
assert.NoError(t, err)
expectedIntervals := ev.LocateEcShardNeedleInterval(needle.GetCurrentVersion(), offset.ToActualOffset(), size)
assert.Equal(t, expectedIntervals, intervals)
}
@@ -38,6 +38,10 @@ type Node struct {
rack string // composite rack key (e.g. "dc1:rack1")
freeSlots int
// noReceive marks a node that refused shards (e.g. answered "no space
// left"): it stays usable as a move source but is never picked as a target,
// regardless of what its freeSlots say.
noReceive bool
disks map[uint32]*disk
shards map[volKey]*volumeShards
}
@@ -47,7 +51,8 @@ type disk struct {
diskType string
tags []string // placement tags, for preferred-tag tiering
freeSlots int
shardCount int // total EC shards on this disk across all volumes
shardCount int // total EC shards on this disk across all volumes
noReceive bool // this disk refused a shard; siblings may still receive
}
type volumeShards struct {
@@ -156,6 +161,39 @@ func (n *Node) SetHost(host string) {
}
}
// SetNoReceive marks the node ineligible as a move destination. Use when the
// node itself refused shards (e.g. answered "no space left") so the heartbeat's
// free-slot count cannot be trusted; the node can still shed shards as a source.
func (n *Node) SetNoReceive() {
n.noReceive = true
}
// SetDiskNoReceive marks one disk on the node ineligible as a move destination.
// An explicit-disk copy can be refused while the node's other disks still have
// room; a disk_id 0 refusal means the server searched every disk and is a
// whole-node refusal (SetNoReceive) instead.
func (n *Node) SetDiskNoReceive(diskID uint32) {
if d, ok := n.disks[diskID]; ok {
d.noReceive = true
}
}
// receiveSlots is the node's receiving capacity: its aggregate free slots with
// refused disks subtracted, and 0 for a refused node. Slots a disk just
// refused are phantom capacity whatever the snapshot reported.
func (n *Node) receiveSlots() int {
if n.noReceive {
return 0
}
free := n.freeSlots
for _, d := range n.disks {
if d.noReceive {
free -= d.freeSlots
}
}
return max(free, 0)
}
// AddDisk registers a physical disk. shardCount is the disk's total EC shard
// count across all volumes (used for disk scoring); freeSlots is the per-disk
// free EC shard slots.
@@ -296,7 +334,7 @@ func buildRacks(nodes map[string]*Node) map[string]*rack {
racks[n.rack] = r
}
r.nodes[n.id] = n
r.freeSlots += n.freeSlots
r.freeSlots += n.receiveSlots()
}
return racks
}
@@ -440,7 +478,7 @@ func planRackTotalCap(vk volKey, racks map[string]*rack, rackShardCount map[stri
continue
}
for _, n := range r.nodes {
room[rk] += min(max(n.freeSlots, 0), max(rp.SameRackCount-volumeShardCount(n, vk), 0))
room[rk] += min(n.receiveSlots(), max(rp.SameRackCount-volumeShardCount(n, vk), 0))
}
}
return rackTotalCap(sortedKeys(racks), nil, room, totalShards)
@@ -575,12 +613,15 @@ func pickBestNodeForVolumeOnDisk(nodes []*Node, vk volKey, rp *super_block.Repli
// metadata is authoritative when present; without it the node's aggregate
// stands in so the volume server's own disk selection can place the shard.
func freeSlotsForType(n *Node, diskType string) int {
if n.noReceive {
return 0
}
if len(n.disks) == 0 {
return n.freeSlots
}
free := 0
for _, d := range n.disks {
if d.freeSlots > 0 && (diskType == "" || d.diskType == diskType) {
if d.freeSlots > 0 && !d.noReceive && (diskType == "" || d.diskType == diskType) {
free += d.freeSlots
}
}
@@ -604,7 +645,7 @@ func pickBestNodeForVolume(nodes []*Node, vk volKey, rp *super_block.ReplicaPlac
bestFree := -1
bestScore := -1
for _, node := range nodes {
if node.freeSlots <= 0 {
if node.receiveSlots() <= 0 {
continue
}
count := volumeShardCount(node, vk)
@@ -909,7 +950,7 @@ func detectGlobalImbalance(nodes map[string]*Node, racks map[string]*rack, diskT
continue
}
util := float64(count) / float64(capacity)
if util < minUtil && node.freeSlots > 0 {
if util < minUtil && node.receiveSlots() > 0 {
minUtil, minCount, minNode = util, count, node
}
if util > maxUtil {
@@ -1091,7 +1132,7 @@ func pickBestDiskOnNode(node *Node, vk volKey, diskType string, shardID, dataSha
if diskType != "" && d.diskType != diskType {
continue
}
if d.freeSlots <= 0 {
if d.freeSlots <= 0 || d.noReceive {
continue
}
@@ -926,3 +926,67 @@ func TestGlobalPrefersVolumeAbsentFromDestinationMachine(t *testing.T) {
t.Errorf("first global move is volume %d, want 200 (vol100 already on the destination machine)", moves[0].volumeID)
}
}
// TestNoReceiveNodeNeverTargeted: a node that refused a copy ("no space left")
// must not be picked as a destination no matter how many free slots it reports,
// but it still sheds its own over-cap shards as a source.
func TestNoReceiveNodeNeverTargeted(t *testing.T) {
const vid = uint32(1)
topo := NewTopology()
src := topo.AddNode("n1", "dc1", "dc1:r1", 50)
src.AddShards(vid, "c", 0, bits(0, 1, 2, 3, 4, 5, 6, 7, 8, 9))
full := topo.AddNode("n2", "dc1", "dc1:r1", 50)
full.AddShards(vid, "c", 0, bits(10, 11, 12, 13)) // all parity, over the 2/node cap
full.SetNoReceive()
topo.AddNode("n3", "dc1", "dc1:r1", 50)
moves := Plan(topo, Options{Ratio: ratio(10, 4)})
if len(moves) == 0 {
t.Fatal("expected moves off the over-cap nodes")
}
shed := false
for _, m := range moves {
if m.TargetNode == "n2" {
t.Fatalf("move targets the refusing node: %+v", m)
}
if m.SourceNode == "n2" {
shed = true
}
}
if !shed {
t.Error("the refusing node should still shed its over-cap shards as a source")
}
}
// TestNoReceiveDiskLeavesSiblingsEligible: an explicit-disk refusal marks only
// that disk, not the node — the planner may still target the node's other
// disks, but never the refused one.
func TestNoReceiveDiskLeavesSiblingsEligible(t *testing.T) {
const vid = uint32(1)
topo := NewTopology()
src := topo.AddNode("n1", "dc1", "dc1:r1", 50)
src.AddShards(vid, "c", 0, bits(0, 1, 2, 3, 4, 5, 6, 7, 8, 9))
n2 := topo.AddNode("n2", "dc1", "dc1:r1", 50)
// The refused disk scores better (empty); without the noReceive check the
// picker would choose it again.
n2.AddDisk(1, "hdd", 50, 0)
n2.AddDisk(2, "hdd", 50, 50)
n2.AddShards(vid, "c", 2, bits(10, 11, 12, 13))
n2.SetDiskNoReceive(1)
topo.AddNode("n3", "dc1", "dc1:r1", 50)
moves := Plan(topo, Options{Ratio: ratio(10, 4)})
targeted := false
for _, m := range moves {
if m.TargetNode != "n2" {
continue
}
targeted = true
if m.TargetDisk == 1 {
t.Fatalf("move targets the refused disk: %+v", m)
}
}
if !targeted {
t.Fatal("a node with one refused disk should still receive on its other disks")
}
}
@@ -430,6 +430,9 @@ func rackTotalCap(rackKeys []string, held, room map[string]int, totalShards int)
// its free slots, bounded by the per-disk durability cap (maxPerDisk shards of
// one volume per disk; <= 0 disables it).
func diskShardRoom(n *Node, d *disk, vk volKey, maxPerDisk int) int {
if d.noReceive {
return 0
}
room := d.freeSlots
if maxPerDisk > 0 {
held := 0
@@ -444,13 +447,16 @@ func diskShardRoom(n *Node, d *disk, vk volKey, maxPerDisk int) int {
// nodeShardRoom returns how many more shards of the volume the node's eligible
// disks can take, bounded by the node's free slots.
func nodeShardRoom(n *Node, vk volKey, eligible func(*disk) bool, maxPerDisk int) int {
if n.noReceive {
return 0
}
room := 0
for _, d := range n.disks {
if eligible(d) {
room += diskShardRoom(n, d, vk, maxPerDisk)
}
}
return max(min(room, n.freeSlots), 0)
return max(min(room, n.receiveSlots()), 0)
}
// rackShardRoom returns how many more shards of the volume the rack can take.
@@ -532,7 +538,7 @@ func pickBestDiskEligible(node *Node, vk volKey, eligible func(*disk) bool, pref
bestPreferred := false
for _, diskID := range sortedDiskKeys(node.disks) {
d := node.disks[diskID]
if !eligible(d) || d.freeSlots <= 0 {
if !eligible(d) || d.freeSlots <= 0 || d.noReceive {
continue
}
existingShards := 0
+13
View File
@@ -343,6 +343,19 @@ func (dn *DataNode) GetRack() *Rack {
return dn.Parent().(*NodeImpl).value.(*Rack)
}
func (dn *DataNode) matchesLocation(option *VolumeGrowOption) bool {
if option.DataCenter != "" && dn.GetDataCenter().Id() != NodeId(option.DataCenter) {
return false
}
if option.Rack != "" && dn.GetRack().Id() != NodeId(option.Rack) {
return false
}
if option.DataNode != "" && dn.Id() != NodeId(option.DataNode) {
return false
}
return true
}
func (dn *DataNode) GetTopology() *Topology {
p := dn.Parent()
for p.Parent() != nil {
+16
View File
@@ -448,6 +448,22 @@ func (t *Topology) GetVolumeLayout(collectionName string, rp *super_block.Replic
}).(*Collection).GetOrCreateVolumeLayout(rp, ttl, diskType)
}
// FindVolumeLayout is the read-only counterpart of GetVolumeLayout: it returns
// nil when the collection or its layout does not exist, instead of creating
// one. Lets callers check a layout's state without polluting the collection
// map with an empty collection.
func (t *Topology) FindVolumeLayout(collectionName string, rp *super_block.ReplicaPlacement, ttl *needle.TTL, diskType types.DiskType) *VolumeLayout {
c, ok := t.collectionMap.Find(collectionName)
if !ok {
return nil
}
vl, ok := c.(*Collection).GetVolumeLayout(rp, ttl, diskType)
if !ok {
return nil
}
return vl
}
// DecayQuietVolumeSizes decays pending assign estimates across every layout.
// A volume that changed reports within a pulse, so two quiet pulses mean the
// size on record is the size there is.
+40 -7
View File
@@ -650,13 +650,7 @@ func (vl *VolumeLayout) PickForWrite(count uint64, option *VolumeGrowOption) (vi
writableVolumeId := vl.writables[(start+i)%lenWriters]
volumeLocationList := vl.vid2location[writableVolumeId]
for _, dn := range volumeLocationList.list {
if option.DataCenter != "" && dn.GetDataCenter().Id() != NodeId(option.DataCenter) {
continue
}
if option.Rack != "" && dn.GetRack().Id() != NodeId(option.Rack) {
continue
}
if option.DataNode != "" && dn.Id() != NodeId(option.DataNode) {
if !dn.matchesLocation(option) {
continue
}
sample[found] = writableVolumeId
@@ -762,6 +756,45 @@ func (vl *VolumeLayout) ShouldGrowVolumes() bool {
return writable <= crowded
}
// WritableAvgFill returns the average fill ratio of the writable volumes and
// how many there are. The fill of a single volume is its tracked effective
// size over the layout's volume size limit; volumes without a size-tracking
// entry yet (freshly created, no heartbeat size report so far) count as
// empty. A non-empty location filter in option narrows the measure to
// writables serving that target, so a targeted grow is not blocked or
// triggered by fill elsewhere in the layout.
func (vl *VolumeLayout) WritableAvgFill(option *VolumeGrowOption) (avgFill float64, writableCount int) {
vl.accessLock.RLock()
defer vl.accessLock.RUnlock()
var total uint64
for _, vid := range vl.writables {
if option != nil && (option.DataCenter != "" || option.Rack != "" || option.DataNode != "") {
matched := false
if loc := vl.vid2location[vid]; loc != nil {
for _, dn := range loc.list {
if dn.matchesLocation(option) {
matched = true
break
}
}
}
if !matched {
continue
}
}
writableCount++
if st := vl.sizeTracking[vid]; st != nil {
total += st.effectiveSize
}
}
if writableCount == 0 || vl.volumeSizeLimit == 0 {
return 0, writableCount
}
avgFill = float64(total) / (float64(writableCount) * float64(vl.volumeSizeLimit))
return avgFill, writableCount
}
func (vl *VolumeLayout) ShouldGrowVolumesByDcAndRack(writables *[]needle.VolumeId, dcId NodeId, rackId NodeId) bool {
// When replication spans multiple racks (DiffRackCount > 0), a writable
// volume's replicas only cover some racks in a DC. It is wrong to
@@ -0,0 +1,179 @@
package topology
import (
"math"
"testing"
"github.com/seaweedfs/seaweedfs/weed/storage/needle"
"github.com/seaweedfs/seaweedfs/weed/storage/super_block"
"github.com/seaweedfs/seaweedfs/weed/storage/types"
)
func TestWritableAvgFill(t *testing.T) {
// 3 writable volumes at 20%, 50%, 80% fill (sizes 2000, 5000, 8000 of
// limit 10000): average fill is 50%.
layout := `
{
"dc1":{
"rack1":{
"server1":{
"volumes":[
{"id":1, "size":2000, "replication":"000"},
{"id":2, "size":5000, "replication":"000"},
{"id":3, "size":8000, "replication":"000"}
],
"limit":10
}
}
}
}
`
_, vl := setupPickTest(t, layout, 10000)
avgFill, writableCount := vl.WritableAvgFill(nil)
if writableCount != 3 {
t.Fatalf("expected 3 writable volumes, got %d", writableCount)
}
if math.Abs(avgFill-0.5) > 1e-9 {
t.Errorf("expected avg fill 0.5, got %v", avgFill)
}
}
func TestWritableAvgFillNoWritables(t *testing.T) {
layout := `
{
"dc1":{
"rack1":{
"server1":{
"volumes":[
{"id":1, "size":2000, "replication":"000"}
],
"limit":10
}
}
}
}
`
_, vl := setupPickTest(t, layout, 10000)
vl.accessLock.Lock()
vl.removeFromWritable(1)
vl.accessLock.Unlock()
avgFill, writableCount := vl.WritableAvgFill(nil)
if writableCount != 0 || avgFill != 0 {
t.Errorf("expected (0, 0) with no writables, got (%v, %d)", avgFill, writableCount)
}
}
// A writable volume without a size-tracking entry yet — freshly created, no
// heartbeat size report so far — counts as empty, not as an error.
func TestWritableAvgFillUntrackedVolumeCountsAsEmpty(t *testing.T) {
layout := `
{
"dc1":{
"rack1":{
"server1":{
"volumes":[
{"id":1, "size":5000, "replication":"000"},
{"id":2, "size":5000, "replication":"000"}
],
"limit":10
}
}
}
}
`
_, vl := setupPickTest(t, layout, 10000)
vl.accessLock.Lock()
delete(vl.sizeTracking, 2)
vl.accessLock.Unlock()
avgFill, writableCount := vl.WritableAvgFill(nil)
if writableCount != 2 {
t.Fatalf("expected 2 writable volumes, got %d", writableCount)
}
if math.Abs(avgFill-0.25) > 1e-9 {
t.Errorf("expected avg fill 0.25 (one of two volumes empty), got %v", avgFill)
}
}
func TestWritableAvgFillScopedToTarget(t *testing.T) {
layout := `
{
"dc1":{
"rack1":{
"server1":{
"volumes":[
{"id":1, "size":9000, "replication":"000"},
{"id":2, "size":9000, "replication":"000"}
],
"limit":10
}
}
},
"dc2":{
"rack2":{
"server2":{
"volumes":[
{"id":3, "size":1000, "replication":"000"}
],
"limit":10
}
}
}
}
`
_, vl := setupPickTest(t, layout, 10000)
avgFill, writableCount := vl.WritableAvgFill(&VolumeGrowOption{DataCenter: "dc2"})
if writableCount != 1 || math.Abs(avgFill-0.1) > 1e-9 {
t.Fatalf("expected dc2 fill (0.1, 1), got (%v, %d)", avgFill, writableCount)
}
// A target with no writable volumes reports none, so growth is not
// blocked by fill elsewhere.
avgFill, writableCount = vl.WritableAvgFill(&VolumeGrowOption{DataCenter: "dc2", DataNode: "missing"})
if writableCount != 0 || avgFill != 0 {
t.Fatalf("expected (0, 0) for a target with no writables, got (%v, %d)", avgFill, writableCount)
}
}
func TestFindVolumeLayout(t *testing.T) {
layout := `
{
"dc1":{
"rack1":{
"server1":{
"volumes":[
{"id":1, "size":2000, "collection":"colA", "replication":"000"}
],
"limit":10
}
}
}
}
`
topo := setupWithLimit(t, layout, 10000)
rp, _ := super_block.NewReplicaPlacementFromString("000")
// Existing layout is found.
if vl := topo.FindVolumeLayout("colA", rp, needle.EMPTY_TTL, types.HardDriveType); vl == nil {
t.Error("expected layout for colA to be found")
}
// A missing collection resolves to nil and must not be created.
if vl := topo.FindVolumeLayout("noSuchCollection", rp, needle.EMPTY_TTL, types.HardDriveType); vl != nil {
t.Error("expected nil layout for unknown collection")
}
if _, ok := topo.collectionMap.Find("noSuchCollection"); ok {
t.Error("FindVolumeLayout must not create the collection")
}
// A missing layout within an existing collection also resolves to nil.
rpOther, _ := super_block.NewReplicaPlacementFromString("001")
if vl := topo.FindVolumeLayout("colA", rpOther, needle.EMPTY_TTL, types.HardDriveType); vl != nil {
t.Error("expected nil layout for unused replication")
}
}
+4
View File
@@ -7,6 +7,10 @@ const (
ErrMsgOperationNotPermitted = "operation not permitted"
ErrMsgBadDigest = "The Content-Md5 you specified did not match what we received."
ErrMsgEntryNameTooLong = "entry name too long"
// Unsupported write-mode combinations; deterministic client errors.
ErrMsgAppendSaveInsideChunked = "append saved inside the entry to a chunked file is not supported"
ErrMsgAppendInlineMode = "append to inline content with this storage mode is not supported"
ErrMsgOffsetWriteInline = "offset write to inline small file is not supported yet"
// Suffix appended to a path, e.g. "/buckets/b/foo/bar is a file"
ErrMsgIsAFile = " is a file"
// Prefix + suffix, e.g. "existing /buckets/b/foo is a directory"
+79
View File
@@ -0,0 +1,79 @@
package util
import (
"net"
"strings"
)
// HostKey is the hostname/IP of an "host:port" address (or the string itself
// when it has no port). Compared case-insensitively so worker Hello
// "10.87.0.3:33646" matches a volume replica on "10.87.0.3:8080".
func HostKey(addr string) string {
addr = strings.TrimSpace(addr)
if addr == "" {
return ""
}
host, _, err := net.SplitHostPort(addr)
if err != nil {
return strings.ToLower(addr)
}
return strings.ToLower(host)
}
// HostKeysFromAddresses returns unique HostKey values, skipping empties.
func HostKeysFromAddresses(addrs []string) []string {
seen := make(map[string]struct{}, len(addrs))
var out []string
for _, a := range addrs {
h := HostKey(a)
if h == "" {
continue
}
if _, ok := seen[h]; ok {
continue
}
seen[h] = struct{}{}
out = append(out, h)
}
return out
}
// AddressIsLocal reports whether addr's host is this process's machine
// (loopback or a configured interface address). Used so an EC worker prefers
// a replica on the same box after replica sync.
func AddressIsLocal(addr string) bool {
host := HostKey(addr)
if host == "" {
return false
}
if host == "localhost" || host == "127.0.0.1" || host == "::1" {
return true
}
ip := net.ParseIP(host)
if ip != nil && ip.IsLoopback() {
return true
}
ifaces, err := net.InterfaceAddrs()
if err != nil {
return false
}
for _, ia := range ifaces {
var got net.IP
switch v := ia.(type) {
case *net.IPNet:
got = v.IP
case *net.IPAddr:
got = v.IP
}
if got == nil {
continue
}
if strings.EqualFold(got.String(), host) {
return true
}
if ip != nil && got.Equal(ip) {
return true
}
}
return false
}
+24
View File
@@ -0,0 +1,24 @@
package util
import "testing"
func TestHostKeyStripsPort(t *testing.T) {
t.Parallel()
if got := HostKey("10.87.0.3:8080"); got != "10.87.0.3" {
t.Fatalf("HostKey replica: got %q", got)
}
if got := HostKey("10.87.0.3:33646"); got != "10.87.0.3" {
t.Fatalf("HostKey worker: got %q", got)
}
if HostKey("10.87.0.3:8080") != HostKey("10.87.0.3:33646") {
t.Fatal("worker and volume server on the same IP must share a HostKey")
}
}
func TestHostKeysFromAddressesDedups(t *testing.T) {
t.Parallel()
got := HostKeysFromAddresses([]string{"10.0.0.1:8080", "10.0.0.1:8081", "10.0.0.2:8080", ""})
if len(got) != 2 || got[0] != "10.0.0.1" || got[1] != "10.0.0.2" {
t.Fatalf("got %v", got)
}
}
@@ -116,17 +116,29 @@ func (t *ECBalanceTask) Execute(ctx context.Context, params *worker_pb.TaskParam
return nil
}
err := mover.MoveEcShards(ctx, volume_move.EcShardMove{
move := volume_move.EcShardMove{
VolumeId: needle.VolumeId(params.VolumeId),
Collection: params.Collection,
ShardIds: shardIds,
Source: sourceAddr,
Target: targetAddr,
TargetDisk: target.DiskId,
}, volume_move.EcMoveOptions{
}
opts := volume_move.EcMoveOptions{
IoBytePerSecond: ecParams.GetIoBytePerSecond(),
Progress: t.reportProgress,
})
}
err := mover.MoveEcShards(ctx, move, opts)
// A "no space left" refusal to a pinned disk convicts only that disk;
// retry once letting the server pick any disk on the same node. The
// planner's node choice stands; only the intra-node disk defers to the
// server's live view. When the node is truly full, the retry fails with
// the same error.
if err != nil && target.DiskId > 0 && volume_move.IsNoSpaceLeftError(err) {
glog.Infof("EC balance: %s disk %d has no space left; retrying with server-picked disk", target.Node, target.DiskId)
move.TargetDisk = 0
err = mover.MoveEcShards(ctx, move, opts)
}
if err != nil {
return err
}
@@ -6,6 +6,8 @@ import (
"testing"
"github.com/seaweedfs/seaweedfs/weed/pb/worker_pb"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
)
func TestECBalanceTaskRejectsOutOfRangeShardIds(t *testing.T) {
@@ -25,3 +27,32 @@ func TestECBalanceTaskRejectsOutOfRangeShardIds(t *testing.T) {
t.Fatalf("Execute: expected out-of-range rejection, got: %v", err)
}
}
// A refusal against the pinned disk convicts only that disk, not the node:
// the task retries once letting the server pick a disk, and the move completes.
func TestECBalanceTaskRetriesServerPickOnPinnedDiskRefusal(t *testing.T) {
source := startFakeEcVolumeServer(t, 7, 5)
target := startFakeEcVolumeServer(t, 7)
target.refuseCopyOnDisk = 2
task := NewECBalanceTask("t2", 7, "c1",
grpc.WithTransportCredentials(insecure.NewCredentials()))
params := &worker_pb.TaskParams{
VolumeId: 7,
Sources: []*worker_pb.TaskSource{{Node: source.address(), ShardIds: []uint32{5}}},
Targets: []*worker_pb.TaskTarget{{Node: target.address(), DiskId: 2, ShardIds: []uint32{5}}},
}
if err := task.Execute(context.Background(), params); err != nil {
t.Fatalf("Execute: %v", err)
}
if got := target.copyDiskIds; len(got) != 2 || got[0] != 2 || got[1] != 0 {
t.Fatalf("expected copies to disk 2 then server-picked 0, got %v", got)
}
if source.has(7, 5) {
t.Fatalf("source still holds shard 5")
}
if !target.has(7, 5) {
t.Fatalf("target never received shard 5")
}
}
@@ -32,6 +32,12 @@ type fakeEcVolumeServer struct {
// the request even when it was a no-op against the inventory.
deleted []string
unmounted []string
// refuseCopyOnDisk, when nonzero, makes VolumeEcShardsCopy fail requests
// pinning that disk id with "no space left" — a full disk; a request with
// DiskId 0 (server picks) still succeeds.
refuseCopyOnDisk uint32
// copyDiskIds records the DiskId each copy request carried.
copyDiskIds []uint32
// grpcAddr is host:port; the code under test dials by address string.
grpcAddr string
stop func()
@@ -46,11 +52,31 @@ func (f *fakeEcVolumeServer) VolumeEcShardsInfo(ctx context.Context, req *volume
VolumeId: req.VolumeId,
ShardId: sid,
Collection: f.collection,
Size: 1,
})
}
return resp, nil
}
// VolumeEcShardsCopy records the request's DiskId and adds the shards to this
// server's inventory, unless the pinned disk is the configured refusing disk.
func (f *fakeEcVolumeServer) VolumeEcShardsCopy(ctx context.Context, req *volume_server_pb.VolumeEcShardsCopyRequest) (*volume_server_pb.VolumeEcShardsCopyResponse, error) {
f.mu.Lock()
defer f.mu.Unlock()
f.copyDiskIds = append(f.copyDiskIds, req.DiskId)
if f.refuseCopyOnDisk != 0 && req.DiskId == f.refuseCopyOnDisk {
return nil, fmt.Errorf("disk %d: no space left", req.DiskId)
}
for _, sid := range req.ShardIds {
f.shards[req.VolumeId][sid] = true
}
return &volume_server_pb.VolumeEcShardsCopyResponse{}, nil
}
func (f *fakeEcVolumeServer) VolumeEcShardsMount(ctx context.Context, req *volume_server_pb.VolumeEcShardsMountRequest) (*volume_server_pb.VolumeEcShardsMountResponse, error) {
return &volume_server_pb.VolumeEcShardsMountResponse{}, nil
}
func (f *fakeEcVolumeServer) VolumeEcShardsDelete(ctx context.Context, req *volume_server_pb.VolumeEcShardsDeleteRequest) (*volume_server_pb.VolumeEcShardsDeleteResponse, error) {
f.mu.Lock()
defer f.mu.Unlock()
@@ -21,6 +21,7 @@ import (
storagetypes "github.com/seaweedfs/seaweedfs/weed/storage/types"
"github.com/seaweedfs/seaweedfs/weed/storage/volume_info"
"github.com/seaweedfs/seaweedfs/weed/storage/volume_replica"
"github.com/seaweedfs/seaweedfs/weed/util"
"github.com/seaweedfs/seaweedfs/weed/wdclient"
"github.com/seaweedfs/seaweedfs/weed/worker/types"
"github.com/seaweedfs/seaweedfs/weed/worker/types/base"
@@ -36,6 +37,9 @@ type ErasureCodingTask struct {
workDir string
progress float64
grpcDialOption grpc.DialOption
// workerAddress is weed worker -address (host:port). Not a volume URL;
// late-bind matches replica hostnames against HostKey(workerAddress).
workerAddress string
// EC parameters
dataShards int32
@@ -403,9 +407,75 @@ func (t *ErasureCodingTask) syncAndSelectSourceReplica() error {
glog.Infof("EC encode: using best replica %s as source for volume %d (was %s)", best.Url, t.volumeID, t.server)
t.server = best.Url
}
// Late-bind from after consistency: worker -address hostname, not Sources[0]
// and not AddressIsLocal (replica URLs are hostnames like full-02:8081).
workerAddr := strings.TrimSpace(t.workerAddress)
if workerAddr == "" {
workerAddr, _ = os.Hostname()
}
if local := preferWorkerHostReplicaIfComplete(t.grpcDialOption, needle.VolumeId(t.volumeID), locs, t.server, workerAddr); local != "" && local != t.server {
glog.Infof("EC encode: preferring worker-host replica %s as source for volume %d (was %s)", local, t.volumeID, t.server)
t.server = local
}
return nil
}
type replicaFileCount struct {
url string
fileCount uint64
err error
}
// preferWorkerHostReplicaIfComplete rebinds from to a same-hostname replica
// only when live FileCounts are equal. Diverged counts keep current (max).
func preferWorkerHostReplicaIfComplete(dial grpc.DialOption, vid needle.VolumeId, locs []wdclient.Location, current, workerAddress string) string {
statuses := volume_replica.GetReplicaStatuses(dial, vid, locs)
replicas := make([]replicaFileCount, 0, len(statuses))
for _, s := range statuses {
replicas = append(replicas, replicaFileCount{url: s.Location.Url, fileCount: s.FileCount, err: s.Error})
}
return lateBindSourceToWorkerHost(current, workerAddress, replicas)
}
// lateBindSourceToWorkerHost: equal FileCount → replica on worker -address host
// (any port). Diverge → keep current (already max FileCount). Worker address
// is not a volume URL.
func lateBindSourceToWorkerHost(current, workerAddress string, replicas []replicaFileCount) string {
workerHost := util.HostKey(workerAddress)
if workerHost == "" {
return current
}
var live []replicaFileCount
for _, r := range replicas {
if r.err != nil || r.url == "" {
continue
}
live = append(live, r)
}
if len(live) == 0 {
return current
}
maxCount := live[0].fileCount
allSame := true
for _, r := range live[1:] {
if r.fileCount != live[0].fileCount {
allSame = false
}
if r.fileCount > maxCount {
maxCount = r.fileCount
}
}
if !allSame {
return current
}
for _, r := range live {
if util.HostKey(r.url) == workerHost {
return r.url
}
}
return current
}
// rollbackReadonly is a best-effort restore of every replica markReplicasReadonly
// touched, used when the EC task fails before the originals are deleted. Logs but
// does not return errors; uses a fresh context since the caller's may be cancelled.
@@ -0,0 +1,31 @@
package erasure_coding
import "testing"
func TestLateBindSourceToWorkerHostEqualFileCountPrefersWorkerHost(t *testing.T) {
a := replicaFileCount{url: "full-01:8080", fileCount: 21}
b := replicaFileCount{url: "full-02:8081", fileCount: 21}
worker := "full-02:9334"
// Sources [A,B] — topology-first is A; worker on B must still copy B.
if got := lateBindSourceToWorkerHost(a.url, worker, []replicaFileCount{a, b}); got != b.url {
t.Fatalf("Sources [A,B] worker B: got %s want %s", got, b.url)
}
// Invert Sources — still B.
if got := lateBindSourceToWorkerHost(b.url, worker, []replicaFileCount{b, a}); got != b.url {
t.Fatalf("Sources [B,A] worker B: got %s want %s", got, b.url)
}
}
func TestLateBindSourceToWorkerHostDivergeKeepsMaxFileCount(t *testing.T) {
a := replicaFileCount{url: "full-01:8080", fileCount: 40}
b := replicaFileCount{url: "full-02:8081", fileCount: 21}
worker := "full-02:9334"
if got := lateBindSourceToWorkerHost(a.url, worker, []replicaFileCount{a, b}); got != a.url {
t.Fatalf("diverge [A,B]: got %s want max FileCount %s", got, a.url)
}
if got := lateBindSourceToWorkerHost(a.url, worker, []replicaFileCount{b, a}); got != a.url {
t.Fatalf("diverge [B,A]: got %s want max FileCount %s", got, a.url)
}
}
@@ -26,7 +26,9 @@ func init() {
Category: pluginworker.CategoryHeavy,
Aliases: []string{"erasure-coding", "erasure.coding", "ec"},
Build: func(opts pluginworker.HandlerBuildOptions) (pluginworker.JobHandler, error) {
return NewErasureCodingHandler(opts.GrpcDialOption, opts.WorkingDir), nil
h := NewErasureCodingHandler(opts.GrpcDialOption, opts.WorkingDir)
h.workerAddress = strings.TrimSpace(opts.WorkerAddress)
return h, nil
},
})
}
@@ -39,6 +41,7 @@ type erasureCodingWorkerConfig struct {
type ErasureCodingHandler struct {
grpcDialOption grpc.DialOption
workingDir string
workerAddress string
}
func NewErasureCodingHandler(grpcDialOption grpc.DialOption, workingDir string) *ErasureCodingHandler {
@@ -519,6 +522,7 @@ func (h *ErasureCodingHandler) Execute(
params.Collection,
h.grpcDialOption,
)
task.workerAddress = h.workerAddress
execCtx, execCancel := context.WithCancel(ctx)
defer execCancel()
task.SetProgressCallback(func(progress float64, stage string) {
@@ -670,8 +674,15 @@ func buildErasureCodingProposal(
}
sourceNode := ""
if len(params.Sources) > 0 {
sourceNode = strings.TrimSpace(params.Sources[0].Node)
var replicaServers []string
for _, s := range params.Sources {
if s == nil || strings.TrimSpace(s.Node) == "" || len(s.ShardIds) > 0 {
continue
}
replicaServers = append(replicaServers, s.Node)
if sourceNode == "" {
sourceNode = strings.TrimSpace(s.Node)
}
}
summary := fmt.Sprintf("Erasure code volume %d", result.VolumeID)
@@ -702,6 +713,9 @@ func buildErasureCodingProposal(
"source_server": {
Kind: &plugin_pb.ConfigValue_StringValue{StringValue: sourceNode},
},
"replica_servers": {
Kind: &plugin_pb.ConfigValue_StringList{StringList: &plugin_pb.StringList{Values: replicaServers}},
},
"collection": {
Kind: &plugin_pb.ConfigValue_StringValue{StringValue: result.Collection},
},