mirrors/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-07-27 02:23:23 +00:00
Code Issues Packages Projects Releases Wiki Activity
11,434 Commits 157 Branches 353 Tags
5aff086aa4e9ec0070650ccd8b70d9c67b7f3968
Commit Graph
892 Commits
Author SHA1 Message Date
NyaMistyandGitHub 8d0e6f1ead fix: volume.list volume info output not in order (#6737) 2025-04-27 08:52:49 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>chrislu
 		216c52e377 chore(deps): bump gocloud.dev from 0.40.0 to 0.41.0 (#6679) 
* chore(deps): bump gocloud.dev from 0.40.0 to 0.41.0

Bumps [gocloud.dev](https://github.com/google/go-cloud) from 0.40.0 to 0.41.0.
- [Release notes](https://github.com/google/go-cloud/releases)
- [Commits](https://github.com/google/go-cloud/compare/v0.40.0...v0.41.0)

---
updated-dependencies:
- dependency-name: gocloud.dev
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix error

* fix printing errors

* Update go.mod

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
 2025-03-31 21:42:54 -07:00
chrislu c45b8bd6ac add more help message 
fix https://github.com/seaweedfs/seaweedfs/issues/6625
 2025-03-13 09:11:16 -07:00
Lisandro PinandGitHub c07596691c ec.encode: Fix resolution of target collections. (#6585) 
* Don't ignore empty (`""`) collection names when computing collections for a given volume ID.

* `ec.encode`: Fix resolution of target collections.

When no `volumeId` parameter is provided, compute volumes
based on the provided collection name, even if it's empty (`""`).

This restores behavior to before recent EC rebalancing rework. See also
https://github.com/seaweedfs/seaweedfs/blob/ec30a504bae6cad75f859964e14c60d39cc43709/weed/shell/command_ec_encode.go#L99 .
 2025-02-28 11:42:19 -08:00
Lisandro PinandGitHub 76a111f0a2 Fix calculation of node's free EC shard slots. (#6584) 2025-02-28 07:35:28 -08:00
Changrui ChenandGitHub be74548cb5 fix: error info size bug in command_fs_merge_volumes.go (#6567) 2025-02-23 06:36:47 -08:00
Lisandro PinandGitHub 392656d59e ec.encode: Explictly mount EC shards after volume conversion. (#6528) 
This guarantees EC shards are immediately available after encoding,
even if not affected by subsequent re-balancing.
 2025-02-10 09:49:58 -08:00
Lisandro PinandGitHub e8d8bfcccc Nit: remove missing newlines on weed shell commands output. (#6524) 
Nit: remove missing newlines on `weed` commands output.
 2025-02-07 10:27:04 -08:00
Lisandro PinGitHubChris Lu
 		29c2d9b965 Remove warning on EC balancing if no replica placement settings are found. (#6516) 
Effectively undoes c9399a68; with ff8bd862, a replica placement type `000`
will no longer break shards re-balancing.

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2025-02-06 09:19:28 -08:00
Lisandro PinandGitHub 68f547bdf2 Nit: fix missing newline on EC balancing warnings regarding replica settings (#6509) 
Nit: fix missing newline on EC balancing warnings regarding replica settings.

See 79136812.
 2025-02-04 10:59:25 -08:00
Lisandro PinandGitHub 331c1f0f3f Improve EC shards balancing logic regarding replica placement settings. (#6491) 
The replica placement type specifies numebr of _replicas_ on the same/different rack;
that means we can have one EC shard copy on each, even if the replica setting is zero.

This PR reworks replica placement parsing for EC rebalancing, so we check allow
(replica placement + 1) when selecting racks and nodes to balance EC shards into.
 2025-01-30 09:26:45 -08:00
Lisandro PinandGitHub 250fbbb3db ec.balance: Allow EC balancing without collections. (#6488) 2025-01-29 08:51:59 -08:00
Lisandro PinandGitHub 7913681297 ec.encode: Display a warning on EC balancing if no replica placement settings are found. (#6487) 2025-01-29 08:50:19 -08:00
Chris LuandGitHub cc05874d06 Add message queue agent (#6463) 
* scaffold message queue agent

* adjust proto, add mq_agent

* add agent client implementation

* remove unused function

* agent publish server implementation

* adding agent
 2025-01-20 22:19:27 -08:00
Hadi ZamaniandGitHub b2f56d9add Add JWT authentication to fs.mergeVolumes command (#6461) 
Add jwt authentication to fs.mergeVolumes command
 2025-01-20 18:16:46 -08:00
Lisandro PinandGitHub eab2e0e112 ec.encode: Fix bug causing source volumes not being deleted after EC conversion. (#6447) 
This logic was originally part of `spreadEcShards()`, which got removed during
the unification effort with `ec.balance` (https://github.com/seaweedfs/seaweedfs/pull/6344),
accidentally breaking functionality in the process.

The commit restores the deletion code for EC'd volumes - with parallelization support.
 2025-01-17 01:02:30 -08:00
ftong2020andGitHub e7f2936dcc fix force arg dropped during volume balance command (#6432) 2025-01-12 23:30:18 -08:00
dsdandGitHub da2a234b00 [weed] change -n to -force (#6421) 2025-01-08 09:57:18 -08:00
Brad MurrayandGitHub bc3640ee64 Update command_fs_merge_volumes.go (#6406) 2025-01-02 08:57:26 -08:00
Guang Jiong LouandGitHub 3b1ac77e1f worm grace period and retention time support (#6404) 
Signed-off-by: lou <alex1988@outlook.com>
 2024-12-31 18:41:43 -08:00
dsdandGitHub 20cbc9e4eb skip error while executing volume.fix.replication (#6382) 2024-12-20 07:36:13 -08:00
chrislu ec155022e7 "golang.org/x/exp/slices" => "slices" and go fmt 2024-12-19 19:25:06 -08:00
Lisandro PinandGitHub 4d91ec359b Fix volume replica parallelization within ec.encode. (#6377) 
See 826edd5d.
 2024-12-19 17:46:11 -08:00
Lisandro PinandGitHub ba0707af64 Allow configuring the maximum number of concurrent tasks for EC parallelization. (#6376) 
Follow-up to b0210df0.
 2024-12-18 13:26:26 -08:00
Lisandro PinandGitHub 44c48c929a Parallelize volume replica operations within ec.encode. (#6374) 2024-12-18 11:59:48 -08:00
Lisandro PinandGitHub 9fbc4ea417 Rework shell.EcBalance()'s waitgroup code into a standalone type. (#6373) 
Rework `shell.EcBalance()`'s waitgroup with errors code into a standalone type.

We'll re-use this for other EC jobs - for example, volume creation. Also fixes
potential concurrency issues when collecting error results.
 2024-12-17 09:39:51 -08:00
Lisandro PinandGitHub 9b48ce0613 Parallelize EC shards balancing within racks (#6354) 
Parallelize EC shards balancing within racks.
 2024-12-15 13:36:23 -08:00
Lisandro PinandGitHub 926cfea3dc Parallelize EC shards balancing across racks. (#6352) 2024-12-13 06:05:32 -08:00
Lisandro PinandGitHub b81def5e5c Parallelize EC balancing for racks. (#6351) 2024-12-13 05:33:53 -08:00
Konstantin LebedevandGitHub 0a4b1909a2 [shell] only apply the balancing for writable volumes (#6346) 2024-12-13 01:10:00 -08:00
Lisandro PinandGitHub b0210df081 Begin implementing EC balancing parallelization support. (#6342) 
* Begin implementing EC balancing parallelization support.

Impacts both `ec.encode` and `ec.balance`,

* Nit: improve type naming.

* Make the goroutine workgroup handler for `EcBalance()` a bit smarter/error-proof.

* Nit: unify naming for `ecBalancer` wait group methods with the rest of the module.

* Fix concurrency bug.

* Fix whitespace after Gitlab automerge.

* Delete stray TODO.
 2024-12-12 09:14:44 -08:00
Lisandro PinandGitHub 23ffbb083c Limit EC re-balancing for ec.encode to relevant collections when a volume ID argument is provided. (#6347) 
Limit EC re-balancing for `ec.encode` to relevant collections when a volume ID is provided.
 2024-12-12 08:41:33 -08:00
Lisandro PinandGitHub 6320036c56 Delete legacy balancing code for ec.encode. (#6344) 2024-12-12 07:42:03 -08:00
Konstantin LebedevandGitHub 700b95304b [shell] volume.list show only writable volumes (#6338) 
* show only writable volumes

* fix import
 2024-12-11 09:06:15 -08:00
Konstantin LebedevandGitHub c37281735e volume.list avoid output empty data center and rack and disk info (#6341) 2024-12-11 09:03:16 -08:00
Lisandro PinandGitHub 8c82c037b9 Unify the re-balancing logic for ec.encode with ec.balance. (#6339) 
Among others, this enables recent changes related to topology aware
re-balancing at EC encoding time.
 2024-12-10 13:30:13 -08:00
Konstantin LebedevandGitHub ff1392f7f4 [shell] use constant for hdd of type (#6337) 
use constant for hdd of type
 2024-12-10 08:43:59 -08:00
Lisandro PinandGitHub 522a25790a Remove average constraints when selecting nodes/racks to balance EC shards into. (#6325) 2024-12-06 09:00:06 -08:00
Lisandro PinandGitHub 34cdbdd279 Share common parameters for EC re-balancing functions under a single struct. (#6319) 
TODO cleanup for https://github.com/seaweedfs/seaweedfs/discussions/6179.
 2024-12-05 09:00:46 -08:00
Lisandro PinandGitHub edef485333 Account for replication placement settings when balancing EC shards within the same rack. (#6317) 
* Account for replication placement settings when balancing EC shards within racks.

* Update help contents for `ec.balance`.

* Add a few more representative test cases for `pickEcNodeToBalanceShardsInto()`.
 2024-12-04 10:47:51 -08:00
Lisandro PinandGitHub 351efa134d Account for replication placement settings when balancing EC shards across racks. (#6316) 2024-12-04 09:00:55 -08:00
Lisandro PinandGitHub b2ba7d7408 Resolve replica placement for EC volumes from master server defaults. (#6303) 2024-12-02 08:44:07 -08:00
Lisandro PinandGitHub 9a741a61b1 Display details upon failures to re-balance EC shards racks. (#6299) 2024-11-28 08:42:41 -08:00
Lisandro PinandGitHub 559a1fd0f4 Improve EC shards rebalancing logic across nodes (#6297) 
* Improve EC shards rebalancing logic across nodes.

- Favor target nodes with less preexisting shards, to ensure a fair distribution.
- Randomize selection when multiple possible target nodes are available.
- Add logic to account for replication settings when selecting target nodes (currently disabled).

* Fix minor test typo.

* Clarify internal error messages for `pickEcNodeToBalanceShardsInto()`.
 2024-11-27 11:51:57 -08:00
Trim21andGitHub d43fa07f06 use readable bytes size string in shell output (#6288) 2024-11-25 17:25:17 -08:00
chrislu 04081128a9 use math rand v2 2024-11-21 08:54:03 -08:00
Lisandro PinandGitHub ca499de1cb Improve EC shards rebalancing logic across racks (#6270) 
Improve EC shards rebalancing logic across racks.

  - Favor target shards with less preexisting shards, to ensure a fair distribution.
  - Randomize selection when multiple possible target shards are available.
  - Add logic to account for replication settings when selecting target shards (currently disabled).
 2024-11-21 08:46:24 -08:00
Konstantin LebedevandGitHub a143c888e5 [shell] don't require lock when there are no changes for volume.fix.replication (#6266) 
* don't require lock when there are no changes

* revert takeAction
 2024-11-21 08:17:25 -08:00
Konstantin LebedevandGitHub 254ed8897e [shell] add noLock param for volume.move (#6261) 2024-11-20 08:35:24 -08:00
chrislu e28f55eb08 typo 2024-11-19 08:32:31 -08:00