Improve EC shards rebalancing logic across racks.
- Favor target shards with less preexisting shards, to ensure a fair distribution.
- Randomize selection when multiple possible target shards are available.
- Add logic to account for replication settings when selecting target shards (currently disabled).
* Rename `command_ec_encode_test.go` to `command_ec_common_test.go`.
All tests defined in this file are now for `command_ec_common.go`.
* Minor code cleanups.
- Fix broken `ec.balance` test.
- Rework integer ceiling division to not use floats, which can introduce precision errors.
* Introduce logic to resolve volume replica placement within EC rebalancing.
This will be used to make rebalancing logic topology-aware.
* Give shell.EcNode.dc a dedicated DataCenterId type.
* fix: parse filename in PUT + refactor
* fix: master iu public url with http
* fix: better parsing and handle disposition header
* fix: take mime type from file extension if not set
* Refactor `ec.balance` logic into a `weeed/shell/command_ec_common.go` standalone function.
This is a prerequisite to unify the balance logic for `ec.balance` and `ec.encode'.
* s/Balance()/EcBalance()/g
* listing files to convert to parquet
* write parquet files
* save logs into parquet files
* pass by value
* compact logs into parquet format
* can skip existing files
* refactor
* refactor
* fix compilation
* when no partition found
* refactor
* add untested parquet file read
* rename package
* refactor
* rename files
* remove unused
* add merged log read func
* parquet wants to know the file size
* rewind by time
* pass in stop ts
* add stop ts
* adjust log
* minor
* adjust log
* skip .parquet files when reading message logs
* skip non message files
* Update subscriber_record.go
* send messages
* skip message data with only ts
* skip non log files
* update parquet-go package
* ensure a valid record type
* add new field to a record type
* Update read_parquet_to_log.go
* fix parquet file name generation
* separating reading parquet and logs
* add key field
* add skipped logs
* use in memory cache
* refactor
* refactor
* refactor
* refactor, and change compact log
* refactor
* rename
* refactor
* fix format
* prefix v to version directory
fix deadlock when broadcast to clients
when master thransfer leader, the old master will disconnect with all
filers and volumeServers, if the cluster is a big , the broadcast
messages may be more big than the max of the channel len 100, then if the
KeepConnect was not listen on the channel in disconnect, it will
deadlock. and the whole cluster will not serve!
* set larger buf size for LogBuffer
* jump to next day when no more entry found
* Update weed/filer/filer_notify_read.go
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Konstantin LebedevandGitHub
Trim21andGitHub
chrislu
Lisandro PinandGitHub
Kevin BulteelandGitHub
Eugeniy E. MikhailovandGitHub
BruceandGitHub
wyangandGitHub
hanandGitHub