* fix(volume): track EC shard read errors and unmount on faulty media
Extract the volume EIO tracker into a reusable IoErrorTracker and add the
same tracking to EcVolume. Sustained EIO on .ecx lookups or .ecd shard
reads now unmounts the EC volume in the heartbeat (without deleting
files) so the master re-replicates from healthy peers, mirroring the
existing volume replica quarantine.
Closes#11227 (EC shard unmount).
* rust(volume): mirror EC shard read error tracking and unmount
Add EIO tracking to the Rust EcVolume mirroring Go: a streak counter
with IO_ERROR_TOLERANCE, a sticky quarantine flag, and unmount (not
file deletion) in the heartbeat so the master re-replicates from
healthy peers.
* feat(metrics): expose storage IO error counter and quarantine gauge
Add a storage_io_error_total counter incremented on every EIO recorded
by the volume or EC shard tracker, and an io_quarantine gauge labelled
by kind (volume/ec_shard) reflecting the count of replicas suppressed
in the heartbeat. Mirrored in Go and Rust.
* feat(healthz): report 503 when local replicas are IO-quarantined
Add Store.HasIoQuarantine (Go) / Store::has_io_quarantine (Rust) and
have /healthz return 503 when any local volume or EC shard is
quarantined due to sustained storage-media EIO, so a load balancer
can drain a server whose underlying media is faulty. Mirrored in Go
and Rust.
* fix(volume): keep quarantined EC volumes in memory and reset EIO on success
Address review feedback: instead of unloading quarantined EC volumes
(which discards the quarantine state healthz needs), keep them in
memory and just skip them from heartbeat reporting, mirroring the
regular volume quarantine. Also clear the EIO streak on successful
.ecx reads in Rust so a transient error does not accumulate, and add
an ec_shard label to the io_quarantine gauge in both Go and Rust.
* fix(volume): exclude quarantined EC shards from heartbeat and add Rust volume tolerance
Address review feedback:
- Filter quarantined EC volumes from CollectErasureCodingHeartbeat
(Go) and collect_ec_shard_delta_messages / collect_live_ec_shards
(Rust) so the master stops advertising faulty shards and
re-replicates from healthy peers.
- Add consecutive EIO count and sticky quarantine to the Rust
regular Volume, mirroring Go IoErrorTracker: a single EIO no
longer deletes the replica; the heartbeat quarantines after the
tolerance threshold and keeps the volume in memory.
- Use the quarantine flag (not last_io_error) in has_io_quarantine
so /healthz reflects sustained, not transient, failures.
* fix(volume): make Rust quarantined volumes read-only and wire recovery
Address Devin review:
- Set no_write_or_delete on Rust volumes when quarantined in the
heartbeat, so cached or direct clients cannot mutate a faulty
replica after the master removes it (mirrors Go).
- Wire reset_io_error_state into Volume::set_writable so an operator
making a volume writable again clears the sticky quarantine and
the volume re-enters heartbeat rotation.
* fix(volume): clear EC quarantine on shard re-mount for operator recovery
Address Greptile review: re-mounting EC shards (Go loadEcShardWithIdxDir
/ Rust mount_ec_shards_with_idx_dir) now calls ResetIoErrorState on the
existing EcVolume, giving operators a documented recovery path that
clears the sticky quarantine and returns the EC volume to heartbeat
rotation. Mirrored in Go and Rust.
* fix(volume): do not clear EC quarantine on routine shard mounts
Address review feedback: clearing the EC IO quarantine on every mount
(including duplicate, retry, sibling-shard, and reconciliation mounts)
is too aggressive and can re-advertise known-bad shards before the
storage media has been validated. Remove the automatic reset from the
mount path; quarantine clears naturally on restart or full unmount
when a fresh EcVolume is created with clean state.
* test(volume): update Rust IO error test for quarantine semantics
The heartbeat now quarantines a volume with sustained EIO (keeps it
mounted, makes it read-only, omits it from heartbeat) instead of
deleting it. Update test_collect_heartbeat_deletes_io_error_volume to
assert the volume stays in the store with no_write_or_delete set, and
update set_last_io_error_for_test to set the consecutive error count
at the tolerance threshold so the test reflects a sustained error.
* fix(volume): reset EIO streak after full write and match Windows media errors
Move the success-side EIO reset from append_needle (after write_all only)
to the end of do_write_request, after flush_dat/flush_idx complete, so a
successful write_all followed by a failed fsync no longer resets the
counter before the EIO is recorded. Repeated fsync EIOs now accumulate
toward the quarantine threshold as intended.
Recognize Windows storage-media failure codes ERROR_CRC (23) and
ERROR_IO_DEVICE (1117) in addition to Unix EIO (errno 5), so quarantined
heartbeat behavior is preserved on Windows. Mirrors the change in both
Go and Rust volume servers.
* fix(volume): preserve checkpoint EIO and clear streak on successful delete
maybe_checkpoint_index now returns whether the checkpoint succeeded;
the success-side EIO reset in do_write_request and do_delete_request
only fires when it did, so a checkpoint media failure is no longer
erased by the unconditional reset that followed it. do_delete_request
also gains the success reset that was lost when append_needle stopped
clearing the streak, so a successful delete still clears an earlier
failure streak.
is_storage_io_error now uses libc::EIO on Unix instead of a hard-coded
5, and the ECX binary-search read path gains a Windows fallback
(seek + read_exact) so the buffer is no longer zeroed on non-Unix
targets.
* chore: remove unreachable dead code across the codebase
Remove ~50,000 lines of unreachable code identified by static analysis.
Major removals:
- weed/filer/redis_lua: entire unused Redis Lua filer store implementation
- weed/wdclient/net2, resource_pool: unused connection/resource pool packages
- weed/plugin/worker/lifecycle: unused lifecycle plugin worker
- weed/s3api: unused S3 policy templates, presigned URL IAM, streaming copy,
multipart IAM, key rotation, and various SSE helper functions
- weed/mq/kafka: unused partition mapping, compression, schema, and protocol functions
- weed/mq/offset: unused SQL storage and migration code
- weed/worker: unused registry, task, and monitoring functions
- weed/query: unused SQL engine, parquet scanner, and type functions
- weed/shell: unused EC proportional rebalance functions
- weed/storage/erasure_coding/distribution: unused distribution analysis functions
- Individual unreachable functions removed from 150+ files across admin,
credential, filer, iam, kms, mount, mq, operation, pb, s3api, server,
shell, storage, topology, and util packages
* fix(s3): reset shared memory store in IAM test to prevent flaky failure
TestLoadIAMManagerFromConfig_EmptyConfigWithFallbackKey was flaky because
the MemoryStore credential backend is a singleton registered via init().
Earlier tests that create anonymous identities pollute the shared store,
causing LookupAnonymous() to unexpectedly return true.
Fix by calling Reset() on the memory store before the test runs.
* style: run gofmt on changed files
* fix: restore KMS functions used by integration tests
* fix(plugin): prevent panic on send to closed worker session channel
The Plugin.sendToWorker method could panic with "send on closed channel"
when a worker disconnected while a message was being sent. The race was
between streamSession.close() closing the outgoing channel and sendToWorker
writing to it concurrently.
Add a done channel to streamSession that is closed before the outgoing
channel, and check it in sendToWorker's select to safely detect closed
sessions without panicking.
This fixes issue #6823 where a single volume server shutdown would cause
other healthy volume servers to fail their health checks and get restarted
by Kubernetes, causing a cascading failure.
Previously, the healthz handler checked if all replicated volumes could
reach their remote replicas via GetWritableRemoteReplications(). When a
volume server went down, the master would remove it from the volume
location list. Other volume servers would then fail their healthz checks
because they couldn't find all required replicas, causing Kubernetes to
restart them.
The healthz endpoint now only checks local conditions:
1. Is the server shutting down?
2. Is the server heartbeating with the master?
This follows the principle that a health check should only verify the
health of THIS server, not the overall cluster state.
Fixes#6823
2025-12-02 23:19:14 -08:00
Chris LuGitHubgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* initial design
* added simulation as tests
* reorganized the codebase to move the simulation framework and tests into their own dedicated package
* integration test. ec worker task
* remove "enhanced" reference
* start master, volume servers, filer
Current Status
✅ Master: Healthy and running (port 9333)
✅ Filer: Healthy and running (port 8888)
✅ Volume Servers: All 6 servers running (ports 8080-8085)
🔄 Admin/Workers: Will start when dependencies are ready
* generate write load
* tasks are assigned
* admin start wtih grpc port. worker has its own working directory
* Update .gitignore
* working worker and admin. Task detection is not working yet.
* compiles, detection uses volumeSizeLimitMB from master
* compiles
* worker retries connecting to admin
* build and restart
* rendering pending tasks
* skip task ID column
* sticky worker id
* test canScheduleTaskNow
* worker reconnect to admin
* clean up logs
* worker register itself first
* worker can run ec work and report status
but:
1. one volume should not be repeatedly worked on.
2. ec shards needs to be distributed and source data should be deleted.
* move ec task logic
* listing ec shards
* local copy, ec. Need to distribute.
* ec is mostly working now
* distribution of ec shards needs improvement
* need configuration to enable ec
* show ec volumes
* interval field UI component
* rename
* integration test with vauuming
* garbage percentage threshold
* fix warning
* display ec shard sizes
* fix ec volumes list
* Update ui.go
* show default values
* ensure correct default value
* MaintenanceConfig use ConfigField
* use schema defined defaults
* config
* reduce duplication
* refactor to use BaseUIProvider
* each task register its schema
* checkECEncodingCandidate use ecDetector
* use vacuumDetector
* use volumeSizeLimitMB
* remove
remove
* remove unused
* refactor
* use new framework
* remove v2 reference
* refactor
* left menu can scroll now
* The maintenance manager was not being initialized when no data directory was configured for persistent storage.
* saving config
* Update task_config_schema_templ.go
* enable/disable tasks
* protobuf encoded task configurations
* fix system settings
* use ui component
* remove logs
* interface{} Reduction
* reduce interface{}
* reduce interface{}
* avoid from/to map
* reduce interface{}
* refactor
* keep it DRY
* added logging
* debug messages
* debug level
* debug
* show the log caller line
* use configured task policy
* log level
* handle admin heartbeat response
* Update worker.go
* fix EC rack and dc count
* Report task status to admin server
* fix task logging, simplify interface checking, use erasure_coding constants
* factor in empty volume server during task planning
* volume.list adds disk id
* track disk id also
* fix locking scheduled and manual scanning
* add active topology
* simplify task detector
* ec task completed, but shards are not showing up
* implement ec in ec_typed.go
* adjust log level
* dedup
* implementing ec copying shards and only ecx files
* use disk id when distributing ec shards
🎯 Planning: ActiveTopology creates DestinationPlan with specific TargetDisk
📦 Task Creation: maintenance_integration.go creates ECDestination with DiskId
🚀 Task Execution: EC task passes DiskId in VolumeEcShardsCopyRequest
💾 Volume Server: Receives disk_id and stores shards on specific disk (vs.store.Locations[req.DiskId])
📂 File System: EC shards and metadata land in the exact disk directory planned
* Delete original volume from all locations
* clean up existing shard locations
* local encoding and distributing
* Update docker/admin_integration/EC-TESTING-README.md
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* check volume id range
* simplify
* fix tests
* fix types
* clean up logs and tests
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Chris LuandGitHub
chrislu
Konstantin Lebedev
LazyDBA247-Anyvision
brstgt