chrislu
802267b281
validates all critical upgrade scenarios for the generation-aware EC vacuum system
2025-08-10 17:33:08 -07:00
chrislu
d6e335d6e2
Integration test: end-to-end vacuum G→G+1 with live reads (zero downtime)
2025-08-10 17:23:17 -07:00
chrislu
884da0496c
testing mixed generation
2025-08-10 17:12:20 -07:00
chrislu
60f975aa20
add unit tests
2025-08-10 17:07:29 -07:00
chrislu
a3d0509c92
logs
2025-08-10 16:57:08 -07:00
chrislu
2e51e1dab2
ec volume UI rendering version
2025-08-10 16:54:44 -07:00
chrislu
a187f103d1
normal volume CompactionRevision
2025-08-10 16:22:36 -07:00
chrislu
3087da07db
metrics with generation
2025-08-10 16:17:46 -07:00
chrislu
3ef8a9f3b2
Mixed-version cluster compatibility
2025-08-10 15:54:30 -07:00
chrislu
d326affc4f
default generation
2025-08-10 15:44:08 -07:00
chrislu
d341db8861
Update ec_vacuum_task.go
2025-08-10 15:39:26 -07:00
chrislu
cf5f0b8e74
cleanupGracePeriod
2025-08-10 15:13:12 -07:00
chrislu
0bcb64140f
Add activation step to EC vacuum: call ActivateEcGeneration when G+1 ready
2025-08-10 15:07:57 -07:00
chrislu
62d89fa60b
EC vacuum distribution updated for generation-aware mount/copy RPCs
2025-08-10 15:04:55 -07:00
chrislu
de9399761b
EC vacuum task updated for generation-aware operation
2025-08-10 15:00:22 -07:00
chrislu
8797e73523
cachedLookupEcShardLocations updated for generation-specific caching
2025-08-10 14:51:55 -07:00
chrislu
f00dc46607
VolumeEcShardRead to read from correct (vid, generation) EcVolume
2025-08-10 14:44:27 -07:00
chrislu
1b41544f97
detecting ec volumes
2025-08-10 14:41:06 -07:00
chrislu
e9033136aa
todo
2025-08-10 14:24:59 -07:00
chrislu
56d1ed77c1
copy file
2025-08-10 14:24:30 -07:00
chrislu
9e2e600b6d
VolumeEcShardsGenerate updated for generation-specific file creation
2025-08-10 14:14:37 -07:00
chrislu
99f132729c
MountEcShards/UnmountEcShards updated for generation support
2025-08-10 14:09:07 -07:00
chrislu
8c31d5e331
EcVolume creation properly refactored
2025-08-10 13:45:08 -07:00
chrislu
ef5f9f629a
Generation file layout
2025-08-10 13:32:20 -07:00
chrislu
50cc17e8fa
ActivateEcGeneration RPC implemented
2025-08-10 13:11:47 -07:00
chrislu
870b2ffe07
LookupEcVolume handle generation parameter
2025-08-10 13:06:37 -07:00
chrislu
e4f266d927
Active generation tracking implemented
2025-08-10 13:03:52 -07:00
chrislu
5650b6f505
Add master_pb.ActivateEcGenerationRequest/Response RPC definition
2025-08-10 12:53:10 -07:00
chrislu
09d740db47
Add generation field to volume_server_pb.VolumeEcShardReadRequest
2025-08-10 12:51:27 -07:00
chrislu
8c54b72af3
Add generation field to volume_server_pb VolumeEcShards{Copy,Generate,Rebuild,ToVolume}Request
2025-08-10 12:51:01 -07:00
chrislu
312fee4ddf
Add generation field to volume_server_pb VolumeEcShards{Mount,Unmount}Request
2025-08-10 12:49:43 -07:00
chrislu
03ad8f2490
Add generation field to master_pb.VolumeEcShardInformationMessage
2025-08-10 12:47:47 -07:00
chrislu
fc666e2e48
collect ec volume deleted bytes
2025-08-10 12:37:18 -07:00
chrislu
3a9ec6f30c
Update worker.go
2025-08-10 02:16:23 -07:00
chrislu
8603a3a1e9
ec vacuum task need to register itself
2025-08-10 02:16:15 -07:00
chrislu
bea9ca3616
generate ec data and deleted data
2025-08-10 02:15:55 -07:00
chrislu
32e7ba2f9d
adding ec vacuum
2025-08-10 00:58:26 -07:00
chrislu
05a0cc156b
Self-Contained Design
...
To prove the system is truly self-contained:
To add a new task:
Create a task package (e.g., worker/tasks/compression/)
Import it: _ "github.com/.../worker/tasks/compression"
That's it! No other changes needed.
To remove a task:
Delete the task package directory
Remove the import line
That's it! No other changes needed.
2025-08-10 00:15:26 -07:00
chrislu
96d6d27607
remove ❌ Vacuum - Completely removed ❌ Balance - Completely removed
2025-08-10 00:00:46 -07:00
chrislu
c7f291e4e4
simplify logging
2025-08-09 23:49:45 -07:00
25bbf4c3d4
Admin UI: Fetch task logs ( #7114 )
...
* show task details
* loading tasks
* task UI works
* generic rendering
* rendering the export link
* removing placementConflicts from task parameters
* remove TaskSourceLocation
* remove "Server ID" column
* rendering balance task source
* sources and targets
* fix ec task generation
* move info
* render timeline
* simplified worker id
* simplify
* read task logs from worker
* isValidTaskID
* address comments
* Update weed/worker/tasks/balance/execution.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
* Update weed/worker/tasks/erasure_coding/ec_task.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
* Update weed/worker/tasks/task_log_handler.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
* fix shard ids
* plan distributing shard id
* rendering planned shards in task details
* remove Conflicts
* worker logs correctly
* pass in dc and rack
* task logging
* Update weed/admin/maintenance/maintenance_queue.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* display log details
* logs have fields now
* sort field keys
* fix link
* fix collection filtering
* avoid hard coded ec shard counts
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-08-09 21:47:29 -07:00
3ac2a2e22d
fix tikv listing due to expired entries ( #7115 )
...
* fix tikv listing due to expired entries
When there are many entries with empty fileName values (which can happen after TTL cleanup), the continue statements prevent the loop counter from incrementing, creating an infinite loop.
* address comments
* Update weed/filer/tikv/tikv_store.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
* address comments
Update weed/filer/tikv/tikv_store.go
Co-Authored-By: Copilot <175728472+Copilot@users.noreply.github.com >
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
2025-08-08 12:38:55 -07:00
Devin Lauderdale and GitHub
fae416586b
Move helm templates into folders ( #7113 )
...
* refactor: move helm templates into respective service folders
* fix: update template path reference in filer-statefulset for s3-secret
2025-08-08 10:36:01 -07:00
Devin Lauderdale and GitHub
92cebe12f0
chore: remove default replica count for all-in-one deployment ( #7111 )
2025-08-07 21:18:17 -07:00
Chris Lu and GitHub
535985adb6
Shell: add verbose ec encoding mode ( #7105 )
...
* add verbose ec encoding mode
* address comments
2025-08-07 00:12:05 -07:00
Chris Lu and GitHub
cde2d65c16
ec candidate selection needs to adjust same rack count compare ( #7106 )
...
ec needs to adjust same rack count compare
2025-08-07 00:09:51 -07:00
b4d9618efc
volume server UI: fix ec volume ui ( #7104 )
...
* fix ec volume ui
* Update weed/storage/erasure_coding/ec_volume.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
2025-08-07 00:07:03 -07:00
chrislu
dd4880d55a
fix for baidu cloud storage
2025-08-06 20:53:05 -07:00
4af182f880
Context cancellation during reading range reading large files ( #7093 )
...
* context cancellation during reading range reading large files
* address comments
* cancellation for fuse read
* fix cancellation
* pass in context for each function to avoid racing condition
* Update reader_at_test.go
* remove dead code
* Update weed/filer/reader_at.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
* Update weed/filer/filechunk_group.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
* Update weed/filer/filechunk_group.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
* address comments
* Update weed/mount/weedfs_file_read.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
* Update weed/mount/weedfs_file_lseek.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
* Update weed/mount/weedfs_file_read.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
* Update weed/filer/reader_at.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
* Update weed/mount/weedfs_file_lseek.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
* test cancellation
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
2025-08-06 10:09:26 -07:00
e446234e9c
remove spoof-able request header ( #7103 )
...
* remove spoof-able request header
https://github.com/seaweedfs/seaweedfs/issues/7094#issuecomment-3158320497
* Update weed/security/guard.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
2025-08-06 10:08:30 -07:00
chrislu
Devin LauderdaleandGitHub