mirrors/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-09-12 02:56:10 +00:00
Code Issues Packages Projects Releases Wiki Activity
11,044 Commits 176 Branches 359 Tags
bbc23f04e04bc6fbdd6be55992a0cc8cd8d210fb
Commit Graph
7123 Commits
Author SHA1 Message Date
chrislu 804e765397 remove unused 2024-10-13 23:12:10 -07:00
chrislu 6845e25318 set proxied request content length 2024-10-10 13:55:32 -07:00
chrislu 35fd1e1c9a optimize memory usage for large number of volumes 
1. unwrap the map to avoid extra map object creation
2. fix ec shard counting in UpdateEcShards
 2024-10-10 10:00:30 -07:00
chrislu b28b1a3402 3.77 2024-10-10 01:28:06 -07:00
chrislu 07df5c53de avoid nil 
I1010 03:10:25.220345 mount_std.go:281 This is SeaweedFS version 30GB 3.76 82ed61c655 linux amd64
^[OR^[OR2024/10/10 03:12:56 Unimplemented opcode POLL
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x3983b41]
goroutine 112 [running]:
github.com/seaweedfs/seaweedfs/weed/mount/meta_cache.SubscribeMetaEvents.func2.mergeProceesors.1(0xc000513540)
/github/workspace/weed/mount/meta_cache/meta_cache_subscribe.go:30 +0x41
github.com/seaweedfs/seaweedfs/weed/pb.WithFilerClientFollowMetadata.makeSubscribeMetadataFunc.func1({0x514a560, 0xc0009b06d0})
/github/workspace/weed/pb/filer_pb_tail.go:87 +0x293
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).WithFilerClient.func1.1(0xc000d9e008)
/github/workspace/weed/mount/wfs_filer_client.go:27 +0x68
github.com/seaweedfs/seaweedfs/weed/pb.WithGrpcClient(0xa9?, 0x9d1f90fb?, 0xc00098fbe0, {0xc000e1c078, 0x15}, 0x0?, {0xc000a76bd0?, 0x1?, 0x1?
})
/github/workspace/weed/pb/grpc_client_server.go:155 +0x33a
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).WithFilerClient.func1()
/github/workspace/weed/mount/wfs_filer_client.go:25 +0x14c
github.com/seaweedfs/seaweedfs/weed/util.Retry({0x489a10c, 0xa}, 0xc00098fcf8)
/github/workspace/weed/util/retry.go:16 +0xa7
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).WithFilerClient(0x0?, 0x30?, 0xc00056a568?)
/github/workspace/weed/mount/wfs_filer_client.go:18 +0x65
github.com/seaweedfs/seaweedfs/weed/pb.WithFilerClientFollowMetadata({0x50f5898, 0xc000e2c180}, 0xc00101e000, 0xc000d22030)
/github/workspace/weed/pb/filer_pb_tail.go:51 +0x87
github.com/seaweedfs/seaweedfs/weed/mount/meta_cache.SubscribeMetaEvents.func2()
/github/workspace/weed/mount/meta_cache/meta_cache_subscribe.go:115 +0xcb
github.com/seaweedfs/seaweedfs/weed/util.RetryUntil({0x48b8b35, 0x11}, 0xc00098fef0, 0x4af0048)
/github/workspace/weed/util/retry.go:64 +0x4e
github.com/seaweedfs/seaweedfs/weed/mount/meta_cache.SubscribeMetaEvents(0xc000d1d980, 0x9d1f90fb, {0x50f5898, 0xc000e2c180}, {0x7ffdf6f98dca,
0x4}, 0x17fcf8b7c7ecf8cf, {0xc0000ac208, 0x1, 0x1})
/github/workspace/weed/mount/meta_cache/meta_cache_subscribe.go:113 +0x39b
created by github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).StartBackgroundTasks in goroutine 1
/github/workspace/weed/mount/weedfs.go:150 +0x146
 2024-10-09 22:46:00 -07:00
chrislu 82ed61c655 3.76 2024-10-06 19:37:03 -07:00
chrislu c0e36231ad use only one metadata follow process 2024-10-06 12:55:19 -07:00
Konstantin LebedevGitHubChris Lu
 		44b275879b [s3] add {Get,Put,Delete}BucketTagging and PublicAccessBlock Handlers (#6088) 
* add {Get,Put,Delete}BucketTagging Handlers

* s3 add skip bucket PublicAccessBlock handlers

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2024-10-04 10:59:14 -07:00
chrislu a3a8f6217e fix 2024-10-03 09:03:17 -07:00
chrislu 117c3aebda 3.75 2024-10-03 08:33:35 -07:00
Konstantin LebedevandGitHub a3556acd98 [s3] add skip bucket encryption handlers (#6091) 
s3 add skip bucket encryption handlers
 2024-10-03 08:10:43 -07:00
Er2andGitHub 5644bc8f01 s3api: Fix signature v4 with reverse proxy at sub-path (#6092) 2024-10-03 08:08:19 -07:00
chrislu ff47956530 add helper functions 2024-10-03 00:13:46 -07:00
chrislu 11e53dd8ef skip error if conf file is not created 2024-10-01 22:46:48 -07:00
chrislu f747767aa1 support load balancer in front of s3 2024-10-01 12:57:45 -07:00
chrislu 7dd480acad handle missing leading zero padding in replica 2024-10-01 10:16:21 -07:00
chrislu e3fa87bcc1 3.74 2024-09-30 11:29:01 -07:00
chrislu cdd7fa81ab fix help message 2024-09-29 21:53:07 -07:00
chrislu 20929f2a57 adjust resource heavy for volume.fix.replication 2024-09-29 11:32:18 -07:00
chrislu 6564ceda91 skip resource heavy commands from running on master nodes 2024-09-29 10:51:17 -07:00
chrislu ec30a504ba refactor 2024-09-29 10:38:22 -07:00
chrislu 9cd263b2ce refactor 2024-09-29 10:35:53 -07:00
chrislu 701abbb9df add IsResourceHeavy() to command interface 2024-09-28 20:23:01 -07:00
coffeecloudgitandGitHub dad3a26fb6 Update volume_grpc_erasure_coding.go , fix no space left bug (#6077) 2024-09-27 12:04:27 -07:00
Konstantin LebedevandGitHub bae93e611d [filer] avoid 500 if table doesn't exist (#6075) 2024-09-27 07:15:54 -07:00
coffeecloudgitandGitHub 6cbce110d5 Update volume_grpc_erasure_coding.go (#6073) 2024-09-27 07:02:30 -07:00
NeoandGitHub aa61824442 master:fix empty target in Build() (#6069) 2024-09-27 00:45:18 -07:00
Max DenushevandGitHub f1e700ce2f Fix/copy before delete replication (#6064) 
* fix(shell): volume.fix.replication misplaced volumes unsatisfying replication factor

* fix(shell): simplify replication check

* fix(shell): add test for satisfyReplicaCurrentLocation
 2024-09-26 08:34:13 -07:00
steve.weiandGitHub ae8c6d95a6 fix: Prevent potential metadata change events from being lost. (#6066) 2024-09-26 08:31:57 -07:00
LHHDZandGitHub 4dc33cc143 fix unclaimed spaces calculation when volumePreallocate is enabled (#6063) 
the calculation of `unclaimedSpaces` only needs to subtract `unusedSpace` when `preallocate` is not enabled.

Signed-off-by: LHHDZ <shichanglin5@qq.com>
 2024-09-24 23:04:18 -07:00
Max DenushevandGitHub d056c0ddf2 fix(volume): don't persist RO state in specific cases (#6058) 
* fix(volume): don't persist RO state in specific cases

* fix(volume): writable always persist
 2024-09-24 16:15:54 -07:00
zemulGitHubzemul
 		028ebb1d0d Feat:merge small chunk (#6049) 
* fix:mount deadlock

* feat: merge small chunk

* adjust MergeChunkMinCount

* fix

---------

Co-authored-by: zemul <zhouzemiao@ihuman.com>
 2024-09-23 08:51:03 -07:00
chrislu 4ec6eab004 fix max volume count auto setting 
related to https://github.com/seaweedfs/seaweedfs/issues/6041#issuecomment-2360448182
 2024-09-19 09:13:08 -07:00
dsdGitHub邓书东
 		a3572999bb Vol check disk bug (#6044) 
* fix volume.check.disk

* ensure multiple replica sync

* add comment

---------

Co-authored-by: 邓书东 <shudong_deng@hhnb2024010108.intsig.com>
 2024-09-19 08:32:22 -07:00
Konstantin LebedevandGitHub 6d4f5c94ac [volume] remove truncate idx file if size not healthy (#6043) 2024-09-19 01:04:08 -07:00
chrislu 50e22864cf refactor 2024-09-17 23:10:41 -07:00
chrislu 99ae99e3b9 fix nil filer conf 
fix https://github.com/seaweedfs/seaweedfs/issues/6034
 2024-09-17 11:07:29 -07:00
Konstantin LebedevandGitHub 2b13d2c083 [filer] fix return 204 for DELETE on entry not found (#6023) 
fix return 204 for DELETE on entry not found
 2024-09-16 23:11:03 -07:00
Guang Jiong LouandGitHub 6c986e9d70 improve worm support (#5983) 
* improve worm support

Signed-off-by: lou <alex1988@outlook.com>

* worm mode in filer

Signed-off-by: lou <alex1988@outlook.com>

* update after review

Signed-off-by: lou <alex1988@outlook.com>

* update after review

Signed-off-by: lou <alex1988@outlook.com>

* move to fs configure

Signed-off-by: lou <alex1988@outlook.com>

* remove flag

Signed-off-by: lou <alex1988@outlook.com>

* update after review

Signed-off-by: lou <alex1988@outlook.com>

* support worm hardlink

Signed-off-by: lou <alex1988@outlook.com>

* update after review

Signed-off-by: lou <alex1988@outlook.com>

* typo

Signed-off-by: lou <alex1988@outlook.com>

* sync filer conf

Signed-off-by: lou <alex1988@outlook.com>

---------

Signed-off-by: lou <alex1988@outlook.com>
 2024-09-16 21:02:21 -07:00
Guang Jiong LouandGitHub 367e7638d7 fix invalid file read (#6024) 2024-09-16 08:45:50 -07:00
Eugeniy E. MikhailovandGitHub 0da9e40abb Feature: compress before encryption (#6013) 2024-09-15 08:47:53 -07:00
BruceandGitHub 5428229347 fix file read crash (#6021) 2024-09-14 08:33:35 -07:00
zouyixiongandGitHub 881c9a009e [master] master missing start LoopPushingMetric routine fixed. (#6018) 2024-09-13 20:01:34 -07:00
Aleksey KosovandGitHub 7340c62c47 changing FindEntry error handling in cassandra store (#6015) 2024-09-13 06:53:04 -07:00
chrislu a9c9e1bcb3 refactor 2024-09-12 22:45:30 -07:00
chrislu 6063a889ed 3.73 2024-09-12 13:11:55 -07:00
Eugeniy E. MikhailovandGitHub dab0bb8097 Feature limit caching to prescribed number of bytes per file (#6009) 
* feature: we can check if a fileId is already in the cache

We using this to protect cache from adding the same needle to
the cache over and over.

* fuse mount: Do not start dowloader if needle is already in the cache

* added maxFilePartSizeInCache property to ChunkCache

If file very large only first maxFilePartSizeInCache bytes
are going to be put to the cache (subject to the needle size
constrains).

* feature: for large files put in cache no more than prescribed number of bytes

Before this patch only the first needle of a large file was intended for
caching. This patch uses maximum prescribed amount of bytes to be put in
cache. This allows to bypass default 2MB maximum for a file part stored
in the cache.

* added dummy mock methods to satisfy interfaces of ChunkCache
 2024-09-11 21:09:20 -07:00
dsdGitHublevi
 		151f2ff7a9 prevent dead loop for followers of master node (#6007) 
* prevent dead loop for followers of master node

* fix comments

---------

Co-authored-by: levi <344887649@foxmail.com>
 2024-09-11 10:24:52 -07:00
dsdandGitHub 3b840c20e3 change math/rand => math/rand/v2 in volume_layout.go where is a perfo… (#6006) 2024-09-11 07:39:40 -07:00
Eugeniy E. MikhailovandGitHub c04edeed68 bug fix in the data received from cache processing (#6002) 
The patch addresses #3745.

The cache should return the exact amount of data requested by the buffer.
By construction of the cache it is always all requested data range
or we have error happening.

The old use of minsize miscalculate the requested data size,
if non zero offset is requested.
 2024-09-10 13:30:18 -07:00