Commit Graph
1211 Commits
Author SHA1 Message Date
Chris LuandGitHub 02773a6107 Accumulated changes for message queue (#6600)
* rename

* set agent address

* refactor

* add agent sub

* pub messages

* grpc new client

* can publish records via agent

* send init message with session id

* fmt

* check cancelled request while waiting

* use sessionId

* handle possible nil stream

* subscriber process messages

* separate debug port

* use atomic int64

* less logs

* minor

* skip io.EOF

* rename

* remove unused

* use saved offsets

* do not reuse session, since always session id is new after restart

remove last active ts from SessionEntry

* simplify printing

* purge unused

* just proxy the subscription, skipping the session step

* adjust offset types

* subscribe offset type and possible value

* start after the known tsns

* avoid wrongly set startPosition

* move

* remove

* refactor

* typo

* fix

* fix changed path
2025-03-09 23:49:42 -07:00
61249d8dde clear cache directory when mount exits (#6605)
Signed-off-by: lizhengui <lizhengui@virtaitech.com>
Co-authored-by: lizhengui <lizhengui@virtaitech.com>
2025-03-06 08:14:17 -08:00
chrislu 314f211260 Update filer.toml 2025-02-26 17:52:45 -08:00
chrislu 9bd4c47299 update redis support status 2025-02-26 17:48:44 -08:00
chrislu 4613186e8a not planning to redis3 2025-02-26 17:35:31 -08:00
Chris LuandGitHub df436d342b add cassandra2 (#6582) 2025-02-26 17:29:22 -08:00
Karol BędkowskiandGitHub 441614b386 fix: s3 command ignore -tlsVerifyClientCert and -cacert.file arguments (#6547)
s3 command ignore tlsVerifyClientCert and cacert.file arguments from
command line. On startS3Server instead of use real values (in s3opt),
default values (from s3Options, always empty) are checked.

Now on right values are checked and if user provide this arguments
RequireAndVerifyClientCert is set and/or ca certificate is loaded.
2025-02-13 13:27:38 -08:00
Lisandro PinandGitHub e8d8bfcccc Nit: remove missing newlines on weed shell commands output. (#6524)
Nit: remove missing newlines on `weed` commands output.
2025-02-07 10:27:04 -08:00
Ethan MosbaughandGitHub 9ebc132ffc fix: bucket-hook fails with gnu wget (#6521) 2025-02-06 23:11:17 -08:00
Chris LuandGitHub cc05874d06 Add message queue agent (#6463)
* scaffold message queue agent

* adjust proto, add mq_agent

* add agent client implementation

* remove unused function

* agent publish server implementation

* adding agent
2025-01-20 22:19:27 -08:00
c5f21b2b01 fix S3 per-user-directory Policy (#6443)
* fix S3 per-user-directory Policy

* Delete docker/config.json

* add tests

* remove logs

* undo modifications of weed/shell/command_volume_balance.go

* remove modifications of docker-compose

* fix failing test

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2025-01-17 01:03:17 -08:00
chrislu ec155022e7 "golang.org/x/exp/slices" => "slices" and go fmt 2024-12-19 19:25:06 -08:00
e77e50886e mount metacache add ttl (#6360)
* fix:mount deadlock

* fix

* feat: metaCache ttl

* Update weed/command/mount.go

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>

* fix InodeEntry

---------

Co-authored-by: zemul <zhouzemiao@ihuman.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2024-12-16 20:19:32 -08:00
vadimartynovandGitHub 914ff0e718 Replaced https.client.enabled with false in scaffold/security (#6320) 2024-12-05 10:39:28 -08:00
Konstantin LebedevandGitHub b65eb2ec45 [security] reload whiteList on http seerver (#6302)
* reload whiteList

* white_list add to scaffold
2024-12-02 10:38:10 -08:00
aebf3952b7 filer sync: source path and exclude path support dir suffix (#6268)
filer sync: source path and exclude paht support dir suffix

Co-authored-by: liguowei <liguowei@xinye.com>
2024-11-21 08:25:12 -08:00
6d3e38f439 Adjust threshold config (#6246)
* fix:mount deadlock

* fix

---------

Co-authored-by: zemul <zhouzemiao@ihuman.com>
2024-11-17 20:51:26 -08:00
a5fe6e21bc feat(filer.backup): add ignore errors option (#6235)
* feat(filer.backup): add ignore errors option

* feat(filer.backup): fix 404 error wrap

* feat(filer.backup): fix wrapping function

* feat(filer.backup): fix wrapping errors in genProcessFunction

* Update weed/command/filer_backup.go

* Update weed/command/filer_backup.go

* Update weed/command/filer_backup.go

---------

Co-authored-by: Max Denushev <denushev@tochka.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2024-11-14 08:40:55 -08:00
chrislu 113c9ce6a8 remove the direct_io flag, as it is not well-supported on macOS 2024-11-07 14:18:59 -08:00
Chris LuandGitHub dc784bf217 merge current message queue code changes (#6201)
* listing files to convert to parquet

* write parquet files

* save logs into parquet files

* pass by value

* compact logs into parquet format

* can skip existing files

* refactor

* refactor

* fix compilation

* when no partition found

* refactor

* add untested parquet file read

* rename package

* refactor

* rename files

* remove unused

* add merged log read func

* parquet wants to know the file size

* rewind by time

* pass in stop ts

* add stop ts

* adjust log

* minor

* adjust log

* skip .parquet files when reading message logs

* skip non message files

* Update subscriber_record.go

* send messages

* skip message data with only ts

* skip non log files

* update parquet-go package

* ensure a valid record type

* add new field to a record type

* Update read_parquet_to_log.go

* fix parquet file name generation

* separating reading parquet and logs

* add key field

* add skipped logs

* use in memory cache

* refactor

* refactor

* refactor

* refactor, and change compact log

* refactor

* rename

* refactor

* fix format

* prefix v to version directory
2024-11-04 12:08:25 -08:00
chrislu 6c7824c813 consistent sorted tags elastic,gocdk,rclone,sqlite,tikv,ydb 2024-10-21 22:53:29 -07:00
chrislu 804e765397 remove unused 2024-10-13 23:12:10 -07:00
Guang Jiong LouandGitHub 6c986e9d70 improve worm support (#5983)
* improve worm support

Signed-off-by: lou <alex1988@outlook.com>

* worm mode in filer

Signed-off-by: lou <alex1988@outlook.com>

* update after review

Signed-off-by: lou <alex1988@outlook.com>

* update after review

Signed-off-by: lou <alex1988@outlook.com>

* move to fs configure

Signed-off-by: lou <alex1988@outlook.com>

* remove flag

Signed-off-by: lou <alex1988@outlook.com>

* update after review

Signed-off-by: lou <alex1988@outlook.com>

* support worm hardlink

Signed-off-by: lou <alex1988@outlook.com>

* update after review

Signed-off-by: lou <alex1988@outlook.com>

* typo

Signed-off-by: lou <alex1988@outlook.com>

* sync filer conf

Signed-off-by: lou <alex1988@outlook.com>

---------

Signed-off-by: lou <alex1988@outlook.com>
2024-09-16 21:02:21 -07:00
zouyixiongandGitHub 881c9a009e [master] master missing start LoopPushingMetric routine fixed. (#6018) 2024-09-13 20:01:34 -07:00
chrislu d660d5c7d4 increasing default cache size 2024-09-10 10:30:19 -07:00
chrislu eb02946c97 support write once read many
fix https://github.com/seaweedfs/seaweedfs/issues/5954
2024-09-04 02:25:07 -07:00
chrislu 18afdb15b6 Revert "weed mount, weed dav add option to force cache"
This reverts commit 7367b976b0.
2024-09-04 01:38:29 -07:00
chrislu 7367b976b0 weed mount, weed dav add option to force cache 2024-09-04 01:19:14 -07:00
chrislu 66ac82bb8f default cacheDirWrite to cacheDir 2024-09-04 00:05:58 -07:00
Shaun AlexanderandGitHub 0d435f9da4 filer.remote.gateway not filer.remote.sync for filer.remote.gateway help string. (#5935) 2024-08-23 20:42:59 -07:00
chrislu 4463296811 add parallel vacuuming 2024-08-21 22:53:54 -07:00
chrislu 2074e7db67 fix invalid toml 2024-08-08 00:03:42 -07:00
qinguoyiandGitHub bf2366ba3a Fix KeyMaterial nil bug (#5849) 2024-08-02 11:06:48 -07:00
qinguoyiandGitHub e88da0be1b Fix KeyMaterial nil bug (#5848) 2024-08-02 09:47:11 -07:00
qinguoyiandGitHub 606d0cba64 Fix extractToFile gz file nil bug (#5847) 2024-08-02 09:46:37 -07:00
qinguoyiandGitHub b74b506e52 add command backup destory volume error log (#5846) 2024-08-02 00:04:57 -07:00
chrislu d49ecde535 rename functions 2024-08-01 23:27:23 -07:00
zuzuviewerandGitHub 5ee50d26d8 fix CVE-2016-2183 (#5844)
* fix CVE-2016-2183

* allow tls.min_version and tls.max_version not set
2024-08-01 20:00:22 -07:00
Konstantin LebedevandGitHub e1f9336688 [filer] fix panic filer on webdav.maxMB (#5822)
fix panic filer on webdav.maxMB
2024-07-27 13:40:22 -07:00
vadimartynovandGitHub 86d92a42b4 Added tls for http clients (#5766)
* Added global http client

* Added Do func for global http client

* Changed the code to use the global http client

* Fix http client in volume uploader

* Fixed pkg name

* Fixed http util funcs

* Fixed http client for bench_filer_upload

* Fixed http client for stress_filer_upload

* Fixed http client for filer_server_handlers_proxy

* Fixed http client for command_fs_merge_volumes

* Fixed http client for command_fs_merge_volumes and command_volume_fsck

* Fixed http client for s3api_server

* Added init global client for main funcs

* Rename global_client to client

* Changed:
- fixed NewHttpClient;
- added CheckIsHttpsClientEnabled func
- updated security.toml in scaffold

* Reduce the visibility of some functions in the util/http/client pkg

* Added the loadSecurityConfig function

* Use util.LoadSecurityConfiguration() in NewHttpClient func
2024-07-16 23:14:09 -07:00
chrislu 19d14b4c71 minor 2024-07-16 09:21:01 -07:00
vadimartynovandGitHub b796c21fa9 Added loadSecurityConfigOnce (#5792) 2024-07-16 09:15:55 -07:00
chrislu 3a82f5ffad ensure metadata follow a specific folder
fix https://github.com/seaweedfs/seaweedfs/issues/5774
2024-07-12 11:17:30 -07:00
vadimartynovandGitHub ec9e7493b3 -metricsIp cmd flag (#5773)
* Added/Updated:
- Added metrics ip options for all servers;
- Fixed a bug with the selection of the binIp or ip parameter for the metrics handler;

* Fixed cmd flags
2024-07-12 10:56:26 -07:00
vadimartynovandGitHub de60f383de Added tls for http handlers (#5764)
* Added https handler for filer

* Added example for security.toml
2024-07-11 07:53:18 -07:00
chrislu 8943a572d8 log error message 2024-07-03 09:50:47 -07:00
Konstantin LebedevandGitHub 5ffacbb6ea refactor all methods strings to const (#5726) 2024-07-01 01:00:39 -07:00
chrislu ac6fd36c06 using fixed bootstrap from peer filer 2024-06-30 23:16:54 -07:00
Nikita BorzykhandGitHub 64756c2d79 fix: correct command flag for master.raftBootstrap option in server.go (#5701) 2024-06-22 19:28:37 -07:00
chrislu a829f36d17 adjust error handling
fix https://github.com/seaweedfs/seaweedfs/issues/5677
2024-06-17 10:59:41 -07:00