mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-20 06:54:24 +00:00
filer: widen metadata-subscription readahead buffers (#11237)
The metadata-subscription readahead channels were sized for a low-throughput era and now bottleneck replay catch-up: - ReadPersistedLogBuffer's readaheadSize was 1024 entries: the background visitor fills the channel, then blocks on the consumer's gRPC Send, so volume-server I/O for the next log file never overlaps with delivery of the current one. Each disk pass takes longer, and the subscribe loop re-lists log files (ListDirectoryEntries on the filer store) more often to drain the same backlog. Raised to 8192 so the reader stays ahead of the consumer through a full log file's worth of entries. - readFilersMerged's logEntryChannelSize was 512 entries per filer stream: the same serialization on the client side, where weed mount (chunk mode) reads persisted log chunks directly from volume servers. A small channel means the producer stalls on the merge consumer's processEventFn, and the next log file's chunks are never fetched ahead. Raised to 4096 so volume I/O overlaps with event delivery. The wider buffers keep the producer goroutines reading through a full log file while the consumer is still processing the previous one, turning serial read→process→read into pipelined read∥process. This cuts the per-pass wall time that drives filer store listings and volume-server round-trips, reducing filer workload under backlog catch-up (e.g. CSI deployments where ~200 mounts reconnect on filer restart).
This commit is contained in:
@@ -375,7 +375,7 @@ func (f *Filer) ReadPersistedLogBuffer(ctx context.Context, startPosition log_bu
|
||||
|
||||
// Readahead: run the visitor in a background goroutine so volume server I/O
|
||||
// for the next log file overlaps with event processing and gRPC delivery.
|
||||
const readaheadSize = 1024
|
||||
const readaheadSize = 8192
|
||||
type entryOrErr struct {
|
||||
entry *filer_pb.LogEntry
|
||||
err error
|
||||
|
||||
@@ -19,7 +19,7 @@ import (
|
||||
type LogFileReaderFn func(chunks []*filer_pb.FileChunk) (io.ReadCloser, error)
|
||||
|
||||
// logEntryChannelSize bounds decoded entries in flight per filer stream.
|
||||
const logEntryChannelSize = 512
|
||||
const logEntryChannelSize = 4096
|
||||
|
||||
// maxLogEntrySize guards the per-entry allocation against a corrupt size
|
||||
// prefix, mirroring the filer package's unexported constant.
|
||||
|
||||
Reference in New Issue
Block a user