filer: widen metadata-subscription readahead buffers (#11237)

The metadata-subscription readahead channels were sized for a
low-throughput era and now bottleneck replay catch-up:

- ReadPersistedLogBuffer's readaheadSize was 1024 entries: the
  background visitor fills the channel, then blocks on the consumer's
  gRPC Send, so volume-server I/O for the next log file never overlaps
  with delivery of the current one. Each disk pass takes longer, and
  the subscribe loop re-lists log files (ListDirectoryEntries on the
  filer store) more often to drain the same backlog. Raised to 8192
  so the reader stays ahead of the consumer through a full log file's
  worth of entries.

- readFilersMerged's logEntryChannelSize was 512 entries per filer
  stream: the same serialization on the client side, where weed mount
  (chunk mode) reads persisted log chunks directly from volume
  servers. A small channel means the producer stalls on the merge
  consumer's processEventFn, and the next log file's chunks are never
  fetched ahead. Raised to 4096 so volume I/O overlaps with event
  delivery.

The wider buffers keep the producer goroutines reading through a full
log file while the consumer is still processing the previous one,
turning serial read→process→read into pipelined read∥process. This
cuts the per-pass wall time that drives filer store listings and
volume-server round-trips, reducing filer workload under backlog
catch-up (e.g. CSI deployments where ~200 mounts reconnect on filer
restart).
This commit is contained in:
Chris Lu
2026-09-08 16:58:09 -07:00
committed by GitHub
parent 5f77a0b67e
commit 723f473f02
2 changed files with 2 additions and 2 deletions
+1 -1
View File
@@ -375,7 +375,7 @@ func (f *Filer) ReadPersistedLogBuffer(ctx context.Context, startPosition log_bu
// Readahead: run the visitor in a background goroutine so volume server I/O
// for the next log file overlaps with event processing and gRPC delivery.
const readaheadSize = 1024
const readaheadSize = 8192
type entryOrErr struct {
entry *filer_pb.LogEntry
err error
+1 -1
View File
@@ -19,7 +19,7 @@ import (
type LogFileReaderFn func(chunks []*filer_pb.FileChunk) (io.ReadCloser, error)
// logEntryChannelSize bounds decoded entries in flight per filer stream.
const logEntryChannelSize = 512
const logEntryChannelSize = 4096
// maxLogEntrySize guards the per-entry allocation against a corrupt size
// prefix, mirroring the filer package's unexported constant.