mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-08-16 12:16:36 +00:00
Chris LuandGitHub
46ce8cbe84
* master: stream volume listings A listing of 800k volumes is 36MB on the wire but 305MB as messages, and the master built all of it, then held it while grpc encoded it. Two of those at once is most of a small master's heap, and the maintenance scanner asks every 30 minutes. The topology goes out first, listing nothing, then its volumes in batches, so the master holds a batch rather than a cluster: 341MB of live heap for one listing becomes 4.4MB. It allocates much the same either way -- what changes is how much of it has to be live at once, which is what sets the heap ceiling. Batches are built under their disk's lock and sent outside it, so a slow reader stalls the stream rather than the topology. They therefore do not share one instant, which a single listing did not either: it takes each disk's lock in turn, so a volume moving during either can be seen twice or not at all. The client helper hides which kind of master answered: one too old for the stream is asked the old way and its reply cut into the same batches. Either way the topology handed over lists no volumes, so a caller cannot come to depend on finding them there. * admin: stream the listing the maintenance scan reads It asks for every volume in the cluster every 30 minutes. Reassembling it client-side keeps the scan identical -- ActiveTopology splits disks by the disk ids on the volumes, so it needs them in the topology -- while the master no longer builds the whole reply to send it.