* master: let VolumeList ask for the volumes it wants
The request carried nothing, so every caller was answered with the whole
cluster. A dashboard opening one volume's page, or a capacity probe adding up
one bucket, was served all 800k of them and threw away the rest -- and the
master built every one of those messages first.
The topology, its disks and their counters are still reported in full: a caller
reading free space or replica placement needs the cluster whichever volumes it
asked about. Only what is listed under a disk is selected, ec shards included.
An empty collection and a zero volume id take everything, the way volume.list
already reads its own -collectionPattern and -volumeId, so a caller that
forgets to narrow is answered too much rather than answered wrongly. That
leaves the default collection unnameable, since it is the one the empty string
names, so it gets a field of its own.
An older client sends none of it and is answered exactly as before.
* admin: ask the master for the volume the page is showing
A volume's detail page was pulling every volume in the cluster to find one and
its replicas, and discarding the rest.
* admin: ask the master for the ec volume the page is showing
Same as the volume detail page: one volume's shards were found by pulling every
ec shard in the cluster.
* s3: ask the master for the bucket's own collection
The SOSAPI capacity probe summed one collection's volumes out of a listing of
every volume in the cluster. Cluster capacity still comes out the same: it is
read from the disk counters, which a filtered listing reports in full.
* topology: read the disk usage counters atomically
They are written with atomic.AddInt64 from heartbeats but were read plainly by
the two listings and by FreeSpace, and the map they sit in was iterated without
the lock its neighbour takes. Under -race a listing concurrent with a heartbeat
trips on both.
A master decides nothing from it. Every caller that read it was asking whether
a volume is remote, which the backend name answers, and the value itself is
reported on demand by the server holding the volume, through the volume info in
ReadVolumeFileStatus.
It is also the one string here that cannot be shared: unique per volume, so
unlike the collection and backend names it carries its own characters for every
volume a master tracks.
VolumeInfo goes from 136 bytes to 120. 800k volumes registered from a heartbeat
that has been over the wire go from 214 to 163 B/volume when tiered.
The volume server's own status page keeps showing the key, now read from the
volume it holds rather than relayed through a master, which is also where the
other volume server implementation reads it.
The heartbeat digest drops it on the same grounds: a change to something the
master does not hold cannot make its copy stale. Both implementations and their
shared vectors move together, and the field-coverage test now names what is
deliberately not retained rather than being loosened.
* storage: order VolumeInfo by alignment
The struct is held for every volume replica in the cluster, so the padding the
compiler inserts is multiplied by however many volumes a master tracks. Two
one-byte fields each sat at the head of a word and left the rest of it empty,
which was ten of the eighteen wasted bytes.
Grouping by size rather than by meaning takes the struct from 152 bytes to 136,
and the map holding them shrinks with it, since a Go map's slack scales with
the size of the value.
800k volumes registered from a heartbeat that has been over the wire:
211 -> 195 B/volume, 214 -> 198 tiered.
* trim the comments on this change to the parts that are not evident
* topology: build the volume list without copying the volume map first
ToDiskInfo copied every VolumeInfo on the disk into a fresh slice, walked it to
build a protobuf message for each, and threw the copy away. The copy was as
large as the messages it produced.
Building them straight from the map holds the disk's read lock for the walk
rather than just the copy, so a heartbeat updating that disk waits for it. It
is a read lock on a call that is now infrequent, against an allocation of the
same size as the response.
ToTopologyInfo over 550k volumes 193617502 B/op -> 110011017 B/op, and faster
for not making the copy.
* trim the comments on this change to the parts that are not evident
* ec.balance: add a -volumeIds filter
Collection scope is often too broad for maintenance. -volumeIds narrows the
plan to the given ec volume ids by leaving every other volume out of the
topology handed to the planner, so no phase, dedup included, can plan against
them. Ids with no ec shard in the selected collection, dataCenter and disk type
are rejected rather than silently skipped.
* ec.encode: key the orphan sweep without narrowing the volume id
int is 32-bit on 32-bit builds, so int(vid) wraps for volume ids above
MaxInt32. Format the id as the uint32 it is.
* pb: ask the master what each collection holds
Callers tracking usage were sent every volume in the cluster to add up
themselves, which is the master's largest single allocation.
* topology: summarise what each collection holds
One pass over the topology, allocating per collection rather than per volume.
Regular volumes count once each for logical totals and once per replica for
physical, taken from the lookup index, which is already keyed by volume and so
needs no set of seen ids. Ec shards are node-local so their sizes sum, while
the file and delete counts describe the volume and resolve once every holder
has been seen.
Replicas of one volume disagree while a write is landing or a heartbeat is
late. Walking a full listing took whichever replica the map iteration reached
first, so the answer moved between runs; this takes the largest, which is
stable and never reports usage below what some replica already holds.
* s3: take bucket sizes from the master's summary
The bucket size metrics pulled the whole volume list once a minute and added it
up, which cost the master 184.6MB of allocation and 17.8MB on the wire for six
numbers per collection.
VolumeList over 550k volumes 184.6 MB allocated, 17.8 MB on the wire
CollectionStatistics 176 bytes allocated, 47 bytes on the wire
The aggregation moves to the master with it, so the cases the removed tests
covered are now asserted against it directly.
* topology: count the replica holding the most live data
Quotas are enforced on size less deletions, and the replica with the biggest
raw size can be the one that has deleted the most. Counting it reported a
bucket smaller than it is and would leave one writable over its quota, which is
the opposite of what picking the largest was meant to guarantee.
* topology: cap a volume's deletions at what it holds
Live usage is read as a collection's size less its deletions, so a volume
reporting more deleted bytes than it has cancels live bytes belonging to other
volumes in the same bucket and reports it smaller than it is. Replica selection
already floored that volume's own live size at zero; the totals have to agree
with it.
* storage: share the volume strings a cluster repeats
Decoding a heartbeat allocates a fresh string for the collection, disk type and
remote backend of every volume, and a master holding a million volumes then
holds a million copies of the same handful of names.
Not the remote storage key, which is unique per volume: interning that would
fill the table rather than share anything.
800k volumes registered from a heartbeat that has actually been over the wire:
227 -> 211 B/volume, and 238 -> 214 when the volumes are tiered, since the
backend name shares too.
* storage: hold the interned strings rather than let them be collected
unique.Make clears its entries by weak reference, and its canonical value does
not survive a collection even while a caller still holds the string it handed
back -- so a volume reported later would get a second copy of a name the rest
of the cluster already shares. With only changed volumes reported, most are
interned once and never again, so that is the common case rather than a corner.
The table therefore only grows, which is why it stays restricted to values
drawn from a small set. Ten thousand collections keep a few hundred kilobytes.
* iceberg: stamp a default name mapping on new tables
* iceberg: repair non-compliant manifests at commit
* s3tables: verify ClickHouse writes read back through PyIceberg
* iceberg: carry the manifest-list content into repaired manifests
* iceberg: refresh the default name mapping on schema evolution
* iceberg: merge historical names into the refreshed name mapping
* iceberg: never fail a commit on repair fallout
* iceberg: harden manifest repair against writer dialects
* s3tables: keep PyIceberg reader stderr out of row data
* iceberg: keep name mappings unambiguous across field id reassignment
* iceberg: align existing manifest content metadata with the list entry
* s3: add the RenameObject endpoint
PUT /{bucket}/{key}?renameObject with x-amz-rename-source moves an object
through the filer's AtomicRenameEntry, so no bytes are read or rewritten and
the ETag, tags and SSE keys travel with the entry.
Only unversioned buckets: a versioned rename would have to rebuild the
.versions chain, and AWS offers RenameObject on directory buckets, which
cannot be versioned. The source arrives in a header, so it is authorized
separately for read and delete; both keys are locked, in key order, across the
precondition checks and the move.
* s3: let a matched source ETag precondition settle its date precondition
RFC 7232 has an ETag precondition outrank the date precondition on its own
side, and AWS documents the same for CopyObject: a matching
x-amz-copy-source-if-match with a failing x-amz-copy-source-if-unmodified-since
copies rather than returning 412. The source check evaluated all four headers in
sequence, so the date header could still veto a decided ETag match.
validateConditionalHeadersForReads already applies this precedence; the source
path now matches it.
* s3: cover a rename source named as a directory without a trailing slash
Renaming a directory would move a whole subtree, so it has to stay a missing
key whether or not the caller wrote the trailing slash.
* s3: accept a bare object key as the RenameObject source
AWS spells x-amz-rename-source both ways. Its CLI, Java and Rust examples pass
the bare source key, and only a second CLI example and the boto3 conditional
example pass bucket/key; the API reference's own example is a bare key too. The
header was read as bucket/key only, so the form AWS leads with was rejected with
InvalidArgument and the endpoint was unusable as documented.
A value is now read as a literal key first — the only reading that can never
name the wrong object — and as bucket-qualified second, when it carries the
request's own bucket and the literal key does not exist. That costs one extra
lookup only for a source that starts with the bucket's own name.
Another bucket's name in the source is no longer a distinct error: RenameObject
moves within one bucket, so it is simply part of a key this bucket does not
hold, and it reports NoSuchKey.
* s3: only a proven absence picks the other reading of a rename source
A source that resolves to a directory is not a miss to fall through on: the
literal path is still what the caller named, so answering for it beats renaming
a different object under the bucket-qualified reading. With a directory at
bucket/source.txt and an object at source.txt, a rename naming the former moved
the latter.
A failed lookup is not a proof of absence either, so a blip can no longer
redirect a rename to the other reading.
* topology: keep per-node volume state with the location it describes
The read-only and oversized indexes were maps from volume id to a list of the
nodes reporting that state -- the same key space the lookup index already
holds, kept a second and third time. Nothing ever asked which nodes; both are
only ever asked whether any node does.
So the state rides on the location list as a bit per entry, and the two indexes
go. Removing a location shifts the bits with it, and a node replacing another
at the same address inherits its slot, since that is what happens to the
location too.
800k volumes, 90% read-only: readonly index 40.0MB -> 0, lookup index
42.5MB -> 48.6MB for the bits, 33.9MB net.
* topology: rebuild the location flags when stale entries are dropped
Refresh rebuilds the location list, so leaving the flags alone left bits
describing whoever moved into the dropped entries' place.
* topology: assert the refreshed flag survived, not just that it moved
Clearing the mask rather than rebuilding it would have passed: the check that
the flag lands on the right location is done by clearing it, which an already
empty mask satisfies.
* topology: track volume size only where writes can land
Size tracking decays pending assignment estimates so the master does not
overfill a volume before heartbeats catch up. Nothing is ever assigned to a
read-only volume, so an entry for one can never be consulted -- and in a tiered
cluster that is most of them, which made this the volume layout's largest cost.
A volume held out of the writable list for capacity is not read-only and keeps
its entry: that entry is what enforces the recovery delay.
800k volumes, 90% read-only: sizeTracking 79.1MB -> 8.4MB, and the crowded set
falls out with it because a read-only volume no longer reaches the threshold
check at all.
* topology: decide size tracking per volume, not per reporting replica
A volume is unwritable if any replica is read-only, so asking the replica whose
heartbeat happened to arrive made the answer depend on arrival order: a
writable replica reporting after a read-only one put the tracking back.
Ask the volume instead, which also drops the caller-supplied flag and the churn
it caused. The crowded entry goes with the tracking, since leaving it behind
would only move the memory this releases.
Costs a map lookup per replica on a full-list heartbeat, about 19ms per 100k
volumes and no allocations, on a path that is now rare.
* s3api: fix ListObjectsV2 dropping objects under a partial prefix
ListObjectsV2 with a prefix that names only part of a directory name -
"data/a" matching both "data/a/..." and "data/ab/..." - dropped objects
when paginating with a small max-keys, reporting IsTruncated=false before
all matching keys had been returned.
On resume, normalizePrefixMarker took the listing directory from the
continuation marker ("data/a/") and discarded the "a" name prefix, so once
the marker's subtree was exhausted the sibling directories that also
matched the prefix were never listed.
Resolve the listing directory and name prefix from the prefix instead of
the marker, so siblings sharing the name prefix are still listed after the
marker's subtree.
Fixes#10652
* s3api: cover a sibling-directory marker in the partial prefix tests
* s3api: resume a truncated delimited listing from the emitted CommonPrefix
Rebuilding the continuation token from the request dir plus the listing prefix
assumes that prefix names a directory. With a partial name prefix like "data/a"
it produced "data/a/ab/" for the sibling prefix "data/ab/", so the next page
resumed in the wrong subtree and repeated it forever.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* heartbeat: name departed volumes in delta heartbeats
* master: release the lookup index with a deleted collection
* master: keep a fresh grow safe from the report that raced it
* volume: name the volumes a deleted collection took with it
Deleting a collection left the master to work out what went by omission from
the next full volume list, which it no longer gets: heartbeats carry the whole
list only when the master asks for it. The volumes a bucket's churn creates and
destroys between two of those requests are never named in either direction, so
the master keeps counting their slots as occupied and a cluster that creates
and drops collections quickly runs its free-slot accounting dry -- assigns fail
with no free volumes left while the disk holds a handful of volumes.
The destroy path already knows exactly which volumes it removed, so send them
down the same channel every other deletion uses.
* rust: name the volumes a deleted collection took with it
Mirrors the Go volume server. The notify path derives its deltas by diffing
snapshots, so a collection delete that does not wake it is invisible until the
master next asks for the whole list.
* topology: mark a volume crowded only if it can take writes
Crowding asks for more room to write into, and the writable-volume refresh loop
marked anything past the threshold regardless of whether writes could land
there. Growth already discounts those by intersecting the crowded set with the
writable list, so the entries changed no decision and only took space -- in a
tiered cluster, one for nearly every volume.
* topology: wait for the crowded-volume collector before reading what it saw
Closing the stop channel does not order the collector's writes against the
test's reads.
* topology: drop the sleep from the crowded-volume test
The channels are unbuffered, so every send has been received by the time the
sweep returns, and waiting for the collector covers the recording. The sleep
only suggested the result turned on timing.
* filer: keep the existing peer subscription on a repeated add
A cluster node add for a peer that is already followed restarted the
subscription, dropping the metadata events between the two runs.
* master: tell a connecting client the current cluster membership
Cluster node updates are only broadcast to the clients connected at that
moment. A filer that lost its master stream while a peer came back never
learned about the peer, and stopped replicating its metadata for good.
* test: a filer joining the master learns about the filers already there
* test: a filer resubscribes to a peer that registered while it was disconnected
Runs the reported sequence against real processes: filer2 leaves, filer1
is paused and its master stream is broken, filer2 registers again, and
filer1 has to replicate from it after reconnecting.
* volume: validate the file extension in CopyFile and ReceiveFile
CopyFile and ReceiveFile build an on-disk path from the client-supplied
Ext. Both are intentionally ungated for cluster-internal peers, so a
value like "/../../x" is joined onto the volume directory and, once
path-cleaned, resolves outside it -- an EC-shard receive can then write,
and CopyFile read, anywhere the process can reach.
Constrain Ext to a real suffix (a leading dot followed by alphanumerics)
before it is used to build any path, so it can no longer carry a
separator or a parent reference.
* test: use an alphanumeric missing-file extension in the copy variants
The not-found and stop-offset-zero cases used ".definitely-missing" as a
deliberately absent source. The extension is now validated, and the hyphen
makes it invalid, so switch to ".missing" -- still a nonexistent file, but a
real extension shape.
* volume: validate the collection in CopyFile and ReceiveFile
The client-supplied Collection is folded into the on-disk path as
"<collection>_<vid>" by VolumeFileName and EcShardBaseFileName, both joined
with path.Join / util.Join. A Collection carrying a separator, e.g.
"../../x", therefore path-cleans to a target outside the volume directory,
the same escape the extension check just closed. Reject a collection that is
a bare parent reference or holds a separator; ordinary names ('.', '-' and
all) still pass.
* filer: drain pending log chunk refs when the metadata stream ends
In metadata chunks mode the server sends log file refs in responses of their
own, and the client can only read them once it knows the run of refs is over.
That was inferred solely from the arrival of a normal event, so refs still
pending when the stream ended were dropped: the subscription returned no
events and no error.
A follower never noticed, because it runs forever and a live event always
arrives to close the run. A bounded subscription — StopTsNs set, range already
in the past — can receive nothing but refs and then EOF, and silently reports
that nothing happened. For anything auditing a path that is the worst possible
answer, since an empty result is indistinguishable from a quiet period.
Drain on EOF as well as at the transition point.
* filer: add filer.meta.scan to audit one directory's change history
Reconstructing what happened to a path means replaying the metadata log, and
filer.meta.tail is built for watching rather than auditing: it follows forever
unless given a stop, prints multi-line JSON, and takes ranges only as durations
before now, so an incident timestamp has to be converted by hand.
Its -pattern also cannot find a versioned object. A versioned key is stored as
<key>.versions/v_<id>, so the events carry the names "<key>.versions" and
"v_<id>" and a pattern of the object's own name matches neither — the search
comes back empty while the object is being written continuously.
filer.meta.scan prints one line per change, stops at the end of the range,
accepts absolute -since/-until with an explicit -tz, and reports versioned
writes against the object key with the version id alongside, so -name matches
the key a client would ask for. Delete markers are labelled as such rather than
appearing as zero-length writes, and pointer flips on the .versions container
are distinguished from writes of object data.
* filer.meta.scan: read persisted log chunks from the volume servers
Reading a range through the filer makes it decode every log entry in that
range and filter each one, so the cost lands on the filer and does not shrink
when the prefix is narrow — only the bytes on the wire do. On a cluster whose
metadata log is dense that is the expensive part of a scan, and it is charged
to the process least able to spare it.
Enable metadata chunks mode: the filer hands out log chunk ids and the scan
reads them from the volume servers itself. ReadLogFileRefs re-applies the same
path filter client-side, so the output is unchanged — verified identical to
the filer-read path over the same range, including after a restart drops the
in-memory buffer and the data must come off disk.
Direct read needs a route to the volume servers that the filer does not, so a
failure before anything has been printed retries through the filer; retrying
after partial output would duplicate lines. -directRead=false forces it.
* filer.meta.scan: confirm an empty direct-read result through the filer
An audit that returns nothing is read as "nothing happened here", so it is the
one answer that must not be produced by a bug. Direct read has more ways to
come back empty than the filer path does — it needs a route to the volume
servers, and it depends on the ref-drain contract holding.
When direct read yields no changes, re-run through the filer before reporting
it, and warn if the two disagree. Re-running is safe only because nothing was
printed; after partial output a replay would duplicate lines instead, so that
case reports the error rather than retrying.
In metadata chunks mode the server sends log file refs in responses of their
own, and the client can only read them once it knows the run of refs is over.
That was inferred solely from the arrival of a normal event, so refs still
pending when the stream ended were dropped: the subscription returned no
events and no error.
A follower never noticed, because it runs forever and a live event always
arrives to close the run. A bounded subscription — StopTsNs set, range already
in the past — can receive nothing but refs and then EOF, and silently reports
that nothing happened. For anything auditing a path that is the worst possible
answer, since an empty result is indistinguishable from a quiet period.
Drain on EOF as well as at the transition point.
* master: evict a dead peer via the local raft handle
OnPeerUpdate only runs on the leader, and the AddVoter branch right above
mutates the local raft directly. The remove branch instead dialed our own
RaftRemoveServer back over gRPC. Drop the self-dial and remove the peer
through the local handle, matching the add path. This also leaves operator
tooling as the only caller of the RaftRemoveServer RPC.
* master: require whitelist auth for raft membership RPCs
RaftAddServer, RaftRemoveServer and RaftLeadershipTransfer rewrite raft
quorum but had no caller check beyond "am I the leader". Any client that
could reach the master gRPC port could add an unreachable phantom voter
and stall the write path.
Gate the three on the admin whitelist, mirroring the volume server's
checkGrpcAdminAuth. With no whitelist configured the guard allows every
caller, so default and single-master deployments are unaffected;
operators who set -whiteList get these RPCs locked down to it. The
leader's own dead-peer eviction no longer dials these RPCs, so the only
remaining callers are operator tooling.
* pb: let a heartbeat carry only the volumes that changed
A partial list cannot travel in volumes: a master that did not understand it
would read the absences as deletions. So changes get their own field, used only
once the master has said it compares digests and can tell when it has fallen
behind.
* master: apply the volumes a heartbeat reports as changed
Only the named volumes are touched. A full report says the server holds exactly
these; a changed report says nothing about the ones it leaves out, so absence
must not read as removal.
Also advertises that the master compares digests, which is what lets a server
stop sending its whole list. Advertising it once per connection means a server
reconnecting to a master that does not is back to full lists straight away.
* volume: send only the volumes that changed once the master accepts them
The whole list goes on every heartbeat until the master says it compares
digests, and again whenever it asks, so a master that cannot tell when it has
fallen behind never has to.
has_no_volumes stays derived from a full list alone. Deriving it from what a
heartbeat happens to carry would make a quiet one read as a server that had
lost every volume, and the master would drop them all.
The digest still covers every volume held rather than the ones sent, which is
what lets the master confirm that applying the changes left it current.
Reporting state is per-connection: a server that reconnects, or reaches a
different master, starts again from the full list.
* volume: let the zero reporting state stand for having told no master anything
A Store built as a literal, which tests do, left the reporting state nil and
panicked on the first heartbeat. As a value its zero form already means nothing
has been reported to anyone, which is exactly the state that sends the whole
list.
* rust: send only the volumes that changed once the master accepts them
Mirrors the Go volume server, with one hazard the Go side does not have: mount
and unmount deltas here are derived by diffing successive heartbeats, so a
heartbeat that carries a partial list would report every volume it left out as
unmounted. Collecting now returns the full set alongside the message, and every
site that diffs uses that rather than what went on the wire.
* volume: do not let a full-list request be lost to the heartbeat it raced
The request arrived while a heartbeat was already being built as a delta, and
committing that heartbeat cleared it, so the master waited for another digest
mismatch before asking again. Count the requests and clear only the one the
heartbeat answered.
* rust: stop marking volumes reported by a heartbeat that is thrown away
The state-notify path collected a heartbeat only to diff its volume list, then
sent a delta message of its own and dropped the one it had collected. Once
collecting recorded what the master had been told, every mount or unmount
silently marked the changed volumes as sent, and the master learned of them
only after a digest mismatch.
Snapshotting no longer records anything, and no longer expires ec volumes
whose deletion that path was already discarding.
* master: announce only the volumes a change actually brought
Every changed volume was broadcast as a new location. Volumes grow constantly
and growth moves no location, so on a busy cluster that told every connected
client about volumes it could already reach, filling bounded broadcast queues
and pushing out the topology updates that matter.
* master: ask for the full list when only one can repair the master
Delta heartbeats stop the full report, and with it the only thing that
re-registers a volume the lookup index lost. The volume server cannot see that
divergence and its digest cannot show it, so the master now checks its own two
indexes agree and asks for the list when they do not.
A node reporting one volume id twice is kept on full lists for the same reason
rather than merely skipped: its digest can never be verified, so nothing else
would tell the master what it had stopped holding.
* master: keep the volume options on every heartbeat response
A volume server takes them from whatever response arrives, and preallocate is a
bare bool with no way to tell off from unmentioned. A response sent to ask for
the volume list therefore turned preallocation off until the server reconnected.
Responses sent mid-stream now start from the configured options rather than
being built field by field.
* master: announce a volume the lookup index had lost
Repairing the index makes the volume servable again, but clients were told it
went when the node dropped out and nothing told them otherwise: the disk map
still held it, so it did not count as an arrival.
Reaching the lookup index is what makes a volume servable, so recovering an
entry there is an arrival as far as clients are concerned, on both the full
report and the changed-volume path.
* s3tables: add ClickHouse iceberg catalog integration test
* ci: run the ClickHouse iceberg catalog test
* s3tables: bound setup HTTP calls in the ClickHouse test
* s3tables: pin the ClickHouse writer image dependencies
VolumeLocationList.Stats subtracts the deleted figures from the totals to
report live size and needle count. Both deleted figures are maintained as
counters independent of the totals they come off, so either can transiently
exceed its total, and neither subtraction was clamped.
Unclamped, the size wraps to ~16 EB. The count is signed so it merely goes
negative, but VolumeLayout.Stats converts it with uint64(fileCount), which
turns it into ~1.8e19 just the same. Either one swamps the cluster totals
behind /dir/status, /vol/status and Topology.CollectionVolumeStats.
commandFsMergeVolumes.getVolumeSize had the same unclamped subtraction, where
a wrapped size reads as a volume far too large to join any merge plan.
Clamped to zero, matching the guards already in CollectionInfo.LogicalSize
and the admin server's logical-size accumulator.
WriteFile's own existence probe runs while the file does not exist, and
WinFsp may serve that answer from its metadata cache for up to the
mount's FileInfoTimeout. A reader racing into that window failed its
open with not-found, which is the cache being a cache, not a defect in
concurrent reading. Establish visibility once before racing the readers,
so the test exercises what it is named for.
* wdclient: keep the location of a volume reported added and removed at once
A volume moved between a server's disks arrives in both lists of one message,
and the server still has it. Additions were applied before removals, so the
removal won and the client was left with no location for a volume that never
went anywhere.
Reordering would swap the bug for a window where the volume resolves nowhere,
since the two updates take the lock separately. Skip the removal instead, so
the order the lists are applied in stops mattering.
* wdclient: build each ec update explicitly in the move test
Reusing one response object and adding the deletion to it left the overlap the
test turns on implicit, and reading it as a delete-only update is the natural
mistake.
* topology: follow a volume that moved between a server's disks
The heartbeat diff asked only whether a volume id was reported anywhere on the
node, so a volume that moved to a disk of another type stayed on the disk it
left as well. The master then held two copies of it forever: the volume count
was overstated, and GetVolumesById returned whichever disk the map iterated
first, so lookups could hand back the disk the volume had already left.
Track which disk types the heartbeat named each volume on, and treat a volume
named on another disk as absent from this one. Disk types are interned to an
index because a server reports a handful of them across hundreds of thousands
of volumes.
A volume named on two disks at once is a stale twin rather than a move, and is
still kept on both -- dropping one would tell the master a replica vanished.
Only a volume named twice on one disk type is unrepresentable, so that is now
what marks the node, rather than any repeat of an id.
* master: do not tell clients a moved volume left the node
A volume moved between a node's disks is removed from one and added to the
other, so it lands in both lists of the same heartbeat. Clients apply additions
before deletions, so the removal wins and they end up with no location for a
volume that never went anywhere.
Skip removals for volumes the node still holds, as the ec shard paths already
do, and update the topology before judging the delta removals so an unmount
that really did happen is still reported.
* trim the comments on this change to the parts that are not evident
* master: judge a volume removal on normal replicas alone
HasVolumesById answers for ec shards as well, so a replica encoded into ec
shards looked like it was still on the node and clients were never told the
normal location had gone. They hold normal and ec locations separately and
prefer the normal one from the same generation, so that location would have
gone on shadowing the shards.
Every enumeration of a directory walked the whole FUSE machinery, so
reopening a folder cost what opening it did. The kernel has a cache for
exactly this: with FOPEN_CACHE_DIR the listing lives in the directory's
page cache and a repeat enumeration never reaches the mount at all.
Local mutations already drop that cache in the kernel. Remote ones
arrive through the metadata subscription, so the entry invalidation
worker now also tells the kernel which directory changed. The worker is
the one place this is safe from: notifying from a thread serving a
kernel request can deadlock against the page it holds, which is why the
file paths deliberately avoid InodeNotify.
Measured in a Linux container, 20k-entry directory, ls repeated:
warm listing before 199-355ms after 6-9ms
A file written from outside the mount appeared in the next listing
within a second, through the subscription notify, and the listing
re-cached after.
The memory is the kernel's page cache: reclaimed under pressure, owned
per-directory, and covering read-through directories the mount-side
caches never see.
* mount: read oversized directories through instead of caching them
Visiting a directory pulls every child from the filer into the local
LevelDB before the first listing returns. For a directory of a few
million entries that is minutes of streaming, gigabytes of local store,
and gigabytes of decoded entries in flight -- paid by a mount that may
only walk the directory once.
A build that crosses -cacheDirMaxEntries (default ten thousand) now
stops, cleans up, and marks the directory read-through: listings stream
from the filer with pagination, the way update-hot directories already
do, and lookups in it consult the filer per entry as any uncached
directory does. The refusal is remembered, so the next visit fails fast
instead of streaming to the limit again, and an oversized ancestor is
stepped over when caching its subdirectories rather than wedging every
listing beneath it.
The direct path keeps the same pagination state on the handle, so a walk
that crosses the limit mid-flight carries on from where the cached walk
reached.
* mount: an ancestor found oversized must not fail its descendants
Visiting a directory builds its whole uncached ancestor chain in one
group, so the first discovery that an ancestor is oversized cancelled the
group and surfaced as the listed directory's own refusal: the descendant
build was aborted and the caller marked the descendant read-through,
leaving a perfectly cacheable directory streaming from the filer until
its inode was forgotten. The earlier test missed this by pre-marking the
ancestor, which exercises only the fast path.
The refusal of any directory other than the one being listed is now kept
out of the group's result; it is already remembered for the next visit.
collectVolumeStats walked only DataNode.GetVolumes(), which returns the
regular volumes on each disk. An encoded volume leaves that set and is
reported through GetEcShards instead, so total_disk_bytes and
total_volume_count silently excluded every erasure-coded volume: a cluster
that encoded everything reported zero bytes and zero volumes while still
counting as a volume server.
Sum each holder's shard sizes into the byte total, parity and extra copies
included, matching how a replicated volume's used size counts every replica
and how CollectionEcVolumeStats already reports EC footprint. Count volume
ids rather than shard entries, since one volume's shards are spread over
many nodes and would otherwise multiply the volume count by the number of
holders.
* pb: carry a volume digest on the heartbeat
The full volume list is the only way a master notices a volume that vanished
without a delta, so it cannot simply be dropped. A digest gives the same
guarantee without the list, and a way back to the list when they disagree.
The digest has explicit presence: a server holding no volumes reports 0, which
has to stay distinguishable from a server that does not compute one at all.
* volume: report a digest of the volumes each heartbeat carries
Digests exactly what goes on the wire: volumes skipped as quarantined, phantom
or expired are absent from both the list and the digest, so the master compares
against the same set the server meant to report.
Runs the master's own hash over the master's own conversion of the message, so
the two ends cannot drift into disagreeing about a field.
* master: check the reported volume digest and ask for the list on a mismatch
Compared after everything the heartbeat carried has been applied, so agreement
means the master is current rather than that nothing changed.
Servers reporting no digest are untouched, and a mismatch on a heartbeat that
already carried the full list is reported rather than answered: there is
nothing further to ask for, so asking again would loop. Nodes reporting one
volume id twice are skipped for the same reason.
* rust: report the heartbeat volume digest
Mirrors the Go volume server. The master compares this against a digest it
computes itself, so the hash has to agree byte for byte across the two
implementations, not merely be a hash of the same fields: report_hash_vectors
pins it against values generated by the Go side, and the ttl and replica
placement narrowing the master applies when it decodes a message is applied
here too rather than assumed away.
A drift there would not corrupt anything, but every volume server on this
implementation would report a digest the master can never match and fall back
to sending its whole volume list forever, which is the cost the digest exists
to avoid.
* master: pin what the digest check does to each kind of report
The upgrade story rests on these: a server that reports no digest is never
asked for anything, so the two sides can be upgraded in either order, and a
disagreement that resending cannot fix is reported rather than re-asked, so it
cannot loop.
* topology: enumerate the digest coverage test from the message
The list of fields was written out by hand, so a field added to
VolumeInformationMessage later would fall outside the digest while the test
went on passing, and a change to it would never reach the master. Walk the
message descriptor instead.
Some fields are narrowed or normalised on the way into VolumeInfo, so the
smallest change to the wire value can land back on the stored one; the test
offers several values per field and asks only that some change is visible.
* telemetry: put the version pie back, beside the stacked chart
The two answer different questions and the pie was the better answer to one
of them: what the fleet is on right now, at a glance. Restore it under its
old name and give the stack its own card as Versions Over Time, so the pie
is the last day of the chart below it.
* telemetry: draw the distribution pies at the size of their cards
Both pies kept the canvas tag's 2:1 ratio at the card's full width, so they
came out around 560px tall and spilled past the card they sit in. Give them a
height to fill instead, and build every chart after the dashboard is shown: a
canvas in a display:none container measures zero, and a pie sized from that
never grows back.
* topology: digest the volumes a master believes each node holds
A volume server resends its whole volume list every heartbeat because that list
is the only way the master can notice a volume that vanished without a delta.
A digest gives the master the same guarantee without the list: the two ends
agree iff the master's copy is current.
VolumeInfo.ReportHash covers every field of VolumeInformationMessage, so a
change the hash misses is a change the master would never hear about. Both ends
run it over the same converted VolumeInfo, so they cannot drift apart.
Disk keeps the xor of its volumes' hashes, which is order-independent and its
own inverse, so add, update and remove each stay O(1) and the running value
needs no per-volume storage.
Nothing reads the digest yet; the heartbeat protocol change comes next.
* topology: test that a changed-volumes-only heartbeat reconciles
The digest is not a change detector -- in a live cluster some volumes always
have changed. It answers whether the master holds what the volume server holds
once the heartbeat's own changes are applied, so reporting three volumes out of
fifty has to reconcile while a volume lost without a delta must not.
* topology: digest the lookup index too, not just the disk maps
The reported digest answers whether the master holds what the volume server
holds. It cannot answer whether the master can serve those volumes: the disk
map and the lookup index are maintained separately, and a disconnect racing a
reconnect drops a volume from the index while leaving it on the node. The
server's report is identical either way, so a digest built from the disk maps
alone matches while the volume answers 'volume id not found'.
Track a second digest over volume ids on both sides of that split, so the
master can see its own indexes disagree without the volume server's help, and
without the O(volumes) scan the full heartbeat currently relies on.
* topology: exclude nodes reporting a duplicate volume id from the digest
A volume id can end up mounted on two disks of one server -- a stale twin
re-attached after a disk repair, which the store handles rather than rejects.
The server reports both copies with different disk ids, but the master keys
volumes by id alone within a disk type and keeps only the last one. Its digest
can then never equal the server's, and no amount of resending the full list
would fix it.
Detect it from the report itself, where deduplicating the ids already tells us
the count, and mark the node. A marked node has to keep sending full lists;
representing both copies is a separate question, and nesting the volume map by
disk id would cost more memory than the digest saves.
* topology: move the lookup digest with the entry, not the node passed in
Two volume servers can hold one address: GetOrCreateDataNode keys on the id a
server reports and refuses to merge a new id onto an address an older node
still claims, while the lookup list keys on address alone. Registering the
second server therefore displaces the first from the entry, and unregistering
through either removes whichever node the entry named.
Crediting the node handed to Set and Remove instead of the one actually
displaced or removed left the digest on the wrong node. A displaced node went
on reporting a consistent index while it could no longer serve the volume,
which is exactly the silent unavailability the digest exists to catch.
Set and Remove now return the node they displaced and removed, so ownership
can be transferred rather than assumed.
* mount: size the direct listing slice from the batch, not the offset
The limit passed here is skipCount+batchSize when a client resumes a
fresh handle partway through a directory, so preallocating for it turns
the client's cookie into an allocation: a readdir at offset 3,000,000
reserves 24MB before the first entry arrives, and an offset near the
uint32 ceiling asks makeslice for ~4.29e9 elements.
* mount: stop replaying a directory that shrank past the resume offset
A client that opens a fresh handle and resumes at a cookie from an
earlier, larger listing gets a preload that cannot reach the entry before
that offset. The resume name was then left empty and the follow-up batch
listed from the directory's first child again, so the client was handed
every name a second time.
The stream always runs from the first child, so failing to reach that
entry means the directory is simply shorter than the offset. That is the
end of it.
* mount: page a directory from where the store reached
The batch loader treated a short batch as the end of the directory, but
the meta cache drops an expired child after the store has already spent
it against the limit, so a batch that filled up could still deliver fewer
entries than asked for. A directory with a handful of expired children
would stop listing early and hide every child behind them; a whole batch
of expired ones truncated the listing to nothing.
ListDirectoryEntries now reports the name the store itself reached. That
is both the sound end-of-directory signal -- the store returning nothing
-- and the right cursor, since resuming from the last visible name would
re-read the dropped children every round and never get past a batch that
was entirely expired.
* mount: drop the entries a directory walk has already passed
entryStreamOffset was only ever written by reset, so the stream a handle
holds grew for the life of the walk and a directory was retained whole
even though nothing could read the entries behind the client's position
again. A 10M-entry walk parked millions of entries per handle, and
NFS-Ganesha opens several on the same directory.
Offsets index into the stream from entryStreamOffset, so advancing the
two together keeps them lined up. One entry is held back because the next
batch resumes from the name immediately before the offset. Seeking back
behind what is still held now restarts the directory, which is what the
offset scheme can honestly support -- it previously returned nothing.
* telemetry: keep the reported version in the daily history
The version only ever lived on the instance record, which holds a cluster's
latest report, so there was no way to ask what anything ran last Tuesday.
Record it per sample, and let the daily axis carry strings as well as counts.
State written before this has no version on its samples. The newest sample is
the report the instance record itself came from, so fill that one in on load
rather than starting a version series a day late.
* telemetry: serve the fleet's version make-up over time
/api/versions gives how many clusters ran each release per day, on the same
axis and hold-forward rule as the cluster sizes. Releases are ordered by
number rather than by size: the caller stacks them, and a stack whose order
changes with the counts is unreadable over time. The tail past the limit is
summed into "other" so the stack still adds up.
Days with no version are dropped before the axis is built, so the series
spans the days it knows a version for instead of climbing out of blanks.
* telemetry: draw version distribution as a stacked growth chart
The pie only ever showed today. Stacked over 30 days the height is the
confirmed fleet and each band is a release, so one chart carries the growth
and the rollouts at once. Newest release on the floor, so the band being read
is anchored to the axis instead of riding on everything below it.
Eight fixed hues instead of the evenly spaced ones the cluster stacks use:
evenly spaced put a green and a cyan close enough to be hard to tell apart,
which matters for a set you read rather than a wall of anonymous ids.
Versions past the eighth fold into "other", and each band carries its own
number so the chart reads without matching colours against the legend.
* filer: decode a listed entry without building its chunk list
A readdir reads attributes and never looks at chunks, but decoding an
entry builds the whole chunk list first: four allocations per chunk, all
of it thrown away. On a directory of ordinary 4MB-chunked files that is
most of what listing costs.
DecodeAttributesOnly walks the wire format and hands everything except
the chunks to the generated unmarshaller, so new fields in filer.proto
need no attention here. The chunks are still measured, because the S3
copy and multipart paths deliberately store a zero FileSize and let the
chunk extents define the size, but nothing is allocated to do it.
The blob is only re-encoded once a chunk is actually seen, so an entry
without any -- every directory, for one -- is unmarshalled where it lies
and pays nothing for the walk.
Listings opt in through the context, the way the lazy remote paths
already do; a store that ignores it stays correct.
chunks full attrs-only allocs
0 312.8n 310.1n ~ 1 -> 1
1 686.1n 411.1n -40.07% 7 -> 1
4 1.742u 667.4n -61.69% 24 -> 1
16 5.770u 1.544u -73.25% 86 -> 1
64 25.23u 6.004u -76.20% 328 -> 1
* mount: list directories with chunk lists omitted
The two meta cache listings behind a readdir are the only callers, and
neither reads a chunk. On 200k single-chunk files one enumeration goes
from 364ms to 277ms and drops a million allocations.
The read-through listing still fetches whole entries from the filer,
which would need the request to say it wants attributes only.
* mount: give the readdir benchmark's entries a chunk
Chunkless entries made the decode look far cheaper than it is, which is
the part of a listing worth measuring.
* filer: let a listing ask for entries without their chunk lists
The read-through readdir fetches whole entries over gRPC, and for a wide
directory the chunk lists are most of what crosses the wire and most of
what the client then unmarshals. A 4MB-chunked file is 113 bytes of
entry against 46 without its chunk.
ListEntriesRequest gains omit_chunks. The size a client needs is already
in the attributes, where the store decode folded the chunk extents in,
so dropping the list costs the client nothing.
The filer still reads the entries whole. A listing is where a TTL-expired
entry gets collected and deleted, and deleting one needs its chunks to
find the data, so omitting them there would leak. Only the response is
trimmed.
The hint moves to filer_pb so one context flag serves both transports:
the gRPC request sets omit_chunks, and a listing served from the local
store skips building the chunks. Cache population is unaffected either
way, since EnsureVisited starts from its own context.
* filer: reject a chunk the full decoder would reject
The walk skipped a chunk's bytes without looking inside them, so a
FileChunk carrying a corrupt nested fid, or a string that is not valid
UTF-8, sailed past the listing decoder while every other read of the same
entry still failed. The file listed with a plausible size and then gave
EIO on open, and corruption that used to fail the listing loudly was
hidden instead.
The chunk bytes are the one part of the blob the generated unmarshaller
never sees, so the two checks it would have made are made here: a
submessage has to parse, and a proto3 string has to be valid UTF-8.
FileChunk's only submessages are FileIds of scalars, so walking them is a
complete check. A descriptor-driven test fails if FileChunk ever gains a
field of either kind that the walk does not know to check, which is the
part that keeps this honest as filer.proto grows.
Taking the scratch buffer lazily, only once a chunk is actually dropped,
also takes the pool out of the path for entries that have none. Those
were measurably slower than the full decoder before; they are now level
with it. Each chunk's length prefix is parsed once rather than twice.
chunks full attrs-only vs base
0 171.4n 176.9n ~ (p=0.670)
1 366.6n 259.4n -29.24%
4 1.034u 500.2n -51.60%
16 3.905u 1.464u -62.52%
64 13.48u 5.195u -61.46%
* filer: carry the size before dropping chunks over the wire
Dropping the chunk list assumed every store folds the chunk extents into
FileSize when it decodes. A store that keeps entries as JSON rather than
as an encoded Entry never re-derives it, so an object written with a zero
FileSize kept its real size only in the chunks, and stripping them left
the client reading the file as empty. Stamp the size into the attributes
first, which costs nothing and does not depend on how the store loaded
the entry.
* mount: test that the readdir context reaches the store decode
Everything else exercises the decoder directly, so a refactor that
stopped threading the context would have reverted the whole thing with
every test still passing.
The benchmark's chunks also carried a constant legacy FileId, which
BeforeEntrySerialization reparses over Fid on the way in, so all 200k
entries stored one byte-identical chunk rather than the varying fixture
it looked like.
GetVolumeIds ranged over a slice and collected the loop indices, so it reported
0-99 rather than the node's volume ids. Nothing calls it: the disk-level
GetVolumeIds, which ranges over a map and is correct, is what ToDiskInfo and
ToMap use. Its private getVolumes helper went with it, having no other caller.
* perf(weed/topology): preallocate the node's volume concatenation
A node's volumes are gathered per disk and concatenated into a slice grown from
nil, so a server with several disks reallocates and copies its way up. The
writable-volume refresh loop does this for every node every few seconds.
BenchmarkDataNodeGetVolumes/8Disks 322551844 B/op -> 121602326 B/op
* perf(weed/topology): fill one slice across a node's disks
Each disk built its own right-sized copy of its volumes, and the node then
copied all of them again into the combined slice. Appending into the caller's
slice makes it one allocation whatever the disk count, which halves even the
single-disk case.
BenchmarkDataNodeGetVolumes
1Disks 121602326 B/op 2 allocs/op -> 60801314 B/op 1 allocs/op
8Disks 121602326 B/op 9 allocs/op -> 60801024 B/op 1 allocs/op
* perf(weed/topology): preallocate the /dir/status volume list
ToVolumeMap boxes every volume on a node into an []interface{} grown from nil,
so the slice reallocates its way up alongside the boxing. The count is known.
* perf(weed/topology): preallocate the volume id list sent to clients
Every filer, s3 gateway, and mount that connects to the master gets one
VolumeLocation per data node carrying that node's whole volume id list, grown
from nil. The count is known.
The ec ids are left alone: shards of one volume can span disks, so the shard
count is an upper bound on the deduped vid count, not the count itself.
ToDiskInfo builds a protobuf message per volume and per ec shard on the disk,
growing both lists from nil. Every VolumeList call runs it for every disk in
the cluster, and the admin dashboard, the plugin worker, several shell commands
and the s3 gateway's per-minute bucket metrics all call VolumeList. Both counts
are already in hand.
ToTopologyInfo over 550k volumes 202.2 MB -> 184.6 MB
A reconnecting volume server reports every volume it has as new, so newVolumes
grew from nil to one entry per volume, reallocating and copying its way there.
Sizing it to len(actualVolumes) instead would allocate the whole list on every
steady-state heartbeat, where nothing is new.
After the deletion pass everything left on the node is also in this heartbeat,
so the difference is exactly what the node is about to gain: all of them on a
reconnect, none in steady state.
First registration of 550k volumes 1041.7 MB -> 667.4 MB
* mount: readdir enters a child in the inode table only when it takes a reference
Only readdirplus into the kernel takes a reference on the children it
reports, and only that reference brings a FORGET later to take the entry
back out. Every other listing was inserting all its children anyway.
On WinFsp that meant a listing looked each child up, took a reference,
and immediately gave it back, so a walk of a wide directory paid three
write-lock acquisitions per entry to leave the table exactly as it found
it. On a plain kernel readdir nothing gives the entry back at all, so
listing a directory of 200k files grew both maps by 200k entries that
were never reclaimed.
A dirent's inode number is informational either way: the kernel must
LOOKUP before it can use a nodeid, and the WinFsp adapter re-resolves
every operation by path. So report the number and let the mapping be
built when something actually looks the entry up.
* mount: take the readdirplus reference without a second full lookup
The entry has just been resolved a few lines above, so redoing the whole
lookup only rebuilds the child path and walks both maps again to reach a
counter. Bump it directly, falling back to the full lookup if a Forget
removed the entry in between.
* mount: benchmark a readdir over a 200k directory
Drives doReadDirectory against a meta cache holding 200k entries, one
round of 4096 at a time, for the three front ends that behave
differently: a plain kernel readdir, kernel readdirplus, and a WinFsp
listing that gets attributes but never returns a reference. Reports what
each leaves behind in the inode table alongside the usual metrics.
The sink declares TakesLookupRef as an ordinary method rather than
through the interface, so the same file runs unchanged against an older
tree for comparison.
* mount: stamp an inode on the benchmark's entries
The filer stores one on every entry it writes, so a real listing arrives
with an inode and never derives its own. Leaving it zero made every child
in the benchmark fall through to the MD5 in AsInode, work no filer-backed
mount does, and charged it to both sides of the comparison.
NewReplicaPlacementFromByte formatted the byte with fmt.Sprintf and parsed the
result back, allocating a string and a ReplicaPlacement every call. The master
calls it once per volume in every heartbeat, and keeps the pointer for the
lifetime of the volume, so a cluster with 1.6M volume replicas carries 1.6M of
these where a handful of distinct values exist.
The table is a flat pointer-free array, so it costs 6KB of static data and no
heap objects however few placements a cluster actually uses.
A byte only ever decodes to a valid placement, so the table is complete and the
error return stays nil.
BenchmarkSyncDataNodeRegistration/100000Volumes 500601 allocs/op -> 300589 allocs/op
The master decodes a TTL per volume in every heartbeat and keeps it for the
volume's lifetime, so a cluster using TTLs carries one two-byte object per
volume replica where at most 256 counts times 7 units exist. Share them, and
decode the uint32 form directly instead of staging it through a byte slice.
Clusters that set no TTL are unaffected; that path already returned the shared
EMPTY_TTL.
BenchmarkSyncDataNodeRegistration/100000Volumes, volumes carrying a ttl
600600 allocs/op -> 500597 allocs/op
2026-08-07 00:51:52 -07:00
Chris LuGitHubcoderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* perf(weed/topology): log writable-state changes, not every check
ensureCorrectWritables ran its three diagnostics whenever it was asked, so a
volume that had always been read-only re-announced that on every registration.
A volume server reconnecting with 550k read-only volumes made the master format
over a million log lines before it could serve anything, which is exactly when
it is already at its memory peak rebuilding the topology.
removeFromWritable already reports the transition, and only when there is one.
Explain it only then. Dropped the separate 'remove from writable' line, which
said nothing that 'becomes unwritable' does not.
BenchmarkRegisterReadOnlyVolumes, 100k volumes
285791480 B/op 1902069 allocs/op -> 234592088 B/op 1302572 allocs/op
* Update weed/topology/volume_layout.go
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Disk.GetVolumes copies the disk's whole volume map into a fresh slice, and every
caller that walks a node's volumes goes through it: the writable-volume refresh
loop every few seconds, ToTopologyInfo on each VolumeList, telemetry, and node
unregistration. Growing from nil reallocates and copies about twice the final
size each time, which at 100k volumes per disk is 60MB of garbage per call.
BenchmarkSyncDataNodeRegistration/100000Volumes 199670102 B/op -> 137728051 B/op
* perf(weed/topology): keep only volume ids in the heartbeat membership set
The map is used solely to test whether a known volume is still present, but it
copied the whole 152-byte VolumeInfo for every volume in the heartbeat. Presize
it too, since the count is known.
BenchmarkSyncDataNodeRegistration/100000Volumes 199670102 B/op -> 180157310 B/op
* perf(weed/topology): diff a heartbeat without copying the volume map
To find volumes the data node no longer reports, UpdateVolumes copied every
VolumeInfo on the node into a fresh slice, then deleted the missing ones one at
a time. At 100k volumes that is a 15MB copy per heartbeat to usually find
nothing. Scan the disk maps in place instead and return only what was removed.
BenchmarkSyncDataNodeRegistration/100000Volumes 180157310 B/op -> 87806436 B/op
* test(weed/topology): benchmark the per-heartbeat volume sync
A volume server re-sends its entire volume list every VolumePulsePeriod, so
SyncDataNodeRegistration is the master's steady-state per-server cost. Give it
a benchmark so allocation regressions show up.
* perf(weed/topology): preallocate the heartbeat volume conversion slice
The slice grows to one entry per volume on the data node, so at 100k volumes
the doubling copies allocate 60MB of garbage per heartbeat. The final length is
known up front.
BenchmarkSyncDataNodeRegistration/100000Volumes 199670102 B/op -> 137725872 B/op
Chris LuandGitHub
Mohit TalniyaandGitHub
Dmitriy PavlovandGitHub
