mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-08-16 04:06:44 +00:00
567052bfb669cd8ea2dbe323493142ab9410ec66
14703
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Chris LuandGitHub
|
567052bfb6 |
s3: take bucket sizes from the master's summary (#10664)
* pb: ask the master what each collection holds Callers tracking usage were sent every volume in the cluster to add up themselves, which is the master's largest single allocation. * topology: summarise what each collection holds One pass over the topology, allocating per collection rather than per volume. Regular volumes count once each for logical totals and once per replica for physical, taken from the lookup index, which is already keyed by volume and so needs no set of seen ids. Ec shards are node-local so their sizes sum, while the file and delete counts describe the volume and resolve once every holder has been seen. Replicas of one volume disagree while a write is landing or a heartbeat is late. Walking a full listing took whichever replica the map iteration reached first, so the answer moved between runs; this takes the largest, which is stable and never reports usage below what some replica already holds. * s3: take bucket sizes from the master's summary The bucket size metrics pulled the whole volume list once a minute and added it up, which cost the master 184.6MB of allocation and 17.8MB on the wire for six numbers per collection. VolumeList over 550k volumes 184.6 MB allocated, 17.8 MB on the wire CollectionStatistics 176 bytes allocated, 47 bytes on the wire The aggregation moves to the master with it, so the cases the removed tests covered are now asserted against it directly. * topology: count the replica holding the most live data Quotas are enforced on size less deletions, and the replica with the biggest raw size can be the one that has deleted the most. Counting it reported a bucket smaller than it is and would leave one writable over its quota, which is the opposite of what picking the largest was meant to guarantee. * topology: cap a volume's deletions at what it holds Live usage is read as a collection's size less its deletions, so a volume reporting more deleted bytes than it has cancels live bytes belonging to other volumes in the same bucket and reports it smaller than it is. Replica selection already floored that volume's own live size at zero; the totals have to agree with it. |
||
|
Chris LuandGitHub
|
38db7e1493 |
storage: share the volume strings a cluster repeats (#10665)
* storage: share the volume strings a cluster repeats Decoding a heartbeat allocates a fresh string for the collection, disk type and remote backend of every volume, and a master holding a million volumes then holds a million copies of the same handful of names. Not the remote storage key, which is unique per volume: interning that would fill the table rather than share anything. 800k volumes registered from a heartbeat that has actually been over the wire: 227 -> 211 B/volume, and 238 -> 214 when the volumes are tiered, since the backend name shares too. * storage: hold the interned strings rather than let them be collected unique.Make clears its entries by weak reference, and its canonical value does not survive a collection even while a caller still holds the string it handed back -- so a volume reported later would get a second copy of a name the rest of the cluster already shares. With only changed volumes reported, most are interned once and never again, so that is the common case rather than a corner. The table therefore only grows, which is why it stays restricted to values drawn from a small set. Ten thousand collections keep a few hundred kilobytes. |
||
|
Chris LuandGitHub
|
923d0bd20c |
iceberg: repair non-compliant manifests at commit (#10641)
* iceberg: stamp a default name mapping on new tables * iceberg: repair non-compliant manifests at commit * s3tables: verify ClickHouse writes read back through PyIceberg * iceberg: carry the manifest-list content into repaired manifests * iceberg: refresh the default name mapping on schema evolution * iceberg: merge historical names into the refreshed name mapping * iceberg: never fail a commit on repair fallout * iceberg: harden manifest repair against writer dialects * s3tables: keep PyIceberg reader stderr out of row data * iceberg: keep name mappings unambiguous across field id reassignment * iceberg: align existing manifest content metadata with the list entry |
||
|
Chris LuandGitHub
|
2d9ea0285c |
s3: add the RenameObject endpoint (#10659)
* s3: add the RenameObject endpoint
PUT /{bucket}/{key}?renameObject with x-amz-rename-source moves an object
through the filer's AtomicRenameEntry, so no bytes are read or rewritten and
the ETag, tags and SSE keys travel with the entry.
Only unversioned buckets: a versioned rename would have to rebuild the
.versions chain, and AWS offers RenameObject on directory buckets, which
cannot be versioned. The source arrives in a header, so it is authorized
separately for read and delete; both keys are locked, in key order, across the
precondition checks and the move.
* s3: let a matched source ETag precondition settle its date precondition
RFC 7232 has an ETag precondition outrank the date precondition on its own
side, and AWS documents the same for CopyObject: a matching
x-amz-copy-source-if-match with a failing x-amz-copy-source-if-unmodified-since
copies rather than returning 412. The source check evaluated all four headers in
sequence, so the date header could still veto a decided ETag match.
validateConditionalHeadersForReads already applies this precedence; the source
path now matches it.
* s3: cover a rename source named as a directory without a trailing slash
Renaming a directory would move a whole subtree, so it has to stay a missing
key whether or not the caller wrote the trailing slash.
* s3: accept a bare object key as the RenameObject source
AWS spells x-amz-rename-source both ways. Its CLI, Java and Rust examples pass
the bare source key, and only a second CLI example and the boto3 conditional
example pass bucket/key; the API reference's own example is a bare key too. The
header was read as bucket/key only, so the form AWS leads with was rejected with
InvalidArgument and the endpoint was unusable as documented.
A value is now read as a literal key first — the only reading that can never
name the wrong object — and as bucket-qualified second, when it carries the
request's own bucket and the literal key does not exist. That costs one extra
lookup only for a source that starts with the bucket's own name.
Another bucket's name in the source is no longer a distinct error: RenameObject
moves within one bucket, so it is simply part of a key this bucket does not
hold, and it reports NoSuchKey.
* s3: only a proven absence picks the other reading of a rename source
A source that resolves to a directory is not a miss to fall through on: the
literal path is still what the caller named, so answering for it beats renaming
a different object under the bucket-qualified reading. With a directory at
bucket/source.txt and an object at source.txt, a rename naming the former moved
the latter.
A failed lookup is not a proof of absence either, so a blip can no longer
redirect a rename to the other reading.
|
||
|
Chris LuandGitHub
|
67b0cc0706 |
topology: keep per-node volume state with the location it describes (#10654)
* topology: keep per-node volume state with the location it describes The read-only and oversized indexes were maps from volume id to a list of the nodes reporting that state -- the same key space the lookup index already holds, kept a second and third time. Nothing ever asked which nodes; both are only ever asked whether any node does. So the state rides on the location list as a bit per entry, and the two indexes go. Removing a location shifts the bits with it, and a node replacing another at the same address inherits its slot, since that is what happens to the location too. 800k volumes, 90% read-only: readonly index 40.0MB -> 0, lookup index 42.5MB -> 48.6MB for the bits, 33.9MB net. * topology: rebuild the location flags when stale entries are dropped Refresh rebuilds the location list, so leaving the flags alone left bits describing whoever moved into the dropped entries' place. * topology: assert the refreshed flag survived, not just that it moved Clearing the mask rather than rebuilding it would have passed: the check that the flag lands on the right location is done by clearing it, which an already empty mask satisfies. |
||
|
Chris LuandGitHub
|
2dc59c9b51 |
topology: track volume size only where writes can land (#10653)
* topology: track volume size only where writes can land Size tracking decays pending assignment estimates so the master does not overfill a volume before heartbeats catch up. Nothing is ever assigned to a read-only volume, so an entry for one can never be consulted -- and in a tiered cluster that is most of them, which made this the volume layout's largest cost. A volume held out of the writable list for capacity is not read-only and keeps its entry: that entry is what enforces the recovery delay. 800k volumes, 90% read-only: sizeTracking 79.1MB -> 8.4MB, and the crowded set falls out with it because a read-only volume no longer reaches the threshold check at all. * topology: decide size tracking per volume, not per reporting replica A volume is unwritable if any replica is read-only, so asking the replica whose heartbeat happened to arrive made the answer depend on arrival order: a writable replica reporting after a read-only one put the tracking back. Ask the volume instead, which also drops the caller-supplied flag and the churn it caused. The crowded entry goes with the tracking, since leaving it behind would only move the memory this releases. Costs a map lookup per replica on a full-list heartbeat, about 19ms per 100k volumes and no allocations, on a path that is now rare. |
||
 
|
0dd33bd7ec |
s3api: fix ListObjectsV2 dropping objects under a partial prefix (#10656)
* s3api: fix ListObjectsV2 dropping objects under a partial prefix
ListObjectsV2 with a prefix that names only part of a directory name -
"data/a" matching both "data/a/..." and "data/ab/..." - dropped objects
when paginating with a small max-keys, reporting IsTruncated=false before
all matching keys had been returned.
On resume, normalizePrefixMarker took the listing directory from the
continuation marker ("data/a/") and discarded the "a" name prefix, so once
the marker's subtree was exhausted the sibling directories that also
matched the prefix were never listed.
Resolve the listing directory and name prefix from the prefix instead of
the marker, so siblings sharing the name prefix are still listed after the
marker's subtree.
Fixes #10652
* s3api: cover a sibling-directory marker in the partial prefix tests
* s3api: resume a truncated delimited listing from the emitted CommonPrefix
Rebuilding the continuation token from the request dir plus the listing prefix
assumes that prefix names a directory. With a partial name prefix like "data/a"
it produced "data/a/ab/" for the sibling prefix "data/ab/", so the next page
resumed in the wrong subtree and repeated it forever.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
|
||
|
Chris LuandGitHub
|
a2ffc7aadf |
heartbeat: keep the master current through collection churn (#10657)
* heartbeat: name departed volumes in delta heartbeats * master: release the lookup index with a deleted collection * master: keep a fresh grow safe from the report that raced it * volume: name the volumes a deleted collection took with it Deleting a collection left the master to work out what went by omission from the next full volume list, which it no longer gets: heartbeats carry the whole list only when the master asks for it. The volumes a bucket's churn creates and destroys between two of those requests are never named in either direction, so the master keeps counting their slots as occupied and a cluster that creates and drops collections quickly runs its free-slot accounting dry -- assigns fail with no free volumes left while the disk holds a handful of volumes. The destroy path already knows exactly which volumes it removed, so send them down the same channel every other deletion uses. * rust: name the volumes a deleted collection took with it Mirrors the Go volume server. The notify path derives its deltas by diffing snapshots, so a collection delete that does not wake it is invisible until the master next asks for the whole list. |
||
|
Chris LuandGitHub
|
25d7f62749 |
topology: mark a volume crowded only if it can take writes (#10655)
* topology: mark a volume crowded only if it can take writes Crowding asks for more room to write into, and the writable-volume refresh loop marked anything past the threshold regardless of whether writes could land there. Growth already discounts those by intersecting the crowded set with the writable list, so the entries changed no decision and only took space -- in a tiered cluster, one for nearly every volume. * topology: wait for the crowded-volume collector before reading what it saw Closing the stop channel does not order the collector's writes against the test's reads. * topology: drop the sleep from the crowded-volume test The channels are unbuffered, so every send has been received by the time the sweep returns, and waiting for the collector covers the recording. The sleep only suggested the result turned on timing. |
||
|
Chris LuandGitHub
|
3a61debaa5 |
filer: rebuild peer metadata subscriptions after a master reconnect (#10648)
* filer: keep the existing peer subscription on a repeated add A cluster node add for a peer that is already followed restarted the subscription, dropping the metadata events between the two runs. * master: tell a connecting client the current cluster membership Cluster node updates are only broadcast to the clients connected at that moment. A filer that lost its master stream while a peer came back never learned about the peer, and stopped replicating its metadata for good. * test: a filer joining the master learns about the filers already there * test: a filer resubscribes to a peer that registered while it was disconnected Runs the reported sequence against real processes: filer2 leaves, filer1 is paused and its master stream is broken, filer2 registers again, and filer1 has to replicate from it after reconnecting. |
||
|
Chris LuandGitHub
|
37f3dff677 |
volume: validate the file extension in CopyFile and ReceiveFile (#10644)
* volume: validate the file extension in CopyFile and ReceiveFile
CopyFile and ReceiveFile build an on-disk path from the client-supplied
Ext. Both are intentionally ungated for cluster-internal peers, so a
value like "/../../x" is joined onto the volume directory and, once
path-cleaned, resolves outside it -- an EC-shard receive can then write,
and CopyFile read, anywhere the process can reach.
Constrain Ext to a real suffix (a leading dot followed by alphanumerics)
before it is used to build any path, so it can no longer carry a
separator or a parent reference.
* test: use an alphanumeric missing-file extension in the copy variants
The not-found and stop-offset-zero cases used ".definitely-missing" as a
deliberately absent source. The extension is now validated, and the hyphen
makes it invalid, so switch to ".missing" -- still a nonexistent file, but a
real extension shape.
* volume: validate the collection in CopyFile and ReceiveFile
The client-supplied Collection is folded into the on-disk path as
"<collection>_<vid>" by VolumeFileName and EcShardBaseFileName, both joined
with path.Join / util.Join. A Collection carrying a separator, e.g.
"../../x", therefore path-cleans to a target outside the volume directory,
the same escape the extension check just closed. Reject a collection that is
a bare parent reference or holds a separator; ordinary names ('.', '-' and
all) still pass.
|
||
|
Chris LuandGitHub
|
9d11278d95 |
filer: add filer.meta.scan to audit one directory's change history (#10645)
* filer: drain pending log chunk refs when the metadata stream ends In metadata chunks mode the server sends log file refs in responses of their own, and the client can only read them once it knows the run of refs is over. That was inferred solely from the arrival of a normal event, so refs still pending when the stream ended were dropped: the subscription returned no events and no error. A follower never noticed, because it runs forever and a live event always arrives to close the run. A bounded subscription — StopTsNs set, range already in the past — can receive nothing but refs and then EOF, and silently reports that nothing happened. For anything auditing a path that is the worst possible answer, since an empty result is indistinguishable from a quiet period. Drain on EOF as well as at the transition point. * filer: add filer.meta.scan to audit one directory's change history Reconstructing what happened to a path means replaying the metadata log, and filer.meta.tail is built for watching rather than auditing: it follows forever unless given a stop, prints multi-line JSON, and takes ranges only as durations before now, so an incident timestamp has to be converted by hand. Its -pattern also cannot find a versioned object. A versioned key is stored as <key>.versions/v_<id>, so the events carry the names "<key>.versions" and "v_<id>" and a pattern of the object's own name matches neither — the search comes back empty while the object is being written continuously. filer.meta.scan prints one line per change, stops at the end of the range, accepts absolute -since/-until with an explicit -tz, and reports versioned writes against the object key with the version id alongside, so -name matches the key a client would ask for. Delete markers are labelled as such rather than appearing as zero-length writes, and pointer flips on the .versions container are distinguished from writes of object data. * filer.meta.scan: read persisted log chunks from the volume servers Reading a range through the filer makes it decode every log entry in that range and filter each one, so the cost lands on the filer and does not shrink when the prefix is narrow — only the bytes on the wire do. On a cluster whose metadata log is dense that is the expensive part of a scan, and it is charged to the process least able to spare it. Enable metadata chunks mode: the filer hands out log chunk ids and the scan reads them from the volume servers itself. ReadLogFileRefs re-applies the same path filter client-side, so the output is unchanged — verified identical to the filer-read path over the same range, including after a restart drops the in-memory buffer and the data must come off disk. Direct read needs a route to the volume servers that the filer does not, so a failure before anything has been printed retries through the filer; retrying after partial output would duplicate lines. -directRead=false forces it. * filer.meta.scan: confirm an empty direct-read result through the filer An audit that returns nothing is read as "nothing happened here", so it is the one answer that must not be produced by a bug. Direct read has more ways to come back empty than the filer path does — it needs a route to the volume servers, and it depends on the ref-drain contract holding. When direct read yields no changes, re-run through the filer before reporting it, and warn if the two disagree. Re-running is safe only because nothing was printed; after partial output a replay would duplicate lines instead, so that case reports the error rather than retrying. |
||
|
Chris LuandGitHub
|
344ac7684e |
filer: drain pending log chunk refs when the metadata stream ends (#10647)
In metadata chunks mode the server sends log file refs in responses of their own, and the client can only read them once it knows the run of refs is over. That was inferred solely from the arrival of a normal event, so refs still pending when the stream ended were dropped: the subscription returned no events and no error. A follower never noticed, because it runs forever and a live event always arrives to close the run. A bounded subscription — StopTsNs set, range already in the past — can receive nothing but refs and then EOF, and silently reports that nothing happened. For anything auditing a path that is the worst possible answer, since an empty result is indistinguishable from a quiet period. Drain on EOF as well as at the transition point. |
||
|
Chris LuandGitHub
|
e9cde3e4b1 |
master: gate raft membership RPCs behind the admin whitelist (#10649)
* master: evict a dead peer via the local raft handle OnPeerUpdate only runs on the leader, and the AddVoter branch right above mutates the local raft directly. The remove branch instead dialed our own RaftRemoveServer back over gRPC. Drop the self-dial and remove the peer through the local handle, matching the add path. This also leaves operator tooling as the only caller of the RaftRemoveServer RPC. * master: require whitelist auth for raft membership RPCs RaftAddServer, RaftRemoveServer and RaftLeadershipTransfer rewrite raft quorum but had no caller check beyond "am I the leader". Any client that could reach the master gRPC port could add an unreachable phantom voter and stall the write path. Gate the three on the admin whitelist, mirroring the volume server's checkGrpcAdminAuth. With no whitelist configured the guard allows every caller, so default and single-master deployments are unaffected; operators who set -whiteList get these RPCs locked down to it. The leader's own dead-peer eviction no longer dials these RPCs, so the only remaining callers are operator tooling. |
||
 Mohit TalniyaandGitHub
|
5b9236c76d | storage: fix corrupted leveldb detection in DoOffsetLoading (#10650) | ||
|
Chris LuandGitHub
|
ce7d388639 |
heartbeat: send only the volumes that changed (#10640)
* pb: let a heartbeat carry only the volumes that changed A partial list cannot travel in volumes: a master that did not understand it would read the absences as deletions. So changes get their own field, used only once the master has said it compares digests and can tell when it has fallen behind. * master: apply the volumes a heartbeat reports as changed Only the named volumes are touched. A full report says the server holds exactly these; a changed report says nothing about the ones it leaves out, so absence must not read as removal. Also advertises that the master compares digests, which is what lets a server stop sending its whole list. Advertising it once per connection means a server reconnecting to a master that does not is back to full lists straight away. * volume: send only the volumes that changed once the master accepts them The whole list goes on every heartbeat until the master says it compares digests, and again whenever it asks, so a master that cannot tell when it has fallen behind never has to. has_no_volumes stays derived from a full list alone. Deriving it from what a heartbeat happens to carry would make a quiet one read as a server that had lost every volume, and the master would drop them all. The digest still covers every volume held rather than the ones sent, which is what lets the master confirm that applying the changes left it current. Reporting state is per-connection: a server that reconnects, or reaches a different master, starts again from the full list. * volume: let the zero reporting state stand for having told no master anything A Store built as a literal, which tests do, left the reporting state nil and panicked on the first heartbeat. As a value its zero form already means nothing has been reported to anyone, which is exactly the state that sends the whole list. * rust: send only the volumes that changed once the master accepts them Mirrors the Go volume server, with one hazard the Go side does not have: mount and unmount deltas here are derived by diffing successive heartbeats, so a heartbeat that carries a partial list would report every volume it left out as unmounted. Collecting now returns the full set alongside the message, and every site that diffs uses that rather than what went on the wire. * volume: do not let a full-list request be lost to the heartbeat it raced The request arrived while a heartbeat was already being built as a delta, and committing that heartbeat cleared it, so the master waited for another digest mismatch before asking again. Count the requests and clear only the one the heartbeat answered. * rust: stop marking volumes reported by a heartbeat that is thrown away The state-notify path collected a heartbeat only to diff its volume list, then sent a delta message of its own and dropped the one it had collected. Once collecting recorded what the master had been told, every mount or unmount silently marked the changed volumes as sent, and the master learned of them only after a digest mismatch. Snapshotting no longer records anything, and no longer expires ec volumes whose deletion that path was already discarding. * master: announce only the volumes a change actually brought Every changed volume was broadcast as a new location. Volumes grow constantly and growth moves no location, so on a busy cluster that told every connected client about volumes it could already reach, filling bounded broadcast queues and pushing out the topology updates that matter. * master: ask for the full list when only one can repair the master Delta heartbeats stop the full report, and with it the only thing that re-registers a volume the lookup index lost. The volume server cannot see that divergence and its digest cannot show it, so the master now checks its own two indexes agree and asks for the list when they do not. A node reporting one volume id twice is kept on full lists for the same reason rather than merely skipped: its digest can never be verified, so nothing else would tell the master what it had stopped holding. * master: keep the volume options on every heartbeat response A volume server takes them from whatever response arrives, and preallocate is a bare bool with no way to tell off from unmentioned. A response sent to ask for the volume list therefore turned preallocation off until the server reconnected. Responses sent mid-stream now start from the configured options rather than being built field by field. * master: announce a volume the lookup index had lost Repairing the index makes the volume servable again, but clients were told it went when the node dropped out and nothing told them otherwise: the disk map still held it, so it did not count as an arrival. Reaching the lookup index is what makes a volume servable, so recovering an entry there is an arrival as far as clients are concerned, on both the full report and the changed-volume path. |
||
|
Chris LuandGitHub
|
213eb4c23a |
s3tables: add ClickHouse iceberg catalog integration test (#10637)
* s3tables: add ClickHouse iceberg catalog integration test * ci: run the ClickHouse iceberg catalog test * s3tables: bound setup HTTP calls in the ClickHouse test * s3tables: pin the ClickHouse writer image dependencies |
||
|
Chris LuandGitHub
|
cab666fca1 |
filer: configurable TUS max upload size and session expiry (#10638)
* make TUS max upload size and session expiry configurable * default TUS session expiry to 24h |
||
 Dmitriy PavlovandGitHub
|
457277ec9a | reload filer config on local metadata updates (#10622) | ||
|
Chris LuandGitHub
|
08f0ba5564 |
topology: clamp the deleted-vs-total subtractions in volume stats (#10633)
VolumeLocationList.Stats subtracts the deleted figures from the totals to report live size and needle count. Both deleted figures are maintained as counters independent of the totals they come off, so either can transiently exceed its total, and neither subtraction was clamped. Unclamped, the size wraps to ~16 EB. The count is signed so it merely goes negative, but VolumeLayout.Stats converts it with uint64(fileCount), which turns it into ~1.8e19 just the same. Either one swamps the cluster totals behind /dir/status, /vol/status and Topology.CollectionVolumeStats. commandFsMergeVolumes.getVolumeSize had the same unclamped subtraction, where a wrapped size reads as a volume far too large to join any merge plan. Clamped to zero, matching the guards already in CollectionInfo.LogicalSize and the admin server's logical-size accumulator. |
||
|
Chris LuandGitHub
|
4527947afc |
mount: absorb the WinFsp metadata cache window in the concurrent-reader test (#10636)
WriteFile's own existence probe runs while the file does not exist, and WinFsp may serve that answer from its metadata cache for up to the mount's FileInfoTimeout. A reader racing into that window failed its open with not-found, which is the cache being a cache, not a defect in concurrent reading. Establish visibility once before racing the readers, so the test exercises what it is named for. |
||
|
Chris LuandGitHub
|
0b78381513 |
wdclient: keep the location of a volume reported added and removed at once (#10635)
* wdclient: keep the location of a volume reported added and removed at once A volume moved between a server's disks arrives in both lists of one message, and the server still has it. Additions were applied before removals, so the removal won and the client was left with no location for a volume that never went anywhere. Reordering would swap the bug for a window where the volume resolves nowhere, since the two updates take the lock separately. Skip the removal instead, so the order the lists are applied in stops mattering. * wdclient: build each ec update explicitly in the move test Reusing one response object and adding the deletion to it left the overlap the test turns on implicit, and reading it as a delete-only update is the natural mistake. |
||
|
Chris LuandGitHub
|
5ec813b4f1 |
topology: follow a volume that moved between a server's disks (#10628)
* topology: follow a volume that moved between a server's disks The heartbeat diff asked only whether a volume id was reported anywhere on the node, so a volume that moved to a disk of another type stayed on the disk it left as well. The master then held two copies of it forever: the volume count was overstated, and GetVolumesById returned whichever disk the map iterated first, so lookups could hand back the disk the volume had already left. Track which disk types the heartbeat named each volume on, and treat a volume named on another disk as absent from this one. Disk types are interned to an index because a server reports a handful of them across hundreds of thousands of volumes. A volume named on two disks at once is a stale twin rather than a move, and is still kept on both -- dropping one would tell the master a replica vanished. Only a volume named twice on one disk type is unrepresentable, so that is now what marks the node, rather than any repeat of an id. * master: do not tell clients a moved volume left the node A volume moved between a node's disks is removed from one and added to the other, so it lands in both lists of the same heartbeat. Clients apply additions before deletions, so the removal wins and they end up with no location for a volume that never went anywhere. Skip removals for volumes the node still holds, as the ec shard paths already do, and update the topology before judging the delta removals so an unmount that really did happen is still reported. * trim the comments on this change to the parts that are not evident * master: judge a volume removal on normal replicas alone HasVolumesById answers for ec shards as well, so a replica encoded into ec shards looked like it was still on the node and clients were never told the normal location had gone. They hold normal and ec locations separately and prefer the normal one from the same generation, so that location would have gone on shadowing the shards. |
||
|
Chris LuandGitHub
|
75ae33ade8 |
mount: let the kernel cache directory listings (#10634)
Every enumeration of a directory walked the whole FUSE machinery, so
reopening a folder cost what opening it did. The kernel has a cache for
exactly this: with FOPEN_CACHE_DIR the listing lives in the directory's
page cache and a repeat enumeration never reaches the mount at all.
Local mutations already drop that cache in the kernel. Remote ones
arrive through the metadata subscription, so the entry invalidation
worker now also tells the kernel which directory changed. The worker is
the one place this is safe from: notifying from a thread serving a
kernel request can deadlock against the page it holds, which is why the
file paths deliberately avoid InodeNotify.
Measured in a Linux container, 20k-entry directory, ls repeated:
warm listing before 199-355ms after 6-9ms
A file written from outside the mount appeared in the next listing
within a second, through the subscription notify, and the listing
re-cached after.
The memory is the kernel's page cache: reclaimed under pressure, owned
per-directory, and covering read-through directories the mount-side
caches never see.
|
||
|
Chris LuandGitHub
|
dd73fee077 |
mount: read oversized directories through instead of caching them (#10631)
* mount: read oversized directories through instead of caching them Visiting a directory pulls every child from the filer into the local LevelDB before the first listing returns. For a directory of a few million entries that is minutes of streaming, gigabytes of local store, and gigabytes of decoded entries in flight -- paid by a mount that may only walk the directory once. A build that crosses -cacheDirMaxEntries (default ten thousand) now stops, cleans up, and marks the directory read-through: listings stream from the filer with pagination, the way update-hot directories already do, and lookups in it consult the filer per entry as any uncached directory does. The refusal is remembered, so the next visit fails fast instead of streaming to the limit again, and an oversized ancestor is stepped over when caching its subdirectories rather than wedging every listing beneath it. The direct path keeps the same pagination state on the handle, so a walk that crosses the limit mid-flight carries on from where the cached walk reached. * mount: an ancestor found oversized must not fail its descendants Visiting a directory builds its whole uncached ancestor chain in one group, so the first discovery that an ancestor is oversized cancelled the group and surfaced as the listed directory's own refusal: the descendant build was aborted and the caller marked the descendant read-through, leaving a perfectly cacheable directory streaming from the filer until its inode was forgotten. The earlier test missed this by pre-marking the ancestor, which exercises only the fast path. The refusal of any directory other than the one being listed is now kept out of the group's result; it is already remembered for the next visit. |
||
|
Chris LuandGitHub
|
506ce0850b |
telemetry: count erasure-coded volumes in the reported totals (#10632)
collectVolumeStats walked only DataNode.GetVolumes(), which returns the regular volumes on each disk. An encoded volume leaves that set and is reported through GetEcShards instead, so total_disk_bytes and total_volume_count silently excluded every erasure-coded volume: a cluster that encoded everything reported zero bytes and zero volumes while still counting as a volume server. Sum each holder's shard sizes into the byte total, parity and extra copies included, matching how a replicated volume's used size counts every replica and how CollectionEcVolumeStats already reports EC footprint. Count volume ids rather than shard entries, since one volume's shards are spread over many nodes and would otherwise multiply the volume count by the number of holders. |
||
|
Chris LuandGitHub
|
6d08b08f37 |
heartbeat: carry a volume digest and verify it (#10627)
* pb: carry a volume digest on the heartbeat The full volume list is the only way a master notices a volume that vanished without a delta, so it cannot simply be dropped. A digest gives the same guarantee without the list, and a way back to the list when they disagree. The digest has explicit presence: a server holding no volumes reports 0, which has to stay distinguishable from a server that does not compute one at all. * volume: report a digest of the volumes each heartbeat carries Digests exactly what goes on the wire: volumes skipped as quarantined, phantom or expired are absent from both the list and the digest, so the master compares against the same set the server meant to report. Runs the master's own hash over the master's own conversion of the message, so the two ends cannot drift into disagreeing about a field. * master: check the reported volume digest and ask for the list on a mismatch Compared after everything the heartbeat carried has been applied, so agreement means the master is current rather than that nothing changed. Servers reporting no digest are untouched, and a mismatch on a heartbeat that already carried the full list is reported rather than answered: there is nothing further to ask for, so asking again would loop. Nodes reporting one volume id twice are skipped for the same reason. * rust: report the heartbeat volume digest Mirrors the Go volume server. The master compares this against a digest it computes itself, so the hash has to agree byte for byte across the two implementations, not merely be a hash of the same fields: report_hash_vectors pins it against values generated by the Go side, and the ttl and replica placement narrowing the master applies when it decodes a message is applied here too rather than assumed away. A drift there would not corrupt anything, but every volume server on this implementation would report a digest the master can never match and fall back to sending its whole volume list forever, which is the cost the digest exists to avoid. * master: pin what the digest check does to each kind of report The upgrade story rests on these: a server that reports no digest is never asked for anything, so the two sides can be upgraded in either order, and a disagreement that resending cannot fix is reported rather than re-asked, so it cannot loop. * topology: enumerate the digest coverage test from the message The list of fields was written out by hand, so a field added to VolumeInformationMessage later would fall outside the digest while the test went on passing, and a change to it would never reach the master. Walk the message descriptor instead. Some fields are narrowed or normalised on the way into VolumeInfo, so the smallest change to the wire value can land back on the stored one; the test offers several values per field and asks only that some change is visible. |
||
|
Chris LuandGitHub
|
5532a316c5 |
telemetry: put the version pie back beside the stacked chart (#10626)
* telemetry: put the version pie back, beside the stacked chart The two answer different questions and the pie was the better answer to one of them: what the fleet is on right now, at a glance. Restore it under its old name and give the stack its own card as Versions Over Time, so the pie is the last day of the chart below it. * telemetry: draw the distribution pies at the size of their cards Both pies kept the canvas tag's 2:1 ratio at the card's full width, so they came out around 560px tall and spilled past the card they sit in. Give them a height to fill instead, and build every chart after the dashboard is shown: a canvas in a display:none container measures zero, and a pie sized from that never grows back. |
||
|
Chris LuandGitHub
|
553bc5ab90 |
topology: digest the volumes a master believes each node holds (#10619)
* topology: digest the volumes a master believes each node holds A volume server resends its whole volume list every heartbeat because that list is the only way the master can notice a volume that vanished without a delta. A digest gives the master the same guarantee without the list: the two ends agree iff the master's copy is current. VolumeInfo.ReportHash covers every field of VolumeInformationMessage, so a change the hash misses is a change the master would never hear about. Both ends run it over the same converted VolumeInfo, so they cannot drift apart. Disk keeps the xor of its volumes' hashes, which is order-independent and its own inverse, so add, update and remove each stay O(1) and the running value needs no per-volume storage. Nothing reads the digest yet; the heartbeat protocol change comes next. * topology: test that a changed-volumes-only heartbeat reconciles The digest is not a change detector -- in a live cluster some volumes always have changed. It answers whether the master holds what the volume server holds once the heartbeat's own changes are applied, so reporting three volumes out of fifty has to reconcile while a volume lost without a delta must not. * topology: digest the lookup index too, not just the disk maps The reported digest answers whether the master holds what the volume server holds. It cannot answer whether the master can serve those volumes: the disk map and the lookup index are maintained separately, and a disconnect racing a reconnect drops a volume from the index while leaving it on the node. The server's report is identical either way, so a digest built from the disk maps alone matches while the volume answers 'volume id not found'. Track a second digest over volume ids on both sides of that split, so the master can see its own indexes disagree without the volume server's help, and without the O(volumes) scan the full heartbeat currently relies on. * topology: exclude nodes reporting a duplicate volume id from the digest A volume id can end up mounted on two disks of one server -- a stale twin re-attached after a disk repair, which the store handles rather than rejects. The server reports both copies with different disk ids, but the master keys volumes by id alone within a disk type and keeps only the last one. Its digest can then never equal the server's, and no amount of resending the full list would fix it. Detect it from the report itself, where deduplicating the ids already tells us the count, and mark the node. A marked node has to keep sending full lists; representing both copies is a separate question, and nesting the volume map by disk id would cost more memory than the digest saves. * topology: move the lookup digest with the entry, not the node passed in Two volume servers can hold one address: GetOrCreateDataNode keys on the id a server reports and refuses to merge a new id onto an address an older node still claims, while the lookup list keys on address alone. Registering the second server therefore displaces the first from the entry, and unregistering through either removes whichever node the entry named. Crediting the node handed to Set and Remove instead of the one actually displaced or removed left the digest on the wrong node. A displaced node went on reporting a consistent index while it could no longer serve the volume, which is exactly the silent unavailability the digest exists to catch. Set and Remove now return the node they displaced and removed, so ownership can be transferred rather than assumed. |
||
|
Chris LuandGitHub
|
12627d376d |
mount: fix four readdir pagination bugs (#10624)
* mount: size the direct listing slice from the batch, not the offset The limit passed here is skipCount+batchSize when a client resumes a fresh handle partway through a directory, so preallocating for it turns the client's cookie into an allocation: a readdir at offset 3,000,000 reserves 24MB before the first entry arrives, and an offset near the uint32 ceiling asks makeslice for ~4.29e9 elements. * mount: stop replaying a directory that shrank past the resume offset A client that opens a fresh handle and resumes at a cookie from an earlier, larger listing gets a preload that cannot reach the entry before that offset. The resume name was then left empty and the follow-up batch listed from the directory's first child again, so the client was handed every name a second time. The stream always runs from the first child, so failing to reach that entry means the directory is simply shorter than the offset. That is the end of it. * mount: page a directory from where the store reached The batch loader treated a short batch as the end of the directory, but the meta cache drops an expired child after the store has already spent it against the limit, so a batch that filled up could still deliver fewer entries than asked for. A directory with a handful of expired children would stop listing early and hide every child behind them; a whole batch of expired ones truncated the listing to nothing. ListDirectoryEntries now reports the name the store itself reached. That is both the sound end-of-directory signal -- the store returning nothing -- and the right cursor, since resuming from the last visible name would re-read the dropped children every round and never get past a batch that was entirely expired. * mount: drop the entries a directory walk has already passed entryStreamOffset was only ever written by reset, so the stream a handle holds grew for the life of the walk and a directory was retained whole even though nothing could read the entries behind the client's position again. A 10M-entry walk parked millions of entries per handle, and NFS-Ganesha opens several on the same directory. Offsets index into the stream from entryStreamOffset, so advancing the two together keeps them lined up. One entry is held back because the next batch resumes from the name immediately before the offset. Seeking back behind what is still held now restarts the directory, which is what the offset scheme can honestly support -- it previously returned nothing. |
||
|
Chris LuandGitHub
|
a49cf11e16 |
telemetry: version distribution over time (#10625)
* telemetry: keep the reported version in the daily history The version only ever lived on the instance record, which holds a cluster's latest report, so there was no way to ask what anything ran last Tuesday. Record it per sample, and let the daily axis carry strings as well as counts. State written before this has no version on its samples. The newest sample is the report the instance record itself came from, so fill that one in on load rather than starting a version series a day late. * telemetry: serve the fleet's version make-up over time /api/versions gives how many clusters ran each release per day, on the same axis and hold-forward rule as the cluster sizes. Releases are ordered by number rather than by size: the caller stacks them, and a stack whose order changes with the counts is unreadable over time. The tail past the limit is summed into "other" so the stack still adds up. Days with no version are dropped before the axis is built, so the series spans the days it knows a version for instead of climbing out of blanks. * telemetry: draw version distribution as a stacked growth chart The pie only ever showed today. Stacked over 30 days the height is the confirmed fleet and each band is a release, so one chart carries the growth and the rollouts at once. Newest release on the floor, so the band being read is anchored to the axis instead of riding on everything below it. Eight fixed hues instead of the evenly spaced ones the cluster stacks use: evenly spaced put a green and a cyan close enough to be hard to tell apart, which matters for a set you read rather than a wall of anonymous ids. Versions past the eighth fold into "other", and each band carries its own number so the chart reads without matching colours against the legend. |
||
|
Chris LuandGitHub
|
b46946ece5 |
filer: list directories without decoding chunk lists (#10616)
* filer: decode a listed entry without building its chunk list
A readdir reads attributes and never looks at chunks, but decoding an
entry builds the whole chunk list first: four allocations per chunk, all
of it thrown away. On a directory of ordinary 4MB-chunked files that is
most of what listing costs.
DecodeAttributesOnly walks the wire format and hands everything except
the chunks to the generated unmarshaller, so new fields in filer.proto
need no attention here. The chunks are still measured, because the S3
copy and multipart paths deliberately store a zero FileSize and let the
chunk extents define the size, but nothing is allocated to do it.
The blob is only re-encoded once a chunk is actually seen, so an entry
without any -- every directory, for one -- is unmarshalled where it lies
and pays nothing for the walk.
Listings opt in through the context, the way the lazy remote paths
already do; a store that ignores it stays correct.
chunks full attrs-only allocs
0 312.8n 310.1n ~ 1 -> 1
1 686.1n 411.1n -40.07% 7 -> 1
4 1.742u 667.4n -61.69% 24 -> 1
16 5.770u 1.544u -73.25% 86 -> 1
64 25.23u 6.004u -76.20% 328 -> 1
* mount: list directories with chunk lists omitted
The two meta cache listings behind a readdir are the only callers, and
neither reads a chunk. On 200k single-chunk files one enumeration goes
from 364ms to 277ms and drops a million allocations.
The read-through listing still fetches whole entries from the filer,
which would need the request to say it wants attributes only.
* mount: give the readdir benchmark's entries a chunk
Chunkless entries made the decode look far cheaper than it is, which is
the part of a listing worth measuring.
* filer: let a listing ask for entries without their chunk lists
The read-through readdir fetches whole entries over gRPC, and for a wide
directory the chunk lists are most of what crosses the wire and most of
what the client then unmarshals. A 4MB-chunked file is 113 bytes of
entry against 46 without its chunk.
ListEntriesRequest gains omit_chunks. The size a client needs is already
in the attributes, where the store decode folded the chunk extents in,
so dropping the list costs the client nothing.
The filer still reads the entries whole. A listing is where a TTL-expired
entry gets collected and deleted, and deleting one needs its chunks to
find the data, so omitting them there would leak. Only the response is
trimmed.
The hint moves to filer_pb so one context flag serves both transports:
the gRPC request sets omit_chunks, and a listing served from the local
store skips building the chunks. Cache population is unaffected either
way, since EnsureVisited starts from its own context.
* filer: reject a chunk the full decoder would reject
The walk skipped a chunk's bytes without looking inside them, so a
FileChunk carrying a corrupt nested fid, or a string that is not valid
UTF-8, sailed past the listing decoder while every other read of the same
entry still failed. The file listed with a plausible size and then gave
EIO on open, and corruption that used to fail the listing loudly was
hidden instead.
The chunk bytes are the one part of the blob the generated unmarshaller
never sees, so the two checks it would have made are made here: a
submessage has to parse, and a proto3 string has to be valid UTF-8.
FileChunk's only submessages are FileIds of scalars, so walking them is a
complete check. A descriptor-driven test fails if FileChunk ever gains a
field of either kind that the walk does not know to check, which is the
part that keeps this honest as filer.proto grows.
Taking the scratch buffer lazily, only once a chunk is actually dropped,
also takes the pool out of the path for entries that have none. Those
were measurably slower than the full decoder before; they are now level
with it. Each chunk's length prefix is parsed once rather than twice.
chunks full attrs-only vs base
0 171.4n 176.9n ~ (p=0.670)
1 366.6n 259.4n -29.24%
4 1.034u 500.2n -51.60%
16 3.905u 1.464u -62.52%
64 13.48u 5.195u -61.46%
* filer: carry the size before dropping chunks over the wire
Dropping the chunk list assumed every store folds the chunk extents into
FileSize when it decodes. A store that keeps entries as JSON rather than
as an encoded Entry never re-derives it, so an object written with a zero
FileSize kept its real size only in the chunks, and stripping them left
the client reading the file as empty. Stamp the size into the attributes
first, which costs nothing and does not depend on how the store loaded
the entry.
* mount: test that the readdir context reaches the store decode
Everything else exercises the decoder directly, so a refactor that
stopped threading the context would have reverted the whole thing with
every test still passing.
The benchmark's chunks also carried a constant legacy FileId, which
BeforeEntrySerialization reparses over Fid on the way in, so all 200k
entries stored one byte-identical chunk rather than the varying fixture
it looked like.
|
||
|
Chris LuandGitHub
|
af7cf6ab8a |
chore(weed/topology): drop the unused DataNode volume id listing (#10618)
GetVolumeIds ranged over a slice and collected the loop indices, so it reported 0-99 rather than the node's volume ids. Nothing calls it: the disk-level GetVolumeIds, which ranges over a map and is correct, is what ToDiskInfo and ToMap use. Its private getVolumes helper went with it, having no other caller. |
||
|
Chris LuandGitHub
|
cce3bab0e2 |
perf(weed/topology): gather a node's volumes into one slice (#10617)
* perf(weed/topology): preallocate the node's volume concatenation A node's volumes are gathered per disk and concatenated into a slice grown from nil, so a server with several disks reallocates and copies its way up. The writable-volume refresh loop does this for every node every few seconds. BenchmarkDataNodeGetVolumes/8Disks 322551844 B/op -> 121602326 B/op * perf(weed/topology): fill one slice across a node's disks Each disk built its own right-sized copy of its volumes, and the node then copied all of them again into the combined slice. Appending into the caller's slice makes it one allocation whatever the disk count, which halves even the single-disk case. BenchmarkDataNodeGetVolumes 1Disks 121602326 B/op 2 allocs/op -> 60801314 B/op 1 allocs/op 8Disks 121602326 B/op 9 allocs/op -> 60801024 B/op 1 allocs/op |
||
|
Chris LuandGitHub
|
228e850da1 |
perf(weed/topology): preallocate the client-facing topology snapshots (#10615)
* perf(weed/topology): preallocate the /dir/status volume list
ToVolumeMap boxes every volume on a node into an []interface{} grown from nil,
so the slice reallocates its way up alongside the boxing. The count is known.
* perf(weed/topology): preallocate the volume id list sent to clients
Every filer, s3 gateway, and mount that connects to the master gets one
VolumeLocation per data node carrying that node's whole volume id list, grown
from nil. The count is known.
The ec ids are left alone: shards of one volume can span disks, so the shard
count is an upper bound on the deduped vid count, not the count itself.
|
||
|
Chris LuandGitHub
|
9f1e21e73f |
perf(weed/topology): preallocate the per-disk VolumeList payload (#10614)
ToDiskInfo builds a protobuf message per volume and per ec shard on the disk, growing both lists from nil. Every VolumeList call runs it for every disk in the cluster, and the admin dashboard, the plugin worker, several shell commands and the s3 gateway's per-minute bucket metrics all call VolumeList. Both counts are already in hand. ToTopologyInfo over 550k volumes 202.2 MB -> 184.6 MB |
||
|
Chris LuandGitHub
|
0cfca436f1 |
perf(weed/topology): size the new-volume list from the actual delta (#10613)
A reconnecting volume server reports every volume it has as new, so newVolumes grew from nil to one entry per volume, reallocating and copying its way there. Sizing it to len(actualVolumes) instead would allocate the whole list on every steady-state heartbeat, where nothing is new. After the deletion pass everything left on the node is also in this heartbeat, so the difference is exactly what the node is about to gain: all of them on a reconnect, none in steady state. First registration of 550k volumes 1041.7 MB -> 667.4 MB |
||
|
Chris LuandGitHub
|
8aa57bef78 |
mount: stop churning the inode table on every readdir (#10606)
* mount: readdir enters a child in the inode table only when it takes a reference Only readdirplus into the kernel takes a reference on the children it reports, and only that reference brings a FORGET later to take the entry back out. Every other listing was inserting all its children anyway. On WinFsp that meant a listing looked each child up, took a reference, and immediately gave it back, so a walk of a wide directory paid three write-lock acquisitions per entry to leave the table exactly as it found it. On a plain kernel readdir nothing gives the entry back at all, so listing a directory of 200k files grew both maps by 200k entries that were never reclaimed. A dirent's inode number is informational either way: the kernel must LOOKUP before it can use a nodeid, and the WinFsp adapter re-resolves every operation by path. So report the number and let the mapping be built when something actually looks the entry up. * mount: take the readdirplus reference without a second full lookup The entry has just been resolved a few lines above, so redoing the whole lookup only rebuilds the child path and walks both maps again to reach a counter. Bump it directly, falling back to the full lookup if a Forget removed the entry in between. * mount: benchmark a readdir over a 200k directory Drives doReadDirectory against a meta cache holding 200k entries, one round of 4096 at a time, for the three front ends that behave differently: a plain kernel readdir, kernel readdirplus, and a WinFsp listing that gets attributes but never returns a reference. Reports what each leaves behind in the inode table alongside the usual metrics. The sink declares TakesLookupRef as an ordinary method rather than through the interface, so the same file runs unchanged against an older tree for comparison. * mount: stamp an inode on the benchmark's entries The filer stores one on every entry it writes, so a real listing arrives with an inode and never derives its own. Leaving it zero made every child in the benchmark fall through to the MD5 in AsInode, work no filer-backed mount does, and charged it to both sides of the comparison. |
||
|
Chris LuandGitHub
|
ee54fd6c08 |
perf(weed/storage/super_block): intern the byte-encoded replica placements (#10610)
NewReplicaPlacementFromByte formatted the byte with fmt.Sprintf and parsed the result back, allocating a string and a ReplicaPlacement every call. The master calls it once per volume in every heartbeat, and keeps the pointer for the lifetime of the volume, so a cluster with 1.6M volume replicas carries 1.6M of these where a handful of distinct values exist. The table is a flat pointer-free array, so it costs 6KB of static data and no heap objects however few placements a cluster actually uses. A byte only ever decodes to a valid placement, so the table is complete and the error return stays nil. BenchmarkSyncDataNodeRegistration/100000Volumes 500601 allocs/op -> 300589 allocs/op |
||
|
Chris LuandGitHub
|
33c36fc7a3 |
perf(weed/storage/needle): intern the stored ttl values (#10611)
The master decodes a TTL per volume in every heartbeat and keeps it for the volume's lifetime, so a cluster using TTLs carries one two-byte object per volume replica where at most 256 counts times 7 units exist. Share them, and decode the uint32 form directly instead of staging it through a byte slice. Clusters that set no TTL are unaffected; that path already returned the shared EMPTY_TTL. BenchmarkSyncDataNodeRegistration/100000Volumes, volumes carrying a ttl 600600 allocs/op -> 500597 allocs/op |
||
|
4f0322af86 |
perf(weed/topology): log writable-state changes, not every check (#10612)
* perf(weed/topology): log writable-state changes, not every check ensureCorrectWritables ran its three diagnostics whenever it was asked, so a volume that had always been read-only re-announced that on every registration. A volume server reconnecting with 550k read-only volumes made the master format over a million log lines before it could serve anything, which is exactly when it is already at its memory peak rebuilding the topology. removeFromWritable already reports the transition, and only when there is one. Explain it only then. Dropped the separate 'remove from writable' line, which said nothing that 'becomes unwritable' does not. BenchmarkRegisterReadOnlyVolumes, 100k volumes 285791480 B/op 1902069 allocs/op -> 234592088 B/op 1302572 allocs/op * Update weed/topology/volume_layout.go Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> --------- Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> |
||
|
Chris LuandGitHub
|
1d8d9570eb |
perf(weed/topology): preallocate the disk volume snapshot (#10609)
Disk.GetVolumes copies the disk's whole volume map into a fresh slice, and every caller that walks a node's volumes goes through it: the writable-volume refresh loop every few seconds, ToTopologyInfo on each VolumeList, telemetry, and node unregistration. Growing from nil reallocates and copies about twice the final size each time, which at 100k volumes per disk is 60MB of garbage per call. BenchmarkSyncDataNodeRegistration/100000Volumes 199670102 B/op -> 137728051 B/op |
||
|
Chris LuandGitHub
|
2ec899bdee |
perf(weed/topology): diff a heartbeat without copying the volume map (#10608)
* perf(weed/topology): keep only volume ids in the heartbeat membership set The map is used solely to test whether a known volume is still present, but it copied the whole 152-byte VolumeInfo for every volume in the heartbeat. Presize it too, since the count is known. BenchmarkSyncDataNodeRegistration/100000Volumes 199670102 B/op -> 180157310 B/op * perf(weed/topology): diff a heartbeat without copying the volume map To find volumes the data node no longer reports, UpdateVolumes copied every VolumeInfo on the node into a fresh slice, then deleted the missing ones one at a time. At 100k volumes that is a 15MB copy per heartbeat to usually find nothing. Scan the disk maps in place instead and return only what was removed. BenchmarkSyncDataNodeRegistration/100000Volumes 180157310 B/op -> 87806436 B/op |
||
|
Chris LuandGitHub
|
3fce1a938d |
perf(weed/topology): preallocate the heartbeat volume conversion slice (#10607)
* test(weed/topology): benchmark the per-heartbeat volume sync A volume server re-sends its entire volume list every VolumePulsePeriod, so SyncDataNodeRegistration is the master's steady-state per-server cost. Give it a benchmark so allocation regressions show up. * perf(weed/topology): preallocate the heartbeat volume conversion slice The slice grows to one entry per volume on the data node, so at 100k volumes the doubling copies allocate 60MB of garbage per heartbeat. The final length is known up front. BenchmarkSyncDataNodeRegistration/100000Volumes 199670102 B/op -> 137725872 B/op |
||
 Lars LehtonenandGitHub
|
2ff3dda7cd | chore(weed/s3api/policy_engine): prune dead code (#10599) | ||
|
Chris LuandGitHub
|
fb92d46e2d |
helm: enterprise license Secret, and a persistent-claim option for master data (#10601)
* helm: mount an enterprise license Secret into every component
Running the enterprise image under this chart meant hand-rolling
extraVolumes and extraVolumeMounts on every component. Missing one is
easy and quiet: a component without the license silently drops to
community mode, and on the admin that surfaces only as Data Recovery and
Point-in-Time Recovery refusing to enable, with the master looking fine.
Add global.seaweedfs.license.existingSecret. The Secret is mounted
read-only into master, volume, filer, s3, sftp, admin, worker and
all-in-one, and SEAWEED_LICENSE points every process at the file rather
than relying on the binary's search paths, which depend on the working
directory.
The mount is a directory, never a subPath: kubelet refreshes Secret
contents in place, but a subPath is resolved once at container start and
never updates, which would break license renewal. There is deliberately
no checksum annotation on the pod template either — that would roll every
pod on renewal, the opposite of what is wanted. Verified on kind: the
renewed file reached a running master ~70s after the Secret was patched,
same pod UID, restartCount 0.
Also documents that only the master re-reads the license on a timer
today; the other components pick a renewal up on their next restart.
* helm: keep master data on a claim by default
The master's -mdir holds its Raft log and snapshots, and with them the
cluster's topology UUID — the identity an enterprise license is issued
against. It defaulted to a hostPath under /ssd, which does not follow a
rescheduled pod: the master came back with an empty data directory, a
freshly generated cluster UUID, and a license that no longer matched.
With the chart's default of a single master replica there is no peer to
recover the identity from either.
Default master.data.type to persistentVolumeClaim, sized 1Gi (Raft state
is small). hostPath stays available for anyone who wants it.
This is breaking for existing releases: volumeClaimTemplates is immutable,
so helm upgrade on a release installed with the old default fails with
"updates to statefulset spec for fields other than ... are forbidden".
Verified on kind, along with both ways out — pinning
master.data.type=hostPath upgrades cleanly, and the documented migration
(stop the master, pre-seed a claim named after the StatefulSet, upgrade)
preserves the cluster UUID. Seeding has to happen while the master is
stopped; copying into a live pod loses the state, since the running
master rewrites its Raft files before the restart.
* helm: mount the license on masters only
The master is what reads the license file: it validates it, enforces the
capacity limit and binds it to the cluster UUID. Mounting the Secret on
volume, filer, s3, sftp, admin and worker put it in six more containers
that never look at it, so drop it there and keep master plus all-in-one,
which runs `weed server -master`.
Two fixes from review while here:
- project only the configured key out of the Secret, so an unrelated
key in the same Secret is not exposed to the container. Verified the
key-scoped projection still updates in place: patching the Secret
reached the running master in ~50s, same pod UID, restartCount 0.
- drop SEAWEED_LICENSE from merged extraEnvironmentVars while a
license Secret is configured. It used to be possible to render the
key twice in one container, with the user's value winning over the
path the chart actually mounts.
CI now pins the scope (master only, all-in-one separately), the
key-scoped projection, readOnly, and that SEAWEED_LICENSE renders once.
* helm: fix the documented master-data migration
The seed pod in the migration never mounted the claim it was supposed to
seed, so following the steps verbatim copied the Raft state onto the
pod's ephemeral filesystem and threw it away with the pod — landing the
reader in exactly the empty-claim, new-cluster-UUID state the section
exists to avoid. Give the pod the volume.
The names were assembled as <release>-seaweedfs-*, which is wrong
whenever the release name already contains the chart name or an override
is set; read the StatefulSet name from the cluster instead and derive the
claim from it. Also scope the procedure to the chart's single-master
default, and create the Secret in the release namespace.
Trims the enterprise prose this section had accumulated: this is the OSS
chart, and the master-data default is a durability fix that stands on its
own.
* helm: quote the projected license key, reserve it on the secret env path
A secretKey that YAML reads as a non-string (123, yes, no) rendered
unquoted into the volume's items, so the projection would not match the
Secret's key. Quote both key and path.
all-in-one renders secretExtraEnvironmentVars itself, outside the merge
helper that already drops SEAWEED_LICENSE, so an entry there could still
render the variable twice. Skip it there too while a license Secret is
configured. The master template has no such block, so this is the only
remaining path.
* helm: correct the license helper comments after scoping to masters
* helm: keep hostPath as the master data default
Defaulting master.data.type to a claim broke every existing release:
volumeClaimTemplates is immutable on a StatefulSet, so helm upgrade
failed with "updates to statefulset spec for fields other than ... are
forbidden" before it changed anything.
Keep hostPath as the default and document the claim as the option to
choose — for a new install, or for an existing one via the migration
already in the README. The chart supported both types all along; only
the default moves back.
Every immutable field of every rendered StatefulSet is now identical to
upstream under default values, so an in-place upgrade cannot trip the
API. Verified on kind: install with the unmodified upstream chart,
upgrade to this branch (ok), upgrade again turning the license Secret on
(ok, volume added in place). A fresh install with
master.data.type=persistentVolumeClaim binds its claim as before.
The whole PR is additive now: nothing renders differently until a value
is set.
* helm: scope the migration's StatefulSet lookup to the release
* helm: trim the comments added by this change
|
||
|
Chris LuandGitHub
|
a8c8372b99 |
rust: stop quarantining v2 volumes, load a disk's volumes concurrently (#10602)
* rust: only compare the .dat tail on v3 volumes Go's verifyNeedleIntegrity does the "does .dat end exactly at the last indexed needle" comparison inside its v3 branch -- it rides along with the v3 append-timestamp read -- so a v1/v2 volume carrying an unindexed trailing record loads read-write and silent. The Rust check ran it at every version, so booting the Rust server on a legacy cluster warned on and quarantined volumes the Go server had been serving happily. * rust: load a disk's volumes concurrently Opening a volume is dominated by reading its .idx into the needle map, and the loader did them one at a time, so a disk holding thousands of volumes needed thousands of serial index reads before the server came up. Go's concurrentLoadingVolumes spreads the same work over max(cores, 10) workers; do the same, keeping the directory pre-pass and the insert serial so only the open is parallel. * rust: let a failed volume open fall back to the next candidate Two collections can name the same volume id on one disk. Deduping the load queue by id claimed the id for whichever candidate the scan saw first, so a corrupt one shadowed a good one behind it; the serial loader this replaced only claimed an id once a volume had actually opened. Carry every claiming collection per id and try them in scan order until one loads. * rust: trim the new comments in the volume loader |
||
|
Chris LuandGitHub
|
0cf62a921a |
admin: dashboard counts chunks, not files (#10598)
* admin: count each chunk once in the dashboard total The dashboard summed file_count from every node's volume list, so a chunk was counted once per replica and deleted chunks were never subtracted. Reuse the collection aggregation, which dedupes replicas and EC shard holders and nets out tombstones. * admin: the dashboard card counts chunks, so name it that Volumes store chunks, and a file is split into one or more of them, so the 'Total Files' card always read far higher than the number of files in the filer. Rename it to 'Total Chunks' and say so in the tooltip. * admin: collections pages count chunks once and say so The collections list and detail pages summed file_count straight off the topology, so replicas multiplied the count, tombstones stayed in it, and the detail page ignored EC volumes entirely. Take the numbers from the shared collection aggregation and label them chunks. * admin: dedupe replica chunk counts per volume instead of dividing Dividing each replica's live count by the copy count truncated a chunk per odd-sized volume, and reported half the count while a volume's second replica had not checked in yet. Replicas mirror each other's needles and deletes, so keep the fullest report per volume id. * admin: fix the collections CSV export column mapping The exporter read chunks from the EC-volume cell and shifted size and disk types with it. Read every column the table actually has. |
||
 Andrei StroianandGitHub
|
aa04889f05 |
fix(helm): mark filer db-init ConfigMap volume optional (#10596)
The filer StatefulSet declares db-schema-config-volume unconditionally, referencing <release>-seaweedfs-db-init-config. The chart never renders that ConfigMap and the README documents it as operator-supplied, so any deployment that has not pre-created it ends up with a pod spec pointing at a non-existent object. No container mounts the volume, so this is currently inert, but it is misleading and would become a pod-start failure if a volumeMount is ever added. |
||
 github-actions[bot]
|
de34a1a87c | 4.41 4.41 |