* admin: make paths relative
* admin: make filer browser link and nav path checks prefix-relative
* admin: add isCurrentPath and currentPathStartsWith helpers
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* benchmark tool for mounted filesystems
* ci: on-demand mount benchmark, native WinFsp vs rclone plus a Linux reference
* windows mount: let the Windows cache manager cache file data
WinFsp only turns the cache manager on for a file when FileInfoTimeout
is infinite; at any finite value every application read and write is a
synchronous trip into the mount process at whatever size the application
issued. Metadata events already reach FspFileSystemNotify, which purges
a changed file's cached pages and attributes, so an infinite timeout
stays coherent. The dir listing, volume info and EA timeouts are pinned
to one second so they do not silently inherit the infinity.
* windows mount: cache resolved paths and attributes in the adapter
WinFsp addresses every operation by path and has no FORGET, so the
adapter walked the whole path through Lookup on each one, and in a
directory the filer has not listed yet every walk was a filer round
trip; nothing played the part of the kernel's dentry and attribute
caches. The path cache owns one lookup reference per entry the way the
kernel holds one until FORGET, serves attribute reads for files without
an open handle, and is purged by the mount's own mutations and by
metadata events, with the timeout as backstop.
* windows mount: keep a closed file's attributes cached
Open steals the path's cache entry for its handle and Release returned
the reference with a purge, so the stat that follows every copied file
walked to the filer again. Reading the handle's final attributes before
it goes away and moving the reference back into the cache serves that
stat locally, the way the kernel's attribute cache does after a close.
Only if the path still names that inode, though: WinFsp reports the
path the handle opened with, and after a delete-on-close or a rename
caching it would resurrect an entry that is gone.
* windows mount: persist entries at create, and let the flush stay at close
WinFsp posts the cleanup and close that carry the flush after
CloseHandle has returned, so deferring the filer entry to the flush let
everything that reads through the filer race an unflushed close: a
listing missed just-written files, and a directory rename moved a
directory on the filer before its newest child existed there, leaving
the straggler flush to recreate the child under the dead path.
Flush-at-cleanup is not the answer either: it makes every handle's
cleanup flush, and those flushes race the unlinks of delete-on-close,
re-inserting the entry the unlink just removed. Persisting the entry at
create takes the ordering question away.
* mount: flush written pages before a truncate shrinks past them
The shrink trims chunks, but written pages that have not become chunks
yet are invisible to it, so the next flush wrote them back and the file
grew again, resurrecting the truncated bytes. Windows hits this on
every write-then-shrink because its flush runs after CloseHandle, but
the gap is platform-neutral.
* mount: order a file's unlink against its in-flight flush
Unlink set the handle's deleted flag bare, so a flush already past its
own check of that flag wrote the entry back right after the delete
removed it, and a delete-on-close file outlived its last handle. The
flag is now set under the handle's flush lock and re-checked under it,
so a flush either completes before the delete or sees the flag and
skips. An eagerly created handle also starts clean: the dirty mark
existed to make the deferred filer create happen at flush, and eager
creates have nothing to flush.
* shell: volume.move restores source writability when aborted after the copy phase
* shell: volume.move removes the incomplete target copy when aborted before the source delete
* shell: give each abort cleanup RPC its own timeout
* filer: TUS creation accepts Upload-Concat partial uploads
* filer: TUS final uploads concatenate completed partials
* filer: TUS concatenation tests
* filer: consumed marker pins TUS chunk ownership on completion
* filer: TUS session delete decides chunk ownership after removing the session info
* filer: TUS completion persists the consumed marker before creating the entry
* filer: TUS completion re-verifies the session after persisting the consumed marker
* filer: serialize TUS session ownership transitions per filer
* filer: surface failed TUS consumed-marker rollbacks
* s3: a null object wins over a rescan when the latest-version pointer is absent
The read path already resolves an absent pointer this way; the listing-path
counterpart scanned .versions/ first and could surface an old version or
delete marker over the current suspended-versioning null object.
* s3: dedup a key against its .versions sibling in suspended buckets too
A suspended bucket keeps its .versions directories, so a suspended-versioning
null object and its .versions sibling emitted the same key twice.
* s3: retract a null object from the listing when a delete marker shadows it
Deleting a key whose null version predates versioning leaves the base-path
entry in place and records the delete marker under <key>.versions. The
listing appended the base-path entry and relied on the .versions sibling to
replace it, but a delete-marker current version emitted nothing, so the
deleted key stayed visible to ListObjects while GET and HEAD returned 404.
* s3: keep a key's .versions sibling on the same page as the key
When the page quota ran out between a base-path entry and its .versions
directory, the page ended with the stale entry and the next page skipped the
directory as a marker echo, so the replacement or retraction never happened.
* s3: the null version is not latest when the .versions pointer names a newer one
ListObjectVersions stamped IsLatest on every base-path null object, so a key
deleted after enabling versioning reported IsLatest on both the delete marker
and the null version.
* s3: test listing after a pre-versioning null object is delete-marked
* s3: find a key's earlier page entry by scan, not by adjacency
A key such as k.bak sorts between k and k.versions, so the entry a .versions
sibling replaces or retracts is not always the last one on the page. Scan
back through the page for the key, and insert a late resolution in sorted
position instead of at the end.
* s3: settle trailing null objects by lookup when a page fills
The quota can run out while keys still sit between a null object and its
.versions sibling, and the sibling-adjacent page-boundary exception never
fires for those. Track the trailing null objects whose sibling has not been
ruled out and look each one up before declaring the page full; a retraction
reopens the quota.
* s3: do not resolve a .versions sibling its page has already moved past
A page resuming from a marker inside the base key's extension region has
already listed and settled the base null object on an earlier page, so
resolving the .versions directory again re-emitted the key.
* s3: test listing with keys between a null object and its .versions sibling
* s3: pick the newer of the null object and the scanned versions
Making the null object win outright whenever the pointer is absent misread
multi-filer pointer lag: version files replicate ahead of the pointer, and a
key overwritten or delete-marked after pre-versioning days would list its
stale null again. The suspended-versioning write that legitimately makes the
null current is also the newer entry, so mtime tells the two apart.
* s3: a delete-marked null object no longer keeps its prefix alive
The hidden-entries probe took any plain file as proof of a listable key, but
a null object shadowed by its .versions sibling's delete marker is not one.
Hold plain files pending until the sibling settles them either way.
* s3: settle an evicted pending null instead of dropping it
Nested keys like k, k!, k!! can hold more pending nulls than the cap. A
silently evicted one could close the page unsettled, and the resume skip
would then keep the stale entry for good.
* s3: test deleted-prefix hiding and the pending-null cap
* s3: cover the reported '!' intervening key with a live version
* s3: an unstamped same-second version outranks the null object
Second-resolution mtimes cannot order same-second writes, so the tie went to
the stale null when the pointer lagged. The suspended write that makes a null
current stamps the version it displaces before clearing the pointer, so the
stamp is the authoritative signal and a tie without it goes to the version.
* s3: a pointer-less versions listing still checks what replicated
ListObjectVersions took a missing pointer as proof the null object is latest,
but under pointer lag the sibling can already hold newer replicated versions
or markers. Apply the same nullObjectWins rule as the listing recovery.
* s3: a failed null-object settlement fails the listing
Every getEntry error read as a missing sibling, so a transient filer error at
a page boundary committed the unsettled null and the next page skipped its
sibling for good. Only a definitive not-found means the null is live; other
failures are retained on eviction and fail the request at page close.
* s3: retract a CommonPrefix whose only backers were delete-marked nulls
The directory probe settles this for the / delimiter, but any other delimiter
derives prefixes from base-path keys directly, and a prefix built solely from
null objects survived their delete markers. Count the unsettled null backers
behind the newest prefix and retract it when the last one settles as a marker;
a live resolution or any listable contributor confirms the prefix instead.
* s3: test custom-delimiter prefix retraction
* s3: an explicit signal marks the null object current, not the demotion stamp
The NoncurrentSinceNs stamp survives promotion: delete the version that
demoted another and the promoted one is current yet still stamped, so a
lagging replica would resurrect the stale null. A suspended-versioning write
now records Seaweed-X-Amz-Null-Version-Is-Latest on the .versions directory
when it clears the pointer, every pointer update removes it, and the
recovery paths trust the signal instead of the stamp.
* s3: a filer failover retry rebuilds the listing page from scratch
The failover wrapper reruns the callback on another filer after a transport
error, and the partially built page, spent quota, and advanced marker leaked
into the retry, which could then return a stale or duplicated page as
success.
* s3: only a prefix's own backers can debit it
A delete marker for a version-only key (no base object) derived the same
prefix as its neighbors and decremented backing it never contributed,
retracting a prefix that a live null object still backed. Track backers by
key so settlement is idempotent and only debits what was counted.
* s3: test a version-only marker against a null-backed prefix
* s3: a pointer recompute clears the null-current signal
The routed finalize for delete markers, COPY, and multipart rewrites the
.versions pointer through RECOMPUTE_LATEST, which left a suspended-era
null-current signal in place. Version files never carry the signal, so
mapping it in CopyExtended deletes it whenever the pointer recomputes.
* s3: the pointer outranks the null-current signal in the versions listing
The signal check guarded the pointer check, so a stale signal a recompute
had not cleared yet would have let the null claim IsLatest alongside the
pointed-at version.
* topology: name a node's volumes without copying them
ToVolumeLocations reads a volume id off every volume in the cluster, and got
there through GetVolumes, which copies a whole storage.VolumeInfo per volume to
be read for four bytes of it. Every client that connects asks for this.
At 800k volumes the walk goes from 94.6MB to 16.0MB, which is the ids
themselves.
* master: log why a client send failed, not what was sent
The message names every volume on a newly connected node, so a client going
away had the master format a protobuf that size into text -- through the one
log level that is always on. The error is the part worth having.
Go stores a map value inline once it fits in 128 bytes, and indirectly above
that. Shrinking storage.VolumeInfo to 120 bytes therefore moved 1.6M of them
into the map's own slots, which are allocated to capacity rather than to
occupancy, and the map grew by 149MB even though each volume got smaller.
Hold pointers, so the size of a volume record stops deciding how it is stored.
Updates are written through the pointer already there, so a heartbeat that
changes a volume allocates nothing, as it did when the struct was above the
threshold and Go was doing this itself.
At 800k volume ids across 3 servers the resident topology goes from 705.3MB to
546.6MB, with heartbeat cost unchanged at 111.6MB.
* master: stream volume listings
A listing of 800k volumes is 36MB on the wire but 305MB as messages, and the
master built all of it, then held it while grpc encoded it. Two of those at
once is most of a small master's heap, and the maintenance scanner asks every
30 minutes.
The topology goes out first, listing nothing, then its volumes in batches, so
the master holds a batch rather than a cluster: 341MB of live heap for one
listing becomes 4.4MB. It allocates much the same either way -- what changes is
how much of it has to be live at once, which is what sets the heap ceiling.
Batches are built under their disk's lock and sent outside it, so a slow reader
stalls the stream rather than the topology. They therefore do not share one
instant, which a single listing did not either: it takes each disk's lock in
turn, so a volume moving during either can be seen twice or not at all.
The client helper hides which kind of master answered: one too old for the
stream is asked the old way and its reply cut into the same batches. Either way
the topology handed over lists no volumes, so a caller cannot come to depend on
finding them there.
* admin: stream the listing the maintenance scan reads
It asks for every volume in the cluster every 30 minutes. Reassembling it
client-side keeps the scan identical -- ActiveTopology splits disks by the
disk ids on the volumes, so it needs them in the topology -- while the master
no longer builds the whole reply to send it.
* topology: report a disk id that does not depend on map order
A topology disk that fronts several physical disks took its reported id from
whichever volume the map yielded first, so two listings of an unchanged disk
could disagree. Take the smallest instead.
* topology: test that a streamed listing rebuilds to the whole one
The callers that stream now rebuild the listing from a topology sent without
volumes plus the batches after it, so that has to come out the same as being
sent it whole, at every batch size and under a filter.
* clients: stream the volume listings that ask for everything
The dashboard's list and export pages, the collection and ec shard pages, the
topology view, the worker metrics and two shell commands each asked the master
to build all 800k volumes into one reply. They read the same listing as before,
rebuilt on their side, so the master no longer holds it.
The three that already ask for one volume or one collection stay as they are:
their replies are small, and streaming one costs a round trip to say so.
* master: stream volume listings
A listing of 800k volumes is 36MB on the wire but 305MB as messages, and the
master built all of it, then held it while grpc encoded it. Two of those at
once is most of a small master's heap, and the maintenance scanner asks every
30 minutes.
The topology goes out first, listing nothing, then its volumes in batches, so
the master holds a batch rather than a cluster: 341MB of live heap for one
listing becomes 4.4MB. It allocates much the same either way -- what changes is
how much of it has to be live at once, which is what sets the heap ceiling.
Batches are built under their disk's lock and sent outside it, so a slow reader
stalls the stream rather than the topology. They therefore do not share one
instant, which a single listing did not either: it takes each disk's lock in
turn, so a volume moving during either can be seen twice or not at all.
The client helper hides which kind of master answered: one too old for the
stream is asked the old way and its reply cut into the same batches. Either way
the topology handed over lists no volumes, so a caller cannot come to depend on
finding them there.
* admin: stream the listing the maintenance scan reads
It asks for every volume in the cluster every 30 minutes. Reassembling it
client-side keeps the scan identical -- ActiveTopology splits disks by the
disk ids on the volumes, so it needs them in the topology -- while the master
no longer builds the whole reply to send it.
Volume growth registers a provisional record before it can know which
directory the server chose, while the server's own report -- pushed
during the AllocateVolume RPC -- carries the real disk id. The merge is
last-writer-wins, so whichever lands second sticks, and fresh volumes
nondeterministically show disk 0 on multi-dir servers. Keep the reported
disk id when the provisional update carries none, before the report
digest is computed so the stored record stays consistent with what the
server keeps reporting.
Claude-Session: https://claude.ai/code/session_01QdTEEPbg4MtcoEGwqbgtZC
The volume server names the directory index in every
VolumeShortInformationMessage, but NewVolumeInfoFromShort dropped it, so
volumes registered through the incremental new-volume path showed
disk_id 0 at the master until a full report -- misreporting multi-dir
servers in volume.list and the per-physical-disk topology views.
Claude-Session: https://claude.ai/code/session_01QdTEEPbg4MtcoEGwqbgtZC
* volume: decode IPv6 transition addresses in the remote-endpoint guard
checkBlockedIP normalized only ::ffff: mapped IPv4, so NAT64 (64:ff9b::/96),
6to4 (2002::/16), Teredo (2001:0000::/32), and IPv4-compatible (::/96) addresses
that embed an internal IPv4 (loopback, 169.254.169.254, RFC 1918) passed the
endpoint guard even though the plain IPv4 forms are refused. Extract the
embedded IPv4 from those forms and re-check it against the deny list, which
covers both the up-front validation and the dial-time guard. Mirrored in the
Rust volume server.
* volume: require the full NAT64 well-known prefix before decoding
Only 64:ff9b::/96 carries the embedded IPv4 in the low 32 bits, so also require
bytes 4-11 to be zero before treating an address as NAT64; other 64:ff9b:
prefixes place the IPv4 elsewhere and are left untouched. Add public-target
coverage for 6to4, Teredo, and IPv4-compatible so every decoder is exercised on
both a blocked and an allowed destination. Mirrored in the Rust volume server.
* master: let VolumeList ask for the volumes it wants
The request carried nothing, so every caller was answered with the whole
cluster. A dashboard opening one volume's page, or a capacity probe adding up
one bucket, was served all 800k of them and threw away the rest -- and the
master built every one of those messages first.
The topology, its disks and their counters are still reported in full: a caller
reading free space or replica placement needs the cluster whichever volumes it
asked about. Only what is listed under a disk is selected, ec shards included.
An empty collection and a zero volume id take everything, the way volume.list
already reads its own -collectionPattern and -volumeId, so a caller that
forgets to narrow is answered too much rather than answered wrongly. That
leaves the default collection unnameable, since it is the one the empty string
names, so it gets a field of its own.
An older client sends none of it and is answered exactly as before.
* admin: ask the master for the volume the page is showing
A volume's detail page was pulling every volume in the cluster to find one and
its replicas, and discarding the rest.
* admin: ask the master for the ec volume the page is showing
Same as the volume detail page: one volume's shards were found by pulling every
ec shard in the cluster.
* s3: ask the master for the bucket's own collection
The SOSAPI capacity probe summed one collection's volumes out of a listing of
every volume in the cluster. Cluster capacity still comes out the same: it is
read from the disk counters, which a filtered listing reports in full.
* topology: read the disk usage counters atomically
They are written with atomic.AddInt64 from heartbeats but were read plainly by
the two listings and by FreeSpace, and the map they sit in was iterated without
the lock its neighbour takes. Under -race a listing concurrent with a heartbeat
trips on both.
A master decides nothing from it. Every caller that read it was asking whether
a volume is remote, which the backend name answers, and the value itself is
reported on demand by the server holding the volume, through the volume info in
ReadVolumeFileStatus.
It is also the one string here that cannot be shared: unique per volume, so
unlike the collection and backend names it carries its own characters for every
volume a master tracks.
VolumeInfo goes from 136 bytes to 120. 800k volumes registered from a heartbeat
that has been over the wire go from 214 to 163 B/volume when tiered.
The volume server's own status page keeps showing the key, now read from the
volume it holds rather than relayed through a master, which is also where the
other volume server implementation reads it.
The heartbeat digest drops it on the same grounds: a change to something the
master does not hold cannot make its copy stale. Both implementations and their
shared vectors move together, and the field-coverage test now names what is
deliberately not retained rather than being loosened.
* storage: order VolumeInfo by alignment
The struct is held for every volume replica in the cluster, so the padding the
compiler inserts is multiplied by however many volumes a master tracks. Two
one-byte fields each sat at the head of a word and left the rest of it empty,
which was ten of the eighteen wasted bytes.
Grouping by size rather than by meaning takes the struct from 152 bytes to 136,
and the map holding them shrinks with it, since a Go map's slack scales with
the size of the value.
800k volumes registered from a heartbeat that has been over the wire:
211 -> 195 B/volume, 214 -> 198 tiered.
* trim the comments on this change to the parts that are not evident
* topology: build the volume list without copying the volume map first
ToDiskInfo copied every VolumeInfo on the disk into a fresh slice, walked it to
build a protobuf message for each, and threw the copy away. The copy was as
large as the messages it produced.
Building them straight from the map holds the disk's read lock for the walk
rather than just the copy, so a heartbeat updating that disk waits for it. It
is a read lock on a call that is now infrequent, against an allocation of the
same size as the response.
ToTopologyInfo over 550k volumes 193617502 B/op -> 110011017 B/op, and faster
for not making the copy.
* trim the comments on this change to the parts that are not evident
* ec.balance: add a -volumeIds filter
Collection scope is often too broad for maintenance. -volumeIds narrows the
plan to the given ec volume ids by leaving every other volume out of the
topology handed to the planner, so no phase, dedup included, can plan against
them. Ids with no ec shard in the selected collection, dataCenter and disk type
are rejected rather than silently skipped.
* ec.encode: key the orphan sweep without narrowing the volume id
int is 32-bit on 32-bit builds, so int(vid) wraps for volume ids above
MaxInt32. Format the id as the uint32 it is.
* pb: ask the master what each collection holds
Callers tracking usage were sent every volume in the cluster to add up
themselves, which is the master's largest single allocation.
* topology: summarise what each collection holds
One pass over the topology, allocating per collection rather than per volume.
Regular volumes count once each for logical totals and once per replica for
physical, taken from the lookup index, which is already keyed by volume and so
needs no set of seen ids. Ec shards are node-local so their sizes sum, while
the file and delete counts describe the volume and resolve once every holder
has been seen.
Replicas of one volume disagree while a write is landing or a heartbeat is
late. Walking a full listing took whichever replica the map iteration reached
first, so the answer moved between runs; this takes the largest, which is
stable and never reports usage below what some replica already holds.
* s3: take bucket sizes from the master's summary
The bucket size metrics pulled the whole volume list once a minute and added it
up, which cost the master 184.6MB of allocation and 17.8MB on the wire for six
numbers per collection.
VolumeList over 550k volumes 184.6 MB allocated, 17.8 MB on the wire
CollectionStatistics 176 bytes allocated, 47 bytes on the wire
The aggregation moves to the master with it, so the cases the removed tests
covered are now asserted against it directly.
* topology: count the replica holding the most live data
Quotas are enforced on size less deletions, and the replica with the biggest
raw size can be the one that has deleted the most. Counting it reported a
bucket smaller than it is and would leave one writable over its quota, which is
the opposite of what picking the largest was meant to guarantee.
* topology: cap a volume's deletions at what it holds
Live usage is read as a collection's size less its deletions, so a volume
reporting more deleted bytes than it has cancels live bytes belonging to other
volumes in the same bucket and reports it smaller than it is. Replica selection
already floored that volume's own live size at zero; the totals have to agree
with it.
* storage: share the volume strings a cluster repeats
Decoding a heartbeat allocates a fresh string for the collection, disk type and
remote backend of every volume, and a master holding a million volumes then
holds a million copies of the same handful of names.
Not the remote storage key, which is unique per volume: interning that would
fill the table rather than share anything.
800k volumes registered from a heartbeat that has actually been over the wire:
227 -> 211 B/volume, and 238 -> 214 when the volumes are tiered, since the
backend name shares too.
* storage: hold the interned strings rather than let them be collected
unique.Make clears its entries by weak reference, and its canonical value does
not survive a collection even while a caller still holds the string it handed
back -- so a volume reported later would get a second copy of a name the rest
of the cluster already shares. With only changed volumes reported, most are
interned once and never again, so that is the common case rather than a corner.
The table therefore only grows, which is why it stays restricted to values
drawn from a small set. Ten thousand collections keep a few hundred kilobytes.
* iceberg: stamp a default name mapping on new tables
* iceberg: repair non-compliant manifests at commit
* s3tables: verify ClickHouse writes read back through PyIceberg
* iceberg: carry the manifest-list content into repaired manifests
* iceberg: refresh the default name mapping on schema evolution
* iceberg: merge historical names into the refreshed name mapping
* iceberg: never fail a commit on repair fallout
* iceberg: harden manifest repair against writer dialects
* s3tables: keep PyIceberg reader stderr out of row data
* iceberg: keep name mappings unambiguous across field id reassignment
* iceberg: align existing manifest content metadata with the list entry
* s3: add the RenameObject endpoint
PUT /{bucket}/{key}?renameObject with x-amz-rename-source moves an object
through the filer's AtomicRenameEntry, so no bytes are read or rewritten and
the ETag, tags and SSE keys travel with the entry.
Only unversioned buckets: a versioned rename would have to rebuild the
.versions chain, and AWS offers RenameObject on directory buckets, which
cannot be versioned. The source arrives in a header, so it is authorized
separately for read and delete; both keys are locked, in key order, across the
precondition checks and the move.
* s3: let a matched source ETag precondition settle its date precondition
RFC 7232 has an ETag precondition outrank the date precondition on its own
side, and AWS documents the same for CopyObject: a matching
x-amz-copy-source-if-match with a failing x-amz-copy-source-if-unmodified-since
copies rather than returning 412. The source check evaluated all four headers in
sequence, so the date header could still veto a decided ETag match.
validateConditionalHeadersForReads already applies this precedence; the source
path now matches it.
* s3: cover a rename source named as a directory without a trailing slash
Renaming a directory would move a whole subtree, so it has to stay a missing
key whether or not the caller wrote the trailing slash.
* s3: accept a bare object key as the RenameObject source
AWS spells x-amz-rename-source both ways. Its CLI, Java and Rust examples pass
the bare source key, and only a second CLI example and the boto3 conditional
example pass bucket/key; the API reference's own example is a bare key too. The
header was read as bucket/key only, so the form AWS leads with was rejected with
InvalidArgument and the endpoint was unusable as documented.
A value is now read as a literal key first — the only reading that can never
name the wrong object — and as bucket-qualified second, when it carries the
request's own bucket and the literal key does not exist. That costs one extra
lookup only for a source that starts with the bucket's own name.
Another bucket's name in the source is no longer a distinct error: RenameObject
moves within one bucket, so it is simply part of a key this bucket does not
hold, and it reports NoSuchKey.
* s3: only a proven absence picks the other reading of a rename source
A source that resolves to a directory is not a miss to fall through on: the
literal path is still what the caller named, so answering for it beats renaming
a different object under the bucket-qualified reading. With a directory at
bucket/source.txt and an object at source.txt, a rename naming the former moved
the latter.
A failed lookup is not a proof of absence either, so a blip can no longer
redirect a rename to the other reading.
* topology: keep per-node volume state with the location it describes
The read-only and oversized indexes were maps from volume id to a list of the
nodes reporting that state -- the same key space the lookup index already
holds, kept a second and third time. Nothing ever asked which nodes; both are
only ever asked whether any node does.
So the state rides on the location list as a bit per entry, and the two indexes
go. Removing a location shifts the bits with it, and a node replacing another
at the same address inherits its slot, since that is what happens to the
location too.
800k volumes, 90% read-only: readonly index 40.0MB -> 0, lookup index
42.5MB -> 48.6MB for the bits, 33.9MB net.
* topology: rebuild the location flags when stale entries are dropped
Refresh rebuilds the location list, so leaving the flags alone left bits
describing whoever moved into the dropped entries' place.
* topology: assert the refreshed flag survived, not just that it moved
Clearing the mask rather than rebuilding it would have passed: the check that
the flag lands on the right location is done by clearing it, which an already
empty mask satisfies.
* topology: track volume size only where writes can land
Size tracking decays pending assignment estimates so the master does not
overfill a volume before heartbeats catch up. Nothing is ever assigned to a
read-only volume, so an entry for one can never be consulted -- and in a tiered
cluster that is most of them, which made this the volume layout's largest cost.
A volume held out of the writable list for capacity is not read-only and keeps
its entry: that entry is what enforces the recovery delay.
800k volumes, 90% read-only: sizeTracking 79.1MB -> 8.4MB, and the crowded set
falls out with it because a read-only volume no longer reaches the threshold
check at all.
* topology: decide size tracking per volume, not per reporting replica
A volume is unwritable if any replica is read-only, so asking the replica whose
heartbeat happened to arrive made the answer depend on arrival order: a
writable replica reporting after a read-only one put the tracking back.
Ask the volume instead, which also drops the caller-supplied flag and the churn
it caused. The crowded entry goes with the tracking, since leaving it behind
would only move the memory this releases.
Costs a map lookup per replica on a full-list heartbeat, about 19ms per 100k
volumes and no allocations, on a path that is now rare.
* s3api: fix ListObjectsV2 dropping objects under a partial prefix
ListObjectsV2 with a prefix that names only part of a directory name -
"data/a" matching both "data/a/..." and "data/ab/..." - dropped objects
when paginating with a small max-keys, reporting IsTruncated=false before
all matching keys had been returned.
On resume, normalizePrefixMarker took the listing directory from the
continuation marker ("data/a/") and discarded the "a" name prefix, so once
the marker's subtree was exhausted the sibling directories that also
matched the prefix were never listed.
Resolve the listing directory and name prefix from the prefix instead of
the marker, so siblings sharing the name prefix are still listed after the
marker's subtree.
Fixes#10652
* s3api: cover a sibling-directory marker in the partial prefix tests
* s3api: resume a truncated delimited listing from the emitted CommonPrefix
Rebuilding the continuation token from the request dir plus the listing prefix
assumes that prefix names a directory. With a partial name prefix like "data/a"
it produced "data/a/ab/" for the sibling prefix "data/ab/", so the next page
resumed in the wrong subtree and repeated it forever.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* heartbeat: name departed volumes in delta heartbeats
* master: release the lookup index with a deleted collection
* master: keep a fresh grow safe from the report that raced it
* volume: name the volumes a deleted collection took with it
Deleting a collection left the master to work out what went by omission from
the next full volume list, which it no longer gets: heartbeats carry the whole
list only when the master asks for it. The volumes a bucket's churn creates and
destroys between two of those requests are never named in either direction, so
the master keeps counting their slots as occupied and a cluster that creates
and drops collections quickly runs its free-slot accounting dry -- assigns fail
with no free volumes left while the disk holds a handful of volumes.
The destroy path already knows exactly which volumes it removed, so send them
down the same channel every other deletion uses.
* rust: name the volumes a deleted collection took with it
Mirrors the Go volume server. The notify path derives its deltas by diffing
snapshots, so a collection delete that does not wake it is invisible until the
master next asks for the whole list.
* topology: mark a volume crowded only if it can take writes
Crowding asks for more room to write into, and the writable-volume refresh loop
marked anything past the threshold regardless of whether writes could land
there. Growth already discounts those by intersecting the crowded set with the
writable list, so the entries changed no decision and only took space -- in a
tiered cluster, one for nearly every volume.
* topology: wait for the crowded-volume collector before reading what it saw
Closing the stop channel does not order the collector's writes against the
test's reads.
* topology: drop the sleep from the crowded-volume test
The channels are unbuffered, so every send has been received by the time the
sweep returns, and waiting for the collector covers the recording. The sleep
only suggested the result turned on timing.
* filer: keep the existing peer subscription on a repeated add
A cluster node add for a peer that is already followed restarted the
subscription, dropping the metadata events between the two runs.
* master: tell a connecting client the current cluster membership
Cluster node updates are only broadcast to the clients connected at that
moment. A filer that lost its master stream while a peer came back never
learned about the peer, and stopped replicating its metadata for good.
* test: a filer joining the master learns about the filers already there
* test: a filer resubscribes to a peer that registered while it was disconnected
Runs the reported sequence against real processes: filer2 leaves, filer1
is paused and its master stream is broken, filer2 registers again, and
filer1 has to replicate from it after reconnecting.
* volume: validate the file extension in CopyFile and ReceiveFile
CopyFile and ReceiveFile build an on-disk path from the client-supplied
Ext. Both are intentionally ungated for cluster-internal peers, so a
value like "/../../x" is joined onto the volume directory and, once
path-cleaned, resolves outside it -- an EC-shard receive can then write,
and CopyFile read, anywhere the process can reach.
Constrain Ext to a real suffix (a leading dot followed by alphanumerics)
before it is used to build any path, so it can no longer carry a
separator or a parent reference.
* test: use an alphanumeric missing-file extension in the copy variants
The not-found and stop-offset-zero cases used ".definitely-missing" as a
deliberately absent source. The extension is now validated, and the hyphen
makes it invalid, so switch to ".missing" -- still a nonexistent file, but a
real extension shape.
* volume: validate the collection in CopyFile and ReceiveFile
The client-supplied Collection is folded into the on-disk path as
"<collection>_<vid>" by VolumeFileName and EcShardBaseFileName, both joined
with path.Join / util.Join. A Collection carrying a separator, e.g.
"../../x", therefore path-cleans to a target outside the volume directory,
the same escape the extension check just closed. Reject a collection that is
a bare parent reference or holds a separator; ordinary names ('.', '-' and
all) still pass.
* filer: drain pending log chunk refs when the metadata stream ends
In metadata chunks mode the server sends log file refs in responses of their
own, and the client can only read them once it knows the run of refs is over.
That was inferred solely from the arrival of a normal event, so refs still
pending when the stream ended were dropped: the subscription returned no
events and no error.
A follower never noticed, because it runs forever and a live event always
arrives to close the run. A bounded subscription — StopTsNs set, range already
in the past — can receive nothing but refs and then EOF, and silently reports
that nothing happened. For anything auditing a path that is the worst possible
answer, since an empty result is indistinguishable from a quiet period.
Drain on EOF as well as at the transition point.
* filer: add filer.meta.scan to audit one directory's change history
Reconstructing what happened to a path means replaying the metadata log, and
filer.meta.tail is built for watching rather than auditing: it follows forever
unless given a stop, prints multi-line JSON, and takes ranges only as durations
before now, so an incident timestamp has to be converted by hand.
Its -pattern also cannot find a versioned object. A versioned key is stored as
<key>.versions/v_<id>, so the events carry the names "<key>.versions" and
"v_<id>" and a pattern of the object's own name matches neither — the search
comes back empty while the object is being written continuously.
filer.meta.scan prints one line per change, stops at the end of the range,
accepts absolute -since/-until with an explicit -tz, and reports versioned
writes against the object key with the version id alongside, so -name matches
the key a client would ask for. Delete markers are labelled as such rather than
appearing as zero-length writes, and pointer flips on the .versions container
are distinguished from writes of object data.
* filer.meta.scan: read persisted log chunks from the volume servers
Reading a range through the filer makes it decode every log entry in that
range and filter each one, so the cost lands on the filer and does not shrink
when the prefix is narrow — only the bytes on the wire do. On a cluster whose
metadata log is dense that is the expensive part of a scan, and it is charged
to the process least able to spare it.
Enable metadata chunks mode: the filer hands out log chunk ids and the scan
reads them from the volume servers itself. ReadLogFileRefs re-applies the same
path filter client-side, so the output is unchanged — verified identical to
the filer-read path over the same range, including after a restart drops the
in-memory buffer and the data must come off disk.
Direct read needs a route to the volume servers that the filer does not, so a
failure before anything has been printed retries through the filer; retrying
after partial output would duplicate lines. -directRead=false forces it.
* filer.meta.scan: confirm an empty direct-read result through the filer
An audit that returns nothing is read as "nothing happened here", so it is the
one answer that must not be produced by a bug. Direct read has more ways to
come back empty than the filer path does — it needs a route to the volume
servers, and it depends on the ref-drain contract holding.
When direct read yields no changes, re-run through the filer before reporting
it, and warn if the two disagree. Re-running is safe only because nothing was
printed; after partial output a replay would duplicate lines instead, so that
case reports the error rather than retrying.
In metadata chunks mode the server sends log file refs in responses of their
own, and the client can only read them once it knows the run of refs is over.
That was inferred solely from the arrival of a normal event, so refs still
pending when the stream ended were dropped: the subscription returned no
events and no error.
A follower never noticed, because it runs forever and a live event always
arrives to close the run. A bounded subscription — StopTsNs set, range already
in the past — can receive nothing but refs and then EOF, and silently reports
that nothing happened. For anything auditing a path that is the worst possible
answer, since an empty result is indistinguishable from a quiet period.
Drain on EOF as well as at the transition point.
* master: evict a dead peer via the local raft handle
OnPeerUpdate only runs on the leader, and the AddVoter branch right above
mutates the local raft directly. The remove branch instead dialed our own
RaftRemoveServer back over gRPC. Drop the self-dial and remove the peer
through the local handle, matching the add path. This also leaves operator
tooling as the only caller of the RaftRemoveServer RPC.
* master: require whitelist auth for raft membership RPCs
RaftAddServer, RaftRemoveServer and RaftLeadershipTransfer rewrite raft
quorum but had no caller check beyond "am I the leader". Any client that
could reach the master gRPC port could add an unreachable phantom voter
and stall the write path.
Gate the three on the admin whitelist, mirroring the volume server's
checkGrpcAdminAuth. With no whitelist configured the guard allows every
caller, so default and single-master deployments are unaffected;
operators who set -whiteList get these RPCs locked down to it. The
leader's own dead-peer eviction no longer dials these RPCs, so the only
remaining callers are operator tooling.
* pb: let a heartbeat carry only the volumes that changed
A partial list cannot travel in volumes: a master that did not understand it
would read the absences as deletions. So changes get their own field, used only
once the master has said it compares digests and can tell when it has fallen
behind.
* master: apply the volumes a heartbeat reports as changed
Only the named volumes are touched. A full report says the server holds exactly
these; a changed report says nothing about the ones it leaves out, so absence
must not read as removal.
Also advertises that the master compares digests, which is what lets a server
stop sending its whole list. Advertising it once per connection means a server
reconnecting to a master that does not is back to full lists straight away.
* volume: send only the volumes that changed once the master accepts them
The whole list goes on every heartbeat until the master says it compares
digests, and again whenever it asks, so a master that cannot tell when it has
fallen behind never has to.
has_no_volumes stays derived from a full list alone. Deriving it from what a
heartbeat happens to carry would make a quiet one read as a server that had
lost every volume, and the master would drop them all.
The digest still covers every volume held rather than the ones sent, which is
what lets the master confirm that applying the changes left it current.
Reporting state is per-connection: a server that reconnects, or reaches a
different master, starts again from the full list.
* volume: let the zero reporting state stand for having told no master anything
A Store built as a literal, which tests do, left the reporting state nil and
panicked on the first heartbeat. As a value its zero form already means nothing
has been reported to anyone, which is exactly the state that sends the whole
list.
* rust: send only the volumes that changed once the master accepts them
Mirrors the Go volume server, with one hazard the Go side does not have: mount
and unmount deltas here are derived by diffing successive heartbeats, so a
heartbeat that carries a partial list would report every volume it left out as
unmounted. Collecting now returns the full set alongside the message, and every
site that diffs uses that rather than what went on the wire.
* volume: do not let a full-list request be lost to the heartbeat it raced
The request arrived while a heartbeat was already being built as a delta, and
committing that heartbeat cleared it, so the master waited for another digest
mismatch before asking again. Count the requests and clear only the one the
heartbeat answered.
* rust: stop marking volumes reported by a heartbeat that is thrown away
The state-notify path collected a heartbeat only to diff its volume list, then
sent a delta message of its own and dropped the one it had collected. Once
collecting recorded what the master had been told, every mount or unmount
silently marked the changed volumes as sent, and the master learned of them
only after a digest mismatch.
Snapshotting no longer records anything, and no longer expires ec volumes
whose deletion that path was already discarding.
* master: announce only the volumes a change actually brought
Every changed volume was broadcast as a new location. Volumes grow constantly
and growth moves no location, so on a busy cluster that told every connected
client about volumes it could already reach, filling bounded broadcast queues
and pushing out the topology updates that matter.
* master: ask for the full list when only one can repair the master
Delta heartbeats stop the full report, and with it the only thing that
re-registers a volume the lookup index lost. The volume server cannot see that
divergence and its digest cannot show it, so the master now checks its own two
indexes agree and asks for the list when they do not.
A node reporting one volume id twice is kept on full lists for the same reason
rather than merely skipped: its digest can never be verified, so nothing else
would tell the master what it had stopped holding.
* master: keep the volume options on every heartbeat response
A volume server takes them from whatever response arrives, and preallocate is a
bare bool with no way to tell off from unmentioned. A response sent to ask for
the volume list therefore turned preallocation off until the server reconnected.
Responses sent mid-stream now start from the configured options rather than
being built field by field.
* master: announce a volume the lookup index had lost
Repairing the index makes the volume servable again, but clients were told it
went when the node dropped out and nothing told them otherwise: the disk map
still held it, so it did not count as an arrival.
Reaching the lookup index is what makes a volume servable, so recovering an
entry there is an arrival as far as clients are concerned, on both the full
report and the changed-volume path.
* s3tables: add ClickHouse iceberg catalog integration test
* ci: run the ClickHouse iceberg catalog test
* s3tables: bound setup HTTP calls in the ClickHouse test
* s3tables: pin the ClickHouse writer image dependencies
VolumeLocationList.Stats subtracts the deleted figures from the totals to
report live size and needle count. Both deleted figures are maintained as
counters independent of the totals they come off, so either can transiently
exceed its total, and neither subtraction was clamped.
Unclamped, the size wraps to ~16 EB. The count is signed so it merely goes
negative, but VolumeLayout.Stats converts it with uint64(fileCount), which
turns it into ~1.8e19 just the same. Either one swamps the cluster totals
behind /dir/status, /vol/status and Topology.CollectionVolumeStats.
commandFsMergeVolumes.getVolumeSize had the same unclamped subtraction, where
a wrapped size reads as a volume far too large to join any merge plan.
Clamped to zero, matching the guards already in CollectionInfo.LogicalSize
and the admin server's logical-size accumulator.
Chris LuandGitHub
Mohit TalniyaandGitHub
Dmitriy PavlovandGitHub