Sniff identifies TS assets for the coming policy-driven alignment even
though nothing reaches it through repack today. Compile-time assertions
now state each adapter's capabilities explicitly.
Sniff sat in the mandatory adapter interface but has exactly one
caller, the repack gate, which only Indexer adapters can reach - the
hls-ts implementation was dead code. Move it to a Sniffer capability
discovered by assertion like the others: parquet keeps it, hls-ts
drops it, and repack skips the gate when an adapter cannot sniff.
The stored size is uint64; converting a larger value wrapped negative,
sizing the repack sniff buffer with make([]byte, -1) - a handler panic -
and passing Validate a negative size, which means skip the size check.
Refuse repack and answer views stale instead.
The identity digests covered only chunks, while the view's extent path
would serve from inline Content when present - a gRPC update could set
Content with the chunks and size unchanged and segments were served
from bytes the layout never described. Format entries are never written
with inline content, so treat it as disqualifying: views answer stale,
repack rejects it up front, the extent path no longer reads it, and the
source identity digests it so it cannot appear mid-repack unnoticed.
The commit-time check compared only the chunk list, so a concurrent
change that kept the chunks - clearing the TTL, moving the expiry
anchor, hard-linking, going remote - passed verification, and the swap
paired that fresh metadata with chunks uploaded under the old inputs: a
permanent entry pointing at chunks that still expire. Digest everything
the repack consumed - chunk fingerprint, file size, TTL, both time
anchors, the S3-expiry flag, hard-link and remote state - and answer
409 when any of it moved.
The entry lock is filer-local, so a writer on another filer could
commit between repack's read and its swap, and repack then restored the
old bytes over an acknowledged update. Re-read the entry and revalidate
the chunk fingerprint and WORM right before the swap, answering 409 on
any change, and build the new entry from the fresh read so concurrent
metadata-only updates are carried forward. This shrinks the unguarded
window from the whole repack to the commit itself; closing it entirely
needs owner routing.
The layout binding hashed only offset and file id, so a mutation that
kept both - a truncate shrinking chunk.Size, then a sparse extend back
to the original length - passed both the size and fingerprint checks
and served a stale view. Digest size, modification timestamp, cipher
key, compression, manifest status, and SSE type as well.
The fallback capped the remainder at MaxInt32 seconds, which the volume
TTL grid encodes as 68 years - about 35 days shorter than the entry's
lifetime. For the narrow band nothing can round up within int32, store
the chunks without a volume TTL instead: they outlive the entry rather
than predecease it.
SecondsToTTL truncates to the volume TTL grid, so 3599 remaining
seconds became 59m and anything under a minute became no TTL at all -
permanent chunks under an expiring entry. Round the remaining lifetime
up to the smallest representable value instead, and anchor it the way
FindEntry expires entries: S3-expiring entries age from Mtime, others
from Crtime, so a recently overwritten S3 object is no longer treated
as nearly expired.
Also bind each layout to a digest of the chunk list it described.
Offset writes and appends keep Extended while changing the chunks, so
a same-size partial write used to leave the old playlist and extents
being served over new bytes; the views now detect the mismatch and
answer 404 until the file is re-ingested or repacked.
WORM was checked before the entry lock was acquired, so a concurrent
writer could enable it while an ingest, repack, or plain HTTP overwrite
waited, and the commit then replaced a protected entry. Repack now
checks under its lock, and ingest and saveMetaData recheck at commit
time.
gRPC writers, renames, and repack already took the per-path entry lock,
but plain HTTP overwrites committed without it: an overwrite landing
between repack's read and its update was silently replaced, orphaning
its chunks. Take the lock around the saveMetaData and format-ingest
commits, so repack's exclusive hold spans every writer.
Views validated conditional requests against the media entry's ETag,
so re-ingesting identical bytes with a different sidecar changed the
playlist and segment boundaries while clients kept getting 304s. Fold
the encoded layout and the request's view parameters into the ETag the
view serves and checks.
S3 object versions may share one chunk list, so deleting the old chunks
after a repack could corrupt sibling versions; reject those entries
until chunk ownership is tracked.
New chunks also carried the full original TTL, restarting needle
expiry at repack time while entry expiry stayed anchored to creation: a
nearly expired entry left chunks stored for almost a full extra span.
Assign the remaining lifetime instead, and reject entries already past
it.
Ingest admits mediaSequence up to MaxInt64-(count-1), but the payload
decoder rejected anything above MaxInt64-count, so a boundary playlist
ingested successfully and then failed every view. Mirror the ingest
bound, covered by a round-trip test at the boundary.
- namespace the query parameters as format.ingest, format.repack and
format.view, following the mv.from/cp.from dotted convention, so the
general endpoints cannot collide with pass-through client parameters;
requests naming both ingest and repack are rejected
- state Accept-Ranges: none on view responses, which always answer with
whole documents or whole extents
- derive the small-content permission from the boundary source instead
of a second positional bool that a call site could silently swap
- validate the hls-ts layout before returning it, making the formattest
invariant enforced rather than emergent
DecodeLayout rejected names over 256 bytes while Encode accepted them,
so an oversized name encoded fine and then failed every decode. Enforce
the bound in Validate, shared by both directions.
New chunks were assigned with the TTL the request query implied while
the entry kept its own, so repacking a permanent file with ?ttl= made
its chunks expire under permanent metadata. Force the entry TTL onto
the storage option instead.
Filer.UpdateEntry only writes the store, so metadata subscribers never
heard about the new chunk ids while the old ones were queued for
deletion. Emit the update event the way the gRPC UpdateEntry path does.
The cutter materialized every interior cut up front, so a tiny sidecar
declaring one enormous extent could allocate gigabytes of cut offsets
before any media byte arrived. Keep only the extent start offsets,
bounded by the extent count, and derive each cut arithmetically.
Extent chunks are buffered in memory, so an absent limit must not mean
unlimited. Also close the repack chunk reader to release its private
reader cache, and drop the arithmetic capacity hint on the extended-map
allocation.
A merge failure part-way through returned only the flat data chunks,
dropping the manifests already written: cleanup paths could not delete
those needles, and AppendToEntry, which keeps the returned list after
logging the error, lost the wrapped chunks. Return the manifests plus
the not-yet-wrapped remainder instead - a complete representation of
every byte, safe to delete or to keep.
Three hooks, all on the entry's real path so JWT scopes, WORM, and
read-only rules apply unchanged:
- POST /path?format=<name> ingests a multipart index sidecar plus media
and cuts storage chunks on the extents the sidecar declares
- GET /path?view=<name> serves adapter views; rendered documents and
extent streams both ride the normal prefetch path with entry ETag,
preconditions, and HEAD support
- POST /path?repack=<name> derives the layout from the stored bytes and
rewrites the chunks cut on extent boundaries, swapping the entry under
the entry lock and queueing the old chunks for deletion
The layout is advisory: a stale one 404s its views while plain reads
stay untouched. Repack refuses hard-linked, remote, and SSE entries.
The footer already names every row-group byte range, so the adapter
only reads metadata: one extent per row group, the leading magic riding
with the first, and a trailing extent for the page indexes and footer.
Engines that fetch row groups by offset then read exactly the covering
chunks. Parquet needs no view; alignment alone delivers the benefit.
The ingest sidecar is an FFmpeg-style EXT-X-BYTERANGE media playlist;
its segments become the extents and 188 becomes the align quantum, so
every storage chunk holds whole TS packets of one segment. The view
renders a playback playlist with plain numbered segment URLs for
clients that do not speak byte-range HLS, and maps ?seq=N to the
segment's extent. Playlist state the generated playlist cannot
reproduce (EXT-X-KEY, MAP, DISCONTINUITY, GAP, I-frame-only) is
rejected at ingest.
A format adapter reduces one container format to three things the core
understands: extent sizes, an alignment quantum, and an opaque payload.
Capabilities beyond identity (Indexer, SidecarIndexer, Viewer) are
discovered by type assertion. The layout persists in one compact
extended attribute keyed by extent sizes rather than chunk ids, so it
survives chunk manifest folding, and the Cutter turns it into upload
chunk boundaries clamped by maxMB and the align quantum. The formattest
kit holds every adapter to no-panic parsing of truncated input.
The chunking loop always cut at a fixed size. Accept a ChunkBoundaries
source instead, with the fixed size as the default implementation, so a
caller can align storage chunks to structure inside the file. Inline
small-content storage is disabled in that mode because it would drop the
first boundary.
* pb: ask the master what each collection holds
Callers tracking usage were sent every volume in the cluster to add up
themselves, which is the master's largest single allocation.
* topology: summarise what each collection holds
One pass over the topology, allocating per collection rather than per volume.
Regular volumes count once each for logical totals and once per replica for
physical, taken from the lookup index, which is already keyed by volume and so
needs no set of seen ids. Ec shards are node-local so their sizes sum, while
the file and delete counts describe the volume and resolve once every holder
has been seen.
Replicas of one volume disagree while a write is landing or a heartbeat is
late. Walking a full listing took whichever replica the map iteration reached
first, so the answer moved between runs; this takes the largest, which is
stable and never reports usage below what some replica already holds.
* s3: take bucket sizes from the master's summary
The bucket size metrics pulled the whole volume list once a minute and added it
up, which cost the master 184.6MB of allocation and 17.8MB on the wire for six
numbers per collection.
VolumeList over 550k volumes 184.6 MB allocated, 17.8 MB on the wire
CollectionStatistics 176 bytes allocated, 47 bytes on the wire
The aggregation moves to the master with it, so the cases the removed tests
covered are now asserted against it directly.
* topology: count the replica holding the most live data
Quotas are enforced on size less deletions, and the replica with the biggest
raw size can be the one that has deleted the most. Counting it reported a
bucket smaller than it is and would leave one writable over its quota, which is
the opposite of what picking the largest was meant to guarantee.
* topology: cap a volume's deletions at what it holds
Live usage is read as a collection's size less its deletions, so a volume
reporting more deleted bytes than it has cancels live bytes belonging to other
volumes in the same bucket and reports it smaller than it is. Replica selection
already floored that volume's own live size at zero; the totals have to agree
with it.
* storage: share the volume strings a cluster repeats
Decoding a heartbeat allocates a fresh string for the collection, disk type and
remote backend of every volume, and a master holding a million volumes then
holds a million copies of the same handful of names.
Not the remote storage key, which is unique per volume: interning that would
fill the table rather than share anything.
800k volumes registered from a heartbeat that has actually been over the wire:
227 -> 211 B/volume, and 238 -> 214 when the volumes are tiered, since the
backend name shares too.
* storage: hold the interned strings rather than let them be collected
unique.Make clears its entries by weak reference, and its canonical value does
not survive a collection even while a caller still holds the string it handed
back -- so a volume reported later would get a second copy of a name the rest
of the cluster already shares. With only changed volumes reported, most are
interned once and never again, so that is the common case rather than a corner.
The table therefore only grows, which is why it stays restricted to values
drawn from a small set. Ten thousand collections keep a few hundred kilobytes.
* iceberg: stamp a default name mapping on new tables
* iceberg: repair non-compliant manifests at commit
* s3tables: verify ClickHouse writes read back through PyIceberg
* iceberg: carry the manifest-list content into repaired manifests
* iceberg: refresh the default name mapping on schema evolution
* iceberg: merge historical names into the refreshed name mapping
* iceberg: never fail a commit on repair fallout
* iceberg: harden manifest repair against writer dialects
* s3tables: keep PyIceberg reader stderr out of row data
* iceberg: keep name mappings unambiguous across field id reassignment
* iceberg: align existing manifest content metadata with the list entry
* s3: add the RenameObject endpoint
PUT /{bucket}/{key}?renameObject with x-amz-rename-source moves an object
through the filer's AtomicRenameEntry, so no bytes are read or rewritten and
the ETag, tags and SSE keys travel with the entry.
Only unversioned buckets: a versioned rename would have to rebuild the
.versions chain, and AWS offers RenameObject on directory buckets, which
cannot be versioned. The source arrives in a header, so it is authorized
separately for read and delete; both keys are locked, in key order, across the
precondition checks and the move.
* s3: let a matched source ETag precondition settle its date precondition
RFC 7232 has an ETag precondition outrank the date precondition on its own
side, and AWS documents the same for CopyObject: a matching
x-amz-copy-source-if-match with a failing x-amz-copy-source-if-unmodified-since
copies rather than returning 412. The source check evaluated all four headers in
sequence, so the date header could still veto a decided ETag match.
validateConditionalHeadersForReads already applies this precedence; the source
path now matches it.
* s3: cover a rename source named as a directory without a trailing slash
Renaming a directory would move a whole subtree, so it has to stay a missing
key whether or not the caller wrote the trailing slash.
* s3: accept a bare object key as the RenameObject source
AWS spells x-amz-rename-source both ways. Its CLI, Java and Rust examples pass
the bare source key, and only a second CLI example and the boto3 conditional
example pass bucket/key; the API reference's own example is a bare key too. The
header was read as bucket/key only, so the form AWS leads with was rejected with
InvalidArgument and the endpoint was unusable as documented.
A value is now read as a literal key first — the only reading that can never
name the wrong object — and as bucket-qualified second, when it carries the
request's own bucket and the literal key does not exist. That costs one extra
lookup only for a source that starts with the bucket's own name.
Another bucket's name in the source is no longer a distinct error: RenameObject
moves within one bucket, so it is simply part of a key this bucket does not
hold, and it reports NoSuchKey.
* s3: only a proven absence picks the other reading of a rename source
A source that resolves to a directory is not a miss to fall through on: the
literal path is still what the caller named, so answering for it beats renaming
a different object under the bucket-qualified reading. With a directory at
bucket/source.txt and an object at source.txt, a rename naming the former moved
the latter.
A failed lookup is not a proof of absence either, so a blip can no longer
redirect a rename to the other reading.
* topology: keep per-node volume state with the location it describes
The read-only and oversized indexes were maps from volume id to a list of the
nodes reporting that state -- the same key space the lookup index already
holds, kept a second and third time. Nothing ever asked which nodes; both are
only ever asked whether any node does.
So the state rides on the location list as a bit per entry, and the two indexes
go. Removing a location shifts the bits with it, and a node replacing another
at the same address inherits its slot, since that is what happens to the
location too.
800k volumes, 90% read-only: readonly index 40.0MB -> 0, lookup index
42.5MB -> 48.6MB for the bits, 33.9MB net.
* topology: rebuild the location flags when stale entries are dropped
Refresh rebuilds the location list, so leaving the flags alone left bits
describing whoever moved into the dropped entries' place.
* topology: assert the refreshed flag survived, not just that it moved
Clearing the mask rather than rebuilding it would have passed: the check that
the flag lands on the right location is done by clearing it, which an already
empty mask satisfies.
* topology: track volume size only where writes can land
Size tracking decays pending assignment estimates so the master does not
overfill a volume before heartbeats catch up. Nothing is ever assigned to a
read-only volume, so an entry for one can never be consulted -- and in a tiered
cluster that is most of them, which made this the volume layout's largest cost.
A volume held out of the writable list for capacity is not read-only and keeps
its entry: that entry is what enforces the recovery delay.
800k volumes, 90% read-only: sizeTracking 79.1MB -> 8.4MB, and the crowded set
falls out with it because a read-only volume no longer reaches the threshold
check at all.
* topology: decide size tracking per volume, not per reporting replica
A volume is unwritable if any replica is read-only, so asking the replica whose
heartbeat happened to arrive made the answer depend on arrival order: a
writable replica reporting after a read-only one put the tracking back.
Ask the volume instead, which also drops the caller-supplied flag and the churn
it caused. The crowded entry goes with the tracking, since leaving it behind
would only move the memory this releases.
Costs a map lookup per replica on a full-list heartbeat, about 19ms per 100k
volumes and no allocations, on a path that is now rare.
* s3api: fix ListObjectsV2 dropping objects under a partial prefix
ListObjectsV2 with a prefix that names only part of a directory name -
"data/a" matching both "data/a/..." and "data/ab/..." - dropped objects
when paginating with a small max-keys, reporting IsTruncated=false before
all matching keys had been returned.
On resume, normalizePrefixMarker took the listing directory from the
continuation marker ("data/a/") and discarded the "a" name prefix, so once
the marker's subtree was exhausted the sibling directories that also
matched the prefix were never listed.
Resolve the listing directory and name prefix from the prefix instead of
the marker, so siblings sharing the name prefix are still listed after the
marker's subtree.
Fixes#10652
* s3api: cover a sibling-directory marker in the partial prefix tests
* s3api: resume a truncated delimited listing from the emitted CommonPrefix
Rebuilding the continuation token from the request dir plus the listing prefix
assumes that prefix names a directory. With a partial name prefix like "data/a"
it produced "data/a/ab/" for the sibling prefix "data/ab/", so the next page
resumed in the wrong subtree and repeated it forever.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* heartbeat: name departed volumes in delta heartbeats
* master: release the lookup index with a deleted collection
* master: keep a fresh grow safe from the report that raced it
* volume: name the volumes a deleted collection took with it
Deleting a collection left the master to work out what went by omission from
the next full volume list, which it no longer gets: heartbeats carry the whole
list only when the master asks for it. The volumes a bucket's churn creates and
destroys between two of those requests are never named in either direction, so
the master keeps counting their slots as occupied and a cluster that creates
and drops collections quickly runs its free-slot accounting dry -- assigns fail
with no free volumes left while the disk holds a handful of volumes.
The destroy path already knows exactly which volumes it removed, so send them
down the same channel every other deletion uses.
* rust: name the volumes a deleted collection took with it
Mirrors the Go volume server. The notify path derives its deltas by diffing
snapshots, so a collection delete that does not wake it is invisible until the
master next asks for the whole list.
* topology: mark a volume crowded only if it can take writes
Crowding asks for more room to write into, and the writable-volume refresh loop
marked anything past the threshold regardless of whether writes could land
there. Growth already discounts those by intersecting the crowded set with the
writable list, so the entries changed no decision and only took space -- in a
tiered cluster, one for nearly every volume.
* topology: wait for the crowded-volume collector before reading what it saw
Closing the stop channel does not order the collector's writes against the
test's reads.
* topology: drop the sleep from the crowded-volume test
The channels are unbuffered, so every send has been received by the time the
sweep returns, and waiting for the collector covers the recording. The sleep
only suggested the result turned on timing.
* filer: keep the existing peer subscription on a repeated add
A cluster node add for a peer that is already followed restarted the
subscription, dropping the metadata events between the two runs.
* master: tell a connecting client the current cluster membership
Cluster node updates are only broadcast to the clients connected at that
moment. A filer that lost its master stream while a peer came back never
learned about the peer, and stopped replicating its metadata for good.
* test: a filer joining the master learns about the filers already there
* test: a filer resubscribes to a peer that registered while it was disconnected
Runs the reported sequence against real processes: filer2 leaves, filer1
is paused and its master stream is broken, filer2 registers again, and
filer1 has to replicate from it after reconnecting.
* volume: validate the file extension in CopyFile and ReceiveFile
CopyFile and ReceiveFile build an on-disk path from the client-supplied
Ext. Both are intentionally ungated for cluster-internal peers, so a
value like "/../../x" is joined onto the volume directory and, once
path-cleaned, resolves outside it -- an EC-shard receive can then write,
and CopyFile read, anywhere the process can reach.
Constrain Ext to a real suffix (a leading dot followed by alphanumerics)
before it is used to build any path, so it can no longer carry a
separator or a parent reference.
* test: use an alphanumeric missing-file extension in the copy variants
The not-found and stop-offset-zero cases used ".definitely-missing" as a
deliberately absent source. The extension is now validated, and the hyphen
makes it invalid, so switch to ".missing" -- still a nonexistent file, but a
real extension shape.
* volume: validate the collection in CopyFile and ReceiveFile
The client-supplied Collection is folded into the on-disk path as
"<collection>_<vid>" by VolumeFileName and EcShardBaseFileName, both joined
with path.Join / util.Join. A Collection carrying a separator, e.g.
"../../x", therefore path-cleans to a target outside the volume directory,
the same escape the extension check just closed. Reject a collection that is
a bare parent reference or holds a separator; ordinary names ('.', '-' and
all) still pass.
* filer: drain pending log chunk refs when the metadata stream ends
In metadata chunks mode the server sends log file refs in responses of their
own, and the client can only read them once it knows the run of refs is over.
That was inferred solely from the arrival of a normal event, so refs still
pending when the stream ended were dropped: the subscription returned no
events and no error.
A follower never noticed, because it runs forever and a live event always
arrives to close the run. A bounded subscription — StopTsNs set, range already
in the past — can receive nothing but refs and then EOF, and silently reports
that nothing happened. For anything auditing a path that is the worst possible
answer, since an empty result is indistinguishable from a quiet period.
Drain on EOF as well as at the transition point.
* filer: add filer.meta.scan to audit one directory's change history
Reconstructing what happened to a path means replaying the metadata log, and
filer.meta.tail is built for watching rather than auditing: it follows forever
unless given a stop, prints multi-line JSON, and takes ranges only as durations
before now, so an incident timestamp has to be converted by hand.
Its -pattern also cannot find a versioned object. A versioned key is stored as
<key>.versions/v_<id>, so the events carry the names "<key>.versions" and
"v_<id>" and a pattern of the object's own name matches neither — the search
comes back empty while the object is being written continuously.
filer.meta.scan prints one line per change, stops at the end of the range,
accepts absolute -since/-until with an explicit -tz, and reports versioned
writes against the object key with the version id alongside, so -name matches
the key a client would ask for. Delete markers are labelled as such rather than
appearing as zero-length writes, and pointer flips on the .versions container
are distinguished from writes of object data.
* filer.meta.scan: read persisted log chunks from the volume servers
Reading a range through the filer makes it decode every log entry in that
range and filter each one, so the cost lands on the filer and does not shrink
when the prefix is narrow — only the bytes on the wire do. On a cluster whose
metadata log is dense that is the expensive part of a scan, and it is charged
to the process least able to spare it.
Enable metadata chunks mode: the filer hands out log chunk ids and the scan
reads them from the volume servers itself. ReadLogFileRefs re-applies the same
path filter client-side, so the output is unchanged — verified identical to
the filer-read path over the same range, including after a restart drops the
in-memory buffer and the data must come off disk.
Direct read needs a route to the volume servers that the filer does not, so a
failure before anything has been printed retries through the filer; retrying
after partial output would duplicate lines. -directRead=false forces it.
* filer.meta.scan: confirm an empty direct-read result through the filer
An audit that returns nothing is read as "nothing happened here", so it is the
one answer that must not be produced by a bug. Direct read has more ways to
come back empty than the filer path does — it needs a route to the volume
servers, and it depends on the ref-drain contract holding.
When direct read yields no changes, re-run through the filer before reporting
it, and warn if the two disagree. Re-running is safe only because nothing was
printed; after partial output a replay would duplicate lines instead, so that
case reports the error rather than retrying.
In metadata chunks mode the server sends log file refs in responses of their
own, and the client can only read them once it knows the run of refs is over.
That was inferred solely from the arrival of a normal event, so refs still
pending when the stream ended were dropped: the subscription returned no
events and no error.
A follower never noticed, because it runs forever and a live event always
arrives to close the run. A bounded subscription — StopTsNs set, range already
in the past — can receive nothing but refs and then EOF, and silently reports
that nothing happened. For anything auditing a path that is the worst possible
answer, since an empty result is indistinguishable from a quiet period.
Drain on EOF as well as at the transition point.
* master: evict a dead peer via the local raft handle
OnPeerUpdate only runs on the leader, and the AddVoter branch right above
mutates the local raft directly. The remove branch instead dialed our own
RaftRemoveServer back over gRPC. Drop the self-dial and remove the peer
through the local handle, matching the add path. This also leaves operator
tooling as the only caller of the RaftRemoveServer RPC.
* master: require whitelist auth for raft membership RPCs
RaftAddServer, RaftRemoveServer and RaftLeadershipTransfer rewrite raft
quorum but had no caller check beyond "am I the leader". Any client that
could reach the master gRPC port could add an unreachable phantom voter
and stall the write path.
Gate the three on the admin whitelist, mirroring the volume server's
checkGrpcAdminAuth. With no whitelist configured the guard allows every
caller, so default and single-master deployments are unaffected;
operators who set -whiteList get these RPCs locked down to it. The
leader's own dead-peer eviction no longer dials these RPCs, so the only
remaining callers are operator tooling.
* pb: let a heartbeat carry only the volumes that changed
A partial list cannot travel in volumes: a master that did not understand it
would read the absences as deletions. So changes get their own field, used only
once the master has said it compares digests and can tell when it has fallen
behind.
* master: apply the volumes a heartbeat reports as changed
Only the named volumes are touched. A full report says the server holds exactly
these; a changed report says nothing about the ones it leaves out, so absence
must not read as removal.
Also advertises that the master compares digests, which is what lets a server
stop sending its whole list. Advertising it once per connection means a server
reconnecting to a master that does not is back to full lists straight away.
* volume: send only the volumes that changed once the master accepts them
The whole list goes on every heartbeat until the master says it compares
digests, and again whenever it asks, so a master that cannot tell when it has
fallen behind never has to.
has_no_volumes stays derived from a full list alone. Deriving it from what a
heartbeat happens to carry would make a quiet one read as a server that had
lost every volume, and the master would drop them all.
The digest still covers every volume held rather than the ones sent, which is
what lets the master confirm that applying the changes left it current.
Reporting state is per-connection: a server that reconnects, or reaches a
different master, starts again from the full list.
* volume: let the zero reporting state stand for having told no master anything
A Store built as a literal, which tests do, left the reporting state nil and
panicked on the first heartbeat. As a value its zero form already means nothing
has been reported to anyone, which is exactly the state that sends the whole
list.
* rust: send only the volumes that changed once the master accepts them
Mirrors the Go volume server, with one hazard the Go side does not have: mount
and unmount deltas here are derived by diffing successive heartbeats, so a
heartbeat that carries a partial list would report every volume it left out as
unmounted. Collecting now returns the full set alongside the message, and every
site that diffs uses that rather than what went on the wire.
* volume: do not let a full-list request be lost to the heartbeat it raced
The request arrived while a heartbeat was already being built as a delta, and
committing that heartbeat cleared it, so the master waited for another digest
mismatch before asking again. Count the requests and clear only the one the
heartbeat answered.
* rust: stop marking volumes reported by a heartbeat that is thrown away
The state-notify path collected a heartbeat only to diff its volume list, then
sent a delta message of its own and dropped the one it had collected. Once
collecting recorded what the master had been told, every mount or unmount
silently marked the changed volumes as sent, and the master learned of them
only after a digest mismatch.
Snapshotting no longer records anything, and no longer expires ec volumes
whose deletion that path was already discarding.
* master: announce only the volumes a change actually brought
Every changed volume was broadcast as a new location. Volumes grow constantly
and growth moves no location, so on a busy cluster that told every connected
client about volumes it could already reach, filling bounded broadcast queues
and pushing out the topology updates that matter.
* master: ask for the full list when only one can repair the master
Delta heartbeats stop the full report, and with it the only thing that
re-registers a volume the lookup index lost. The volume server cannot see that
divergence and its digest cannot show it, so the master now checks its own two
indexes agree and asks for the list when they do not.
A node reporting one volume id twice is kept on full lists for the same reason
rather than merely skipped: its digest can never be verified, so nothing else
would tell the master what it had stopped holding.
* master: keep the volume options on every heartbeat response
A volume server takes them from whatever response arrives, and preallocate is a
bare bool with no way to tell off from unmentioned. A response sent to ask for
the volume list therefore turned preallocation off until the server reconnected.
Responses sent mid-stream now start from the configured options rather than
being built field by field.
* master: announce a volume the lookup index had lost
Repairing the index makes the volume servable again, but clients were told it
went when the node dropped out and nothing told them otherwise: the disk map
still held it, so it did not count as an arrival.
Reaching the lookup index is what makes a volume servable, so recovering an
entry there is an arrival as far as clients are concerned, on both the full
report and the changed-volume path.
* s3tables: add ClickHouse iceberg catalog integration test
* ci: run the ClickHouse iceberg catalog test
* s3tables: bound setup HTTP calls in the ClickHouse test
* s3tables: pin the ClickHouse writer image dependencies
VolumeLocationList.Stats subtracts the deleted figures from the totals to
report live size and needle count. Both deleted figures are maintained as
counters independent of the totals they come off, so either can transiently
exceed its total, and neither subtraction was clamped.
Unclamped, the size wraps to ~16 EB. The count is signed so it merely goes
negative, but VolumeLayout.Stats converts it with uint64(fileCount), which
turns it into ~1.8e19 just the same. Either one swamps the cluster totals
behind /dir/status, /vol/status and Topology.CollectionVolumeStats.
commandFsMergeVolumes.getVolumeSize had the same unclamped subtraction, where
a wrapped size reads as a volume far too large to join any merge plan.
Clamped to zero, matching the guards already in CollectionInfo.LogicalSize
and the admin server's logical-size accumulator.
WriteFile's own existence probe runs while the file does not exist, and
WinFsp may serve that answer from its metadata cache for up to the
mount's FileInfoTimeout. A reader racing into that window failed its
open with not-found, which is the cache being a cache, not a defect in
concurrent reading. Establish visibility once before racing the readers,
so the test exercises what it is named for.
* wdclient: keep the location of a volume reported added and removed at once
A volume moved between a server's disks arrives in both lists of one message,
and the server still has it. Additions were applied before removals, so the
removal won and the client was left with no location for a volume that never
went anywhere.
Reordering would swap the bug for a window where the volume resolves nowhere,
since the two updates take the lock separately. Skip the removal instead, so
the order the lists are applied in stops mattering.
* wdclient: build each ec update explicitly in the move test
Reusing one response object and adding the deletion to it left the overlap the
test turns on implicit, and reading it as a delete-only update is the natural
mistake.
* topology: follow a volume that moved between a server's disks
The heartbeat diff asked only whether a volume id was reported anywhere on the
node, so a volume that moved to a disk of another type stayed on the disk it
left as well. The master then held two copies of it forever: the volume count
was overstated, and GetVolumesById returned whichever disk the map iterated
first, so lookups could hand back the disk the volume had already left.
Track which disk types the heartbeat named each volume on, and treat a volume
named on another disk as absent from this one. Disk types are interned to an
index because a server reports a handful of them across hundreds of thousands
of volumes.
A volume named on two disks at once is a stale twin rather than a move, and is
still kept on both -- dropping one would tell the master a replica vanished.
Only a volume named twice on one disk type is unrepresentable, so that is now
what marks the node, rather than any repeat of an id.
* master: do not tell clients a moved volume left the node
A volume moved between a node's disks is removed from one and added to the
other, so it lands in both lists of the same heartbeat. Clients apply additions
before deletions, so the removal wins and they end up with no location for a
volume that never went anywhere.
Skip removals for volumes the node still holds, as the ec shard paths already
do, and update the topology before judging the delta removals so an unmount
that really did happen is still reported.
* trim the comments on this change to the parts that are not evident
* master: judge a volume removal on normal replicas alone
HasVolumesById answers for ec shards as well, so a replica encoded into ec
shards looked like it was still on the node and clients were never told the
normal location had gone. They hold normal and ec locations separately and
prefer the normal one from the same generation, so that location would have
gone on shadowing the shards.
Chris Lu
Chris LuandGitHub
Mohit TalniyaandGitHub
Dmitriy PavlovandGitHub