* master: name the unlabeled disk layout plainly in assign errors
When no volume server serves the layout an assign targets, the error
named the empty disk type as "hdd" (HardDriveType is the empty string),
sending operators looking for servers labeled hdd when the actual
mismatch is labeled (e.g. -disk=ssd) servers versus unlabeled clients.
- describe the layout as "default (unlabeled)" when the disk type is
empty, keep %q naming for labeled types
- log the unserved-layout condition once per option instead of letting
every failing write repeat an unactionable line
Observed in production: volume servers started with -disk=ssd while CSI
mounts assign with the unlabeled layout; the per-write error stream
pointed at a nonexistent hdd fleet.
* master: bound and expire the unserved-layout warning dedupe
The dedupe map retained every distinct option key permanently. Option
keys embed request-derived fields (collection, disk type), so repeated
assignments with distinct options would grow master memory without
bound, and a retained key suppressed the warning if the same option
went unserved again after the topology recovered.
Remember last-warned timestamps instead, expiring after an hour, with a
hard cap that resets the set when a client-driven key flood fills it.
* master: silence per-retry unserved-layout log and name explicit hdd
Addresses Devin Review comments on #11290.
- The unserved-layout branch already rate-limits its warning via
assignUnservedLayoutWarning.Do, but the common epilogue still logged
lastErr at V(0) on every retry, so the flood the dedup was meant to
stop continued. Skip the epilogue log when the unserved-layout branch
owns the logging; the error is still returned to the client.
- describeDiskLayout took the canonicalized option.DiskType, but
ToDiskType folds both "" and "hdd" into HardDriveType, so an explicit
disk=hdd request was mislabeled "default (unlabeled)". Pass the
original request disk type instead: only an empty request is the
unlabeled default; an explicit hdd is named "hdd".
Adds TestAssignFailsFastNamesExplicitHdd covering the explicit-hdd
wording.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* master: scope the startup capacity shed to a truly empty topology
The retryable "no volume server capacity registered yet" shed checked
capacity for the requested disk type, so a cluster serving only other
media -- where that capacity will never register -- shed every assign
until the client's deadline instead of failing fast. An unsteered write
to such a cluster hung for its full HTTP deadline and surfaced "context
deadline exceeded" in place of "No writable volumes". Shed only while
no disk type has any registered capacity, and name the unserved medium
in the fast failure.
Claude-Session: https://claude.ai/code/session_01TF7FQghfDkpdoZgakTMX4R
* master: name the unserved medium for every fail-fast caller
The diagnostic sat in the growth-initiator block, so a follower joining
an in-flight growth and a growth-disabled master failed the same way
with only the generic pick error. Wrap at the fail-fast break instead,
which every caller reaches, and cover all three paths in the test.
Claude-Session: https://claude.ai/code/session_01TF7FQghfDkpdoZgakTMX4R
An assign arriving before any volume server has heartbeated saw zero
available space and failed outright with a plain error no client retries,
so the first write to a fresh bucket answered 500 while the cluster was
still starting. Distinguish a topology with no registered capacity from a
genuinely full one: fail fast only when registered capacity is exhausted,
and shed ResourceExhausted otherwise so the client's retry budget rides
out the startup window.
Claude-Session: https://claude.ai/code/session_018G9kWFgy8BaBAEkYV3YL9n
* fix(master): let the growth initiator wait for the growth it triggered
The growth-in-flight shed also fired on the request that initiated the
growth: it sets the pending flag right before the shed check, so a
cold-start assign enqueued growth and immediately failed itself with
"volume growth in progress". With no concurrent assigns around to pick
up the freshly grown volume, a single writer against an empty cluster
never completes a write despite ample free space.
Claim the pending flag with a compare-and-swap so exactly one request
becomes the initiator, triggering growth at most once, and let it wait
for that growth to land. Everyone else still sheds retryably instead of
pinning a goroutine: followers behind an in-flight growth, an initiator
whose growth concluded without yielding a writable volume, and an
initiator whose growth outlives the 10s wait budget, which previously
surfaced a non-retryable error (gRPC Unknown, HTTP 406) even though a
retry would have succeeded moments later.
* fix(master): stop assign waits when the request is cancelled
The assign retry loops slept through client cancellation, keeping a
goroutine spinning for the rest of the 10s budget after the caller had
gone; StreamAssign also ran assigns on a background context detached
from the stream. Wait on the request context and pass the stream
context through.
* topology: drop the unconditional grow-request setter
Growth is only claimed through AddGrowRequestIfAbsent's compare-and-swap
now; keeping the raw Store(true) around invites the check-then-set race
back.
* test: cover cold-start first write with a real cluster
Boot a fresh master plus three empty volume servers and require the very
first assign - HTTP and gRPC, each on a cold volume layout, no client
retries - to complete a write. The assign that triggers volume growth
must wait for it rather than answering "volume growth in progress";
unit tests stub the topology, so only a real cluster exercises the
assign-grow-wait path end to end.
Under a herd of concurrent assigns with no writable volume, Assign spun
PickForWrite for the full 10s timeout, pinning a goroutine per request and
starving the master of the cycles it needs to process growth and answer
heartbeats. When growth is the relevant remedy and already in flight, stop
spinning: if free space exists, shed with a fast retryable error so clients
back off and retry once growth lands; if the cluster is out of space, fail fast
with the real out-of-space error instead of masking it as retryable.
The gRPC shed uses ResourceExhausted, not Unavailable: operation.Assign retries
it, but the client connection layer doesn't treat it as a dead channel, so a
per-request shed across a herd doesn't tear down the shared master connection
and cancel every other in-flight assign. The HTTP dirAssignHandler sheds with
503 + Retry-After.
Chris LuandGitHub