mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-08-17 12:46:59 +00:00
0de7ff5eb8fa00a2cd0e5b748259b41b83348390
737
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Chris LuandGitHub
|
4f50c5b0d4 |
feat: throughput limits for replicate, EC shard, and worker-driven moves (#10749)
* feat: throughput limits for replicate, EC shard, and worker-driven moves VolumeCopy was the only rate-limitable transfer; EC shard copies, replica creation, and worker-driven moves all ran at whatever the receiving server's maintenance rate allowed, with no per-operation control. - proto: VolumeEcShardsCopyRequest and the balance / ec_balance task params and configs gain io_byte_per_second; 0 keeps today's behavior (the volume server's own maintenance rate governs). - volume server: VolumeEcShardsCopy throttles with one WriteThrottler per request, shared across the shard, .ecx, .ecj, .vif, and .ecsum copies so the limit caps the transfer as a whole - the same shape as VolumeCopy. - volume_move: ReplicateVolume accepts the limit; EcMoveOptions carries it through MoveEcShards/CopyAndMountEcShards into the copy request, with fake-client tests asserting propagation. - shell: ec.balance gains -ioBytePerSecond; volume.tier.move's replication top-up honors the command's existing -ioBytePerSecond instead of running unthrottled. - worker: balance and ec_balance configs gain io_byte_per_second (surfaced in the admin config schema), carried through detection and plugin job parameters into task params and handed to the shared mover; batch balance jobs inherit the limit from their detection results. The limit is per copy stream, so maxParallelization multiplies the aggregate ceiling. * worker plugins: expose io_byte_per_second in the plugin config and derive it The plugin-driven detection path derives its task Config from the plugin configuration values, and both balance and ec_balance left IoBytePerSecond at zero there - a configured limit silently reverted to the server maintenance rate. Both derive functions now read the field (clamped at zero), and the plugin descriptors expose it with defaults so the configuration form carries it. |
||
|
Chris LuandGitHub
|
a7d5443125 |
ec: confirm a surviving copy before deleting a duplicate EC shard (#10719)
* ec: confirm a surviving copy before deleting a duplicate EC shard The dedup phase of EC balancing removes a shard it believes exists elsewhere. It copies nothing first, so the shard surviving on another node is the only thing that makes the delete safe -- and it took the plan's word for that. The plan is built from the master's topology, which can name a location that holds nothing: such a server answers "CopyFile not found ec volume id N" when something later tries to read the shard there. A shard listed on a phantom location and on a real one looks duplicated, so dedup deletes one of them. When it picks the real one the last copy is gone, and the job reports success -- the loss only surfaces later, as a rebuild that cannot assemble enough shards. The move phase already refuses to work on trust: it verifies the shard registered on the destination before removing the source. Dedup now holds to the same standard. The planner records which node it chose to keep, and both executors -- the worker task and the shell's ec.balance -- confirm that node really holds the shard before deleting. A keep node that cannot be queried is unknown rather than confirmed, and blocks the delete. Tests drive the destructive path against an in-process volume server that tracks what is actually on disk separately from what the plan claims, which is the distinction the bug turns on. Without the guard, two of them fail by deleting the only copy and returning success. * ec: check the collection and bound the wait when confirming a survivor Two gaps in the dedup survivor check. The inventory RPC is keyed by volume id alone, so a server holding the same number for a different collection answers "yes, I have that shard" to a question about this one. Accepting that deletes the last real copy on the strength of an unrelated volume. The response already carries the collection, so verify against it rather than widening the RPC. The shell path also queried on a background context, so a keep node that accepts the connection but never answers would hang the whole balance run instead of reporting that the survivor could not be confirmed. Bound it. The check moves into VerifyShardsOnServer next to the existing helper, shared by both executors, so the two paths cannot drift. |
||
|
Chris LuandGitHub
|
c6e1387f59 |
shell: multi-target fs.mergeVolumes and volume.mark -readonlyCanDelete (#10706)
* shell: fs.mergeVolumes distributes one volume across multiple -toVolumeId targets * volume: volume.mark -readonlyCanDelete rejects writes but keeps accepting deletes * seaweed-volume: mirror readonlyCanDelete volume state |
||
|
Chris LuandGitHub
|
46ce8cbe84 |
master: stream volume listings (#10676)
* master: stream volume listings A listing of 800k volumes is 36MB on the wire but 305MB as messages, and the master built all of it, then held it while grpc encoded it. Two of those at once is most of a small master's heap, and the maintenance scanner asks every 30 minutes. The topology goes out first, listing nothing, then its volumes in batches, so the master holds a batch rather than a cluster: 341MB of live heap for one listing becomes 4.4MB. It allocates much the same either way -- what changes is how much of it has to be live at once, which is what sets the heap ceiling. Batches are built under their disk's lock and sent outside it, so a slow reader stalls the stream rather than the topology. They therefore do not share one instant, which a single listing did not either: it takes each disk's lock in turn, so a volume moving during either can be seen twice or not at all. The client helper hides which kind of master answered: one too old for the stream is asked the old way and its reply cut into the same batches. Either way the topology handed over lists no volumes, so a caller cannot come to depend on finding them there. * admin: stream the listing the maintenance scan reads It asks for every volume in the cluster every 30 minutes. Reassembling it client-side keeps the scan identical -- ActiveTopology splits disks by the disk ids on the volumes, so it needs them in the topology -- while the master no longer builds the whole reply to send it. |
||
|
Chris LuandGitHub
|
a2ff9cca27 |
master: let VolumeList ask for the volumes it wants (#10674)
* master: let VolumeList ask for the volumes it wants The request carried nothing, so every caller was answered with the whole cluster. A dashboard opening one volume's page, or a capacity probe adding up one bucket, was served all 800k of them and threw away the rest -- and the master built every one of those messages first. The topology, its disks and their counters are still reported in full: a caller reading free space or replica placement needs the cluster whichever volumes it asked about. Only what is listed under a disk is selected, ec shards included. An empty collection and a zero volume id take everything, the way volume.list already reads its own -collectionPattern and -volumeId, so a caller that forgets to narrow is answered too much rather than answered wrongly. That leaves the default collection unnameable, since it is the one the empty string names, so it gets a field of its own. An older client sends none of it and is answered exactly as before. * admin: ask the master for the volume the page is showing A volume's detail page was pulling every volume in the cluster to find one and its replicas, and discarding the rest. * admin: ask the master for the ec volume the page is showing Same as the volume detail page: one volume's shards were found by pulling every ec shard in the cluster. * s3: ask the master for the bucket's own collection The SOSAPI capacity probe summed one collection's volumes out of a listing of every volume in the cluster. Cluster capacity still comes out the same: it is read from the disk counters, which a filtered listing reports in full. * topology: read the disk usage counters atomically They are written with atomic.AddInt64 from heartbeats but were read plainly by the two listings and by FreeSpace, and the map they sit in was iterated without the lock its neighbour takes. Under -race a listing concurrent with a heartbeat trips on both. |
||
|
Chris LuandGitHub
|
567052bfb6 |
s3: take bucket sizes from the master's summary (#10664)
* pb: ask the master what each collection holds Callers tracking usage were sent every volume in the cluster to add up themselves, which is the master's largest single allocation. * topology: summarise what each collection holds One pass over the topology, allocating per collection rather than per volume. Regular volumes count once each for logical totals and once per replica for physical, taken from the lookup index, which is already keyed by volume and so needs no set of seen ids. Ec shards are node-local so their sizes sum, while the file and delete counts describe the volume and resolve once every holder has been seen. Replicas of one volume disagree while a write is landing or a heartbeat is late. Walking a full listing took whichever replica the map iteration reached first, so the answer moved between runs; this takes the largest, which is stable and never reports usage below what some replica already holds. * s3: take bucket sizes from the master's summary The bucket size metrics pulled the whole volume list once a minute and added it up, which cost the master 184.6MB of allocation and 17.8MB on the wire for six numbers per collection. VolumeList over 550k volumes 184.6 MB allocated, 17.8 MB on the wire CollectionStatistics 176 bytes allocated, 47 bytes on the wire The aggregation moves to the master with it, so the cases the removed tests covered are now asserted against it directly. * topology: count the replica holding the most live data Quotas are enforced on size less deletions, and the replica with the biggest raw size can be the one that has deleted the most. Counting it reported a bucket smaller than it is and would leave one writable over its quota, which is the opposite of what picking the largest was meant to guarantee. * topology: cap a volume's deletions at what it holds Live usage is read as a collection's size less its deletions, so a volume reporting more deleted bytes than it has cancels live bytes belonging to other volumes in the same bucket and reports it smaller than it is. Replica selection already floored that volume's own live size at zero; the totals have to agree with it. |
||
|
Chris LuandGitHub
|
344ac7684e |
filer: drain pending log chunk refs when the metadata stream ends (#10647)
In metadata chunks mode the server sends log file refs in responses of their own, and the client can only read them once it knows the run of refs is over. That was inferred solely from the arrival of a normal event, so refs still pending when the stream ended were dropped: the subscription returned no events and no error. A follower never noticed, because it runs forever and a live event always arrives to close the run. A bounded subscription — StopTsNs set, range already in the past — can receive nothing but refs and then EOF, and silently reports that nothing happened. For anything auditing a path that is the worst possible answer, since an empty result is indistinguishable from a quiet period. Drain on EOF as well as at the transition point. |
||
|
Chris LuandGitHub
|
ce7d388639 |
heartbeat: send only the volumes that changed (#10640)
* pb: let a heartbeat carry only the volumes that changed A partial list cannot travel in volumes: a master that did not understand it would read the absences as deletions. So changes get their own field, used only once the master has said it compares digests and can tell when it has fallen behind. * master: apply the volumes a heartbeat reports as changed Only the named volumes are touched. A full report says the server holds exactly these; a changed report says nothing about the ones it leaves out, so absence must not read as removal. Also advertises that the master compares digests, which is what lets a server stop sending its whole list. Advertising it once per connection means a server reconnecting to a master that does not is back to full lists straight away. * volume: send only the volumes that changed once the master accepts them The whole list goes on every heartbeat until the master says it compares digests, and again whenever it asks, so a master that cannot tell when it has fallen behind never has to. has_no_volumes stays derived from a full list alone. Deriving it from what a heartbeat happens to carry would make a quiet one read as a server that had lost every volume, and the master would drop them all. The digest still covers every volume held rather than the ones sent, which is what lets the master confirm that applying the changes left it current. Reporting state is per-connection: a server that reconnects, or reaches a different master, starts again from the full list. * volume: let the zero reporting state stand for having told no master anything A Store built as a literal, which tests do, left the reporting state nil and panicked on the first heartbeat. As a value its zero form already means nothing has been reported to anyone, which is exactly the state that sends the whole list. * rust: send only the volumes that changed once the master accepts them Mirrors the Go volume server, with one hazard the Go side does not have: mount and unmount deltas here are derived by diffing successive heartbeats, so a heartbeat that carries a partial list would report every volume it left out as unmounted. Collecting now returns the full set alongside the message, and every site that diffs uses that rather than what went on the wire. * volume: do not let a full-list request be lost to the heartbeat it raced The request arrived while a heartbeat was already being built as a delta, and committing that heartbeat cleared it, so the master waited for another digest mismatch before asking again. Count the requests and clear only the one the heartbeat answered. * rust: stop marking volumes reported by a heartbeat that is thrown away The state-notify path collected a heartbeat only to diff its volume list, then sent a delta message of its own and dropped the one it had collected. Once collecting recorded what the master had been told, every mount or unmount silently marked the changed volumes as sent, and the master learned of them only after a digest mismatch. Snapshotting no longer records anything, and no longer expires ec volumes whose deletion that path was already discarding. * master: announce only the volumes a change actually brought Every changed volume was broadcast as a new location. Volumes grow constantly and growth moves no location, so on a busy cluster that told every connected client about volumes it could already reach, filling bounded broadcast queues and pushing out the topology updates that matter. * master: ask for the full list when only one can repair the master Delta heartbeats stop the full report, and with it the only thing that re-registers a volume the lookup index lost. The volume server cannot see that divergence and its digest cannot show it, so the master now checks its own two indexes agree and asks for the list when they do not. A node reporting one volume id twice is kept on full lists for the same reason rather than merely skipped: its digest can never be verified, so nothing else would tell the master what it had stopped holding. * master: keep the volume options on every heartbeat response A volume server takes them from whatever response arrives, and preallocate is a bare bool with no way to tell off from unmentioned. A response sent to ask for the volume list therefore turned preallocation off until the server reconnected. Responses sent mid-stream now start from the configured options rather than being built field by field. * master: announce a volume the lookup index had lost Repairing the index makes the volume servable again, but clients were told it went when the node dropped out and nothing told them otherwise: the disk map still held it, so it did not count as an arrival. Reaching the lookup index is what makes a volume servable, so recovering an entry there is an arrival as far as clients are concerned, on both the full report and the changed-volume path. |
||
|
Chris LuandGitHub
|
6d08b08f37 |
heartbeat: carry a volume digest and verify it (#10627)
* pb: carry a volume digest on the heartbeat The full volume list is the only way a master notices a volume that vanished without a delta, so it cannot simply be dropped. A digest gives the same guarantee without the list, and a way back to the list when they disagree. The digest has explicit presence: a server holding no volumes reports 0, which has to stay distinguishable from a server that does not compute one at all. * volume: report a digest of the volumes each heartbeat carries Digests exactly what goes on the wire: volumes skipped as quarantined, phantom or expired are absent from both the list and the digest, so the master compares against the same set the server meant to report. Runs the master's own hash over the master's own conversion of the message, so the two ends cannot drift into disagreeing about a field. * master: check the reported volume digest and ask for the list on a mismatch Compared after everything the heartbeat carried has been applied, so agreement means the master is current rather than that nothing changed. Servers reporting no digest are untouched, and a mismatch on a heartbeat that already carried the full list is reported rather than answered: there is nothing further to ask for, so asking again would loop. Nodes reporting one volume id twice are skipped for the same reason. * rust: report the heartbeat volume digest Mirrors the Go volume server. The master compares this against a digest it computes itself, so the hash has to agree byte for byte across the two implementations, not merely be a hash of the same fields: report_hash_vectors pins it against values generated by the Go side, and the ttl and replica placement narrowing the master applies when it decodes a message is applied here too rather than assumed away. A drift there would not corrupt anything, but every volume server on this implementation would report a digest the master can never match and fall back to sending its whole volume list forever, which is the cost the digest exists to avoid. * master: pin what the digest check does to each kind of report The upgrade story rests on these: a server that reports no digest is never asked for anything, so the two sides can be upgraded in either order, and a disagreement that resending cannot fix is reported rather than re-asked, so it cannot loop. * topology: enumerate the digest coverage test from the message The list of fields was written out by hand, so a field added to VolumeInformationMessage later would fall outside the digest while the test went on passing, and a change to it would never reach the master. Walk the message descriptor instead. Some fields are narrowed or normalised on the way into VolumeInfo, so the smallest change to the wire value can land back on the stored one; the test offers several values per field and asks only that some change is visible. |
||
|
Chris LuandGitHub
|
b46946ece5 |
filer: list directories without decoding chunk lists (#10616)
* filer: decode a listed entry without building its chunk list
A readdir reads attributes and never looks at chunks, but decoding an
entry builds the whole chunk list first: four allocations per chunk, all
of it thrown away. On a directory of ordinary 4MB-chunked files that is
most of what listing costs.
DecodeAttributesOnly walks the wire format and hands everything except
the chunks to the generated unmarshaller, so new fields in filer.proto
need no attention here. The chunks are still measured, because the S3
copy and multipart paths deliberately store a zero FileSize and let the
chunk extents define the size, but nothing is allocated to do it.
The blob is only re-encoded once a chunk is actually seen, so an entry
without any -- every directory, for one -- is unmarshalled where it lies
and pays nothing for the walk.
Listings opt in through the context, the way the lazy remote paths
already do; a store that ignores it stays correct.
chunks full attrs-only allocs
0 312.8n 310.1n ~ 1 -> 1
1 686.1n 411.1n -40.07% 7 -> 1
4 1.742u 667.4n -61.69% 24 -> 1
16 5.770u 1.544u -73.25% 86 -> 1
64 25.23u 6.004u -76.20% 328 -> 1
* mount: list directories with chunk lists omitted
The two meta cache listings behind a readdir are the only callers, and
neither reads a chunk. On 200k single-chunk files one enumeration goes
from 364ms to 277ms and drops a million allocations.
The read-through listing still fetches whole entries from the filer,
which would need the request to say it wants attributes only.
* mount: give the readdir benchmark's entries a chunk
Chunkless entries made the decode look far cheaper than it is, which is
the part of a listing worth measuring.
* filer: let a listing ask for entries without their chunk lists
The read-through readdir fetches whole entries over gRPC, and for a wide
directory the chunk lists are most of what crosses the wire and most of
what the client then unmarshals. A 4MB-chunked file is 113 bytes of
entry against 46 without its chunk.
ListEntriesRequest gains omit_chunks. The size a client needs is already
in the attributes, where the store decode folded the chunk extents in,
so dropping the list costs the client nothing.
The filer still reads the entries whole. A listing is where a TTL-expired
entry gets collected and deleted, and deleting one needs its chunks to
find the data, so omitting them there would leak. Only the response is
trimmed.
The hint moves to filer_pb so one context flag serves both transports:
the gRPC request sets omit_chunks, and a listing served from the local
store skips building the chunks. Cache population is unaffected either
way, since EnsureVisited starts from its own context.
* filer: reject a chunk the full decoder would reject
The walk skipped a chunk's bytes without looking inside them, so a
FileChunk carrying a corrupt nested fid, or a string that is not valid
UTF-8, sailed past the listing decoder while every other read of the same
entry still failed. The file listed with a plausible size and then gave
EIO on open, and corruption that used to fail the listing loudly was
hidden instead.
The chunk bytes are the one part of the blob the generated unmarshaller
never sees, so the two checks it would have made are made here: a
submessage has to parse, and a proto3 string has to be valid UTF-8.
FileChunk's only submessages are FileIds of scalars, so walking them is a
complete check. A descriptor-driven test fails if FileChunk ever gains a
field of either kind that the walk does not know to check, which is the
part that keeps this honest as filer.proto grows.
Taking the scratch buffer lazily, only once a chunk is actually dropped,
also takes the pool out of the path for entries that have none. Those
were measurably slower than the full decoder before; they are now level
with it. Each chunk's length prefix is parsed once rather than twice.
chunks full attrs-only vs base
0 171.4n 176.9n ~ (p=0.670)
1 366.6n 259.4n -29.24%
4 1.034u 500.2n -51.60%
16 3.905u 1.464u -62.52%
64 13.48u 5.195u -61.46%
* filer: carry the size before dropping chunks over the wire
Dropping the chunk list assumed every store folds the chunk extents into
FileSize when it decodes. A store that keeps entries as JSON rather than
as an encoded Entry never re-derives it, so an object written with a zero
FileSize kept its real size only in the chunks, and stripping them left
the client reading the file as empty. Stamp the size into the attributes
first, which costs nothing and does not depend on how the store loaded
the entry.
* mount: test that the readdir context reaches the store decode
Everything else exercises the decoder directly, so a refactor that
stopped threading the context would have reverted the whole thing with
every test still passing.
The benchmark's chunks also carried a constant legacy FileId, which
BeforeEntrySerialization reparses over Fid on the way in, so all 200k
entries stored one byte-identical chunk rather than the varying fixture
it looked like.
|
||
|
Chris LuandGitHub
|
3514925581 |
filer: let a nested path rule turn worm off (#10503)
* filer: let a nested path rule turn worm off mergePathConf ORs the booleans, so worm set on a bucket could never be lifted on a directory under it, while every string field is overridden by the more specific rule. Make worm tri-state instead: unset inherits, set wins. readOnly, fsync and disableChunkDeletion keep the OR, so a nested rule still cannot escape a lock the bucket set. Configurations written before this carry an explicit "worm": false on every rule, because they are marshalled with EmitUnpopulated. Reading those back as an override would quietly drop worm from nested paths, so filer.conf is now stamped with a version and the flag is dropped to unset when the version predates it. * filer: copy the worm value out of the matched rule mergePathConf aliased the pointer into the merged result, so a caller that wrote through it would reach into the stored rule. |
||
|
Chris LuandGitHub
|
9c37e52c9b |
volume: EC decode onto a clean peer via staged-new-volume adopt (Go+Rust) (#10463)
Decoding EC shards back to a normal volume in place reconstructs <vid>.dat
in the shards' own directory, so the vid is momentarily registered as both
an EC and a normal volume in one location — the load/scan path then sees it
as both, risking mount ambiguity and needle loss. VolumeEcShardsToVolume
still supports that in-place path; this adds the primitives to decode onto
a *clean* peer instead:
- ReceiveFile gains a staged-new-volume mode: when the volume does not
exist here and ReceiveFileInfo.disk_type is set, pick a free-slot disk
of that medium and write <base><ext>.copying (not a valid volume name,
so the scanner never half-loads a partial push).
- VolumeEcShardsToVolume gains from_staged: adopt the pushed .dat/.idx/
.vif — rename .copying into place under a .note in-progress marker,
then mount — so <vid> lands on the peer only as a normal volume.
The caller decodes the shards off-box and streams the finished volume to a
peer holding no shard of the vid on the target medium. Go and Rust volume
servers get identical handlers. Proto: ReceiveFileInfo.disk_type (12; 8-11
reserved for versioned-EC), VolumeEcShardsToVolumeRequest.from_staged (3) +
disk_type (4).
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
|
||
|
Chris LuandGitHub
|
5536d88fbb |
azure: let the blob endpoint be configured (#10460)
* azure: let the blob endpoint be configured The service url was always derived as <account>.blob.core.windows.net, which leaves out Azure Government, Azure China, and private endpoints. Name the blob service url instead and those accounts become reachable. The url has to be https, since the account key or the bearer token would otherwise travel in the clear. * azure: reject an endpoint that carries no hostname A url like https://:443/ has a host of ":443", so the emptiness check on Host let it through and the request only failed once it reached Azure. The hostname is what has to be there. |
||
|
Chris LuandGitHub
|
fee3fcb55a |
mount: report data sizes to df with -df.logical (#10459)
df on a mount shows the space the cluster gives up to the data: every replica of a regular volume, every shard of an ec one. That is the honest answer for capacity planning, but it is not the question a user asks when they want to know how much of their data is stored. Add -df.logical. The master reports the logical sizes alongside the raw ones: one replica per regular volume, the data shards of each ec volume counted once. Free space is divided by the copies the requested replication makes, so used plus available stays the amount of data the mount can still write, and it comes off the cluster-wide usage rather than one collection's, since capacity is cluster-wide too. Statistics through a filer resolves an unset replication to the filer's default rather than the master's, matching where the writes it is sizing for actually land. The flag governs the quota check too, so a mount has one notion of how much it is using. A filer that predates the new fields sends zeros, and the mount keeps reporting the raw sizes. |
||
|
Chris LuandGitHub
|
3ae4e9c563 |
azure: authenticate with Entra ID instead of a storage account key (#10456)
* azure: authenticate the blob sink with Entra ID Shared account keys have to be distributed and rotated everywhere a sink runs. Leaving account_key empty now falls back to the identity chain, so a workload identity or managed identity carries the authorization instead. * azure: authenticate remote storage with Entra ID The remote storage client demanded an account key and refused to start without one. Fall back to the identity chain when it is absent, and let azure.client_id pin a user-assigned identity. * azure: reject a malformed storage account name The account name is interpolated into the service URL, so a name carrying a "/", "?" or "@" moves the authority elsewhere and an authenticated request follows it. Hold callers to Azure's own naming rule instead. * azure: keep a leftover environment key off the identity path A configured client id asks for Entra ID, but AZURE_STORAGE_ACCESS_KEY still filled in the account key behind it. An old mounted secret would go on authenticating until it rotated, and the failure then blamed the key. * azure: say what the identity path reads from the environment A pinned client id alone is not enough for workload identity: the tenant and the projected token come from the environment, and missing them only surfaces later, when a token is first requested. |
||
|
Chris LuandGitHub
|
be81b9d5d7 |
volume: fix EC decode/reconstruct index locality under -dir.idx (#10442)
* volume: fix EC decode/reconstruct index locality under -dir.idx EC->replicated decode failed under -dir.idx and on multi-disk with "volume not found on disk". The reconstruct rebuilds the .dat on the data disk but the on-demand VolumeMount scans only the data directory, matching on .idx/.vif; with the rebuilt .idx off in the index directory it matched the volume's leftover EC .vif and skipped the volume as EC metadata. - Resolve the EC .ecx local-first: prefer the copy co-located with the shards over the shared -dir.idx copy, with a non-empty preference so a 0-byte local stub still yields to a valid sibling (the cross-disk fallback). - Co-locate the rebuilt .idx with the .dat at the end of the reconstruct so the mount finds it; sweep .ecx/.ecj from both the data and index directories on Destroy so a stale copy cannot re-mount as a phantom EC volume. - Add VolumeConsolidateIndex: once the EC shards are deleted, unmount, move the .idx/.sdx from the data disk back to the -dir.idx directory (copy fallback across filesystems), and remount. A no-op without -dir.idx. * volume: tests for EC index locality (local-first .ecx, sweep, consolidate) - NewEcVolume prefers a non-empty local .ecx over the shared index dir, and a 0-byte local stub yields to a non-empty shared copy (the #9212 fallback). - Destroy sweeps .ecx/.ecj from both the data and index directories. - ConsolidateVolumeIndex moves a co-located index back to the -dir.idx dir and keeps the volume mounted; no-op without a separate index dir. - RenameOrCopyFile moves a file and drops the source. * volume: relocate the decoded index in place, without a read gap ConsolidateVolumeIndex previously unmounted the volume, moved the index, and remounted it. Between the EC-shard delete and the remount the volume had neither a normal nor an EC form mounted, so a read landing in that window got a not-found (or was proxied away). Move the index in place instead: RelocateIndexTo takes the data-file write lock, closes the needle map and data backend, moves the .idx (and derived .sdx), then retargets dirIdx and reloads — the same close-swap-load CommitCompact uses. The volume never leaves the mounted set, so a concurrent read blocks briefly on the lock rather than failing. The test now writes a needle before consolidating and reads it back after, proving the in-place reload keeps the volume serving. * volume: address review — maintenance guard, no orphan on copy failure - VolumeConsolidateIndex now rejects the request under maintenance mode, like VolumeConfigure and the other mutating volume RPCs. - RenameOrCopyFile rolls the cross-device copy back when the source cannot be removed, so a failed move never leaves two divergent copies (the loader would keep the data-dir one while the idx-dir orphan goes stale). - RelocateIndexTo logs a failed reopen-after-failed-move instead of swallowing it, since that leaves the volume unusable until the next load. |
||
|
Chris LuandGitHub
|
47b491b53c |
mount: version open file handles by filer log position (#10403)
* filer: stamp a log position on lookup and remote-cache responses Metadata events are logged after their store write and stamped with the filer clock. Reading that clock before serving an entry therefore gives a timestamp with a causal guarantee: every event at or below it is reflected in the returned entry. Clients caching filer state can use it as the entry's version to order the response against subscription events, including events committed before the call but delivered after it. * mount: version open file handles by filer log position A subscription event refreshing an open handle did a second lookup; a transient failure left the handle pinned to its old entry with no retry, since the subscription cursor had already advanced. The deeper problem is ordering: the handle is a cache written by three unordered channels — the async invalidation worker, local mutation acks, and open-time lookups — and overwriting cached state safely requires knowing which write is newer. The filer log timestamp is that order, and it now travels with every value instead of being derived out of band. Events carry it natively; lookup and remote-cache responses carry the log position stamped before the serving read; mutation acks carry it in their returned event; and the local store pairs each read with a version cursor advanced under the same lock as the store write. Each handle records the version its entry reflects, and one rule replaces the per-site reasoning: state at or below the handle's version is old news and must not be installed. The invalidation itself applies the event's own entry — no lookup, so no transient-failure window — except under a cached parent, where the store entry is the ordered merge of the event and anything applied since, and its version outranks the event's. An uncached parent receives no store writes, so a hit there would be a stale leftover masking the event. A vacated path (delete, rename away) keeps the last entry so unlinked-but-open reads still work. Directory builds version the completed directory at the listing snapshot and re-invalidate buffered events at that version, since their mid-build refresh ran against an incomplete store. The tests replay every race this replaces machinery for: rollback of a newer local flush (queued, cached, and read-through), stale leftovers under uncached parents, the build window including abort, handles opened after an event was queued, events landing mid-lookup, and undelivered events at remote-cache time across a filer failover. * filer: serialize the log position fence with mutations, stamp mutation acks The fence stamped before an unlocked entry read could precede state the read returned: a mutation writes storage first and assigns its event timestamp only at notify time, so a lookup racing that window handed the mount an entry newer than its fence, and the event's later delivery looked like fresh news — destroying dirty pages for a change the handle already had. The mutation handlers already hold an exclusive per-path lock across read, write, and notify; the lookup and remote-cache reads now take it shared around the stamp and the read, making the fence exact: everything at or below it is in the entry, nothing above it is. A no-change update returns success without an event, leaving the mount nothing to fence with even though the response confirms current state. Create and update acks now carry a log position stamped under the same lock, and the mount falls back to it whenever the ack has no event. Also regenerate the VT marshalers, which the earlier generation missed: without them a VT round-trip silently zeroed every log position. * java: sync filer.proto * mount: scope store versions to what they vouch for; atomic handle install The store's version cursor claimed too much. Advanced by local mutation acks and directory listing snapshots, it inflated the version of store reads for unrelated paths whose events the subscription still owed, and those events were then fenced out permanently. The cursor now tracks subscription progress only — events arrive in log order, so everything at or below it has been delivered for every path — and a completed listing records its snapshot as a per-directory floor instead of a global claim. Local acks never touch it: they version their own handle directly. Buffered build events advance the cursor at delivery, since their store write may never happen (abort) while their invalidation is already queued; their read-through directory pairs no store read with it, and rename fragments are applied first. Concurrent first opens raced: a slower opener's older lookup could overwrite the newer entry a faster opener had installed, while the monotonic version kept the newer timestamp — an old entry fenced at a new version, immune to every correcting event. Entry and version are now installed as one decision under the handle map lock, and an install that does not outrank the handle's version is dropped. The remote-cache commit also escaped the fence: it wrote storage and notified without the path lock, so a lookup's shared-locked fence and read could land between the two and hand out the cached state under-versioned. The commit now re-reads and writes under the exclusive path lock, and backs off entirely when the entry changed during the download — the concurrent writer supersedes the cached content. * mount: floors gate store applies; installs respect handle users; renames join the fence A directory floor certifies the listing state as of its snapshot, but a delayed event at or below the floor was still applied to the store — rolling the content back to pre-snapshot state while the floor kept claiming the snapshot version, so the correcting events were fenced out of every future read. Events are now gated against the affected directory's floor, each half of a rename independently. Fences are lower bounds: a listing or lookup can include a mutation whose event has not been delivered yet, and that event later passes every gate carrying state the handle already holds. Such a re-delivery now advances the version without destroying dirty pages or reinstalling the entry — invalidating local writes over a no-op was the real damage in every remaining under-fence window, including the unlocked listing snapshot, which no per-path lock can serialize. The concurrent-open install moved from the map lock to the handle lock every reader, writer, and invalidation synchronizes on, and rejects what cannot improve the handle: dirty state (local writes would be lost), unversioned lookup responses (they cannot outrank anything, and two zero-version opens must not overwrite each other), and anything not strictly newer. New handles are still fully initialized before the map exposes them. Renames committed metadata and emitted events with no path lock, so a lookup could read the renamed state under a fence preceding its events. Both rename handlers now hold the source and destination locks, ordered by path, across commit and notification; descendants of a renamed directory are not individually locked and rely on the no-op re-delivery handling above. * mount: per-entry store versions replace the cursor and directory floors The store's aggregate versions — a global subscription cursor and per-directory listing floors — were versions at coarser granularity than the values they described, and every over-claiming bug in this series traced to that gap: an aggregate vouching for state its source never saw. Each store entry now carries the filer log position of the write that produced it — the event that applied it, or the listing snapshot that inserted it, recorded in the store's key-value space under the same lock as the entry write. The store becomes what the handle already is: a last-writer-wins register with one rule, install only what outranks the current claim. The cursor, the floors, their advancement rules, the pairing ordering constraint, and the floor gating all collapse into that rule. Applies are gated per entry, each half of a rename independently; an unversioned local write clears the claim its content no longer proves; version records lingering after a bulk folder wipe cannot fence a recreate, since a claim only blocks while its entry exists. Listing inserts are stamped at build completion, before the buffered replay so newer replayed events override the stamp. Filer side, the fence dance every versioned read must perform is now a single choke point, fencedFindEntry, so a future read RPC gets the lock-serialized stamp by construction rather than by convention. * mount: judge no-op re-deliveries against an immutable base, not the live entry The equal-state skip compared the incoming event to the live handle entry, but local writes mutate the live entry — size, timestamps, chunks — so a delayed event re-delivering the base the handle was opened with no longer matched, and the installer destroyed the dirty pages and rolled the entry back over nothing new. The handle now keeps an immutable snapshot of the filer state it last installed or acknowledged, refreshed at every install and mutation ack (flush acks snapshot the request entry before the id mapping mutates it), and the no-op judgment runs against that base: an event carrying the base brings nothing, whatever the live entry has diverged to since. * mount: tombstones for versioned deletes, absence floors, copy enrollment Four gaps in the per-entry version protocol, all the same shape: a versioned fact with nothing carrying its version. A deletion is a fact about a path with no entry left to hold it — clearing the record let a delayed older event resurrect the deleted path, permanently, since the deletion's own redelivery is dedup-suppressed. Versioned deletes now leave a tombstone record that fences without an entry; renames tombstone their source the same way. Plain records still only block while their entry exists, so records lingering after a bulk folder wipe cannot fence a recreate. A completed listing proves absences as well as presences: a name it omitted was deleted as of the snapshot, and a delayed create below the snapshot re-creates it. The snapshot is kept per directory strictly as an absence fence, consulted only when a path has neither an entry nor a version record — present entries carry their own versions and never touch it, which is what separates this from the over-claiming floor it replaces. A rebuild against a pre-upgrade filer returns no snapshot; stamping now clears the children's records in that case, so a reinserted entry cannot reactivate the stale claim its previous incarnation left behind and reject valid events below it. Server-side copies installed the copied entry without enrolling in the base protocol, so the copy's own event differed from the stale pre-copy base and destroyed writes made to the destination after the copy. The install now refreshes the base and takes its version from the fenced readback. * mount: deletion facts outlive the cache's knowledge of the entry A versioned delete of a path the store held no entry for recorded nothing, so a delayed older event recreated the path — permanently, with the deletion's redelivery dedup-suppressed. The tombstone is now written whenever a versioned event vacates a path: the deletion is a fact about the path, not about what this cache happened to hold. For an absent entry, the listing's absence floor now speaks whatever older record remains: a tombstone at one position does not exhaust what is known about the path when a newer snapshot has confirmed the name still absent, and an event between the two was slipping past both. A committed copy whose readback failed installed a synthesized base with local timestamps; the copy's real event legitimately differs from it, and was read as foreign state — destroying writes made to the destination after the copy. The handle now marks that its own event is en route and adopts that event's state as the base without touching the live entry or the dirty pages; the adoption is one-shot, so a genuinely foreign event still invalidates. * mount: authoritative acks cancel pending event adoption; tombstones scoped and pruned The copy-event adoption flag could outlive its purpose: a flush after the failed readback installs a newer base and advances the version, the copy's own event is then version gated without consuming the flag, and the next genuinely foreign event was silently adopted — base advanced, live entry and dirty pages untouched — leaving the mount to later overwrite that remote change. Every local acknowledgment now installs its base through one helper that also cancels any pending adoption: the ack supersedes the mutation the adoption was waiting for. Tombstones were written for every versioned delete under the mount and survived directory eviction by design, growing LevelDB with historical deletions on delete-heavy mounts. They are now scoped to directories whose cached state the fence actually protects — an uncached parent never serves from the store nor applies the resurrecting insert — and a completed listing prunes the direct-child tombstones its absence floor supersedes, leaving only those above the snapshot. The store gains a key-prefix visitor for the sweep. * mount: acked saves install their value; trailer snapshots; direct-child prune range A version must never advance without its value. saveEntry stamped any open handle with the acknowledgment's version, but a handle opened while the save was in flight holds the pre-mutation entry — stamping it fenced out the events carrying the state it lacked, permanently, with the local apply performing no invalidation and the redelivery deduplicated. The acknowledged entry is now installed together with its version, through the same guarded install the racing-open path uses: under the handle lock, only when it outranks the handle, never over dirty local writes. Empty listings return no in-band snapshot — a snapshot-only response would be read as an entry by older consumers — so directories that end empty gained no absence floor and their tombstones were never pruned. The filer now sends the snapshot in the stream trailer, which older clients ignore, and the client reads it when no in-band snapshot arrived. Empty directories get real floors, their tombstones prune, and their buffered replays gain the snapshot filter instead of the replay-all fallback. Version records now encode the parent directory and name separated by a NUL, making a directory's direct children one contiguous key range: the tombstone prune scans exactly them under the cache lock, instead of walking every descendant record — the whole store, for root. * mount: fix dirty-page loss, uid/gid base, download race, copy adopt, leak; dedup Correctness fixes from the versioned-invalidation review: - A foreign delete/rename-away of a file held open with unflushed local writes destroyed the dirty pages unconditionally. A process may keep writing to an unlinked-but-open file and those writes were already acknowledged; preserve the pages when the handle is dirty. - downloadRemoteEntry stored the handle's base with filer-side uid/gid while every candidate it is later compared against is in local form, so under a non-identity UidGidMapper an unchanged re-delivery looked foreign and force-destroyed dirty pages. Map the base to local. - downloadRemoteEntry wrote the entry/base/version triple under only the handle's shared lock, so two concurrent reads of the same remote-only file could tear it. Serialize the install with a dedicated mutex (invalidation is already excluded by the exclusive handle lock). - A committed server-side copy whose readback failed adopted the FIRST event past the version gate as its base; a foreign write delivered first was silently swallowed. Adopt only an event whose content matches the synthesized base — the copy's own event — and install any other normally. - The deferred-create path relied on AcquireFileHandle installing the passed entry on a pre-existing handle, which the version rework dropped. Restore that install in the compat wrapper; the versioned open path keeps its gated install. Growth and hot-path cost: - Per-entry version records and tombstones leaked when a directory was evicted or read-through without a rebuild. An uncached directory gates its own inserts, so its records fence nothing; clear a directory's child version records when it is wiped for eviction. - FindEntry paid for the version KvGet on every lookup/getattr cache hit and threw it away. FindEntry now reads only the entry; the hot lookupEntry cache-hit path skips the version entirely. Cleanups: - Extract ackVersionTsNs over the shared response interface, replacing the metadata-event-else-log-ts snippet copy-pasted at four ack sites. - Extract acquireRenamePathLocks, replacing the verbatim sorted two-path lock fence in both rename handlers. * mount: no resurrection on foreign delete, version no-event acks, gate downloads, tighten copy adopt Follow-ups to the review patches: - Preserving dirty pages on a foreign delete let the next flush pass the isDeleted guard and CreateEntry, resurrecting the remotely-unlinked name. Mark the handle deleted in the vacate branch: the open fd can still read its buffered writes, but a flush no longer recreates the file. - A no-event acknowledgment (log fence only) synthesized a metadata event with TsNs 0, so the cache stored the entry unversioned and an older subscriber event rolled it back. Stamp the synthesized event with the ack's log position at all four ack sites. - downloadRemoteEntry serialized its install but did not check the version, so an older response arriving last overwrote the entry/base while the monotonic version kept the newer value, fencing corrections out. Install only when the response is at least as new as the handle. - sameEntryContent compared only size and chunks, so a foreign chmod with unchanged content was adopted as the copy's own event. Compare everything except server-assigned timestamps, so a metadata-only foreign change installs instead. * mount: trim comments to the non-obvious why The versioning work accumulated multi-line comment blocks restating what the code says. Keep the constraint a reader cannot derive — why a fence is exact, why a version must not advance without its value, why an uncached parent's records fence nothing — and drop the rest. * mount: distinguish rename from delete, tighten the download and adopt gates - A rename emits a nil old-path invalidation just like an unlink, so the vacate branch marked the handle deleted and later writes through the already-open descriptor were skipped instead of persisted. Carry the delete/rename distinction on the invalidation and mark only an actual delete. - The remote-download install accepted an unversioned response regardless of the handle's version, so during a rolling upgrade a delayed response could install stale content under a newer version. Require the response to be at least as new, with one exception: a handle still lacking local chunks takes the content anyway — it cannot read without it — but does not claim the response's log position. - Copy-event adoption returned without installing, so a foreign touch arriving before the copy's own event lost its timestamps. Content is unchanged either way, so the dirty pages stay valid; a clean handle now takes the entry, while a dirty one keeps its diverged version. * mount: one directory floor instead of a record per child; agree on TTL Review feedback: - Build completion wrote one KV record per direct child inside the cache write lock, so a large directory stalled every other cache operation for O(children) store writes. The directory's listing snapshot already covers every child it saw; make that floor the version for any child without a record of its own, and a child earns a record only when a later event touches it. One map write per build replaces the per-child writes, with the same fencing. - The presence probe read the store directly and so counted a TTL-expired entry as present, judging the path by a record describing content that has logically vanished. It now applies the same expiry the read path does, and an expired path falls back to its directory floor. - Preserve ErrNotFound identity when the commit-time re-read finds the object deleted, so callers still surface a 404. - Assert the rename-away source fence timestamp in the invalidation test. Also record the tombstone ceiling: distinct deleted names in a cached directory accumulate until it is rebuilt or evicted, which prunes everything at or below the new snapshot. * mount: pin the fence's clock domain instead of letting skew decide A log-position fence is stamped by one filer's clock under that filer's in-process lock, so comparing it to an event another filer logged is comparing two unrelated clocks. The two error directions are not equally costly: applying an event the fence already covered is a re-apply the base-equality check absorbs, while skipping one it does not cover leaves the handle holding exactly the state the event was meant to correct, with the subscription cursor already past it — the unhealable staleness this whole PR exists to remove. So refuse to guess. Fences now carry the signature of the filer that stamped them, and a handle records it alongside the position. An event is only fenced out when the filer that logged it is the one that stamped the fence — the logging filer appends its own signature, so its presence identifies the clock domain. Events from any other filer are applied. Positions taken from events keep comparing as before; the subscription already delivers those in order. The invalidation callback takes a struct now: it carries the path, entry, position, delete/rename distinction, and signatures, and was about to need a fifth positional parameter. * mount: follow a foreign rename; key page invalidation on content, not equality - A rename's old-path invalidation now carries the destination, and the handle follows the file there: an open fd tracks the inode, and leaving it on the old path made its next flush recreate that name instead of updating the renamed file. - Dirty pages overlay content, so only a content change invalidates them. Keying that on exact equality meant any timestamp-only event destroyed them, which the copy-adoption marker existed to paper over — a foreign touch could consume the marker and leave the copy's own event to drop the post-copy writes. Comparing content instead makes the marker unnecessary, so it is gone: a metadata-only event keeps the overlay, and a dirty handle keeps its diverged entry unless foreign content supersedes it. - A remote download response that is merely older is now refused even when the handle still lacks chunks; only an unversioned one is taken (and claims no position), since an older response's content predates what the handle reflects. - A refused or unversioned download no longer publishes to the metadata cache, where a zero-position event would clear the entry's version and let an older subscriber event roll the cache back. * mount: page invalidation keys on content alone; unversioned writes claim no position - sameEntryContent compared everything but timestamps, so a foreign chmod, chown, or xattr change counted as a content change and destroyed the dirty-page overlay. It was strict only to serve the copy-adoption marker, which is gone; its one caller now asks the question it actually needs — did the bytes change — so metadata-only events leave the overlay alone. - A rename over an existing file destroys that file, but its open handle was left live and still pointed at the name the renamed source now occupies, so its flush could overwrite it. MovePath already reports the displaced inode; mark that handle deleted. - An acknowledgment was refused whenever its position was numerically lower, even when a different filer stamped the fence it lost to. Two known, differing signatures mean unrelated clocks, so the comparison no longer applies there; unknown signatures still compare as before. - A local write with no log position behind it now records that explicitly instead of deleting its version record. Absence means the directory listing covers the path, which is why the snapshot floor applies; local content the listing never saw must not inherit it, or the events that would correct it are fenced out. * mount: widen the existing lookup functions instead of forking WithVersion twins The versioning work grew a parallel function for every accessor that needed to return a log position — lookupEntryWithVersion beside lookupEntry, maybeLoadEntryWithVersion beside maybeLoadEntry, FindEntryWithVersion beside FindEntry, AcquireFileHandleWithVersion beside AcquireFileHandle, advanceEntryVersion beside advanceEntryVersionTsNs, plus a getPbEntryWithVersion wrapper and an InsertListedEntriesForTest hook. Two names for one operation is two places to keep in step, and the split let callers pick the one that happened to compile. Each pair is now the single original name carrying the position, with callers that do not want it discarding it. filer_pb.GetEntry returns the fence its response already carried rather than a mount-side wrapper re-issuing the lookup, and InsertEntry takes the position its content reflects rather than a test-only twin that inserted without one. The one behavioural knot the merge exposed: AcquireFileHandle had been installing the entry on a pre-existing handle only in its unversioned form, which conflated 'the caller is authoritative' with 'the lookup had no version'. Deferred create is the only caller that means the former, so it now installs explicitly and the map function just acquires. |
||
 
|
490379bff3 |
Add codespell support with configuration and typo fixes (#10393)
* Add GitHub Actions workflow for codespell on master * Add rudimentary codespell config * Tune codespell config: skip generated code, ignore camelCase, whitelist domain terms Add camelCase/PascalCase regex to ignore common Go/Rust/JS identifiers like allLocations, publishErr, ReadInside, FlushInterval. Also skip templ-generated *_templ.go files, and whitelist a handful of short/domain-specific words (visibles, fo, te, ser, bject, unparseable, keep-alives, tread, anc, ue) that show up as false positives across the tree. Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Fix ambiguous typos and protect false positives Fixes typos that codespell reports with multiple candidate suggestions (so `codespell -w` cannot auto-apply them), plus one inline pragma and one config entry to protect legitimate identifiers. Manual fixes (single correct answer chosen from context): - pattens -> patterns (5x) in filer/upload/shell flag help strings - finded -> found (2x) in tarantool storage.lua comment - spacify -> specify (2x) in helm chart values.yaml comment - wether -> whether in skiplist.go docstring - simpe -> simple in mq schema test case name False-positive protection: - Add `//codespell:ignore` next to `source GET's` (possessive of HTTP verb) in s3api_object_handlers_copy_stream.go - Whitelist `auther` in .codespellrc — it's a local variable meaning "authenticator" in weed/security/tls.go, not a typo of "author". Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Extend codespell ignore list: .git-meta path and thirdparty groupId Also skip `.git-meta` (scratch dir for commit messages that may contain typo words verbatim) and whitelist `thirdparty` — it appears as the literal Maven groupId `org.apache.hadoop.thirdparty` in hdfs3 poms and cannot be renamed. Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> * [DATALAD RUNCMD] Fix non-ambiguous typos with codespell -w Auto-applied fixes to the 44 remaining single-suggestion typos across docs, comments, log messages, tests, config, and one Java pom. === Do not change lines below === { "chain": [], "cmd": "uvx codespell -w", "exit": 0, "extra_inputs": [], "inputs": [], "outputs": [], "pwd": "." } ^^^ Do not change lines above ^^^ * Revert breaking codespell fixes; whitelist unknwon and atleast Two of the auto-applied `codespell -w` fixes were false positives that would break the build/tests: - go.mod: `github.com/unknwon/goconfig` is a real Go module path — the upstream author's GitHub handle is literally `unknwon`. Renaming to `unknown` would fail dependency resolution. - test/benchmark/fuse_db/bin/{sqlite_verify.py,run_mysql.sh,run_sqlite.sh}: `atleast` is a literal CLI mode value (a string constant compared and passed as a positional argument). Rewriting to `at least` splits it into two arguments and breaks the mode check. Reverted those files and whitelisted both words in .codespellrc so future runs won't re-suggest the same broken fixes. Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris LuandGitHub
|
cdb60069a6 |
filer: conditional UpdateEntry with a chunk-set write condition (#10382)
* filer: accept a WriteCondition on UpdateEntry, under the per-path lock UpdateEntry was a bare read-modify-write: the precondition check, the chunk garbage diff, and the store write could interleave with a concurrent update to the same path. Take the per-path lock CreateEntry already holds, and evaluate an optional CreateEntry-style WriteCondition under it, failing with FailedPrecondition like expected_extended. * filer: IF_CHUNKS_EQUAL write condition compares the stored chunk fid set A chunk-preserving read-modify-write (tagging, setattr, copy-in-place) races UpdateEntry's garbage diff: if a concurrent update empties the chunk list first, the stale writer's commit resurrects fids that are already queued for deletion, stranding the entry on a dead needle once vacuum reclaims it. The reverse also holds: a writer that read an empty chunk list can wipe chunks a concurrent update just added. IF_CHUNKS_EQUAL guards both: the stored chunk fid multiset must still equal what the caller read, order-independent, with an empty fids list expecting no chunks. Absent entry counts as no chunks for CreateEntry overwrites and transactions. * filer: delete and append serialize on the entry path lock DeleteEntry queues the entry's chunks for deletion and AppendToEntry rewrites the chunk list, but neither held the per-path lock, so either could interleave with a conditional update between its precondition check and its write — a passed IF_CHUNKS_EQUAL would then resurrect fids already on the deletion queue, or clobber a freshly appended chunk. AppendToEntry keeps the cluster lock for cross-filer append serialization; the path lock covers the local read-modify-write. * filer: reuse lockPath in UpdateEntry lookup |
||
|
Chris LuandGitHub
|
564803becd |
shell: show who holds the cluster lock (#10353)
* regenerate master_grpc.pb.go with protoc-gen-go-grpc v1.6.2 The other generated pb files are already on v1.6.2; this one was stale. * shell: keep unlock from racing the lease renewal A renewal RPC in flight while ReleaseLock runs re-creates the lock on the master after the release deletes it, and can blank the client name if the renewal reads it mid-release. The stale-token release is then ignored, so the lock stays held (sometimes anonymously) until it expires. Serialize the renew and release RPCs, and set the client name before flipping isLocked so the renewal never sends a partial acquisition. * shell: restart lease renewal after a failed renewal The renewal goroutine exits on error but never cleared its running flag, so later locks in the same process were never renewed and silently expired after ten seconds. * shell: show who holds the cluster lock A blocked lock command gave no hint that another client holds the lock (the refusals only surfaced at -v=2), and cluster.status reported the shell's own lock state as if it were the cluster's. Add a GetAdminLockStatus RPC to the master so lock prints the holder before blocking and cluster.status shows the actual cluster-wide holder. Both degrade silently against masters without the RPC. * shell: bound admin lock RPC attempts with timeouts The lease, renew, release, and holder-status calls all ran without a deadline, so an unresponsive master could hang the renewal goroutine, an unlock (which now waits on the renewal mutex), or the shell prompt. Give each attempt its own short context; the retry loops still resolve a fresh leader on the next try. * master: reject admin token release on non-leaders A follower holds no lock state, so it answered a release with success while the leader kept the lock until expiry. Refuse like LeaseAdminToken does so the client can try the leader instead. * shell: leave the lock release call unbounded A release cut short by a deadline leaves the lock held on the master until it expires, so a slow master would turn every unlock into a ten-second ghost lock. Restore the single fire-and-forget attempt; the timeouts stay on the lease and renew paths, where a stalled call forfeits the lease anyway. * shell: release only the token unlock started with A RequestLock racing a slow release (the admin presence lock does this on shutdown) could have its freshly acquired token sent in the release request or zeroed by the trailing stores. Capture the token once under the mutex and compare on clear so a concurrent acquisition survives an in-flight unlock. |
||
|
Chris LuandGitHub
|
25ab4c3cac |
preserve Content-Encoding for remote-mounted objects (#10340)
* remote storage: carry Content-Encoding into mounted entries A RemoteEntry now records the remote object's Content-Encoding, and every path that materializes a local entry from remote metadata (lazy fetch, lazy listing, remote.mount, remote.meta.sync, remote.cache) stamps it into the entry extended attributes, so HTTP and S3 HeadObject/GetObject return the header. GCS and Azure populate it on listing and stat; S3 only exposes it via HeadObject, so listings leave it empty. * remote storage: set Content-Encoding when uploading to the remote An entry carrying Content-Encoding in its extended attributes (a native S3 upload, or a value pulled from the remote) now keeps it when filer.remote.sync or remote.copy.local writes the object to GCS, S3, or Azure, instead of silently dropping it. * gcs: read remote objects without decompressive transcoding GCS transparently decompresses gzip-encoded objects on download, which ignores range requests and returns byte counts that disagree with the tracked RemoteSize. Request the stored bytes instead; chunked reads of gzip-encoded objects then behave like any other object. * remote storage: track Content-Encoding presence so removals propagate A listing that does not report encodings (S3) leaves the field unset and the local header untouched, while an authoritative report of no encoding (GCS, Azure, any stat) now clears a previously stamped header instead of leaving it stale. remote.cache also schedules a metadata update when only the reported encoding changes. * remote storage: propagate Content-Encoding on metadata-only updates filer.remote.sync routes same-content changes through UpdateFileMetadata, which only touched custom metadata (GCS, Azure) or tags (S3), so a Content-Encoding change in the extended attributes never reached the remote object's real header. GCS now patches contentEncoding alongside the metadata, and Azure reissues the blob's HTTP headers with the new value, carrying the others over since the call replaces the full set. S3 stays tags-only: changing the header there means rewriting the object, which the sync already does whenever content changes. * remote.meta.sync: optional per-file stat for listing-omitted metadata S3 listings carry no Content-Encoding, so entries synced from them never learn it and the lazy-stat path never runs once an entry exists. With -statFiles, each new or changed file whose listing left the encoding unreported is stat-ed before reconciling, and the stat-derived value is persisted so the next run only stats files that changed. Off by default: it costs one remote request per file, and GCS and Azure listings already carry the encoding. * s3: apply metadata-only Content-Encoding changes with an in-place copy Content-Encoding is S3 system metadata, so the tags-only metadata update silently left the object's real header untouched. When the encoding differs, reissue the object as a self-copy with replaced metadata, carrying the content type and configured storage class like a fresh write does. CopyObject caps at 5 GiB; beyond that the change is logged and applies on the next content write. * azure: skip the metadata call when user metadata is unchanged An encoding-only change reissues the blob's HTTP headers; sending the unchanged user metadata alongside it wastes a round trip and bumps the blob's ETag once more than needed. * s3: carry existing object metadata through the encoding copy The replace directive drops everything not resent, and a mounted entry usually has no local mime or user metadata, so the in-place copy wiped the object's Content-Type, Cache-Control, user metadata, encryption settings, and storage class. Read them back with a HeadObject first and carry them over, overriding only what SeaweedFS manages: the encoding, a locally set mime, and the configured storage class. S3 reports Expires as a string while the copy input wants a time, so it is parsed and skipped when malformed. |
||
|
Chris LuandGitHub
|
c015cc3939 |
generate vtproto marshalers for filer_pb and use them on the metadata log path (#10337)
* generate vtproto marshalers for filer_pb and use them on the metadata log path Reflection-based proto.Unmarshal allocates a fresh message tree through reflect.New on every call. On the metadata subscription fan-out the same event is decoded once per subscriber, so reflect.New tops the decode churn under many mounts. Generate MarshalVT/UnmarshalVT/SizeVT for filer.proto (a separate filer_vtproto.pb.go, filer.pb.go untouched) and call them on the log entry marshal and the subscribe/replay decode paths. UnmarshalVT allocates message structs directly and copies byte and string fields, so it stays wire-compatible with proto.Unmarshal and preserves the non-aliasing the persisted-log cache depends on. For SubscribeMetadataResponse this cuts decode allocations 69 -> 50 and ~4.5us -> ~2.1us per event; the win scales with subscriber overlap. * marshal log entries directly into the buffer SizeVT is allocation-free and MarshalToSizedBufferVT writes into a pre-sized slice, so the log entry can be marshaled straight into logBuffer.buf. This drops the per-entry MarshalVT allocation and the follow-up copy on the write path. * expand vtproto benchmarks: marshal, decode, and marshal-into-buffer by chunk count Parametrize by nested-message count (chunks per event) and add encode + zero-alloc marshal-into-buffer benchmarks alongside the decode one, so the write-path win from MarshalToSizedBufferVT is measurable too. * keep proto.Unmarshal for metadata events to preserve UTF-8 validation UnmarshalVT skips proto3's UTF-8 validation of string fields, so a SubscribeMetadataResponse with an invalid-UTF-8 string (e.g. Directory "\xff") that proto.Unmarshal rejects would decode and reach path filtering and subscribers. Decode events with proto.Unmarshal again; UnmarshalVT stays on the log entry paths, whose only variable-length fields are bytes and so carry no UTF-8 constraint. Tests cover the codec difference and that a malformed event is skipped before delivery. |
||
|
Chris LuandGitHub
|
a9cfbd8d3a |
s3: tear down the emptied .versions directory on last-version delete; drain existing residue (#10278)
* s3: routed last-version delete removes the emptied .versions directory The routed versioned delete (routedDeleteSpecificVersion) repoints the latest pointer and deletes the version file, but unlike the lock-path fallback (updateLatestVersionAfterDeletion) it never tears down the .versions/ directory it just emptied. The residue keeps the key's read path in the self-heal rescan loop: every GET of the deleted key logs event=surfaced plus a GetObject error until the background EmptyFolderCleaner gets to the directory — at least two minutes away on its delay queue, and possibly never (the queue is in-memory, bounded, and gated on the bucket's allow-empty-folders policy). Veeam's lock arbitration probes deleted lock keys continuously, so those windows are always open and the log spam is chronic. ObjectMutation DELETE gains remove_empty_parent: after the child delete, the filer best-effort removes the parent directory in the same locked transaction. Non-recursive on purpose — a concurrent write that lands a new version fails the removal instead of being lost with it. The routed last-version delete sets it on the version-file DELETE, matching the lock-path fallback's contract. Claude-Session: https://claude.ai/code/session_014mMYAHXZySkCCUfpRFNtSv * s3: drain empty .versions residue on read heal and in s3.versions.audit Directories already stranded by pre-teardown deletes (or dropped from the EmptyFolderCleaner's bounded in-memory queue) previously re-entered the self-heal rescan on every GET forever: the heal only cleared the pointer and nothing ever removed the directory, and s3.versions.audit counted the state as clean. When the heal rescan finds no remaining version, remove the directory outright (non-recursive, so orphan children still block and fall back to the pointer clear) and log event=healed mode=empty_dir_removed; the next GET takes the clean not-found path. The audit gains an empty category so the residue is visible, and -heal removes such directories in bulk. Claude-Session: https://claude.ai/code/session_014mMYAHXZySkCCUfpRFNtSv |
||
|
Chris LuandGitHub
|
9b1ff91949 |
filer: stream offloaded metadata-log entries to fix concurrent-write OOM (#10203)
* filer: stream offloaded metadata-log entries instead of buffering whole files The client metadata-chunks read path (ReadLogFileRefs, used by the meta aggregator to consume peer filers and by mounts) decoded every entry of a log file into a slice before handing it to the consumer, and prefetched the next whole file the same way. Peak memory scaled with log-file size: under heavy concurrent writes the per-event chunk lists grow and minute-files reach hundreds of MB to GBs, so a filer aggregating a few peers held many GBs of decoded entries at once (heap dominated by readLogFileEntries -> consumeBytesNoZero) and OOMed. Stream entries through a bounded channel: a producer decodes one entry at a time and the next file's read overlaps processing via the channel buffer, so peak memory is bounded by the channel depth rather than O(file size). In a synthetic replay peak live heap dropped from ~1.3x the file size to a flat few MB regardless of file size. * filer: tighten offload replay tests Share one ordered-replay assertion between the merge-order and single-filer tests, assert the callback's own error is what propagates, and drop atomic counters from callbacks that run on a single goroutine. * filer: abort offloaded log replay promptly instead of joining wedged readers Collapse the single-filer fast path into the merged reader: it was a second copy of the producer/stop lifecycle with its own subtler synchronization, and a one-stream merge does the same job. On abort (processing error or a fatal read error from one filer), the consumer used to drain channels and join every producer. A producer blocked in an uncancellable chunk read cannot observe stop until that read returns, so an abort could stall the caller's retry loop behind a dead volume-server connection. Closing stop is now the only cleanup: producers check it at every send and file boundary and exit on their own, and the merge loop's blocking receives also escape on stop. Producers also check stop before opening each file, so an aborted replay no longer keeps reading remaining files whose entries never reach the channel. A mid-file chunk-not-found still skips to the next file, but the log line now reports how many entries were delivered first instead of pretending the whole file was skipped; the redundant error log before setFatal is gone since the error propagates to callers that already log it. * filer: cap offloaded log entry allocation against corrupt size prefix A garbage 4-byte size prefix (torn chunk or stream desync) drove make([]byte, size) up to 4GiB per entry. Reject sizes above the same 1GiB bound the filer-side log readers enforce. |
||
|
Chris LuandGitHub
|
05b4b5bf56 |
ec: expose force_deleted_needles_check in ScrubEcVolume RPC and shell (#10176)
* ec: expose force_deleted_needles_check in ScrubEcVolume RPC and shell FULL EC scrubs can opt into strict deleted-needle verification via the -forceDeletedNeedlesCheck shell flag, off by default since it can report false positives when EC indexes disagree. Rejected for non-FULL modes. The Rust volume server parses the new field and ignores it: its FULL scrub verifies shards via RS parity, not per-needle reads. * volume: require admin auth for ScrubEcVolume ScrubEcVolume ran unauthenticated while its sibling ScrubVolume, and the rest of the mutating volume handlers, gate on checkGrpcAdminAuth. Close the gap so an EC scrub can't be triggered anonymously. * shell: reject ec.scrub -forceDeletedNeedlesCheck outside full mode Fail in the client before fanning out to every volume server, instead of erroring halfway through once the servers reject the request. |
||
|
Chris LuandGitHub
|
77bf2a3ab0 |
volume.balance: gate on real physical disk usage (fixes #10160) (#10162)
* shell: add volume.balance -byDiskUsage to balance by actual data The default balancer ranks servers by slot density, dividing used volumes by MaxVolumeCount. When MaxVolumeCount is configured higher than the disk can hold, a physically near-full server looks nearly empty and gets picked as the move target, so balancing drains less-full servers onto an already-full one. -byDiskUsage ranks servers by the actual data they hold (sum of volume sizes) instead, so the fullest-by-data server is treated as full and balancing drains it. It assumes comparable disk sizes per disk type and still respects each server's free volume slots. Default behavior is unchanged. * plumb physical disk usage into topology, gate volume.balance on it Volume servers now report each disk's filesystem total/free bytes in the heartbeat, and the master stores them in DiskInfo. volume.balance uses them to skip any move target whose disk is already near full (-maxDiskUsagePercent, default 90), so an over-configured maxVolumeCount can no longer make a physically full server look empty and get drained onto. The gate judges each server against its own disk, so heterogeneous disk sizes are fine; servers that do not report bytes fall back to slot-only behavior. Rust seaweed-volume mirrors the heartbeat reporting. * admin: report real physical disk capacity when volume servers provide it The dashboard estimated server capacity as maxVolumeCount * volumeSizeLimit, which overstates it when maxVolumeCount is set higher than the disk holds. Prefer the filesystem capacity now reported per disk, falling back to the estimate for servers that do not report it. * worker: gate automatic balance on physical disk fullness too The maintenance balance worker selects the least slot-utilized server as the move destination, so an over-configured maxVolumeCount makes a physically full server look empty and get drained onto — the same defect as the shell command. Now that DiskInfo carries real disk bytes, skip any destination whose disk is at/above 90% used (per server, against its own disk); a full server can still be a source. When every candidate destination is full, create no tasks. Servers that do not report disk bytes are not gated. * balance: share the physical-disk-fullness gate between shell and worker The shell volume.balance command and the maintenance balance worker each grew their own copy of the disk-fullness gate (targetDiskTooFull / destinationDiskTooFull) and a maxDiskUsagePercent=90 constant. Pull both into weed/topology/balancer (DiskTooFullAfter + DefaultMaxDiskUsagePercent) so the policy has one home and the two balancers can't drift. * balance: harden the physical-disk gate Guard against a nil DiskInfo in the byte/slot lookups. Let a zero disk-capacity report clear previously stored bytes (0 means "not reported" for bytes, unlike maxVolumeCount), so a server that stops reporting falls back to slot-only instead of trusting stale capacity. In the worker, charge each planned move's bytes to its destination within a detection cycle so the gate sees a target fill up rather than only its heartbeat-time free space. Note the per-location capacity summing assumes one location per filesystem (the used ratio the gate relies on stays correct regardless; absolute capacity can over-report). |
||
|
Chris LuandGitHub
|
41d6c821ba |
feat(topology): report empty disks (per-disk type + capacity in heartbeat) (#10166)
* fix(topology): keep physical disk 0 distinct in SplitByPhysicalDisk
DiskId 0 doubles as the first physical disk (Locations[0]) and the
protobuf "unset" default. SplitByPhysicalDisk folded every DiskId-0
record onto the aggregate DiskId whenever that was non-zero, so on a
multi-disk node the first disk's volumes merged into whichever disk
held volumes[0]: the node reported one fewer disk, the sibling showed
~2x volumes, and per-disk max was smeared across the survivors. This
surfaced as cluster.status and volume.list undercounting disks.
Only treat 0 as unset when no record carries a non-zero DiskId; with a
mix, 0 is a real disk and keeps its own entry.
* fix(admin): resolve physical disk 0 in active-topology indexes
rebuildIndexes re-derived each volume/EC record's physical disk id with
the same "DiskId 0 means unset" heuristic SplitByPhysicalDisk used, so
the two agreed only by sharing the bug. Now that SplitByPhysicalDisk
keeps disk 0 distinct, the duplicated heuristic would fold disk-0 records
onto a sibling while at.disks kept them on disk 0; GetVolumeLocations and
GetECShardLocations then matched no record and silently dropped every
volume and EC shard on the first disk, starving balance and EC tasks.
Build the indexes from the same SplitByPhysicalDisk reconstruction that
builds at.disks, so the keys always resolve. One source of truth instead
of a parallel normalize.
* fix(ec): allow physical disk 0 as preferred EC shard target
pickBestDiskOnNode gated its result on bestDiskId != 0, but 0 is both a
valid physical disk and the uint32 zero value, so a best-scoring disk 0
was discarded and the non-matching fallback returned instead. Gate on
bestScore.
* test(admin): cover EC-shard index resolution for physical disk 0
rebuildIndexes builds ecShardIndex the same way as volumeIndex; pin the EC
path too so a shard on disk 0 keeps resolving via GetECShardLocations.
* proto: per-disk type/capacity in DiskTag, DiskInfo.physical_disks
DiskTag gains type + max_volume_count so the heartbeat can describe every
physical disk, including ones holding no volumes or EC shards. DiskInfo
gains physical_disks so the master can hand the full per-type disk set to
per-physical-disk consumers.
* feat(volume): report each physical disk's type and capacity
CollectHeartbeat fills DiskTag.type and the per-disk effective max for
every location, so the master can account for disks that hold no volumes
or EC shards yet. Rust heartbeat mirrors it.
* feat(master): surface empty disks in the per-physical-disk view
The master records each disk's type and max from DiskTags and lists them
on DiskInfo.physical_disks per type, including disks with no volumes or
EC shards. SplitByPhysicalDisk enumerates that full set and gives each
disk its exact max, so cluster.status, volume.list and the admin
topology count and can target empty disks. Without physical_disks the
even-split fallback is unchanged.
* fix(master): clamp per-disk free at zero for over-allocated disks
In the exact-max path FreeVolumeCount could go negative when a disk holds
more volumes than its max; a negative would reduce the node's summed free
and block placement on healthy disks. Clamp at 0.
* fix(master): rebuild disk tags fresh each heartbeat
DiskTags is the full authoritative per-disk list every heartbeat, so
rebuild dn.diskTags from scratch like dn.diskBackends; merging left stale
entries for removed disks.
* fix(master): keep zero-capacity disks in physical_disks
A disk reporting max 0 (an unavailable disk) is a valid physical disk,
not a signal to drop it. List every disk of the type, but only emit
physical_disks when the node reports real per-disk capacity, so an older
server sending all zeros still falls back to the aggregate split.
* test(volume): cover disk-space-low per-disk max in heartbeat
Assert DiskTag.max_volume_count follows the used-slots override when a
location is low on space, matching the per-type max_volume_counts.
* chore: trim comments on the empty-disk change
Drop narration; keep only the non-obvious why (disk-0 sentinel, exact-max
free clamp, EC slots not subtracted, all-zeros fallback).
* refactor(master): merge per-disk tags and capacity into one map
diskTags and diskBackends were parallel maps keyed by the same DiskId and
filled together from DiskTags. Fold them into one diskMetas map of
{tags, type, max}.
* refactor(proto): per-disk max as a map keyed by disk id
physical_disks was a repeated {disk_id, max_volume_count} whose fields
duplicated DiskInfo's own disk_id/max_volume_count. A map<uint32,int64>
keyed by disk id expresses "max per disk" directly, drops the extra
PhysicalDiskInfo message, and the consumer reads it as the disk set.
* docs(proto): note DiskInfo.disk_id's two meanings
Identity on a per-physical-disk DiskInfo (from SplitByPhysicalDisk),
representative fallback on the type-keyed aggregate.
|
||
|
Chris LuandGitHub
|
a9c0ed91b5 |
fix(topology): keep physical disk 0 distinct in SplitByPhysicalDisk (#10161)
* fix(topology): keep physical disk 0 distinct in SplitByPhysicalDisk DiskId 0 doubles as the first physical disk (Locations[0]) and the protobuf "unset" default. SplitByPhysicalDisk folded every DiskId-0 record onto the aggregate DiskId whenever that was non-zero, so on a multi-disk node the first disk's volumes merged into whichever disk held volumes[0]: the node reported one fewer disk, the sibling showed ~2x volumes, and per-disk max was smeared across the survivors. This surfaced as cluster.status and volume.list undercounting disks. Only treat 0 as unset when no record carries a non-zero DiskId; with a mix, 0 is a real disk and keeps its own entry. * fix(admin): resolve physical disk 0 in active-topology indexes rebuildIndexes re-derived each volume/EC record's physical disk id with the same "DiskId 0 means unset" heuristic SplitByPhysicalDisk used, so the two agreed only by sharing the bug. Now that SplitByPhysicalDisk keeps disk 0 distinct, the duplicated heuristic would fold disk-0 records onto a sibling while at.disks kept them on disk 0; GetVolumeLocations and GetECShardLocations then matched no record and silently dropped every volume and EC shard on the first disk, starving balance and EC tasks. Build the indexes from the same SplitByPhysicalDisk reconstruction that builds at.disks, so the keys always resolve. One source of truth instead of a parallel normalize. * fix(ec): allow physical disk 0 as preferred EC shard target pickBestDiskOnNode gated its result on bestDiskId != 0, but 0 is both a valid physical disk and the uint32 zero value, so a best-scoring disk 0 was discarded and the non-matching fallback returned instead. Gate on bestScore. * test(admin): cover EC-shard index resolution for physical disk 0 rebuildIndexes builds ecShardIndex the same way as volumeIndex; pin the EC path too so a shard on disk 0 keeps resolving via GetECShardLocations. |
||
|
Chris LuandGitHub
|
f643893891 |
fix(master): shed assign load when volume growth is already in flight (#10121)
Under a herd of concurrent assigns with no writable volume, Assign spun PickForWrite for the full 10s timeout, pinning a goroutine per request and starving the master of the cycles it needs to process growth and answer heartbeats. When growth is the relevant remedy and already in flight, stop spinning: if free space exists, shed with a fast retryable error so clients back off and retry once growth lands; if the cluster is out of space, fail fast with the real out-of-space error instead of masking it as retryable. The gRPC shed uses ResourceExhausted, not Unavailable: operation.Assign retries it, but the client connection layer doesn't treat it as a dead channel, so a per-request shed across a herd doesn't tear down the shared master connection and cancel every other in-flight assign. The HTTP dirAssignHandler sheds with 503 + Retry-After. |
||
|
Chris LuandGitHub
|
2efc0e1656 |
ec: recover EC shards whose .ecx index lives only on a peer server (#10108)
* ec: recover EC shards whose .ecx index lives only on a peer server A volume server that boots with EC shard files on disk but no .ecx index on any local disk cannot mount the shards, so the master never learns about them. ec.rebuild works off master-registered shards, so it sees the volume as short and gives up even though the shard data is intact. Add an operator-triggered recovery: VolumeEcShardsMount gains a recover_missing_index flag that makes the volume server fetch the missing .ecx (plus .ecj/.vif) from a peer holding it and mount the on-disk shards. ec.rebuild runs this across the cluster before planning, so orphaned shards register and the rebuild sees the true shard set. .ecx is an immutable encode-time index, identical on every holder. .ecj is a per-holder deletion journal that differs across holders, so the recovered node adopts the source peer's deletion view, like a balanced or rebuilt shard does. * ec: mirror missing-index recovery into the Rust volume server Port the #10104 recovery to seaweed-volume so the Rust volume server self-heals the same layout: EC shards on disk with the .ecx index only on a peer. Adds collect_ec_volumes_missing_index / mount_recovered_ec_shards to the store, recover_missing_ec_indexes (master LookupEcVolume + peer CopyFile fetch + mount) to the server, and the recover_missing_index flag on VolumeEcShardsMount. .ecx is the immutable encode-time index, identical on every holder. .ecj is a per-holder deletion journal, so the recovered node adopts the source peer's deletion view, matching the Go path. |
||
|
Chris LuandGitHub
|
96d2d13efe |
s3: replicate by fanning out from the gateway to every holder (#10078)
* s3: replicate by fanning out from the gateway to every holder The S3 gateway uploaded each chunk to one volume server, which then relayed the copies to the other replica holders. The gateway now uploads each chunk to every holder in parallel (type=replicate), removing the primary volume server's receive-then-resend relay. AssignVolume returns every replica holder (new repeated Location replicas, forwarded from the master assign), the s3api captures them, and the chunked uploader fans out whenever a chunk has more than one holder. Cipher uploads keep the server-driven path since per-call encryption would diverge the replicas. * s3: cancel sibling replica uploads on the first failure * s3: trim replica fan-out comments * s3: roll back successful fan-out chunk copies when a holder fails A failed fan-out records no FileChunk, so copies that landed on the holders that finished before the cancel were leaked as orphans the caller could not see. Track the holders that succeeded and delete the needle from each (type=replicate, local-only) on failure, leaving nothing behind. |
||
|
Chris LuandGitHub
|
63f2f0bef5 |
s3: keep a file promoted to a directory retrievable as an object (#10070)
* filer: treat a directory carrying object data as an S3 key object A file promoted to a directory by a child write keeps its chunks, inline content, or remote-tiered entry. Recognize that as a directory key object, not only when a Mime is set, so the object still lists, demotes on delete, and is not reclaimed by cleanup like the object it still is. * filer: keep the empty-folder cleaner from reclaiming a promoted object The cleaner skips directory key objects, but its check only looked at the Mime. Mirror the chunks/content/remote check so a file promoted to a directory is not deleted once its children are gone. * s3: serve ranged GET for a directory that carries object data Reject only zero-size directories so a file promoted to a directory streams range requests instead of returning 404, while empty directories still 404. * s3: return HEAD metadata for a directory that carries object data HEAD now 404s a directory only when it has no data, so a promoted object is retrievable while empty/implicit directories still fall back to LIST. |
||
|
Chris LuandGitHub
|
7df43ad9b5 |
admin: add connected Mount Clients page and dashboard section (#9968)
* admin: add connected mount clients page and dashboard section
The filer is the authority on who is subscribed to its metadata stream
(FUSE/VFS mounts, S3, peer filers, ...), but its in-memory listener
registry only tracked clientId->epoch and was not exposed.
- Enrich the filer subscriber registry with name/type/address/path/
connected-time, populated in addClient and cleared in deleteClient so
it reflects currently-connected clients only.
- Add a ListMetadataSubscribers filer gRPC (optional client-type filter).
- Admin server fans out to every filer, filters to mount types
("mount" Go weed mount, "sw-vfs" Rust VFS), and renders a new
Cluster > Mount Clients page plus a Mount Clients dashboard section.
Read-only; no behavior change to the subscribe hot path.
* admin: address review — parallelize filer fan-out, guard nil map, robust CSV
- GetMountClients now queries filers concurrently, each under a 5s
timeout, so a slow/unreachable filer can't stall the admin dashboard.
- Defensively initialize fs.subscribers before first write.
- Mount Clients CSV export uses a Blob with quote-escaping instead of a
data: URI, so special characters in paths export correctly.
|
||
|
Chris LuandGitHub
|
284796c7b6 |
fix(ec): fence stale-worker EC shard cleanup by encode generation (#9953)
* feat(ec): add encode_ts_ns to the EC task params, shard-unmount, and shard-delete RPCs The generation fence for stale EC-worker cleanup needs the encode generation on three messages: ErasureCodingTaskParams (admin issues it), VolumeEcShardsUnmountRequest, and VolumeEcShardsDeleteRequest (the worker carries it to the volume server). Additive fields only; 0 preserves the existing unfenced behavior. Mirror the two volume-server fields in the Rust volume server's proto copy. * feat(ec): issue the EC encode generation from the admin and carry it on the worker Stamp each EC proposal's encode_ts_ns from the admin's per-cycle DetectionSequence (a single-clock value) so generations are globally ordered even though detection runs on a rotating worker. The worker writes that generation into the distributed .vif and passes it on its shard unmount/delete RPCs; it falls back to a local timestamp for the .vif only on the unfenced legacy/shell path (keeping the read guard on). * fix(ec): fence the stale-worker EC shard unmount and teardown by generation A reaped-but-still-running EC worker's cleanupStaleEcShards issued a generation-blind unmount + full teardown that could unmount and then overwrite a newer run's live shards on a shared node. Both RPCs now carry the encode generation: the volume server unmounts/deletes a disk only when its .vif generation is strictly older than the request, and preserves a same-or-newer generation, a generation-0 (recovered or pre-upgrade) volume, and an unreadable .vif. Unload is per-disk, never node-wide. Request generation 0 keeps the blanket teardown for the shell pre-encode cleanup and pre-upgrade callers. Mirrored in the Rust volume server. * test(ec): cover the generation-fenced teardown and unmount End-to-end volume-server tests: a fenced FullTeardown wipes a strictly- older generation, preserves a newer one, preserves a generation-0 volume, and blanket-wipes on request generation 0; the gen-aware unmount preserves a same-or-newer mounted generation; and the .vif generation reader handles present/absent/no-config cases. * test(ec): pin the fenced .vif==teardown generation and the unreadable-.vif preserve A fenced run must stamp the admin generation verbatim into the .vif so it matches the generation sent on the teardown RPCs; add a regression test that sets the task generation and asserts the .vif carries it exactly. Also cover the present-but-unparseable .vif case (reads as generation 0, preserved) and correct the readEcGenerationTsNs docstring accordingly. * fix(ec): surface EC full-teardown filesystem errors in the Rust volume server remove_ec_volume_files(_full_teardown) discarded every fs::remove_file error, so a teardown that failed on permissions or a full disk still returned full_teardown_done=true and left stale artifacts to collide with the next encode. Return io::Result, ignore NotFound, propagate the first real error, and have the teardown RPC surface it -- matching the Go contract. The best-effort reconcile/load-cleanup callers keep ignoring it. * refactor(ec): reuse the EC volume lookup on unmount and short-circuit the gen read Address review: the Rust unmount fence reuses the ec_vol it already fetched instead of a second find_ec_volume; the Go .vif generation reader breaks out of the data/idx loop early when the two dirs are the same. |
||
|
Chris LuandGitHub
|
da243b9423 |
fix(ec): group orphan-source completeness by encode generation (topology encode_ts_ns) (#9952)
* feat(ec): carry the encode generation through the topology heartbeat Add encode_ts_ns (field 14) to VolumeEcShardInformationMessage and populate it from each EC volume's .vif identity. The volume server emits it on the full and incremental heartbeats; the master stores it on EcVolumeInfo and re-emits it via GetTopologyInfo, so the admin/worker layer can see which encode run produced each shard set. Field 14 avoids the enterprise fork's reserved 10-13. Mirror the proto field and both heartbeat emit sites in the Rust volume server. * fix(ec): group orphan-source shard completeness by encode generation countExistingEcShardsForVolume ORed EcIndexBits across every disk, so two interrupted encode runs whose shard sets overlap unioned into a false-complete set -- triggering the orphaned-source delete while no single generation was actually complete. Group shards by encode_ts_ns and return the largest single generation's count, so the trigger fires only when one run holds the full set. Shards from pre-upgrade servers (encode_ts_ns==0) form their own bucket. The heartbeat carries one encode_ts_ns per (volume, disk), so this separates generations on different disks; same-disk mixing is prevented upstream by the pre-encode artifact wipe and the cross-run read guard. * fix(ec): guard against a nil Ec shard info entry in the generation count Defensive: a manually-constructed or corrupted topology could carry a nil entry in EcShardInfos. Skip it rather than dereference. * fix(ec): carry the encode generation on the EC shard unmount delta The mount delta sets EncodeTsNs; the unmount deletion delta left it 0. Populate it from the Ec volume before unloading so both incremental deltas are consistent (the Rust volume server already does this via its snapshot diff). |
||
|
Chris LuandGitHub
|
79ac279fe1 |
fix(ec): don't mix EC shards from different encode runs (#9880)
* feat(ec): add encode_ts_ns to EC shard metadata and the shard read RPC EcShardConfig and VolumeEcShardReadRequest gain an int64 encode_ts_ns (encode time in unix nanos). It rides in .vif and the read request so a read can be scoped to the encode run that produced the index. * fix(ec): stamp each encode and reject cross-run shard reads Generate stamps EncodeTsNs into the volume's .vif. Reads carry it to the shard's owning volume (resolved together via FindEcVolumeWithShard, so a multi-disk server validates the disk that actually serves the bytes) and reject a shard from a different encode run, recovering from parity. A zero on either side (pre-upgrade volume) skips the guard. * fix(ec): stamp the encode identity on the worker-generated .vif The worker-local encode path now writes EncodeTsNs (and the resolved EC ratio) into the .vif, so the read guard is not silently off for volumes encoded by the maintenance worker. * fix(ec): wipe stale EC artifacts before re-encoding VolumeEcShardsGenerate evicts any in-memory EcVolume for the volume and removes its on-disk shard/index/sidecar files before writing fresh ones, so a retried encode never builds on a partial prior run and the unlink frees the inodes instead of leaving open fds serving old bytes. * fix(ec): unmount EC shards across all disks UnmountEcShards walked only the first disk holding the shard, leaving a duplicate copy mounted on a sibling disk (split-disk reconciled volumes) still serving and heartbeating. Traverse every disk and emit one deletion delta per disk. * fix(ec): delete orphan shards without a local .ecx deleteEcShardIdsForEachLocation gated shard-file removal on a local .ecx, so it could not clean an orphan .ecNN left by a failed copy on a disk with no index. Delete the requested shard files unconditionally; the index-file (.ecx/.ecj/.vif) routing stays gated as before. * fix(ec): clear stale EC shards cluster-wide before re-encoding ec.encode unmounts and deletes EC shards for the target volumes on every node before regenerating: fatal for the shards the topology reports (mounted leftovers), best-effort for the rest (a sweep that catches unmounted failed-copy orphans). A down node is a no-op. * fix(ec): don't nil EC fds on close so reads can't race eviction A reader resolves an EcVolume/shard under the lock then reads after it is released, so an eviction that nils ecxFile/ecdFile would race that read and panic. Close the fds without nilling the fields: the field is now write-once (no data race) and a concurrent read hits a closed fd, getting a clean error that the caller recovers from parity. * fix(ec): wipe stale EC artifacts on every disk and surface failures The pre-encode wipe only deleted beside the source volume, so a stale shard on a sibling disk survived and could be mounted against the new index at reconcile. Sweep every disk. Removal also ignored os.Remove errors, reporting a failed cleanup as success and letting a stale shard join the next generation; surface the first real failure (treating already-gone as success) from removeStaleEcArtifacts and the shard delete. * fix(ec): log when a local shard is skipped for a different encode run The cross-run guard returned errShardNotLocal, indistinguishable in logs from a genuinely-absent shard. Add a V(1) line naming both EncodeTsNs so operators can tell "wrong encode generation" from "shard not here". * fix(ec): surface metadata removal failures in the shard delete path deleteEcShardIdsForEachLocation still dropped os.Remove errors on the .ecx/.ecj/.vif/sidecar cleanup. A surviving stale .ecx is the orphan-index condition this path prevents, so route those through removeFileIfExists and return the first real failure instead of reporting cleanup as success. * fix(ec): fail orphan cleanup when a reachable node's delete fails The pre-encode orphan sweep swallowed every error for unreported (node, volume) pairs. That is only safe for an unreachable node, which cannot receive this encode's new generation. A reachable node whose delete genuinely failed (permission/IO) keeps an orphan shard that a later copy re-stamps with the new run's volume-level .vif identity, so the read guard would accept stale data. Surface those; stay best-effort only for unreachable nodes (gRPC Unavailable / no status). * fix(ec): guard ecjFile under its lock in the EC delete path EcVolume.Close nils ecjFile under ecjFileAccessLock; a delete that resolved its .ecx lookup before a concurrent eviction (the generate-time UnloadEcVolume) could then reach the journal append with a nil fd. Bail with a clear "volume closed" error under the lock instead. * fix(ec): reject an unstamped shard when the caller has an encode identity The read guard required both identities nonzero, so a current (stamped) caller accepted a holder with identity 0 and could be served a stale pre-upgrade shard. Reject when the caller is stamped and the holder differs (including unstamped); stay lenient only when the caller itself has no identity (pre-upgrade reader). A skipped shard recovers from parity. * fix(ec): full-teardown delete so cluster cleanup wipes a whole generation The pre-encode cluster sweep deleted only the listed canonical shards on remote nodes, leaving index/sidecar (and, on builds with versioned generations, those too) behind. Add a full_teardown flag to VolumeEcShardsDelete that evicts the volume and wipes every EC artifact for it on every disk via removeStaleEcArtifacts; the shell and worker pre-encode cleanup paths set it. Other delete callers (balance/decode/repair) are unchanged. * fix(ec): take ecjFileAccessLock before the nil-check in Sync and Close Sync and Close read ev.ecjFile before acquiring ecjFileAccessLock while Close nils it under the lock, a data race on the field. Take the lock first, then nil-check inside, in both. * fix(ec): acknowledge full_teardown so a pre-upgrade server can't fake success An old volume server silently ignores full_teardown and returns success for an ordinary delete, so the caller wrongly believes the generation was wiped and copies a fresh gen-0 onto an unwiped node. Echo full_teardown_done in the response; the worker destination cleanup fails when it is absent, and the shell cluster sweep fails for a reported (mounted) leftover while staying best-effort for an unreported node. encode_ts_ns stays an accepted transient (an old server just skips the new read guard, no regression). * fix(ec): fail the pre-encode sweep for any reachable node that can't ack teardown A reachable pre-upgrade server ignores full_teardown and returns success without wiping an orphan, which a later copy then folds into the new generation. Treat a missing full_teardown_done ack as fatal for every reachable node (best-effort only for a gRPC-unreachable one), not just for topology-reported pairs. * fix(ec): return the served shard identity and validate it client-side The encode identity was only enforced server-side, so a pre-upgrade server ignored the request field and served bytes unchecked. Echo the served shard's EncodeTsNs on every read response chunk and have the client reject a mismatch (including 0 from an old server), so the guard holds regardless of server version; a rejected read recovers from parity. * fix(ec): reject a short/empty remote shard read instead of serving zeros doReadRemoteEcShardInterval accepted an immediate EOF or a short stream and returned success with a partly zero-filled, unvalidated buffer (the server stamps the identity only on chunks that carry bytes). A non-deleted interval must arrive whole: require n == len(buf), exempting the is_deleted short-circuit (n=0), matching readLocalEcShardInterval's local check. A short read now fails so the caller recovers from parity. * test(ec): fake volume server echoes the full_teardown acknowledgement The worker now fails a teardown delete that isn't acknowledged (so a pre-upgrade server can't silently skip the wipe). The fake server's no-op VolumeEcShardsDelete returned an empty response, which the worker read as a skipped teardown and aborted the encode. Echo full_teardown_done. * feat(ec): mirror the encode-run identity guard + full_teardown into the Rust volume server The Go volume server stamps an encode-run identity (encode_ts_ns) into the .vif and rejects a read served from a shard of a different run; full_teardown wipes a whole generation and acknowledges it. The Rust volume server had none of it. Mirror the shared logic: load encode_ts_ns from the .vif onto the EcVolume, stamp it on every read response, and reject a request/response mismatch on both the server and the distributed-read client (recovering from parity); handle full_teardown by evicting the volume and wiping every EC artifact on each disk, echoing full_teardown_done so the caller can detect a server that ignored it. * fix(ec): remove a stale .vif on full teardown of a shard-only node A shard copy installs shards + .ecx before .vif, so an interrupted copy after a teardown could mount the new files under the previous run's identity / version / shard ratio / dat_file_size carried by the surviving .vif. Remove .vif during full teardown, gated on .idx absence so a source-volume holder keeps its live .vif. In Rust this lives in a teardown-only helper so the reconcile / load- fallback paths (which share the base removal) still preserve .vif. * fix(ec): treat a missing teardown ack as fatal, not as an unreachable node isNodeUnreachable returned true for any non-gRPC-status error, so a reachable pre-upgrade server's missing full_teardown_done ack (a plain error) was classified unreachable and the unreported pair was silently skipped. Classify only a real codes.Unavailable as unreachable, and wrap the missing ack in a sentinel the sweep treats as fatal regardless. A genuinely down node still surfaces as Unavailable from the RPC and stays best-effort. * fix(ec): reject a short shard read in the local EC needle reader read_ec_shard_needle ignored the byte count from shard.read_at and appended the whole pre-sized buffer, so a truncated shard's zero-filled tail passed the later length check and parsed as garbage. Require n == buf.len() per interval, erroring on a short read like the local interval reader already does. * fix(ec): probe reachability before skipping a node that returns Unavailable The pre-encode sweep skipped any node whose teardown delete returned codes.Unavailable, but a reachable volume server in maintenance mode also returns that code for the maintenance-gated delete, so its stale EC files were left behind on a node that can still receive the new generation. Confirm with a non-maintenance-gated empty-target Ping: skip only when the node fails the probe too (genuinely unreachable). * fix(ec): use try_exists for the teardown .vif .idx guard The teardown-only .vif removal gated on Path::exists(), which returns false on a permission/IO stat error, so a stat failure on a present .idx would read as a shard-only node and delete the live source volume's .vif. Gate on try_exists() == Ok(false) instead, preserving the sidecar on any stat error. * fix(ec): only skip a sweep node when a Ping confirms it is transport-down The pre-encode sweep skipped a node whenever its teardown delete and a liveness Ping both failed, but it treated ANY Ping error as down — an application-level Internal/ResourceExhausted, or Unimplemented from a pre-Ping server, left a reachable node's stale generation in place. Classify the Ping tri-state and skip only when it transport-fails with codes.Unavailable; a reachable or inconclusive node stays fatal. * fix(ec): exclude sweep-skipped nodes from the encode's rebalance The pre-encode sweep skips a genuinely-down node best-effort, but the rebalance then recollected the current topology — a node that recovered between the two could become a copy target and receive the new generation while still holding its stale, never-cleared shards. Have the sweep return the skipped set and exclude those nodes from the rebalance for this encode, so a node we could not clean cannot receive the new generation. Standalone ec.balance is unaffected. * fix(ec): re-sweep recovered nodes before generation so they aren't stranded A node skipped as down by the pre-encode sweep is excluded from the rebalance, but it can recover and become the generation host — mounting all shards locally, then being excluded from distribution. Union-only verification accepts all shards on one node and deletes the originals: a single point of failure. Re-sweep the skipped nodes just before generation; one whose teardown now succeeds leaves the skipped set and rebalances normally, while a node still down stays skipped. * fix(ec): abort the encode if a selected source is still skipped after re-sweep The re-sweep un-skips a recovered node, but the source was selected before it and a node can stay down through the re-sweep then recover just in time to be the generation host — mounting all shards locally while still excluded from the rebalance, which union-only verification accepts before deleting the originals. Abort the encode when a selected source remains skipped after the re-sweep. * fix(ec): batch delete returns retriable 503 when a volume became EC mid-batch If a volume is not EC at the batch-delete classification but is encoded to EC and its .dat deleted before the regular-volume mutation, the mutation returns an exact "not found" that the filer chunk-GC treats as completed, dropping the delete. Recheck EC presence under the mutation lock and return a retriable 503 with the "try again" token so the filer requeues it onto the EC path. * fix(ec): recheck EC state before the regular batch-delete mutation ec.encode mounts EC shards (copied from the .dat) before deleting the originals, so a volume can be EC while its .dat still exists. The batch delete only rechecked EC after a NotFound, so a successful regular-volume delete in that window wrote a tombstone to the soon-removed .dat — the delete was lost and the needle resurrected from the pre-tombstone shards. Recheck has_ec_volume under the write lock before delete_volume_needle and return a retriable 503 so the filer requeues onto the EC path. * fix(volume): make the metrics push test independent of test order test_push_metrics_once asserted the pushed body contains the request-counter family without ever touching the counter — a CounterVec with no children emits nothing, so the assertion only held when another test had already created a labelset in the shared registry. Create one in the test itself. |
||
|
Chris LuandGitHub
|
594fc667d5 |
Cut per-subscriber replay decode and widen replay concurrency (#9917)
* Filter metadata events before unmarshaling them per subscriber Every subscriber unmarshaled every log entry into a full event just to run the path filter, and entries carry complete chunk lists, so a fleet of path-filtered subscribers spends almost all replay CPU materializing events it then discards. A shallow wire scan now extracts just the directory, entry names and rename destination into a skeleton event, feeds the same matcher, and skips the decode for entries the subscriber cannot match. Any scan surprise (malformed bytes, merged duplicate message fields) falls back to the full decode, and the unsynced-events heartbeat keeps firing for skipped entries. * Raise the legacy replay cap The cap was sized when every replay pinned a private chunk reader per source filer. Replays now share decoded chunks, so sixteen needlessly serializes subscriber catch-up; the expensive part stays bounded by the cache's load gate. * Weight concurrent log-chunk loads by size The flat eight-load gate let eight tiny chunks through as reluctantly as eight full ones. Charge each load's chunk size against a 128MB in-flight budget instead: small chunks decode wide open while full-size ones still serialize enough to cap the transient peak. Oversized weights clamp to the budget so they can always acquire. * Propagate heartbeat send failures and reset the skip counter A failed heartbeat send means the stream is gone, so end the replay instead of scanning on. A delivered event also resets the skip counter, keeping the heartbeat cadence relative to the last thing the client actually received. * Share the unsynced-events counter across the prefilter and delivery Two independent counters could starve the heartbeat: alternating drops reset each side before either reached its threshold. One shared counter increments on every dropped entry, prefiltered or not, and only an actual delivery resets it, restoring the original cadence exactly. * Tighten comments * Benchmark the subscription match paths For a thousand-chunk event that the subscriber filters out, the shallow scan matches in 10us and 9 allocations against 175us and 4031 allocations for the full decode. |
||
|
Chris LuandGitHub
|
048f9ece2d |
Fix filer metadata-replay OOM under mount reconnect storms (#9901)
* fix(filer): propagate multi-filer metadata log read errors A genuine (non not-found) read error in one filer's log stream was logged and skipped, then the merged cursor advanced past the gap, silently dropping that file's events. Abort the whole replay so the subscriber re-reads from the unchanged position; chunk-not-found still skips. * perf(mount): read persisted metadata log chunks directly from volume servers Set LogFileReaderFn so the filer returns log file references and the mount reads the chunk data itself, instead of the filer reading, decoding, and streaming every persisted entry. Keeps a reconnect storm of many mounts from concentrating hundreds of concurrent log replays in filer memory. * perf(filer): pre-size chunk stream reader buffer to view size The chunk size is known up front, so grow the buffer once instead of letting bytes.Buffer double as the streamed pieces arrive (which transiently overshoots to ~2x per reader). * fix(filer): bound concurrent persisted-log replays Each server-side replay holds an open chunk reader per source filer plus a readahead buffer, so a reconnect storm of clients that predate the metadata-chunks offload multiplies into many GB. Gate replays with a semaphore; abort the acquire when the subscriber's stream is gone so cancelled clients do not pile up parked goroutines. |
||
|
Chris LuandGitHub
|
d67fc48fbd |
fix(filer.sync): guard batched events against nil EventNotification (#9877)
* fix(filer.sync): guard batched events against nil EventNotification The server folds a backlog into one response: the first event in the top-level fields, the rest in resp.Events, and the pipelined sender can drain an idle heartbeat (nil EventNotification) into that tail. Only the envelope got the freshness-signal guard, so a batched heartbeat reached AddSyncJob and nil-derefed in IsEmpty while replaying a backlog buffered during a peer outage. Route every event, envelope and batched, through one handler that sends freshness signals (nil heartbeat, empty marker) to OnIdleHeartbeat. * fix(filer): guard MetaAggregator batched events against nil EventNotification The peer subscription's envelope is nil-guarded but its batched tail was not. The aggregator doesn't enable idle heartbeats today, so the server can't fold a nil EventNotification into the batch yet, but make the two loops consistent so it can't nil-deref if that changes. |
||
|
Chris LuandGitHub
|
755af4adf4 |
s3: actually bind outbound connections when -ip.bind is set (#9849)
* s3: set outbound bind IP before the first filer dial Standalone weed s3 dialed the filer for GetFilerConfiguration before SetOutboundLocalIP ran, so that gRPC conn was created with the stock dialer and no source address. gRPC caches conns by address and reuses the original dialer on reconnect, so the s3->filer connection kept leaving from the OS-chosen source for the life of the process even after the bind IP was set a moment later. * grpc: install the outbound-bind dialer unconditionally The dialer was installed only when OutboundLocalAddr was already set at GrpcDial time, baking the source-address decision into the cached conn, so a conn dialed before the bind IP was configured never bound. Install the context dialer always and decide per dial: bind through OutboundDialContext once a source is set, otherwise fall back to the stock net.Dialer so default deployments keep gRPC's dial timeout and keepalive behavior. The bind now applies on the next reconnect regardless of ordering, matching the HTTP transport's unconditional DialContext. |
||
|
Chris LuandGitHub
|
be7f417a03 |
ip.bind: bind outbound connections to the configured address (#9834)
* ip.bind: bind outbound connections to the configured address -ip.bind only governed listeners; outbound gRPC and HTTP connections let the OS pick the source IP, which may not even be able to reach the target. Mirror the bind address into a process-global source address and apply it to outbound TCP dials: the gRPC context dialer, the per-client HTTP transports, and the default transport. Loopback targets and unix sockets keep the OS-chosen source so same-host traffic still works. * ip.bind: first-write-wins source IP, skip on address-family mismatch Make SetOutboundLocalIP first-write-wins so a `weed server` component's own bind setting (run in its goroutine) can't clobber the process-wide source address the top-level -ip.bind already established for the other components. Skip source binding when the target is a literal IP of a different family than the bind address, since forcing a mismatched source fails the dial. |
||
  
|
e3e02d3364 |
[CheckDisk]: implement disk health detection (#9560)
* [CheckDisk][GRPC]: implement MVP for disk health detection, added timeout for new grpc connections * fix(volume): build disk health check on every platform setDiskStatus only existed behind the statfs build tag, so disk.go failed to compile on windows, openbsd, solaris, netbsd and plan9. Move the timeout wrapper and failure tracking into the shared disk.go and have each platform's fillInDiskStatus return an error, so every platform gets the same protection from a stuck filesystem. Also restore the uint64(fs.Bavail) cast: Bavail is int64 on freebsd, so the unguarded multiply broke the freebsd build. * fix(volume): keep one outstanding statfs probe per disk A stuck statfs used to leave isChecking cleared by the timeout path, so the next check spawned another goroutine while the previous one was still blocked in the syscall, leaking one goroutine per minute on a hung disk. Clear the flag only when statfs returns and treat an overlapping check as a failure, so a hung filesystem keeps a single outstanding probe and still gets reported. * fix(volume): assume disk available until the first health check isDiskAvailable defaulted to false, and CollectHeartbeat skips locations that are not available. A freshly started volume server would therefore omit every volume from its first heartbeats until the async CheckDiskSpace ran, so the master could briefly treat all of them as missing. * fix(volume): label the disk error metric by data directory The new gauge tagged the series with IdxDirectory while every neighbouring resource gauge uses Directory, so the error series would not line up with them in dashboards. Also log the underlying error instead of a generic message. * test(volume): cover disk health success and repeated-failure paths * fix(volume): make a healthy disk the zero-value default Track the disk as isDiskUnavailable instead of isDiskAvailable so the safe state is the zero value, matching isDiskSpaceLow. CollectHeartbeat only skips a location once a check has actively marked it unavailable, so any DiskLocation built without running CheckDiskSpace (tests, future call sites) still reports its volumes instead of silently dropping them. * feat(disk): detect degraded disks using IO latency probes * feat(stats): introduce configurable disk I/O health probe with EWMA-based latency detection * feat(disk): replace EWMA with sliding window algorithm for disk health detection and added user-friendly options * feat(disk): improve disk health probing and recovery * feat(volume): configure disk health checks via volume.toml * fix(volume): Remove disk IO probe CLI options --------- Co-authored-by: ptukha <ptukha@tochka.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris LuandGitHub
|
2386fa550a |
grpc: don't tear down the shared master connection on a caller's own timeout (#9775)
A Canceled/DeadlineExceeded from the caller's per-request context was treated like a dead channel: it closed the shared cached ClientConn and cancelled every other in-flight RPC on it with "the client connection is closing". Under a burst of concurrent chunk assigns (e.g. a large S3 multipart upload) one slow assign hitting its 10s attempt timeout could poison the connection for all the rest, cascading into a flood of 500s. Thread the caller's context into shouldInvalidateConnection and only invalidate on Canceled/DeadlineExceeded while that context is still live, which isolates the genuine stale-channel signal (a peer restart behind a k8s Service VIP). To carry the context, add a ctx parameter to the existing WithGrpcClient, WithMasterClient, and WithMasterServerClient; the master assign and volume-lookup paths pass their per-attempt context and every other caller passes context.Background(). |
||
|
Chris LuandGitHub
|
9658f309d2 |
EC bitrot detection: per-shard checksum sidecars (#9761)
* ec: add EC bitrot checksum protobuf EcBitrotProtection/EcShardChecksums/ChecksumAlgorithm sidecar messages, copy_ecsum_file and unsafe_ignore_sidecar fields, and a CHECKSUM scrub mode. * ec: bitrot checksum sidecar format, validation, and per-volume load Per-shard CRC32C block checksums in an optional <base>.ecsum sidecar with a self-integrity header; validation, rolling builder, backfill primitive, and EcVolume load on mount + removal on destroy. * ec: capture per-shard checksums at encode; verify-and-exclude on rebuild WriteEcFilesWithContext returns the protection computed inline during encoding. generateMissingEcFiles verifies present inputs against the sidecar, excludes corrupt ones, regenerates in place, and re-verifies; fail-closed unless unsafe_ignore_sidecar, removing all generated outputs on failure. * ec: read-only checksum scrub with Reed-Solomon arbiter ChecksumScrub verifies each local shard against the sidecar and reconstructs flagged shards from the clean shards so stale-sidecar false positives are not reported. Wired to the gRPC CHECKSUM mode and ec.scrub -mode checksum. * ec: server-side bitrot sidecar write, copy, cleanup, and opportunistic backfill Write .ecsum at fresh encode; propagate it with copy_ecsum_file (tolerant); remove it on full delete and decode; rebuild honors unsafe_ignore_sidecar and opportunistically backfills a sidecar when all shards are reachable. * ec: volume server bitrot config flags -ec.bitrotChecksum (default on) and -ec.bitrotBlockSizeMB (default 16). * fix(ec_bitrot): bound -ec.bitrotBlockSizeMB before the int64 multiply Validate the MiB value is in [1, 1024] before multiplying by 1 MiB, so a huge flag value cannot overflow int64 and slip past the power-of-two check, and a block size cannot collapse a sidecar to a few oversized blocks. * fix(ec_bitrot): distribute the .ecsum sidecar from the worker encode path The worker EC encode wrote the generation-0 sidecar locally but never added it to shardFiles, so DistributeEcShards never shipped it and the distributed holders came up unprotected. Append it to shardFiles and map the ecsum shard type to its extension in the sender so it travels with the shards. * fix(ec_bitrot): remove orphaned sidecars when the generation is gone Gate sidecar removal on existingShardCount==0 alone rather than also requiring a stray .ecx. A sidecar whose shards have all been deleted is orphaned and must be removed even when no .ecx remains, or it leaks. .ecx/.ecj/.vif removal stays gated on hasEcxFile as before. * fix(ec_bitrot): do not fold checksum blocks scanned into TotalFiles ChecksumScrub's first return is blocks scanned, not files. Discard it so the scrub response's TotalFiles (a needle/file count) is not inflated by the block count for CHECKSUM mode. * test(ec_bitrot): clean up generated .ecsum sidecars in removeGeneratedFiles * fix(ec_bitrot): reject an oversized sidecar payload before the uint32 cast The header stores payload_len as a uint32; bound the payload before the conversion so a pathological manifest cannot truncate the length field and corrupt the sidecar. A real manifest is a few KB, so this never trips. * fix(ec_bitrot): cap -ec.bitrotBlockSizeMB at 64 MiB The block size becomes the per-shard scratch buffer the scrub/backfill path allocates, so an over-large value (e.g. 1 GiB) is a memory hazard per concurrent scrub worker. Lower the upper bound from 1024 to 64 MiB. * fix(ec_bitrot): add -ecUnsafeIgnoreSidecar to weed tool fix -ecx The -ecx recovery path reconstructs missing shards via RebuildEcFilesWithContext, which fails closed on a malformed/stale .ecsum. Without an override flag an operator could not complete the rebuild without manually deleting the sidecar. Expose -ecUnsafeIgnoreSidecar (default false) and thread it through. * fix(ec_bitrot): bound sidecar payload with a direct int constant; drop readFull Guard len(payload) against a plain int constant (1 GiB) before the allocation instead of a uint64 MaxUint32 compare, so the allocation-size value is provably bounded (clears the CodeQL overflow alert) and the math import is no longer needed. Inline os.File.ReadAt with io.EOF handling in verifyShardFileBlocks and remove the now-redundant readFull helper (os.File.ReadAt fills the slice or errors). * test(ec_bitrot): use slices.Contains instead of a hand-rolled containsU32 * refactor(ec): fold the EcFiles WithContext variants into the base functions RebuildEcFiles now takes the *ECContext directly (nil => derive from .vif as before) and WriteEcFiles takes it too (nil => default), removing the parallel RebuildEcFilesWithContext / WriteEcFilesWithContext names. Callers that had an explicit context drop the WithContext suffix; the default-context callers pass nil. No behavior change. * refactor(ec): pass BackgroundECContext instead of nil to Write/RebuildEcFiles Add a non-nil BackgroundECContext placeholder (analogous to context.Background()) and have callers with no specific layout pass it instead of a nil *ECContext. WriteEcFiles resolves a zero/background context to the default ratio and RebuildEcFiles resolves it from the .vif, so behavior is unchanged. * fix(ec_bitrot): make BackgroundECContext a func; RebuildEcFiles fails closed on bad .vif - BackgroundECContext is now a function returning a fresh *ECContext, so callers cannot mutate a shared singleton or race on it (and it mirrors context.Background, which is also a function). - RebuildEcFiles now propagates the MaybeLoadVolumeInfo error: a present-but- unreadable .vif fails closed instead of silently rebuilding with the default ratio (which would corrupt a custom-ratio volume). Pass an explicit ctx to override. |
||
|
Chris LuandGitHub
|
c9623007a2 |
fix(filer.sync): keep sync_offset fresh through filtered-event markers (#9733)
On a read-only watched path the idle heartbeat keeps sync_offset fresh, but a busy source filer still emits a MaxUnsyncedEvents marker after many filtered events. The marker has a non-nil but empty EventNotification, so the client routed it to the event path, where it advanced no real watermark yet drove offsetFunc to republish the stale processed watermark — regressing the gauge between heartbeats and spiking the derived lag every time a filtered-event burst landed. Route the empty marker through OnIdleHeartbeat like the idle heartbeat so its fresh timestamp keeps the gauge current; it still advances the in-stream resume cursor. |
||
|
Chris LuandGitHub
|
a3c0baa9b0 |
filer: cooling-off dual-read for POSIX locks during ring changes (#9672)
While the ring changed within the last snapshot interval, a fresh owner asks the key's previous owner (LockRing.PriorOwner) whether it still holds a conflicting lock before granting TRY_LOCK or answering GET_LK, so it does not double-grant before re-assertion rebuilds its local state. The probe is marked cooling_probe so the previous owner answers from local state without recursing. PriorOwner uses the snapshot's prebuilt ring rather than rebuilding a hash ring per call. |
||
|
Chris LuandGitHub
|
f8caaa4464 |
mount,filer: re-assert POSIX locks via keepalive (ownership migration + restart) (#9668)
* mount: renew POSIX lock leases via keepalive The mount tracks the inode keys it holds locks on and a background loop renews its session lease (KEEP_ALIVE) with each key's owner filer every 5s, within the filer's 15s TTL. A live mount is never reaped; a dead one stops renewing and owners reclaim its locks. Tracking is a superset: holds are added on grant and dropped only on owner release, so a still held lock is never under-renewed. * mount,filer: re-assert held POSIX locks via keepalive The owner filer holds POSIX advisory locks as in-memory soft state, so a key's owner change (ring rebalance) or an owner restart lost or stranded them: the new or restarted owner was blind to existing holders and would double-grant. Make the keepalive carry the mount's held lock ranges per key. The mount mirrors its own granted locks (posixOwn), and each tick re-asserts them to the key's current owner, which rebuilds that session's locks from the assertion — self -healing after a takeover or restart. The owner arbitrates re-asserted locks against other sessions so it never double-grants; a lock that lost a migration race is reported, not forced. A bare keepalive (no ranges) still just renews. |
||
|
Chris LuandGitHub
|
c97b69f8a4 |
filer: session lease + reaping for POSIX locks (#9666)
* filer: session lease + reaping for POSIX locks A mount renews its session lease by keepalive (new KEEP_ALIVE op); the owner filer records last-seen per session and a background sweeper reaps the locks of leased sessions that stop renewing — a dead or partitioned mount. Only sessions that have renewed are leased, so this is inert until mounts run with -posixLock. * mount: route POSIX advisory locks to the owner filer (-posixLock) (#9665) mount: route POSIX advisory locks to the owner filer under -dlm With -dlm, GetLk/SetLk/SetLkw and the flush/release cleanup paths go to the inode's owner filer via the PosixLock RPC instead of the local table, so flock/fcntl are honored across mounts. Advisory locking rides the same switch as whole-file write coordination — and is therefore off under writeback cache, which implies single-writer. The mount calls its filer and relies on filer-side forwarding to reach the owner. Keys are the inode identity (HardLinkId else path); SetLkw is client-side polling with the FUSE cancel channel (no server wait queue); a per-mount session id namespaces owners; a local hint avoids a release RPC on every close. * mount,filer: bound posix-lock release RPCs and stop the reaper on shutdown The unlock/release RPCs run off the syscall path (close/flush) and used context.Background() with no deadline, so a slow or unreachable filer could hang close() indefinitely; bound them to 5s (they still aren't cancelled by an interrupt). The lease-reaping sweeper now selects on a stop channel that FilerServer.Shutdown closes, instead of looping for the process lifetime. |
||
|
Chris LuandGitHub
|
fef49c2d75 |
filer: routed PosixLock RPC over the in-memory authority (#9664)
* filer: in-memory POSIX lock authority (Manager) Concurrent multi-inode authority over the per-inode Set: a Set per opaque inode key (path, or hl:<HardLinkId>) plus a session->keys index so a dead mount's locks reap in O(locks held). Lock state stays in memory like the distributed lock manager's, off the replicated meta-log. TryLock/Unlock/ GetLk/ReleasePosixOwner/ReleaseFlockOwner/ReleaseSession; empty sets and stale index entries are pruned on release. * filer: routed PosixLock RPC over the in-memory authority Adds the PosixLock RPC (try/unlock/get_lk + the flush/release owner drops) that the owner filer answers from its in-memory Manager. The request key is the inode identity ring key; a non-owner filer forwards one hop (is_moved-bounded), mirroring ObjectTransaction, so the owner's table stays the single authority under a stale ring view. Strictly non-blocking; SetLkw polling lives in the mount. |
||
|
Chris LuandGitHub
|
2a4923e7e8 |
ObjectTransaction: filer-side forwarding via route_key (#9659)
A non-owner filer forwards the whole transaction to the ring owner of route_key, so the owner's per-path lock stays the single serialization point even when the caller's ring view is stale. is_moved bounds forwarding to one hop. The gateway stamps route_key on every routed builder via the shared objectRouteKey helper. Completes taking S3 object mutations off the distributed lock. |