mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-08-28 20:06:14 +00:00
eb3bbfeb1fffa1447c7924dc9e7c0fc8f69a6a61
14916
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Chris LuandGitHub
|
eb3bbfeb1f |
filer: apply the path's storage rule TTL on every write path (#10963)
* filer: cover the storage rule TTL on the object transaction write path An object written through ObjectTransaction used to land with ttlSec 0 even under an fs.configure TTL rule, while the same object written through CreateEntry got the rule's TTL. Guard the shared stamping so the two paths cannot drift apart again. * filer: apply the path's storage rule to an appended entry AppendToEntry resolved the storage option from the path - so its chunks land on a TTL volume under an fs.configure TTL rule - but never stamped the rule's TTL on the entry it creates, leaving an entry that outlives its data. Route it through applyStorageDefaultsToEntry, which now feeds the entry's own TTL into the option so the placement an existing entry's appended chunks get is unchanged. * filer: apply the path's storage rule to a completed TUS upload The PATCH path resolves the storage option from the target, so a TUS upload into an fs.configure TTL prefix writes its chunks to a TTL volume, but completion built the final entry with ttlSec 0 - the entry outlived the data it pointed at. Stamp it through applyStorageDefaultsToEntry, which also subsumes the hand-rolled read-only check and supplies the rule's name-length limit. * filer: apply the destination's storage option TTL to a copied entry The copy handler re-uploads the source's chunks under the destination's storage option, so a copy into an fs.configure TTL prefix already lands its data on a TTL volume. The entry, though, carried the source's ttlSec - 0 for a source outside the prefix, or the source's own TTL where the two rules differ - so it never expired with the data it pointed at. Take the TTL from the same option the chunks were placed with, after the data-only copy has restored the destination's metadata. |
||
|
Chris LuandGitHub
|
a02c0024e5 |
master: cap the reported capacity at what the disks hold (#10960)
* master: cap the reported capacity at what the disks hold Statistics reported max volume count times the volume size limit, which is how many volumes the cluster is allowed to place, not how much space it has. A cluster given far more slots than its disks can fill reported a capacity it could never reach -- 65536 slots at 30GB read as 1.9PB on a 460GB disk -- and the number never moved, since writing data changes neither the slot count nor the size limit. The volume servers already report each filesystem's total and free bytes in their heartbeats, so bound the answer by what they say is left. * mount: keep the last known sizes when filer statistics fails A failed Statistics call returned before df's answer was filled in, so a mount whose filer or master was briefly unreachable reported an empty filesystem rather than the sizes it already had. * master: drop the disk ceiling when a volume server does not report A cluster part way through an upgrade has volume servers that predate the disk bytes in the heartbeat. Summing only the ones that answered left the quiet server's free space out of the total, and the server holding the room is exactly the one that could make the cluster read as full. Answer with the disks only when every one of them reported. |
||
|
Chris LuandGitHub
|
b77d954f55 |
rust volume: fail closed on sorted-index failures and reconcile tier-up (#10956)
* rust volume: fail closed on sorted-index failures and reconcile tier-up Follow-ups to the .sdx sorted needle map (#10951): - get() folded open/read failures into None, so an EIO, a torn .sdx, or a failed pooled reopen answered reads with NotFound and let do_delete_request acknowledge the delete as Ok(0) without writing a tombstone. It now returns io::Result and every caller propagates; redb's get() had the same shape and is fixed with it. is_file_unchanged cannot propagate, so it reports unknown and logs rather than treating an unreadable index as proof of a change. - A delete whose .idx append landed but whose .sdx mark failed left the map still resolving the old live entry, so deleted content stayed readable until a reload. The map now records the tombstone before touching .sdx and only clears it once the mark lands; lookups consult that first and report the needle deleted, which is what the next reload concludes anyway. - Mode reconciliation ran one way. Entering remote mode made use_sorted_index() true, which returned early, so a volume tiered while the server runs kept its in-memory map and pinned .idx descriptor until restart — the RAM and fd win never applied. It now reconciles in both directions. - Tier-down dropped the remote reference before the fallible refresh, so a failure left volume_info local, the remote backend attached, the .vif still remote, and a retry reporting "already on local disk". The transition is snapshotted and rolled back. - The read-only fallback set no_write_or_delete but left no_write_can_delete, so metrics and mode checks called the volume delete-capable while every delete was refused. * rust volume: count a sorted-map delete against the durable .idx append The deletion counters sat after the in-place .sdx mark, so a mark that failed left them at their pre-delete values while the tombstone was already durable in .idx — and with retries now idempotent, nothing applied them later either. Heartbeats, status responses, and the garbage calculation would report the volume as free of that garbage until a reload. Move them to the append that makes the delete durable, which is also what a reload of .idx would count. Covered by a test that injects a mark failure through a cfg(test) seam: no portable filesystem trick reproduces it, since a read-only .sdx fails the borrow long before the mark. * rust volume: hide a pending tombstone from the sorted-map scans too The overlay that keeps a needle deleted after a failed .sdx mark was only consulted by get(). visit_live_entries still read the stale valid record straight off .sdx, so ascending_visit, iter_entries and save_to_idx all reported the needle live — and compaction takes iter_entries for the complete live set, so it would copy the deleted content forward and save_to_idx would write it back into the rebuilt .idx as live. Snapshot the overlay once per scan and skip its keys, which is the same conclusion the next reload reaches from the .idx tombstone. * rust volume: quarantine a durable write whose index lookup fails The prior-mapping lookup that decides whether to index a fresh append runs after the record is already down and flushed, so a failing lookup leaves exactly the state a failing put leaves: a durable .dat record nothing indexes. The put path marks the volume read only for it; this one returned the error and kept taking writes, and the next append would bury the orphan mid-file where the .dat tail check on reload cannot see it. Give it the same treatment. |
||
|
Chris LuandGitHub
|
7658305c76 |
mount: name the disk after the mounted path (#10958)
* mount: name the disk after the mounted path Finder and Explorer labelled every mount with the filer address, so two mounts from one filer were indistinguishable. Use the mounted path's last segment, the way df already shows it, and keep the filer address only for a whole-tree mount. * mount: let a given mount option override the default The options from -o were placed before the ones this mount derives, so a volname or iosize given on the command line lost to the derived value. Append them last, matching the Windows adapter. * mount: document what labels the disk |
||
|
Chris LuandGitHub
|
627b5e9d59 |
shell: parse every collection filter the same way (#10955)
* worker: move the collection filter parser into weed/util/wildcard
The parser sits beside the volume-list filtering it was written for, in
weed/plugin/worker, which imports weed/shell — so the shell commands that
parse the same filter three other ways can never call it. Move it down to
weed/util/wildcard, next to the comma-separated wildcard helper it already
replaced, leaving the behavior unchanged.
* shell: parse every collection filter the same way
The shell parsed a collection filter three ways: compileCollectionPattern
compiled one regex for ec.encode, ec.decode, volume.balance and the tier
commands; volume.list and volume.deleteEmpty matched a single wildcard; and
volume.tier.move, volume.fix.replication and volume.configure.replication
called filepath.Match on their own. None of them took a list, so
"ec.encode -collection=a,b" selected nothing, the same way the admin UI did.
They all go through the shared matcher now: a comma-separated list of names,
"*" and "?" wildcards, "_default" for the collection with no name, and regex
entries. The one thing that stays per-command is what an empty value means -
every collection for -collectionPattern, the unnamed collection for the ec
and tier -collection flag - so compileCollectionPattern keeps that mapping.
The matchers are compiled once per command instead of once per volume, and a
regex entry now has to match the whole name unless it anchors itself, so
-collection=bucket no longer picks up mybucket2.
* shell: keep dots in collection names, and commas inside a regex
A dot no longer marks an entry as a regex, so a collection named "my.bucket"
matches itself and not "my-bucket" - the difference decides which volumes
volume.deleteEmpty and volume.tier.move touch. A dot still counts when it is
quantified, so "bucket.*" stays a prefix regex.
The comma split also leaves alone the commas inside a character class or a
repetition count, so "bucket[0-9]{1,3}" stays one entry instead of becoming
two broken fragments.
* shell: let a regex entry match its own spelling
A collection named after regex syntax, say "logs(2024)", was unreachable:
the entry compiled to a pattern that matches "logs2024" instead. Match the
entry verbatim as well, so naming a collection always selects it, whatever
characters it holds.
* shell: reject a collection filter that names no collection
A value of "," parsed to no entries and then matched every collection, so a
typo widened ec.encode or volume.deleteEmpty to the whole cluster. Only a
genuinely empty filter means "all collections"; anything else has to name one.
* shell: keep commas inside a regex group out of the entry split
The split already left alone the commas inside a character class or a
repetition count, but not the ones inside a group, so "bucket(foo,bar)"
was cut into two fragments that no longer compile.
* shell: cover escaping a collection name that is not a regex
A name like "logs(2024" does not parse as a regex on its own; escaping it,
"logs\(2024", reaches it. Pin that so the escape hatch does not regress.
* shell: split entries only on commas inside a closed regex construct
An unmatched "{" or "[" made the splitter swallow every comma after it, so
"foo{bar,videos" became one entry that matches neither collection - the
silent no-op this filter work exists to remove. A construct now has to close
before its commas stop separating entries.
* shell: skip character classes while scanning a regex group
A ")" inside a class is a literal, so "(a[)],b)" ended its group early and
split into two fragments that no longer compile.
* shell: cover escaping a comma inside a collection name
A comma separates entries, so a name holding one is reached by escaping it.
* shell: follow the regexp parser when scanning a character class
A "]" leading a class is a member of it, and a POSIX class such as
"[:alpha:]" carries its own "]", so stopping at the first one cut a valid
filter like "(a[]),],b)" into fragments and rejected it.
|
||
|
Chris LuandGitHub
|
368b2035b2 |
s3: deny anonymous access when the identity config loads no identities (#10954)
* s3: deny anonymous requests when the identity config loads no identities Naming a config file is the operator asking for authentication. A file that yields no identity - an unpopulated secret mount, or a mistyped top-level key the proto parser silently drops - left the gateway open to every anonymous caller: ListBuckets returned 200, and anonymous PUT could create buckets and write objects. * s3: name the unknown top-level keys in an identity config The proto parser discards what it does not recognise, so a mistyped "identites" loads as an empty config. Naming the dropped keys at startup turns the resulting lockout into a one-line diagnosis. * s3: isolate the auth-enforcement tests from AWS environment credentials * s3: use a singular "identity" as the unrecognised-key example Codespell rejects the misspelling the example used. * s3: cover the empty identity config alongside the unrecognised key * s3: cover a config file whose body is an empty object |
||
|
Chris LuandGitHub
|
b58d52ac16 |
rust volume: search .sdx for read-only volumes instead of holding the index (#10951)
* rust volume: search .sdx for read-only volumes instead of holding the index The Go volume server loads every read-only volume through SortedFileNeedleMap: the index lives on disk as a sorted .sdx, a lookup is a binary search, and since #10950 no descriptor is held between lookups. The Rust server had no counterpart. Read-only volumes built a full in-memory CompactNeedleMap, and cloud-tiered ones — noWriteCanDelete, so not the read-only branch — went through the writable path and pinned an .idx append handle on top of it. At the hundreds of thousands of tiered volumes a real server carries, that is an index in RAM and a descriptor each, for volumes nobody reads. Port the sorted map and the bounded handle pool. A tiered volume now costs zero descriptors and zero index bytes when idle; the pool keeps the hot handles open so a busy volume does not pay an open() per needle. Handles are Arc<File>, so an eviction cannot close one a reader still holds. The generated .sdx is byte-identical to Go's — same sort, same last-write-wins, same dropped tombstones — so a volume moved between a Go and a Rust server reads whichever copy is already on disk. A test pins the bytes against a Go-generated fixture. * rust volume: fail compaction on an unreadable .sdx, and rebuild the map on tier-down Two ways the sorted map could lose data. iter_entries swallowed read errors and returned however many entries it managed to collect. Compaction takes that vector for the complete live set, so a truncated .sdx or a mid-scan I/O fault would commit a volume missing every needle past the failure. Return a Result instead and abort. redb's collect_entries dropped errors the same way on the same path, so it goes with it. Tier-down clears the remote mode and publishes the volume as writable, but the map it booted with is the read-only sorted one. Its put always fails, so the first write would append to the local .dat and then fail to index it, leaving bytes nothing references — and a non-fsync write repeats it. Fold the reopen_idx_for_write swap into refresh_remote_write_mode so the map always matches the mode it just published; a rebuild that fails pins the volume read-only rather than letting it take writes it cannot record. Go reaches neither: its tier-down leaves noWriteCanDelete set, so the volume stays read-only until a reload or an explicit mark-writable, which already goes through reopenIdxForWrite. * rust volume: keep read-only volumes mountable on a read-only index dir, and batch the .sdx scan Building .sdx writes to the index directory, and load_index_sorted_file also created a missing .idx there. A volume whose index sits on a read-only mount took both paths and failed to load, where before it mounted read-only off an in-memory index and served reads. Create the .idx only where deletes are allowed, and fall back to the in-memory map when the sorted one cannot be built, so a directory nobody can write costs memory rather than availability. The end-to-end scan behind iter_entries, ascending_visit and save_to_idx read one entry per syscall. Read 1024 at a time instead, the batch size idx::walk_index_file uses. Positional reads, not a cursor: the handle is shared with any other borrower. Also gate the Go byte-parity fixture on the 5bytes feature it describes, which is otherwise dead code in a 4-byte-offset build. * rust volume: roll back a failed writable mark, and rebuild a torn .sdx set_writable clears the read-only flags before it can know the rest will succeed, but only the map rebuild rolled them back. An .idx writer that fails to attach left the volume advertising writable over a needle map with no writer, so puts landed in memory and were gone after a restart — the exact failure the function exists to prevent. The read-only-mount fallback made it reachable: that path loads an in-memory map with no writer attached. All three steps now run behind one rollback point. A .sdx whose length is not a whole number of entries was accepted as long as it looked fresh, and truncation is what makes it look fresh. The entry count then floored, hiding the last needle from lookups and from compaction, which would commit the shorter set. Treat a torn file like a stale one and rebuild it from .idx. Go writes .sdx in place rather than through a temporary, so a crash mid-generation is a real way to produce one. Appends now start at the last whole .idx entry too, so a torn tail there is overwritten by the next tombstone instead of misaligning every row after it. * rust volume: trim a torn .idx before writing to it, keep delete-only volumes online, count sorted-map deletes Three from review. Flooring the sorted map's append offset only protected its own positional writes. Every writable path appends at EOF instead, so a partial row left by a short write pushed the next row off alignment and the following load parsed the rest of the file as garbage. Drop the partial row before attaching any writable index writer — it is unrecoverable anyway, and every loader already skips it. Go refuses to load such a volume at all; trimming keeps it mountable with the rows before the tear intact. The unwritable-index-dir fallback stopped one step short for volumes that allow deletes, which is every tiered one: the in-memory loader opens .idx read-write there and fails on the same directory that just refused the .sdx, so the volume stayed offline. Give up the deletes instead — without a writer no tombstone could be recorded anyway — and a remount on a writable directory restores them. Sorted-map deletes left the counters untouched, so a tiered volume reported itself garbage-free until it restarted. They now land where a reload would put them: the tombstone is another .idx row, and both it and the row it supersedes count as deletions under the rule the load-time metric applies. Go skips this too, and should not. |
||
|
Chris LuandGitHub
|
e482e67971 |
admin: accept a list of collections in the task collection filter (#10953)
The collection filter was parsed twice with two syntaxes: the master-side volume listing compiled the whole string as one regex, while EC encode and EC balance detection split it on commas and matched each entry as a wildcard. A volume had to pass both, so "collection-a,collection-b" matched nothing (no collection is named that), and the ALL_COLLECTIONS sentinel, which the master side skips, dropped every volume at the task side. Parse it once, in one place: a comma-separated list where an entry is a name with optional * and ? wildcards, or a regex when it carries regex syntax. A regex entry now has to match the whole name unless it anchors itself, so listing a collection no longer picks up its longer namesakes. |
||
 
|
ef4c9d9178 |
filter volume by local or remote storage name (#10946)
* filter volume by local or remote storage name Signed-off-by: lou <alex1988@outlook.com> * fix SelectsEverything Signed-off-by: lou <alex1988@outlook.com> * keep the proto sync out of this change The branch copied weed/pb/*.proto over their seaweed-volume and Java counterparts and regenerated every .pb.go with a different protoc and protoc-gen-go-grpc. DiskStatus.error arriving that way broke the Rust build, and the rest is toolchain churn in files this change has nothing to say about. --------- Signed-off-by: lou <alex1988@outlook.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris LuandGitHub
|
70c3adb983 |
volume: stop read-only volumes from pinning .idx and .sdx (#10950)
A read-only or cloud-tiered volume loads a SortedFileNeedleMap, which held both its .idx and its .sdx open for the life of the process. On a server with ~600K tiered volumes that is 1.2M descriptors before a single read, enough to exhaust the fd limit and take the listeners down. The .dat is not the problem: a tiered volume serves it from the remote backend. Neither index file is needed except while a lookup is in flight, so borrow them from a bounded process-wide pool instead. An idle volume now holds zero descriptors; a busy one keeps its handles hot rather than paying an open() per needle. Reads borrow O_RDONLY, so a volume on a read-only mount answers lookups that previously failed at load. Sync tracks whether a tombstone was appended, which also drops the fsync-per-volume storm at shutdown. |
||
|
Chris LuandGitHub
|
b77431c142 |
master: stop hintless small-file assigns from marking volumes full (#10944)
* master: estimate a hintless assign's size from the volume's average file size An assign that carries no dataSize hint charged a flat 1MB per file id against the volume's effective size. A small-file workload overpays by orders of magnitude: bulk-writing 4KB files marks volumes holding a few hundred MB of real data as crowded and then full, so the master grows unnecessary volumes and, once every volume is spuriously full, fails all assigns. Estimate from the volume's own average file size instead, and keep the 1MB fallback only for volumes with no history. * master: decay pending assign sizes for volumes gone quiet The decay that corrects pending assign estimates runs only when a heartbeat reports the volume, and a heartbeat only reports a volume whose content changed. A volume held out of the writable list takes no writes, so once inflated estimates mark every volume full, nothing is ever reported again, nothing decays, and the cluster refuses all writes until a restart. Run the decay from the master's periodic loop for volumes no heartbeat has reported within two pulses, feeding the last reported size back through the same path an unchanged heartbeat would take. * master: trim the comments on the assign size estimate * master: keep the periodic decay out of the replica-dedup window UpdateVolumeSize ignores a report arriving within two seconds of the last one, so replicas of the same volume do not each halve the pending estimate. The periodic decay went through the same path and stamped that window, so a real heartbeat landing right behind it was dropped along with its reported size and compact revision. Only a volume whose content changed is reported at all, so nothing would send that size again and the master kept a stale one. Let the dedup window belong to volume server reports alone. * master: let the decay read the size record under the lock it mutates The periodic decay picked its volumes under a read lock and replayed them under a write one, carrying the size it had read across the gap. A heartbeat landing in between was rolled back: the replay wrote the older size and compact revision over the fresh ones, and a compaction report lost that way is never resent, since only a volume whose content changed is reported. The decay has no size of its own to contribute, so it now reads the record under the same lock it mutates. * master: let a heartbeat that beat the decay stand for the cycle The decay chooses its volumes under a read lock and applies them under a write one. A heartbeat landing in that gap already did the halving the cycle owed, so applying the decay on top of it halved twice and forgot pending bytes the volume has not written yet - the double-halving the replica-dedup window exists to prevent. Both callers now give way to a report already handled for this cycle; only a real report still advances lastUpdateTime, so a quiet volume keeps decaying every pulse. * master: keep genuinely full volumes out of the decay pass A volume the disk really did fill keeps its fullSince set for good, so it was selected every pulse for a decay that cannot help it: UpdateVolumeSize refuses to recover a volume whose reported size is at the limit, and replaying a size that cannot move leaves the record as it found it. Full and quiet is the ordinary resting state of a cluster, so this was most of the pass, taking the layout write lock away from the heartbeats to do nothing. On a million tracked volumes with a hundredth of them phantom-full it costs ten thousand write locks a pulse instead of a million. * master: put the stale-replay test back on the path it guards Giving the decay the dedup window left this test short-circuiting there, so it no longer reached the locked read it was written for and passed with that read removed. Age the record past the window, which is the only case where reading it under the lock is what saves the report. |
||
|
Chris LuandGitHub
|
50b388771a |
s3: stop one abandoned request from cancelling every concurrent upload (#10948)
* grpc: a non-cancellable context is no evidence of a stale channel shouldInvalidateConnection only invalidates on Canceled/DeadlineExceeded while the context handed to WithGrpcClient is still live, so that an RPC timing out on its own does not close the shared cached ClientConn and cancel every other in-flight RPC on it. context.Background()/TODO never expire, so Err() stays nil forever and that guard always answered "invalidate" - and Background is what almost every caller passes, the S3 gateway included. One S3 request whose RPC rode an abandoned HTTP request context therefore closed the shared filer connection, and every multipart part in flight died with "the client connection is closing", surfacing to the client as 400 InvalidRequest. Only a cancellable context bounds an RPC attempt, so require one before reading it. A genuinely stale channel (a peer restart behind a stable L4 endpoint) surfaces as Unavailable, which invalidates on its own branch. * grpc: a bystander of a connection teardown is not a stale-channel witness gRPC raises ErrClientConnClosing locally, before an RPC reaches the wire, when this process has already closed the ClientConn. Every caller that touches a channel during another goroutine's teardown gets it, so reading it as a stale-channel signal lets one teardown re-arm itself across the whole herd of callers it just cancelled. The cached-connection version check keeps those callers from closing a replacement channel, but the streaming path invalidates by address alone and has no such guard. * grpc: end a stream without dropping the peer connection under it A streaming caller gets its own ClientConn, but on any error it also drops the cached non-streaming ClientConn every request handler shares with that peer, to recover a peer restart hidden behind a stable L4 endpoint. Any error includes the ordinary ones: a metadata subscription that reached its stop point, a follow callback that refused an event, a caller that gave up. The S3 gateway follows filer metadata on such a stream and reconnects forever, so each ordinary end of it cancelled every S3 request in flight against the filer. Drop the shared channel only for errors that say the peer went away, which is what invalidation is for. * test: close the connections the cascade tests leave cached Each test swaps in a fresh connection cache and restores the previous one, dropping its own entries without closing them, so the ClientConn's transport and reconnect goroutines outlive the fake filer they dialed. * grpc: say why ErrClientConnClosing's deprecation notice does not apply It points at codes.Canceled, which is the code this function exists to disambiguate. Only the message distinguishes a teardown a caller merely walked into, so the sentinel stays. |
||
|
Chris LuandGitHub
|
44115c1051 |
filer: stop TUS uploads from turning into garbage (#10945)
* filer: store TUS sub-chunks through the regular chunk writer A TUS sub-chunk was written with one assigned file id, retried up to three times against that same id, and abandoned on failure: an attempt that had landed on some replicas left a needle no session record and no entry ever references, unreclaimable by vacuum. dataToChunkWithSSE, which the regular write path uses per chunk, assigns a fresh file id per attempt and hands back the file ids of failed attempts, which are now freed the way the regular write path frees them. * filer: retry a chunk write on a fresh volume when the server 5xxs The filer's chunk writer assigns a fresh file id per attempt but only retried transient network errors, so a volume filling up and turning read-only mid-write failed the whole request even though the very next assignment would have landed elsewhere. Every other write client already routes this through ShouldReassignUpload; the filer's own write path now does the same, for regular uploads and TUS sub-chunks alike. * filer: export the chunk deletion queue The filer test harness in weed/server builds filer.Filer as a struct literal, so any code path reaching DeleteChunks dereferenced a nil queue. Exported like the neighboring DeletionRetryQueue so the harness can arm it. * filer: complete a TUS upload whose chunk records overlap A PATCH retried while its predecessor was still storing a sub-chunk - a proxy timeout with an immediate retry is enough - records the same range twice. HEAD computes Upload-Offset as the covered watermark and reported the upload fully received, but completion demanded exactly adjacent records and failed every attempt: the client concluded success from offset == length, no entry was created, and the session eventually expired, turning the entire upload into deleted needles for the vacuum to chew through. Completion now validates gapless coverage with the same watermark HEAD uses. A record extending coverage joins the entry - the read path resolves partial overlaps by ModifiedTsNs, and the raced copies carry identical bytes - while a fully covered duplicate is freed once the entry lands. * filer: allow one mutating TUS request per session at a time Nothing stopped two PATCHes from writing the same range concurrently: both loaded the same offset, both passed the conflict check, and both recorded their sub-chunks. A client whose request timed out in a proxy retries immediately while the server side is still storing the buffered sub-chunk, which is exactly that race. A session now accepts one PATCH or DELETE at a time, the way tusd locks uploads; a concurrent one is refused with 423 Locked, which TUS clients retry, and HEAD keeps answering so progress polling is unaffected. The chunk state is loaded under the claim, so a retried PATCH sees every record its predecessor left and conflicts cleanly instead of duplicating data. * test: cover a TUS PATCH raced by its own retry Stalls a PATCH mid-body over a raw connection, retries the same range while it is in flight, and expects the retry refused with 423 Locked; the upload then resumes from the reported offset and the final content must be intact. * filer: never free a TUS duplicate the entry still references Coverage is computed from ranges, so a record fully covered by another is treated as a duplicate no matter which needle it names. A malformed record naming a file id the entry keeps would have had that needle freed right after the entry landed - the corruption this change set exists to stop. The duplicates are now freed in one batch, skipping any file id the entry references; their records go with the session directory. * test: bound the raw TUS connection reads http.ReadResponse on the stalled PATCH's connection blocked until the whole go test timeout if the filer never answered. * filer: free the needles of chunk write attempts a retry replaced A volume server stores the needle locally and only then fans out to the replicas, so a replication failure 5xxs with the data already written. Each attempt assigns its own file id, so once a later attempt lands elsewhere nothing references the earlier ones: the caller only sees the chunk that succeeded, and the failed ids were dropped. They are now freed the way the caller frees them when the whole write fails. Retrying on a 5xx makes this reachable on every read-only or full volume, which is exactly the condition that filled the reporter's volumes. |
||
  |
c69bb10407 |
build(deps): bump github.com/getsentry/sentry-go from 0.44.1 to 0.48.0 (#10921)
* build(deps): bump github.com/getsentry/sentry-go from 0.44.1 to 0.48.0 Bumps [github.com/getsentry/sentry-go](https://github.com/getsentry/sentry-go) from 0.44.1 to 0.48.0. - [Release notes](https://github.com/getsentry/sentry-go/releases) - [Changelog](https://github.com/getsentry/sentry-go/blob/master/CHANGELOG.md) - [Commits](https://github.com/getsentry/sentry-go/compare/v0.44.1...v0.48.0) --- updated-dependencies: - dependency-name: github.com/getsentry/sentry-go dependency-version: 0.48.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * bump cockroachdb/errors to v1.14.0 for sentry-go 0.48.0 sentry-go 0.48.0 removed Event.Extra, which cockroachdb/errors v1.11.3 still references; v1.14.0 builds against the new API. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris LuandGitHub
|
68f0793b6f |
mount: register UNC mount points as WinFsp network file systems (#10943)
A \\server\share -dir was passed to WinFsp as a plain mount point, which treats it as a directory path on an actual remote server and fails. Turn it into the VolumePrefix option instead, so the mount registers with the WinFsp network provider: the UNC path is then reachable from every logon session, which a drive letter mounted from a service is not, and each user can map their own drive letter to it. |
||
|
Chris LuandGitHub
|
40f77503d0 |
helm: trim the Lance chart comments (#10940)
Comments only, no rendering change: the values paragraphs compress to the density of the file around them, the env-var note becomes a template comment instead of leaking into the rendered manifest, and the two spots that invite a wrong simplification - the unconditionally rendered -port.lance and the empty-placeholder platform guard - each get their one-line why. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm |
||
|
Chris LuandGitHub
|
d9d7d0be74 |
helm: serve the Lance catalog and deploy the Rust worker (#10936)
* helm: serve the S3 gateway's Lance Namespace, on by default Standalone `weed s3` serves the Lance Namespace API on 9101 unless told not to, so the chart defaulting s3.lancePort to 9101 matches weed's own posture instead of hiding the port behind a null. The flag is always rendered, so lancePort: 0 reaches weed as -port.lance=0 and genuinely disables the namespace rather than silently falling back to the binary default; 0 also drops the service port and the optional lanceIngress, which otherwise mirror the iceberg wiring. The NetworkPolicy admits the port the same way it admits icebergPort. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * helm: run the Lance maintenance worker beside the Go worker The Go and Rust workers have no overlapping jobs - Go serves vacuum, balance, EC and iceberg_maintenance, only /usr/bin/weed-worker serves the lance_* family - so a cluster serving Lance tables needs both, not an either/or switch. The worker deployment now adds a worker-lance container whenever the namespace is reachable: worker.namespaceUrl, or derived from the release's S3 service and s3.lancePort. Untouched Go container; admin address derived the same way; mTLS flags point at the already-mounted worker cert when security is on; metrics on their own worker.lanceMetricsPort (9328, next in the 932x convention) with the same health probes, service port and scrape endpoint the Go container gets, and the worker NetworkPolicy admits that port exactly when the container renders. The image carries an empty placeholder on armv7/386 where exec falls back to the shell and exits 0, so the command refuses those platforms by name; s3.lancePort: 0 is the escape hatch there. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm |
||
|
Chris LuandGitHub
|
b3be2f5449 |
filer.backup, filer.sync: stop sharing resume checkpoints across destinations (#10934)
* filer.backup: key the checkpoint by source path and sink destination The checkpoint id hashed only sink name + directory, so two backups to different buckets or endpoints sharing a directory layout advanced one checkpoint: whichever job was running pushed the shared offset forward, and a stopped or failing job later resumed from the other's position, silently skipping changes. Backups of different source paths to the same destination shared a checkpoint the same way. Each sink now reports a destination identity (endpoint or account, bucket or container, directory) and the checkpoint is keyed by the source path plus that identity. Reads fall back to the historical name+directory key when the new key has no value, so existing backups resume where they left off; writes go only to the new key. * filer.sync: include the target path in the offset key The offset stored on the target filer was keyed by source path and source filer signature only, so two syncs from the same source cluster and path to different directories on the same target cluster advanced one shared checkpoint, and the slower one could resume past events it never applied. The target path now participates in the key; "/" keeps the historical form, and a sync with a non-root target path falls back to the historical key once when its own key has no value yet. * join checkpoint key fields with NUL so they cannot alias A path or configuration value spelling out the separator could concatenate two different field tuples to the same checkpoint key. NUL cannot appear in a CLI path argument or any sane configuration value, making the encoding injective. |
||
|
Chris LuandGitHub
|
4a2879abad |
admin: show a copyable S3 object URL in the bucket file browser (#10933)
* admin: offer copyable S3 object URLs in the bucket file browser * admin: hide object urls when the bucket type lookup fails * admin: ignore an s3.public_endpoint that is not an absolute http url * mini: build the seeded s3 endpoint with JoinHostPort for ipv6 * admin: reject a query or fragment in s3.public_endpoint * mini: drop the seeded s3 endpoint when a later run disables s3 * admin: reject userinfo and bare delimiters in s3.public_endpoint, redact the warning * mini: pass its s3 endpoint as an admin option instead of mutating viper * admin: keep the rejected s3.public_endpoint value out of the log |
||
|
Chris LuandGitHub
|
2a70532d0d |
s3: log each request at -v=2 (#10931)
* s3: log each request at -v=2 * s3: quote requester and path in the access log line * s3: record the post-policy signing identity as the requester |
||
|
Chris LuandGitHub
|
d2c470af1b |
S3: commit SSE GET status only after the first read succeeds (#10935)
The SSE streaming path kept writing 200/206 from filer metadata before fetching or decrypting anything, so a missing needle or failed decrypt setup surfaced as a broken 200 body. Same deferral as the plain path: the status commits on the first body write, and every failure before that returns to the handler for a clean S3 error response. |
||
|
Chris LuandGitHub
|
115756dd41 | helm: expose loadBalancerClass, loadBalancerIP, loadBalancerSourceRanges on services (#10929) | ||
|
Chris LuandGitHub
|
d9d5fab35b |
S3: commit GET status only after the first read succeeds (#10930)
streamFromVolumeServers wrote the 200/206 status from filer metadata before any byte had been fetched from a volume server, so a missing or corrupted needle surfaced as a broken 200 body and the request metrics recorded a success. Defer the status commit to the first body write: a failed first read now returns a clean 500 before headers, while the wire timing of successful responses is unchanged since net/http buffers the status line until body bytes arrive anyway. |
||
|
|
d8545997c2 |
build(deps): bump github.com/seaweedfs/goexif from 1.0.3 to 2.0.0+incompatible (#10917)
build(deps): bump github.com/seaweedfs/goexif Bumps [github.com/seaweedfs/goexif](https://github.com/seaweedfs/goexif) from 1.0.3 to 2.0.0+incompatible. - [Release notes](https://github.com/seaweedfs/goexif/releases) - [Commits](https://github.com/seaweedfs/goexif/commits) --- updated-dependencies: - dependency-name: github.com/seaweedfs/goexif dependency-version: 2.0.0+incompatible dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
|
9121177f48 |
build(deps): bump github.com/shirou/gopsutil/v4 from 4.26.6 to 4.26.7 (#10918)
Bumps [github.com/shirou/gopsutil/v4](https://github.com/shirou/gopsutil) from 4.26.6 to 4.26.7. - [Release notes](https://github.com/shirou/gopsutil/releases) - [Commits](https://github.com/shirou/gopsutil/compare/v4.26.6...v4.26.7) --- updated-dependencies: - dependency-name: github.com/shirou/gopsutil/v4 dependency-version: 4.26.7 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
|
9ba3d473d7 |
build(deps): bump cloud.google.com/go/pubsub from 1.51.0 to 1.51.1 (#10919)
Bumps [cloud.google.com/go/pubsub](https://github.com/googleapis/google-cloud-go) from 1.51.0 to 1.51.1. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.51.0...pubsub/v1.51.1) --- updated-dependencies: - dependency-name: cloud.google.com/go/pubsub dependency-version: 1.51.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
|
e36b01c18f |
build(deps): bump github.com/rabbitmq/amqp091-go from 1.13.0 to 1.14.0 (#10920)
Bumps [github.com/rabbitmq/amqp091-go](https://github.com/rabbitmq/amqp091-go) from 1.13.0 to 1.14.0. - [Release notes](https://github.com/rabbitmq/amqp091-go/releases) - [Changelog](https://github.com/rabbitmq/amqp091-go/blob/main/CHANGELOG.md) - [Commits](https://github.com/rabbitmq/amqp091-go/compare/v1.13.0...v1.14.0) --- updated-dependencies: - dependency-name: github.com/rabbitmq/amqp091-go dependency-version: 1.14.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
|
dfa75ce231 |
build(deps): bump actions/upload-artifact from 4 to 7 (#10922)
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 4 to 7. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](https://github.com/actions/upload-artifact/compare/v4...v7) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
|
61d588e455 |
build(deps): bump github/codeql-action from 4.37.6 to 4.37.8 (#10923)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.37.6 to 4.37.8. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/v4.37.6...v4.37.8) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 4.37.8 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
 
|
a3afe4460b |
tarantool: fix upsert data corruption and missing context propagation (#10926)
Co-authored-by: Marat Karimov <karimov_m@inbox.ru> |
||
|
|
a0ddf22f17 |
Bump Tarantool client library from 3.0.0 to 3.0.1 (#10925)
Co-authored-by: Marat Karimov <karimov_m@inbox.ru> |
||
|
Chris LuandGitHub
|
863fec6c3f |
S3: let a key that is a prefix of other keys be an object (#10912)
* filer: keep the sentinel when CreateEntry reports an update failure CreateEntry flattened the error UpdateEntry wraps, so errors.Is stopped matching and ErrExistingIsDirectory and ErrExistingIsFile never reached the S3 mapper, which answered a retryable 500 instead. * s3: let a key that is a prefix of other keys be an object S3 keys are flat, so "a/b" and "a/b/c" are independent objects that coexist in either write order. The filer stores a key as a path, so one of them has to live on the directory the other is nested under. Writing the nested key first refused the prefix key outright. Writing it second promoted the file to a directory, which kept its data but lost the key: an empty object left nothing to recognise it by and disappeared, and one with data listed under a trailing slash it never had. Mark the directory that carries such a key, and write the object onto it when the path is already a directory. The mark makes an empty prefix object visible to listings and readable by GET and HEAD, keeps the empty folder cleaner off it, and lists it under the key it was written with. Deleting the key strips the mark back off along with the data. * filer: keep a TTL off a directory that stands for an object An expired entry is deleted a row at a time, so expiring a directory removes it and leaves everything under it unreachable. Promoting a file to a directory carried its TTL across, and a promoted file is exactly the one that has keys nested under it. Drop the TTL on promotion, and leave one an older build wrote alone. The lifecycle worker still expires the object, through the delete that leaves the directory behind. * s3: delete the null version of a key other keys are nested under The routed delete cannot remove an entry that other keys live under, and answered a retryable 500 rather than falling back to the lock path the unversioned delete already falls back to. That path then looked the entry up under the bucket with the whole key as its name, so the demote wrote it back one directory too high and failed as not found. Fall back on any non-precondition error, and split the key before deleting it. Trailing-slash directory markers with children reach the same delete. * filer: keep the sentinel when MkFile and Mkdir report a create failure Same flattening one layer out: every mkFile caller lost the sentinel, so a CopyObject onto a key that other keys are nested under answered a retryable 500 where a PutObject of the same key answers 409. * s3: copy and rename a key that other keys are nested under Such a key is stored on the directory those keys live in, and copy and rename both refused it: the source lookup maps every directory entry to NoSuchKey, so a key a plain GET serves could not be copied or moved, and the destination side refused it as a directory conflict. The source is read through a view of the entry as the object it names. The destination is written the way a PutObject of that key writes it. A rename at either end copies the object's own data across and strips it off the source key rather than going through AtomicRenameEntry, which moves a directory by moving everything under it - the nested keys are not part of what is being renamed. |
||
|
Chris LuandGitHub
|
46ce2c45a2 |
mini: reserve the admin gRPC port instead of binding it late (#10928)
* mini: reserve the admin gRPC port instead of binding it late Port selection probes every port with a throwaway listener and closes it. Master, filer, volume and S3 bind a moment later, but the admin waits for all of them first and only then binds its worker gRPC port, roughly two seconds in. That port defaults to the admin http port + 10000, which lands inside the Linux ephemeral range, so one of the cluster's own outgoing gRPC dials can take it during the gap and the admin dies on bind, taking the worker with it. Keep the listener from the availability check and hand it to the admin. * mini: clear the admin gRPC reservation before retaking it A rerun inside one process would otherwise inherit the closed listener of the previous run whenever the reservation fails, and the admin would accept it and only find out inside Serve. * mini: snapshot the admin options for the startup goroutine The cleanup path read the package-level options long after the goroutine started, so a later in-process run could have its reserved listener closed by the previous run. |
||
|
Chris LuandGitHub
|
51eb5333d3 |
ec: read a needle's intervals in parallel (#10911)
* ec: read a needle's intervals in parallel A needle spanning more than one EC block gets one interval per block, and consecutive blocks live on different shards. We read those intervals in sequence, so a 4MB chunk landing in a volume's 1MB small-block region cost five round trips to five different servers. Read them concurrently into disjoint slices of a single buffer, at most 8 in flight. Same change in the Rust volume server's phase C. * ec test: seed the random payload instead of the deprecated rand.Read |
||
|
Chris LuandGitHub
|
69cc2869ad |
Fixes from the review of the admin bucket policy UI (#10907)
* admin: treat a missing S3 Tables policy as an empty load, not an error
The bucket/table policy GET relayed the backend's 404 NoSuchPolicy to the
dialog, whose loader treats any non-OK response as a load failure and
keeps Save and Delete blocked. A bucket or table without a policy could
never be given one. Return policy null instead, the same contract
ShowBucketPolicy uses for classic buckets.
* admin: reject policy documents the structured editor would misread
A top-level JSON array passed the object guard (typeof [] is 'object')
and loaded as a zero-statement policy, which the next commit would
rewrite to an empty document. Object elements in Action/Resource were
coerced to '[object Object]' and saved that way on the s3tables surface,
which stores policies verbatim. Both now throw, which routes the
document to the JSON tab like other unrepresentable shapes.
* admin: let the JSON tab save documents the structured editor can't model
Save with the JSON tab active required a round-trip through
policyDocToEditorState, so exactly the documents the dialogs shunt to
'JSON tab only' mode (unrepresentable Effect, Resource+NotResource, and
the like) could never be saved - Delete was the only mutation left.
Invalid JSON still blocks; an unrepresentable document now saves and the
editor state stays marked unparsed.
* admin: pin the policy editor to what each consumer's backend supports
The s3tables evaluator has no NotResource/NotPrincipal fields - it
silently drops them, turning Allow+NotResource into allow-everything and
making Deny+NotPrincipal inert - and it only matches s3tables: actions
against s3tables ARNs, while the editor suggested s3: actions and
arn:aws:s3::: resources. New registerPolicyEditor knobs: allowNegation
hides the Not* modes and routes documents using them to the JSON tab;
resourceSuggestions pins the Resource autocomplete to the open
resource's ARN; the S3 Tables dialogs get an s3tables-only action
datalist. requirePrincipal now also hides NotPrincipal, which
policy_engine.ValidateBucketPolicy always rejects, and the client-side
check requires Principal specifically to match that server rule.
* admin: save S3 Tables policies from a button, not form submission
The multi-input structured editor sits inside a form whose Save button
was type=submit, so Enter in any single-line editor input - accepting an
autocomplete suggestion, say - implicitly submitted whatever half-built
statement the editor held, and the backend stores the document verbatim.
A lone statement with no Principal matches nobody, locking out every
non-owner. Save is now an ordinary button and the form ignores
submission.
* admin: block zero-statement policy saves
Committing the active tab before the emptiness check made 'Policy JSON
is required' dead code: an empty editor serializes to {"Statement":[]},
which the s3tables backend stores verbatim - evaluated default-deny for
every non-owner, while the statement-count column keeps showing 'Not
configured'. All three policy dialogs now refuse a save with no
statements and point at Delete instead. The classic bucket modal only
gained a clearer message; the server already rejected the document.
* admin: guard S3 Tables policy mutations against stale and overlapping requests
The save/delete completions ran against whatever resource the shared
modal happened to show by then: a slow PUT for one bucket would hide the
modal mid-edit of another and misattribute its alerts, a late DELETE
cleared the shared textarea over the newly opened resource with its
loaded flag set, and nothing stopped a double-click from firing two
overlapping mutations. Ported the classic modal's pattern: capture the
target on start, flag the mutation in flight with the buttons disabled,
and only touch the UI when the completion still matches the open
resource. Success now reloads the page, which also keeps the Policy
column's statement count honest.
* admin: confirm before deleting an S3 Tables policy
Delete Policy sat next to Save and fired on a single click; with
default-allow enabled one stray click silently dropped the resource
policy and left the bucket open to every principal. Same confirmation
the classic bucket modal already has.
* admin: let a corrupt stored bucket policy be shown, fixed, and deleted
A stored document the decoder rejects made the policy GET 500, and with
the loaded flag never set the modal blocked both Save and Delete - the
one policy an operator most needs to remove was the one they couldn't,
even though the delete path never reads the document. The GET now
returns the raw bytes alongside a null policy; the dialog hands them to
the JSON tab and unblocks the buttons.
* admin: url-encode the bucket name in the policy API calls
The filer lists any directory under the buckets path, names S3 would
never allow included; one carrying '#' or '%' broke the fetch URL or
addressed a different name than the modal shows.
* admin: drop stale edit-policy responses on the IAM policies page
The same race the bucket and S3 Tables dialogs already guard against:
open one policy's editor while its GET stalls, open another, and the
late response populates the editor under the second policy's name -
Update then saves the first policy's statements over the second.
* admin: warn before a bucket policy save drops unsupported fields
The editor tracks unmodeled top-level keys precisely so
confirmPolicyFieldDiscard can warn before the server's Version+Statement
decode discards them, but only the IAM page called it; the bucket modal
saved a pasted document with e.g. a console-generated Id without a word
while the editor kept displaying the field.
* s3: enforce the bucket policy size cap on both surfaces
The 20KB cap lived only in the admin UI, so a larger policy stored via
the S3 API displayed there but could never be re-saved, desyncing the
two writers the cap comment claimed could not desync. The constant now
lives in policy_engine next to the shared validator and PutBucketPolicy
rejects oversized documents with PolicyTooLarge, matching AWS.
* admin: ship the policy editor's fieldset styles with the editor
The .policy-stmt-* rules that undo Bootstrap's full-width legend reset
stayed behind in policies.templ when the editor markup moved to the
shared script, so the bucket and S3 Tables dialogs rendered Actions/
Resource/Principal as full-width jumbo headings. PolicyDatalists is the
component every consumer already renders once; the styles live there
now.
* s3: mirror bucket policy changes into the IAM store from the metadata subscription
The advanced-IAM path appends the bucket-policy:<bucket> document to
every STS/session evaluation, but only this gateway's own PutBucketPolicy
maintained that mirror - a policy tightened or created through the admin
UI (or another gateway) never reached it, so revoked access stayed live
indefinitely, and the delete side was an unimplemented TODO in any case.
The metadata subscription now diffs the stored policy on every bucket
entry change and updates or removes the mirror, covering all writers and
deletion with one mechanism; IAMManager gains the missing
RemoveBucketPolicy.
* admin: deduplicate the bucket policy write path
Set and Delete carried line-for-line identical filer closures;
bucketPolicyMutation already treats nil as clear-the-key. The shared
helper sits below Set's validation, since ValidatePolicy cannot take the
nil document Delete passes.
* s3: drop ValidateBucketPolicy's re-checks of ValidatePolicy rules
Both callers run ValidatePolicy first, which already enforces the
version and at-least-one-statement rules; the duplicates were dead code
with drifted error text.
* admin: seed a new statement's Resource from the pinned suggestions
A fresh statement on the S3 Tables dialogs started with no resource row
at all; seed it with the broadest pinned ARN the same way cfg.bucket
already seeds the classic modal.
* admin: refuse to save Not* fields the backend would silently drop
Hiding the NotResource/NotPrincipal modes was not enough where negation
is disallowed: the JSON tab accepts any valid document (that is its
job), and a statement's Advanced-fields box can reintroduce the keys, so
an s3tables save could still store fields the evaluator drops - turning
Allow+NotResource into allow-everything. commitPolicyActiveTab now runs
a final document-level check over what would actually be saved; Delete
stays available for cleanup.
* s3: move the IAM bucket policy mirror on a bucket rename
A same-directory rename delivers one event carrying both entries, and
the byte-equality short-circuit skipped the new name's mirror when the
policy was unchanged - while the replayed delete for the old name
removed its mirror, leaving the renamed bucket unmirrored. The mirror
decision is now a pure function that removes the old name and writes the
new one regardless of byte equality, with the rename cases unit tested.
* s3: backfill the IAM bucket policy mirror on lazy bucket loads
The metadata subscription only mirrors changes, so a policy that
predates the IAM integration never reached the bucket-policy:<bucket>
mirror and its grants did not bind on the IAM path until the policy was
next modified. The gateway is deliberately lazy at startup (nothing
lists all buckets), so the backfill hooks the same place a bucket's
policy first becomes known: the cold bucket-config load. EnsureBucketPolicy
writes only when no mirror is stored, so repeat loads cost one cached
read.
* s3: reconcile the bucket policy backfill against concurrent changes
The backfill's check-then-write could race an event-driven mirror update
or removal and re-store bytes that were already stale, with no later
event to heal it. EnsureBucketPolicy now reports whether it wrote, and a
write is reconciled against a fresh authoritative entry read: a changed
policy is re-mirrored, a removed one is removed. Anything changing after
that read fires its own event, which finds the backfill's write already
present and supersedes it. The backfill also carries the entry's raw
bytes rather than a re-marshaled document, so the reconcile can
byte-compare.
* s3: prime the bucket policy mirror before advanced-IAM authorization
The backfill ran from the lazy bucket-config load, but IAM authorization
evaluates the bucket-policy:<bucket> mirror before any handler runs - a
grant carried only by a not-yet-mirrored policy denied forever, and the
denied request never reached the code that would have loaded the bucket.
authorizeWithIAM now primes the bucket config first (an in-memory cache
hit once warm), and the backfill runs synchronously on the cold load so
the very first authorization already sees the mirror.
|
||
|
Chris LuandGitHub
|
68ec8ca655 |
admin: honor a persisted or admin.toml maintenance enabled=false (#10909)
* admin: honor a persisted or admin.toml maintenance enabled=false The startup path discarded an operator's enabled=false twice over: ApplyDefaultsToProtobuf treated the bool zero value as unset and applied the schema default of true, and a force-enable migration block flipped any survivor. With the legacy /maintenance UI routes gone, nothing could write the config either, so the maintenance system ran unconditionally. Keep the persisted enabled flag across schema-default application in LoadMaintenanceConfig, drop the force-enable block, and add a top-level [maintenance] enabled key to admin.toml as the config surface, persisted through SaveMaintenanceConfig like the per-task settings. Absent config still defaults to enabled. * admin: track presence on the maintenance enabled flag A plain proto3 bool cannot distinguish an operator's persisted false from a legacy file that simply omits the field, so honoring false would have silently switched maintenance off for configs written before the toggle could be persisted. Make the field optional: files that predate presence tracking keep the enabled default, while a file that explicitly persists the toggle is honored either way. |
||
 Mathieu ArnoldandGitHub
|
e931cccc7b |
Manage bucket policies via the admin ui (#10895)
* admin: manage S3 bucket policies from the admin UI
Bucket policies were only manageable through the S3 PutBucketPolicy API;
the admin UI had no equivalent to the quota/owner/lifecycle editors it
already offers. Add GET/PUT/DELETE for a bucket's policy, sharing the
exact validation the S3 gateway uses.
- Extract validateBucketPolicy/validateResourceForBucket out of
s3api_bucket_policy_handlers.go into policy_engine.ValidateBucketPolicy /
ResourceMatchesBucket so both the S3 API and the admin UI enforce
identical rules.
- weed/admin/dash/bucket_policy.go: Get/Set/DeleteBucketPolicy, writing
through ObjectTransaction + PATCH_EXTENDED (the lifecycle pattern) so a
concurrent owner/quota/lifecycle change on the same bucket entry isn't
clobbered. Propagation to every S3 gateway is automatic via the existing
filer metadata log subscription. The S3 gateway's IAM policy mirror is
deliberately not replicated here (its delete path is already an
unimplemented TODO on the S3 side).
- New GET/PUT/DELETE /api/s3/buckets/{bucket}/policy routes, CSRF-guarded
on writes.
- Bucket list and details modal now show a statement-count badge, read
from the entry already fetched (no extra RPC).
- UI: a JSON-textarea policy editor modal, matching the lifecycle modal's
structure.
* admin: reuse the visual policy editor for bucket policies
Extract the structured policy editor (add/remove statement, action/
resource/principal rows with autocomplete, JSON tab kept in sync) out of
policies.templ's inline script into a shared
weed/admin/static/js/policy_editor.js, and wire the bucket policy modal
in s3_buckets.templ up to it instead of a bare JSON textarea.
- registerPolicyEditor(which, config) replaces the hardcoded create/edit
id derivation with a per-instance config (textarea/tab/body ids,
datalist ids, requirePrincipal, bucket). The IAM policies page keeps its
exact pre-extraction ids via two registerPolicyEditor calls, so its
markup is unchanged.
- New policy_datalists.templ exposes the three shared <datalist>s
(actions/resources/principals) as @PolicyDatalists(), now rendered by
both policies.templ and s3_buckets.templ.
- requirePrincipal seeds new bucket-policy statements with Principal: "*"
and adds a client-side check before save (the server, via
policy_engine.ValidateBucketPolicy, remains the actual authority); the
bucket config pins the Resource autocomplete to the open bucket instead
of fetching every bucket in the cluster.
- layout.templ loads policy_editor.js globally, after admin.js/
modal-alerts.js (basePath/escapeHtml/showAlert) which it depends on.
3a (the extraction) is a byte-preserving move verified against the
unchanged policies.templ behavior before layering 3b's parameterization
and the bucket-policy wiring on top.
* admin: migrate S3 Tables bucket/table policy editors to the shared editor
Third consumer of the shared visual policy editor: the S3 Tables bucket
and table policy modals (a bare JSON textarea each) now get the same
structured Editor/JSON tabs as the bucket policy and IAM policy pages,
via registerPolicyEditor('s3tablesBucketPolicy'/'s3tablesTablePolicy',
{ textareaId: ... }). Storage and validation are untouched - S3 Tables
policies still go through their own s3tables.PolicyDocument type and the
s3tables.policy extended attribute, unrelated to policy_engine and
s3-bucket-policy; only the editor UI is shared.
Fix a real bug surfaced by adding this second load path: the bucket
policy modal (and the naive first draft of this s3tables port) called
commitPolicyTextareaToEditor() right after a GET and then force-switched
to the Editor tab. commitPolicyTextareaToEditor() is designed to leave
the current tab in place and the editor state untouched when a document
fails to parse (so an in-progress edit survives a bad tab switch), so
forcing the Editor tab afterwards could show empty/stale editor state
that a careless Save would then serialize over a perfectly valid but
structurally-unusual stored policy. Add
loadPolicyTextareaIntoEditor(which) to policy_editor.js, which has no
"current tab" to defer to and instead falls back to the JSON tab with an
alert on a document the structured editor can't represent - the same
safety editPolicy already had in policies.templ - and use it at all three
"populate the editor right after a GET" call sites (bucket policy,
S3 Tables bucket policy, S3 Tables table policy).
* admin: show policy statement count on the S3 Tables buckets page
Mirrors the "Policy" column already added to the classic S3 buckets
list: a clickable badge with the statement count when the table bucket
has a resource policy, "Not configured" otherwise. S3 Tables policies
are a separate mechanism (s3tables.PolicyDocument under the
s3tables.policy extended attribute) from the S3 bucket policy work
elsewhere in this branch (policy_engine.PolicyDocument /
s3-bucket-policy), so this is a parallel implementation of the same
pattern rather than shared code.
- S3TablesBucketSummary gains PolicyStatementCount, populated in
GetS3TablesBucketsData from entry.Entry.Extended[s3tables.ExtendedKeyPolicy]
via the new extractS3TablesPolicyStatementCountFromEntry - no extra RPC,
the entry is already fetched for ExtendedKeyMetadata.
- The badge reuses the existing .s3tables-bucket-policy-btn class, so it
opens the same policy modal as the row's action button with no JS
changes.
* admin: don't let a failed policy GET open the door to an empty overwrite
loadS3TablesBucketPolicy/loadS3TablesTablePolicy cleared the textarea,
then unconditionally called loadPolicyTextareaIntoEditor() regardless of
whether the GET actually succeeded - including when fetch() rejected or
the response was not ok, silently logged to console only. That leaves
the structured editor holding a legitimate-looking empty policy
({version, statements: []}), with the Editor tab active by default.
If Save is then clicked, commitPolicyActiveTab() serializes that empty
state into the textarea as `{"Version":"2012-10-17","Statement":[]}` -
a non-empty string - before the "Policy JSON is required" guard ever
sees it, so the guard passes and the transient load failure gets
written over whatever policy was actually stored.
Add s3tablesBucketPolicyLoaded/s3tablesTablePolicyLoaded, set true only
once a GET has actually completed (ok, including a genuinely empty
policy) and false on any failure path (fetch rejection or a non-ok
response, which previously fell through silently). Both submit handlers
now check the flag before touching the editor at all, and a failed load
surfaces via alert() instead of only a console.error - the user
previously had no visible indication the load had failed.
Verified with a jsdom simulation driving the real rendered page against
a stubbed fetch: a failed GET followed by Save now sends no PUT at all
(previously it sent Statement: []); a successful GET followed by Save
still PUTs the loaded policy unchanged.
* admin: address code review findings on the policy editor
1. policy_editor.js: policyEditors is only pre-populated for 'create'/
'edit'; every other `which` (bucket, s3tablesBucket, s3tablesTable)
stays undefined until its first successful async load. Nothing in
this file enforces that a page hide its Editor/JSON tabs and
Add-statement button until that load completes - the S3 Tables policy
modals don't - so a click in that window (e.g. Add statement, or
switching to the JSON tab) threw "Cannot read properties of undefined
(reading 'unparsed')". Add policyEditorState(which), which lazily
initializes a default state, and route addPolicyStatement, the
jsonTabBtn 'show.bs.tab' handler, commitPolicyActiveTab, and
renderPolicyEditor through it. Verified with a jsdom simulation
against a never-resolving fetch: the exact click threw on the
pre-fix code and no longer does.
2. s3_buckets.templ: the bucket-policy Save handler checked the
textarea for emptiness before calling commitPolicyActiveTab(), which
is what actually serializes the structured Editor tab's fields into
that textarea. A policy entered entirely through the Editor tab (the
primary path - never touching the JSON tab) left the textarea at
whatever it was at load time, so creating a new policy this way hit
"Enter a policy document" and Save silently did nothing. Move the
commit before the emptiness check, preserving the existing alert and
early-return. Verified with a jsdom simulation: Add-statement then
Save (no tab switch) now PUTs the entered statement; before the fix
the same sequence never reached fetch().
3. s3tables_buckets.templ / s3tables_tables.templ: the policy Editor/
JSON nav-tabs were missing the ARIA roles Bootstrap's own tab pattern
expects (role="tab"/"tabpanel", aria-selected, aria-controls,
aria-labelledby) - screen readers had no way to tell these were tabs
or which pane went with which button. Added the standard Bootstrap 5
tab markup to both.
* admin: guard policy load/save flows against overlapping requests
1. s3tables.js: loadS3TablesBucketPolicy/loadS3TablesTablePolicy had no
protection against overlapping loads. Opening one bucket's (or
table's) policy dialog and then another's before the first GET
resolved let the late response write its document into the shared
textarea and mark the dialog "loaded" while it was now targeting the
second resource - a subsequent Save would then push the first
resource's policy onto the second. Add a per-load monotonic sequence
number (s3tablesBucketPolicyRequestSeq / s3tablesTablePolicyRequestSeq,
the same pattern already used for the classic bucket-policy load in
s3_buckets.templ); a response is only applied - textarea, loaded flag,
editor state - if its captured sequence still matches the latest one
issued.
Verified with a jsdom simulation: bucket A's policy load (artificially
slow) followed immediately by bucket B's (fast) previously left A's
policy in the textarea once A's late response landed; it now correctly
keeps B's.
2. s3_buckets.templ: the bucket-policy Save button lives outside the
(initially hidden) editor wrapper, so it stays clickable while a load
is still in flight - the existing policyRequestSeq guard only protects
the *load* from a stale response, not Save from firing before any
load for the current bucket has completed. Add bucketPolicyLoaded,
reset before each GET and set only once the matching response lands,
and check it at the top of the Save handler.
Verified with a jsdom simulation: clicking Save immediately after
opening the dialog, before a (deliberately never-resolving) GET
settles, now sends no PUT; a normal load-then-save sequence still
PUTs the loaded policy unchanged.
* admin: address further code review findings on the policy editor
1. s3tables.js: loadS3TablesBucketPolicy/loadS3TablesTablePolicy only
reset the JSON textarea when a new load starts; the structured editor
kept showing the previously loaded resource's statements (Editor tab
is the default active one) until the new fetch resolved. Call
loadPolicyTextareaIntoEditor() against the now-cleared textarea
immediately, so switching resources visibly resets the editor right
away instead of only once its own load completes. Verified with jsdom:
opening bucket A (loads fully) then bucket B (GET never resolves) no
longer leaves A's statements visible in B's editor.
2. s3tables.js: deleteS3TablesBucketPolicy/deleteS3TablesTablePolicy had
no loaded-state check, so a failed GET (which already blocks Save)
left Delete fully able to remove the resource's stored policy sight
unseen. Add the same s3tablesBucketPolicyLoaded/s3tablesTablePolicyLoaded
guard Save already uses. Verified with jsdom: delete after a failed
load now sends no DELETE; delete after a successful load is unaffected.
3. s3_buckets.templ: the bucket-policy Editor/JSON nav-tabs were missing
the same ARIA roles already added to the S3 Tables policy tabs in an
earlier round (role="tab"/"tabpanel", aria-selected, aria-controls,
aria-labelledby) - this instance was out of scope for that review
comment but is the same gap. Bootstrap's own tab.js already manages
aria-selected on tab switch once the attribute exists, so no extra JS
was needed.
4. s3_buckets.templ: neither the bucket-policy Save nor Delete handler
guarded against a double-click, or against firing while the other was
still in flight - two overlapping PUT/DELETE requests for the same
bucket could land in either order. Add a shared
bucketPolicyMutationInFlight flag: set (and both buttons disabled)
before each fetch, cleared (and buttons re-enabled) on failure so the
user can retry, left set through the existing success hide-and-reload
path, and also reset when a new bucket's dialog opens so an abandoned
in-flight request from a closed dialog can't leave the buttons stuck
disabled. Verified with jsdom: double-clicking Save now sends exactly
one PUT, and a Delete click while that PUT is still pending sends no
DELETE.
* admin: scope bucket-policy mutation completions to the bucket that started them
1. The previous round's fix reset bucketPolicyMutationInFlight whenever a
new bucket's policy dialog opened, to avoid leaving Save/Delete stuck
disabled if the modal was closed mid-request. That traded one bug for
a worse one: if bucket A's PUT/DELETE was still in flight when the
user opened bucket B's dialog, the reset let B's Save/Delete fire
immediately, and A's completion handler - unaware anything had
changed - would still hide the (now B's) modal and reload the page
out from under whatever the user was doing with B, on success, or
alert a message with no bucket context, on failure.
Stop resetting on reopen, so a pending mutation for a previous bucket
keeps this bucket's Save/Delete blocked until it settles (matches the
"preventing overlapping mutations" the review comment describes).
Instead, capture policyEditorBucket as targetBucket right before each
fetch and compare it against policyEditorBucket again in the
completion handler: the in-flight flag is always released so the
buttons never get stuck, but the modal-hide/reload/alert only fire if
this bucket is still the one showing; a stale completion for an
abandoned bucket just logs to the console instead.
Verified with a jsdom simulation: opening bucket B while bucket A's
Save is still pending leaves B's Save button disabled and a click on
it a no-op; once A's PUT resolves, B's button re-enables but no
modal.hide()/reload() fires (previously both fired unconditionally).
2. bucketPolicyDeleteBtn had no bucketPolicyLoaded check, unlike Save -
a failed GET blocked Save but left Delete free to remove a policy the
client never actually saw (the same gap already fixed for the S3
Tables policy modals in an earlier round). Added the same guard,
ahead of the confirm() dialog. Verified with jsdom: Delete after a
failed load now sends no DELETE request.
* admin: fix spelling mistake
|
||
 孙超andGitHub
|
c80664ec21 |
s3: propagate storage rule fsync to volume server uploads (#10906)
The storage rule's fsync decision was computed by the filer (detectStorageOption -> rule.Fsync) and applied on the filer's own HTTP write path, but was never carried onto the chunk uploads S3 issues: the AssignVolumeResponse had no fsync field, so the s3api client could not learn the decision, and the chunked upload URL was hardcoded without it. Every S3 write to a path with fsync configured went to the volume server as a non-fsync write. Carry the decision through the assign response: - filer.proto: AssignVolumeResponse gains bool fsync, filled from the storage option the assign resolved. - operation.AssignResult gains Fsync, so uploadChunk can append ?fsync=true to the volume server upload URL (single and replica fan-out paths). - The S3 PUT/UploadPart assignFunc, the S3 copy path, the admin file browser upload, and the Iceberg worker assign functions all forward the response field. Adds TestUploadReaderInChunksAppendsFsyncWhenAssigned. |
||
|
Chris LuandGitHub
|
71a8c77a36 |
telemetry: let the dashboard pick the confirmation window (#10904)
* telemetry: let the dashboard pick the confirmation window * telemetry: cover the serialized threshold map through the stats handler |
||
|
Chris LuandGitHub
|
9c8d3b6a81 |
ec: refund the cleared leftover shards' slots in the encode source health check (#10903)
* erasure_coding: one home for the shard-count to volume-slots conversion * ec: refund the cleared leftover shards' slots in the encode source health check |
||
|
Chris LuandGitHub
|
36c97344ef |
s3: confine a Lance catalog table location to the caller's own bucket (#10901)
The Lance namespace gateway took the request-body location field, trimmed a trailing slash, and passed it straight to the marker sink. That location feeds TableDataDirFromMetadataLocation, which joins it under /buckets and collapses any ../ segments, and writeMarker's CreateEntry then auto-creates every missing parent. A caller could point the location at another tenant's bucket, or escape /buckets entirely, and plant a fixed-name marker (recursively creating the parents) or hide a victim's live table with .lance-deregistered. Confine the declared location the way the Iceberg gateway already does: require an s3:// URI whose bucket is the caller's own and whose path carries no traversal segment, on both the declare and register handlers. |
||
|
Chris LuandGitHub
|
74038e1b14 |
master: don't let a dead KeepConnected handler close its successor's channel (#10900)
A client that reconnects before the old handler exits re-registers the same client name, and addClient overwrites the map entry. The old handler's deferred deleteClient then closed whatever channel the map held under that name: the new, live stream's. Receiving from a closed channel returns nil immediately and forever, so the new handler's send loop degenerated into sending empty responses at wire speed, pinning a core on each side until the client killed the connection. deleteClient now closes the channel its own handler registered and leaves the map entry alone unless it still points to that channel. This also closes the previously orphaned old channel, whose drain goroutine used to leak. The send loop treats a closed channel as an exit instead of a message stream. |
||
|
Chris LuandGitHub
|
cf0dba334c |
s3api: no filer failover after the callback has consumed part of a response (#10902)
s3api: no filer failover after fn has consumed part of a response withFilerClientFailover replays fn verbatim on the next filer, so a filer that died mid-stream followed by a healthy peer returned success with the callback's closure-captured accumulator holding the dead filer's prefix twice; the per-attempt accumulator in listWithRetry could not close this, because the replay happens inside a single attempt. Track delivery on the connection handed to fn: once a unary reply or streamed message has reached the callback, surface the transport error unwrapped instead of failing over, and let callers replay from a clean slate. A filer that fails before delivering anything fails over exactly as before. |
||
|
Chris LuandGitHub
|
c167af541e |
telemetry: confirm a cluster after a week of reports, not two days (#10899)
* telemetry: sync the server module to go 1.26 The root module moved to go 1.26 but the telemetry server module, which replaces seaweedfs with the repo root, stayed on 1.25.8, so go refuses to build or test it until the directive catches up. * telemetry: confirm a cluster after a week of reports, not two days Two days of history still lets recurring CI and demo clusters into the confirmed fleet: anything torn down and rebuilt across a UTC midnight counts. Requiring seven distinct UTC days keeps the fleet charts and the version/OS distributions to clusters that actually stay up; real clusters qualify after their first week, and the fallback to all active clusters while none is confirmed is unchanged. |
||
|
Chris LuandGitHub
|
0f85d005ad |
server: 416 only when no requested range overlaps, with Content-Range, and the Rust mirror (#10889)
* filer, volume server: return 416 when no requested range overlaps the content * seaweed-volume: return 416 when no requested range overlaps the content * server: check the range test error, use the request context, fix the no-overlap comment boundary |
||
|
Chris LuandGitHub
|
173adbc291 |
master: never re-seed a raft cluster over committed state under -raftBootstrap (#10883)
* master: never re-seed a raft cluster over committed state -raftBootstrap deleted logs.dat, stable.dat and snapshots on every start and then bootstrapped a fresh cluster. Since hashicorp raft only snapshots after 8192 log entries, the TopologyId lives in the log, not in a snapshot, so the pre-wipe snapshot recovery found nothing and each restart minted a new cluster identity. A master that came up while it could not reach its peers seeded a rival cluster; when the two logs met, SetTopologyId's split-brain guard fatally stopped every master holding the other id, and the master layer crash-looped with no quorum. Bootstrapping is genesis. Drop the wipe and the inline bootstrap. The first master in -peers already mints a cluster once it has confirmed no peer has a leader, so the flag has nothing left to do and is now ignored; keeping that one master the sole bootstrap authority is what stops a partition from minting two clusters, so the flag must not widen it either. A master with state rejoins its peers, and one whose data dir was reset is admitted by the sitting leader instead of forking again. * test: cover -raftBootstrap restarts in the multi-master suite Three masters start with -raftBootstrap, the way the helm chart renders it on every master on every roll, and the cluster has to hold one TopologyId after they all restart. /dir/status is proxied to the leader, so each master's own view of the identity is read out of its log, which is where a fork shows up. Before the fix the hashicorp case minted a new id on each restart. |
||
 Junker der ProvinzandGitHub
|
fa3bd5b5a7 |
mount: use the kernel-resolved node id in Link, not the persisted attribute (#10885)
* fix(mount): reply to LINK with the kernel node id, not the stored inode Link() answered the kernel with out.NodeId = oldEntry.Attributes.Inode. That attribute is a mount-runtime number and only entries created through a mount carry one. An entry written by the S3 API, WebDAV or a direct filer call persists inode 0, so the LINK reply named node id 0, which the kernel rejects as invalid_nodeid and reports as EIO. The hard link itself had already been written to the filer, which is why it looked correct again after a mount restart. The same stale number was also used as an inodeToPath key. AddPath(0, path) filed the new link under inode 0, so a later Lookup on that name handed the kernel node id 0 as well, and a LOOKUP reply carrying node id 0 means no such entry. in.Oldnodeid is the node id the kernel already holds for the source, and it is the key inodeToPath is indexed by, so use it for the reply, for AddPath and for the sibling sync. Fixes #8404 * test(mount): cover the sibling sync in Link with a third hard link The two existing cases never reach the body of syncHardLinkSiblings: with two links the source alias and the name just created are both in skipPaths, so the loop iterates over nothing and a change to that site goes unnoticed. A third link leaves one name that no other part of Link() writes. The new case drives three links off one source. It guards against covering nothing (it fails if every path turns out to be a skipPath), checks that every name of the file reports nlink 3, and then drives the sync with both candidate keys to pin down which one it has to be: keyed by the source's persisted Attributes.Inode, which is 0 for an entry written outside a mount, GetAllPaths has no path to walk, while the kernel node id reaches the sibling. That second half is driven directly because Link() alone cannot tell the two keys apart. The meta cache keeps one blob per hard link id (FilerStoreWrapper setHardLink/maybeReadHardLink), so a read of any sibling returns the attributes of the last write to any of them whether or not the sync ran. |
||
|
Junker der ProvinzandGitHub
|
5ebc9c9f4b | server: reject a Range start offset equal to the file size (#10898) | ||
|
Chris LuandGitHub
|
3b10e43d5d | test: wait for volume server registration in the FUSE p2p harness (#10897) | ||
|
Chris LuandGitHub
|
9d06f2c378 | test: keep per-test log directories in the FUSE DLM harness (#10893) |