mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-08-28 11:56:07 +00:00
cf0dba334c429d19e176f8d13eb7e513a338b8f0
14874
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Chris LuandGitHub
|
cf0dba334c |
s3api: no filer failover after the callback has consumed part of a response (#10902)
s3api: no filer failover after fn has consumed part of a response withFilerClientFailover replays fn verbatim on the next filer, so a filer that died mid-stream followed by a healthy peer returned success with the callback's closure-captured accumulator holding the dead filer's prefix twice; the per-attempt accumulator in listWithRetry could not close this, because the replay happens inside a single attempt. Track delivery on the connection handed to fn: once a unary reply or streamed message has reached the callback, surface the transport error unwrapped instead of failing over, and let callers replay from a clean slate. A filer that fails before delivering anything fails over exactly as before. |
||
|
Chris LuandGitHub
|
c167af541e |
telemetry: confirm a cluster after a week of reports, not two days (#10899)
* telemetry: sync the server module to go 1.26 The root module moved to go 1.26 but the telemetry server module, which replaces seaweedfs with the repo root, stayed on 1.25.8, so go refuses to build or test it until the directive catches up. * telemetry: confirm a cluster after a week of reports, not two days Two days of history still lets recurring CI and demo clusters into the confirmed fleet: anything torn down and rebuilt across a UTC midnight counts. Requiring seven distinct UTC days keeps the fleet charts and the version/OS distributions to clusters that actually stay up; real clusters qualify after their first week, and the fallback to all active clusters while none is confirmed is unchanged. |
||
|
Chris LuandGitHub
|
0f85d005ad |
server: 416 only when no requested range overlaps, with Content-Range, and the Rust mirror (#10889)
* filer, volume server: return 416 when no requested range overlaps the content * seaweed-volume: return 416 when no requested range overlaps the content * server: check the range test error, use the request context, fix the no-overlap comment boundary |
||
|
Chris LuandGitHub
|
173adbc291 |
master: never re-seed a raft cluster over committed state under -raftBootstrap (#10883)
* master: never re-seed a raft cluster over committed state -raftBootstrap deleted logs.dat, stable.dat and snapshots on every start and then bootstrapped a fresh cluster. Since hashicorp raft only snapshots after 8192 log entries, the TopologyId lives in the log, not in a snapshot, so the pre-wipe snapshot recovery found nothing and each restart minted a new cluster identity. A master that came up while it could not reach its peers seeded a rival cluster; when the two logs met, SetTopologyId's split-brain guard fatally stopped every master holding the other id, and the master layer crash-looped with no quorum. Bootstrapping is genesis. Drop the wipe and the inline bootstrap. The first master in -peers already mints a cluster once it has confirmed no peer has a leader, so the flag has nothing left to do and is now ignored; keeping that one master the sole bootstrap authority is what stops a partition from minting two clusters, so the flag must not widen it either. A master with state rejoins its peers, and one whose data dir was reset is admitted by the sitting leader instead of forking again. * test: cover -raftBootstrap restarts in the multi-master suite Three masters start with -raftBootstrap, the way the helm chart renders it on every master on every roll, and the cluster has to hold one TopologyId after they all restart. /dir/status is proxied to the leader, so each master's own view of the identity is read out of its log, which is where a fork shows up. Before the fix the hashicorp case minted a new id on each restart. |
||
 Junker der ProvinzandGitHub
|
fa3bd5b5a7 |
mount: use the kernel-resolved node id in Link, not the persisted attribute (#10885)
* fix(mount): reply to LINK with the kernel node id, not the stored inode Link() answered the kernel with out.NodeId = oldEntry.Attributes.Inode. That attribute is a mount-runtime number and only entries created through a mount carry one. An entry written by the S3 API, WebDAV or a direct filer call persists inode 0, so the LINK reply named node id 0, which the kernel rejects as invalid_nodeid and reports as EIO. The hard link itself had already been written to the filer, which is why it looked correct again after a mount restart. The same stale number was also used as an inodeToPath key. AddPath(0, path) filed the new link under inode 0, so a later Lookup on that name handed the kernel node id 0 as well, and a LOOKUP reply carrying node id 0 means no such entry. in.Oldnodeid is the node id the kernel already holds for the source, and it is the key inodeToPath is indexed by, so use it for the reply, for AddPath and for the sibling sync. Fixes #8404 * test(mount): cover the sibling sync in Link with a third hard link The two existing cases never reach the body of syncHardLinkSiblings: with two links the source alias and the name just created are both in skipPaths, so the loop iterates over nothing and a change to that site goes unnoticed. A third link leaves one name that no other part of Link() writes. The new case drives three links off one source. It guards against covering nothing (it fails if every path turns out to be a skipPath), checks that every name of the file reports nlink 3, and then drives the sync with both candidate keys to pin down which one it has to be: keyed by the source's persisted Attributes.Inode, which is 0 for an entry written outside a mount, GetAllPaths has no path to walk, while the kernel node id reaches the sibling. That second half is driven directly because Link() alone cannot tell the two keys apart. The meta cache keeps one blob per hard link id (FilerStoreWrapper setHardLink/maybeReadHardLink), so a read of any sibling returns the attributes of the last write to any of them whether or not the sync ran. |
||
|
Junker der ProvinzandGitHub
|
5ebc9c9f4b | server: reject a Range start offset equal to the file size (#10898) | ||
|
Chris LuandGitHub
|
3b10e43d5d | test: wait for volume server registration in the FUSE p2p harness (#10897) | ||
|
Chris LuandGitHub
|
9d06f2c378 | test: keep per-test log directories in the FUSE DLM harness (#10893) | ||
|
Chris LuandGitHub
|
9d4270f118 | test: wait for volume server registration in the FUSE DLM harness (#10891) | ||
 Chris Lu
|
3b8931c2f6 | admin: address review feedback on the maintenance scanner fix | ||
|
Chris Lu
|
8d8a25b1cf | s3api: remove the duplicated listing retry helpers left by overlapping merges | ||
|
c58795354a |
s3api: retry a transient filer failure on metadata listings (#10890)
* s3api: retry a transient failure when listing multipart uploads/parts A blip on the way to the filer failed the whole ListMultipartUploads or ListParts request. Both reported failure points sit inside one streaming listing: the ListEntries call that opens the stream, and the stream.Recv calls that drain it. Neither retried, so a single Unavailable answer from a filer that was restarting turned into a 500 for the S3 client. Replay the listing instead, bounded to three attempts with a 100ms backoff that doubles. Only a transient failure is replayed. A not-found answer stays authoritative so the empty-list branch still works, and every other error still reaches the client on the first attempt. This is scoped to (*S3ApiServer).list rather than added inside DoSeaweedListWithSnapshot, which mount, the shell and the other object listings share, and where a retry after a partial stream would re-deliver entries the callback had already seen. Within one call to list, a replay is safe: it collects into a fresh slice each time, so it can neither duplicate nor drop entries. That guarantee does not extend past this function. withFilerClientFailover already re-runs its callback against the next filer on any non-NotFound error without resetting the caller's accumulator, so on a multi-filer gateway a mid-listing failover can itself produce a duplicated result with err == nil, independent of this change and not fixed by it. Noted in the PR rather than silently left for someone to rediscover. Fixes #7221 References #7235 * s3api: move the listing retry inside list itself --------- Co-authored-by: Junker der Provinz <jdp@braethoria.com> |
||
|
Junker der ProvinzandGitHub
|
8d2c0273bd |
admin: stop the maintenance scanner pinning itself to one scan per second after a transient failure (#10887)
* admin: honour persisted task configs when building the maintenance policy
buildPolicyFromTaskConfigs passed a literal nil to vacuum, erasure_coding
and balance LoadConfigFromPersistence. Those functions look for their
LoadXTaskPolicy() accessor via a type assertion, which a nil interface can
never satisfy, so every call fell through to NewDefaultConfig() and the
policy came back with the compiled-in defaults - Enabled: true among them.
A task disabled on disk was therefore still scheduled, and the only trace
was a glog.V(1) "Using default ... configuration" line.
Thread the real ConfigPersistence through instead. There are two copies of
this function: the one in weed/admin/dash builds config.Policy on the
normal admin startup path and can simply take cp as its receiver, and the
one in weed/admin/maintenance is the fallback used when the config carries
no policy yet, which now receives the store from NewMaintenanceManager.
weed/admin/dash already imports weed/admin/maintenance, so the maintenance
side has to keep the duck-typed interface{} parameter that the task
loaders already use rather than importing the concrete type back.
The store is only handed over when a data directory is configured: an
unconfigured one has nothing to read, and a typed nil pointer would pass
the loaders' type assertion and then panic on first use.
Fixes #10874
* admin: restore the maintenance scan cadence after an error backoff
scanLoop shortens its ticker to the error backoff delay after a failed
scan, but it decided whether to replace the ticker by comparing the
target interval against the configured scan interval instead of against
the interval the ticker was actually running at. Once the errors stopped,
getScanInterval returned the configured interval again, the comparison
came out false, and the ticker was left at the backoff delay - so a
single transient scan failure pinned the scanner to one scan per second
for the rest of the process lifetime. That is the ~1/second cadence in
issue #10874: 658 KB/s of "Cancelled N stale pending balance tasks
before re-detection" and 193k orphaned task files over two days.
Track the interval the ticker is running at and compare against that, so
both entering the backoff and returning to the normal cadence replace the
ticker.
While in here:
- defer ticker.Stop() bound the ticker that was current when the defer
was registered, so every replacement ticker leaked on return. Wrap it
in a closure.
- running was written by Start/Stop and read by all three background
loops without synchronisation. Guard it with the existing mutex, fold
the running check in triggerScanInternal into the lock it already
takes, and make Stop a no-op when not running so a second call cannot
close the stop channel twice.
Refs #10874
* admin: make the maintenance policy actually reach the task detectors
Loading the persisted task configs into the maintenance policy only
matters if something reads that policy, and nothing did.
MaintenanceIntegration pushes the policy into every registered detector
and scheduler through interface{ SetEnabled(bool) } and
interface{ SetMaxConcurrent(int) } type assertions. Every task registered
through base.RegisterTask is backed by base.GenericDetector and
base.GenericScheduler, and neither implemented either method, so all four
assertions failed silently for every task on every startup. The policy's
enabled flag reached nothing: ScanWithTaskDetectors gates on
detector.IsEnabled(), and the queue's policy lookups for max concurrent
and repeat interval are fallbacks that only fire when the scheduler
reports zero, which the generic scheduler never does.
Add the setters, delegating to the TaskConfig.SetEnabled the interface
already declares and to TaskDefinition.MaxConcurrent, which is what
GetMaxConcurrent returns.
Applying the policy required three more fixes, because with the
assertions working the policy could now do damage as well as good:
- IsTaskEnabled reports false for a task type the policy has no entry
for, so applying it unconditionally would have disabled every task the
policy does not list. Skip task types with no policy entry: no entry
means no opinion, not disabled.
- ec_balance was exactly such a task. It is registered like the other
three but had no entry in the policy builder and no accessor on
ConfigPersistence at all, so its configuration could never be
persisted. Add SaveEcBalanceTaskPolicy/LoadEcBalanceTaskPolicy, the
task_ec_balance.pb file, the SaveTaskPolicy dispatcher case, and the
policy entry.
- InitMaintenanceManager ran before loadTaskConfigurationsFromPersistence,
which replaces each task's whole config object, so the policy was
applied and then immediately thrown away. Swap the order. Both read the
same files, so the policy is now the last writer and stays
authoritative.
MaintenanceManager.UpdateConfig also updated the queue's and the
scanner's policy but not the integration's, so a policy changed at
runtime never reached the detectors. Add MaintenanceIntegration.SetPolicy
and call it.
While building the policy, stop hand-copying each task's fields and use
the task's own ToTaskPolicy(). The hand-written version was a second
definition of every task's policy and had already lost the erasure coding
preferred tags and replica placement and the balance IO rate limit. For
the same reason, the "nothing persisted yet" branches of
LoadVacuumTaskPolicy, LoadErasureCodingTaskPolicy and
LoadBalanceTaskPolicy now derive from each task's NewDefaultConfig()
instead of a third hand-written copy. Those copies had drifted, so with a
data directory but no config file on disk the effective defaults differed
from what the task and the admin UI schema both advertise:
vacuum scan interval 24h -> 2h
balance scan interval 6h -> 30m
balance imbalance 0.1 -> 0.2
erasure coding scan interval 168h -> 1h
erasure coding fullness 0.90 -> 0.95
erasure coding min volume 1024MB -> 30MB
Finally, weed/admin/dash and weed/admin/maintenance each carried a copy
of the policy builder and they had already diverged. Export the
maintenance one as BuildPolicyFromTaskConfigs and have dash call it.
Refs #10874
* worker: warn when a config store cannot supply a task's persisted config
LoadConfigFromPersistence logged a single glog.V(1) "Using default X
configuration" for every way of not loading anything, so the bug in
issue #10874 - a store handed in that the type assertion rejects, leaving
a task running on compiled-in defaults - looked exactly like the normal
"no data directory configured" case. The reporter had to read the source
to work out why their disabled task kept running, and asked for this
specifically.
Separate the cases. A non-nil store that does not provide the accessor is
always a wiring bug and is now logged at warning level, naming the type
and the missing method. A read error or a policy that will not apply is
also a warning. No persistence configured, and a store with nothing saved
yet, stay at V(1): those are normal.
Refs #10874
* admin: stop GetTaskPolicy panicking on a maintenance policy that is nil
GetTaskPolicy dereferenced its MaintenancePolicy argument to look at
TaskPolicies, so IsTaskEnabled, GetMaxConcurrent and GetRepeatInterval
all took the admin process down when handed a nil policy. A nil policy is
not a programming error here: MaintenanceConfig.Policy is unset until
something builds one, DefaultMaintenanceConfig returns a config with no
policy at all, and UpdateConfig installs whatever config it is given.
Found by calling IsTaskEnabled with the policy from a freshly defaulted
MaintenanceConfig.
Treat a nil policy as "no entry": no task enabled, the safe concurrency
default of 1, and a repeat interval of 0 so callers fall back to their
own default instead of reading DefaultRepeatIntervalSeconds off nil.
Also add the startup test this was found with. It walks the admin
server's startup sequence over a data directory that has balance saved as
disabled and checks the state that decides whether issue #10874 happens:
the balance detector reports disabled, vacuum stays enabled, and tasks
whose config was never saved keep their compiled-in default.
Refs #10874
* admin: document the synchronisation SetPolicy would need beyond startup
ConfigureTasksFromPolicy now really writes TaskDefinition.Config and
TaskDefinition.MaxConcurrent, which the scan loop reads through
detector.IsEnabled() with nothing synchronising the two. Every caller
runs during admin server startup today, before the scan loop exists, so
there is no live race - but the next caller has to add the locking, and
the same already applies to UpdateAllConfigs replacing the whole config
object. Write it down at the seam instead of leaving it to be
rediscovered.
Refs #10874
|
||
|
Junker der ProvinzandGitHub
|
f710b6003a |
s3api: retry a transient failure when listing multipart uploads/parts (RFC on layering) (#10886)
s3api: retry a transient failure when listing multipart uploads/parts A blip on the way to the filer failed the whole ListMultipartUploads or ListParts request. Both reported failure points sit inside one streaming listing: the ListEntries call that opens the stream, and the stream.Recv calls that drain it. Neither retried, so a single Unavailable answer from a filer that was restarting turned into a 500 for the S3 client. Replay the listing instead, bounded to three attempts with a 100ms backoff that doubles. Only a transient failure is replayed. A not-found answer stays authoritative so the empty-list branch still works, and every other error still reaches the client on the first attempt. This is scoped to (*S3ApiServer).list rather than added inside DoSeaweedListWithSnapshot, which mount, the shell and the other object listings share, and where a retry after a partial stream would re-deliver entries the callback had already seen. Within one call to list, a replay is safe: it collects into a fresh slice each time, so it can neither duplicate nor drop entries. That guarantee does not extend past this function. withFilerClientFailover already re-runs its callback against the next filer on any non-NotFound error without resetting the caller's accumulator, so on a multi-filer gateway a mid-listing failover can itself produce a duplicated result with err == nil, independent of this change and not fixed by it. Noted in the PR rather than silently left for someone to rediscover. Fixes #7221 References #7235 |
||
|
Junker der ProvinzandGitHub
|
f3caf6e7da |
admin: count plugin-runtime workers in worker metrics (#10884)
* admin: count plugin-runtime workers in worker metrics The admin server keeps two worker registries: the legacy maintenance-worker map, filled by workers registering over the worker gRPC stream, and the plugin worker registry, filled by workers started as `weed worker`. Both the SeaweedFS_admin_workers_connected / SeaweedFS_admin_worker_slots gauges and the dashboard's Workers card read only the legacy map, so a cluster that runs the admin and its workers as separate components reported 0 workers even while its workers showed up on the plugin pages and ran scheduled jobs. Aggregate both registries instead. The two are merged by worker ID: `weed mini` starts both runtimes out of one working directory, so they share the persisted worker ID and must not be counted twice. For such a worker the slot numbers still come from the legacy registry, which keeps mini's existing readings. Plugin workers report their slots in the heartbeat, so detection and execution slots are summed from there; a worker that has connected but not yet sent a heartbeat counts as connected with zero slots. Fixes #10525 * admin: clamp negative worker-reported slot values in metrics merge A plugin worker's self-reported heartbeat slot counts are untrusted input; clamp them to 0 before summing so a stale or misbehaving worker can't drive the aggregate gauge negative, matching the same defensiveness already used in registry.go's own slot arithmetic. |
||
  |
004fc32503 |
build(deps): bump github.com/moby/go-archive from 0.2.0 to 0.3.0 (#10826)
Bumps [github.com/moby/go-archive](https://github.com/moby/go-archive) from 0.2.0 to 0.3.0. - [Release notes](https://github.com/moby/go-archive/releases) - [Changelog](https://github.com/moby/go-archive/blob/main/changes_test.go) - [Commits](https://github.com/moby/go-archive/compare/v0.2.0...v0.3.0) --- updated-dependencies: - dependency-name: github.com/moby/go-archive dependency-version: 0.3.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
 
|
641fc8b031 |
admin: add visual iam policy editor (#10878)
* admin: add visual iam policy editor Add a structured, tabbed editor (Editor / JSON) for creating and editing IAM policies in the admin dashboard, alongside the existing raw-JSON textarea: - policies.templ: per-statement cards for Sid, Effect, Action, and Resource, with unmanaged fields (Principal, NotPrincipal, NotResource, Condition, or anything else) preserved verbatim in a per-statement "advanced fields" JSON box so nothing is lost on round-trip. Switching tabs commits and reparses in both directions. Restored the "Use Sample Policy" button, now filling both the structured editor and the JSON tab. The "Validate" button now calls the existing but previously unused POST /api/object-store/policies/validate endpoint instead of doing JS-only checks. - Progressive Resource ARN autocomplete: suggests bucket names first, then once "bucket/" is typed, suggests bucket/* plus the bucket's direct subfolders, drilling down one path segment at a time as the user types further "/" characters. - New GET /api/files/list-folders endpoint (file_browser_handlers.go) backing the folder autocomplete: wraps the existing file browser data function and returns just the subdirectory names as JSON, scoped to paths under /buckets. - Action-name suggestions (datalist) for the Action field, sourced from the existing s3_constants.S3_ACTION_* constants plus new s3_constants.S3TABLES_ACTION_* constants (extracted from the s3tables operation dispatch switch) so the suggestion list can't drift from the strings the engines actually understand. - policy_handlers.go: ValidatePolicy now accepts a statement with only NotResource set (previously required Resource), matching policy_engine.validateStatement and the fact the new editor makes such statements reachable from the UI. - Tests: ValidatePolicy behavior, route registration for the policy API and the new list-folders endpoint, list-folders path scoping, and the action-suggestion list's shape. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * admin: fix XSS, cache poisoning, and cap overshoot in policy editor Address code review findings on the IAM policy editor added in the previous commit: - policies.templ (displayPolicyDetails): escape every interpolated policy value (Sid, Effect, Action, Resource, policy name, and the raw JSON document) before assigning to innerHTML. Policy documents can come from other admins or an import, so an unescaped field could execute script when the "View" modal renders it. - policies.templ (policyEditorStateToDoc): reject JSON arrays in a statement's "advanced fields" box, not just invalid JSON. `typeof [] === 'object'` was true, so a JSON array was assigned to the statement; subsequent property assignments (Sid, Effect, ...) landed on the array object but JSON.stringify of an array only serializes numeric indices, silently dropping them. - policies.templ (loadPolicyFolderNames): on a failed folder lookup, remove the cache entry instead of permanently caching the empty fallback, so a transient network/server error doesn't block retries for the rest of the page's lifetime. - file_browser_handlers.go (ListFolders): stop appending directory names as soon as the running count reaches maxListFoldersEntries, instead of only checking the cap after a full page is processed, so the returned list never exceeds the configured cap. Regenerated policies_templ.go with the already-stamped templ v0.3.1001 to keep the diff scoped to this file. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * admin: stop policy editor from clobbering the active tab and dropping malformed advanced fields Address two review findings on the IAM policy editor (Issue 3, stored-XSS in displayPolicyDetails, was already fixed by the previous commit and is unchanged here): - createPolicy, updatePolicy, and validatePolicyDocument always committed the structured editor's (possibly stale) state into the JSON textarea before submitting, even when the user had just edited the JSON tab directly. That silently discarded the user's JSON edits and validated/saved the old structured-editor state instead, which could leave broader permissions in force than intended. Added commitPolicyActiveTab(which), which commits whichever tab is currently visible into the other side instead of unconditionally overwriting the JSON tab from the editor: if the JSON tab is active it parses that JSON back into the structured editor (without touching the textarea itself), otherwise it serializes the structured editor into the textarea as before. All three call sites, plus the JSON-tab "show.bs.tab" handler, now use this and abort with an alert if the currently active tab's content can't be committed. - policyEditorStateToDoc silently continued with an empty object when a statement's "advanced fields" box held invalid JSON, so switching tabs, validating, or saving would drop Principal/NotResource/Condition from that statement without telling the user. It now throws (with the statement number and parse error) on invalid or non-object JSON there, and callers surface that via showAlert and abort instead of proceeding. Regenerated policies_templ.go with the already-stamped templ v0.3.1001. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * admin: keep unmanaged top-level policy fields across editor tab switches policyDocToEditorState only carried Version and Statement into editor state, so any other top-level key (e.g. Id) present in the JSON tab was silently rewritten away as soon as the user switched to the Editor tab and back. Capture those keys in state.otherFields and merge them back in policyEditorStateToDoc before Version and Statement are written, so the two tabs stay faithful to each other and the editor never rewrites text the user typed. Note this is editor fidelity only: the admin API's policy_engine.PolicyDocument carries just Version and Statement, and DocumentJSON is never populated, so such fields are still discarded by the server once a policy is saved. Making them survive a save would require a backend change, which is out of scope here. Regenerated policies_templ.go with the already-stamped templ v0.3.1001. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * admin: warn before a policy save discards unsupported top-level fields The editor round-trips unmanaged top-level keys (e.g. Id) between the Editor and JSON tabs, but the admin API's policy_engine.PolicyDocument carries only Version and Statement, so the server drops them on save and the user saw no indication. Added confirmPolicyFieldDiscard(), called from createPolicy and updatePolicy after the active tab is committed (so the field list is accurate whichever tab is showing). It names the fields that will be lost and lets the user confirm or cancel. Not wired into validatePolicyDocument, which doesn't persist anything. Chose the warning over the alternative of persisting these fields through the backend: policy_engine.PolicyDocument is shared by the S3 bucket-policy engine and IAM evaluation, so extending it would change the stored document shape for every policy in the codebase - far beyond the scope of this editor. Regenerated policies_templ.go with the already-stamped templ v0.3.1001. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * admin: reject malformed Effect and Resource/NotResource conflicts in policy editor Two review findings on the IAM policy editor: - policyDocToEditorState defaulted any non-"Deny" Effect (missing, misspelled, wrong case) to "Allow". A statement meant to be "Deny" with a typo like "deny" would silently become a permissive "Allow" instead of being rejected. It now throws on anything but an exact "Allow" or "Deny", naming the offending statement and value. commitPolicyTextareaToEditor catches this the same way it already catches invalid JSON: alert the user and keep the JSON tab active instead of switching to the Editor tab with wrong data. - policyEditorStateToDoc could save a statement with both Resource (from the structured field) and NotResource (surviving in the "advanced fields" extras from before the user switched to using Resource) set at once - a contradictory combination neither the admin's ValidatePolicy handler nor policy_engine's evaluator rejected. When the structured Resource field is non-empty it now deletes any leftover NotResource from extras, consistent with the file's existing rule that structured fields take precedence over extras. Mirrored the existing Principal/NotPrincipal exclusivity check in weed/admin/handlers/policy_handlers.go's ValidatePolicy to reject the same combination server-side, since create/update perform no validation at all. Deliberately left policy_engine.validateStatement (used by the S3 bucket-policy PUT handler for every bucket policy in the product) unchanged - extending that shared validator is a larger, separate change outside this admin-editor fix's scope. Added a handler test for the new Resource+NotResource rejection. Regenerated policies_templ.go with the already-stamped templ v0.3.1001. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * admin: add NotResource support to the visual policy editor Since Resource and NotResource are mutually exclusive (enforced by a previous fix), NotResource could previously only be set through the raw JSON in a statement's "advanced fields" box. Promote it to a first-class mode of the structured editor: - The static "Resources" label is now a Resource/NotResource dropdown; the same list of values underneath is reused for either key depending on the selected mode, with a short form-text explaining the semantics. - NotResource is added to POLICY_STATEMENT_KNOWN_KEYS, since it's now a managed field like Resource rather than something that falls through to extras. - policyDocToEditorState derives resourceMode from which key is present on load, and throws (same handling as the existing malformed-Effect case: alert, keep the JSON tab active) if a hand-edited document has both Resource and NotResource on one statement, since that can't be represented by the dropdown. - policyEditorStateToDoc writes only the key matching the selected mode, replacing the previous one-directional "delete NotResource whenever Resource is set" fix with mode-driven logic that also deletes Resource when NotResource is selected. - displayPolicyDetails (the read-only View modal) now shows the actual NotResource values with a distinct label instead of a static "(NotResource used instead)" placeholder. No backend changes: the server-side "cannot specify both" check added previously in policy_handlers.go's ValidatePolicy already covers this. Regenerated policies_templ.go with the already-stamped templ v0.3.1001. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe * admin: reject non-object policy documents; catch Principal/NotPrincipal conflicts server-side Two review findings: - policyDocToEditorState treated a top-level JSON value that wasn't an object (null, or a bare string/number/boolean) as an empty statement list instead of failing explicitly. If the user typed e.g. "hello" or 42 in the JSON tab and switched to the Editor tab, their input was silently discarded and replaced with an empty policy - the same class of "guess instead of reject" bug fixed for malformed Effect and Resource/NotResource conflicts previously. Added an explicit check that throws for null/scalar input, while leaving array and object document shapes accepted exactly as before. - weed/admin/handlers/policy_handlers.go's ValidatePolicy checked the Resource/NotResource conflict by non-empty length (len(...Strings()) > 0), which misses a statement where Resource is explicitly present but an empty list (e.g. "Resource": []) alongside a non-empty NotResource. Switched that check to field presence (!= nil), matching how policy_engine's own validateStatement already treats Principal/NotPrincipal exclusivity. Also added the equivalent Principal/NotPrincipal presence check to this handler, which had none before - the advanced-fields box in the visual editor lets a user set both today, and nothing server-side caught it. The existing non-empty "Resource or NotResource is required" check is left as a length check, since an empty array shouldn't count as "provided". Added test cases for both conflict checks in policy_handlers_test.go. Regenerated policies_templ.go with the already-stamped templ v0.3.1001. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe * admin: add Principal/NotPrincipal support to the visual policy editor (v1, AWS-only) Adds a first, deliberately narrow structured editor for a statement's Principal/NotPrincipal, left out when NotResource support was added: - A Principal/NotPrincipal mode dropdown mirrors the existing Resource/NotResource one (same mutual-exclusivity handling: the two fields can't be set at once, and switching modes reuses the same value list). - A simple repeatable text-value list feeds a single {"AWS": [...]} object on save - always the AWS type, never the "bare" (untyped) SeaweedFS-extension shape. Per policy_engine's allowedPrincipalKeys, Service/Federated/CanonicalUser also parse successfully, but nothing in the S3 bucket-policy evaluation path ever sets a real caller's principal to a service name, an OIDC provider ARN, or a canonical user ID, so only AWS is functionally meaningful today - out of scope for this v1. - On load, only the exact {"AWS": ...} single-key shape is unwrapped into the structured field and removed from "extras". Anything else (bare string/array, a different single type key, or several type keys at once) is left untouched in "extras" exactly as before, with a visible warning under the dropdown so the user knows a Principal/NotPrincipal exists but isn't shown there. Saving with the structured field left empty never touches whatever's already in extras, so a preserved complex form isn't silently dropped just because the user didn't touch this field. - The read-only View modal now displays Principal/NotPrincipal for any shape (via a small generic summarizer), not just the AWS-simple one. - Generalized the action/resource field-to-state-key mapping (used by commitPolicyEditorForm and the add/remove-item click handler) into a shared lookup table instead of stacking another ternary, now that a third field (principal) exists. Regenerated policies_templ.go with the already-stamped templ v0.3.1001. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe * admin: support the bare "*" wildcard Principal in the visual editor "Principal": "*" (and NotPrincipal: "*") is the standard AWS shorthand for "everyone" and is common in real bucket policies, but the v1 Principal/NotPrincipal editor only recognized the {"AWS": ...} object form, leaving a bare "*" statement's principal hidden in Advanced fields. parseSimpleAwsPrincipal now also accepts the bare string "*" as a simple, structurally-editable value. On save, a principal value list containing exactly ["*"] is written back as the bare "*" string (matching the common convention) rather than wrapped as {"AWS": "*"}; anything else still wraps under AWS as before. Updated the field's form-text hint accordingly. Regenerated policies_templ.go with the already-stamped templ v0.3.1001. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe * admin: add Principal field autocomplete backed by users + IAM roles Adds a datalist-backed autocomplete for the policy editor's Principal/ NotPrincipal text fields, sourced from a new API listing existing identities: - weed/admin/dash/principal_suggestions.go: AdminServer.GetPrincipalSuggestions combines S3 user ARNs (via the existing GetObjectStoreUsers + iam.UserArn) with IAM role ARNs (via integration.NewFilerRoleStore / ListRoles, reusing the exact same construction already used in iam_manager.go - no new dependency risk introduced). Role ARNs are reconstructed from the role name using SeaweedFS's default arn:aws:iam::role/<name> convention rather than fetching each role's stored definition, since this only backs a suggestion list. Role listing failures are logged and swallowed rather than failing the whole request - an incomplete suggestion list is fine, blocking policy editing over it is not. Service accounts are deliberately not listed separately: a service account's ARN is identical to its parent user's, already covered by the user list. - weed/admin/handlers/policy_handlers.go: GetPrincipalSuggestions handler exposing this as {"principals": [...]}. - Route registered at the API root (GET /api/principals) rather than under policyApi's "/object-store/policies" prefix, since that subrouter's existing "/{name}" GET route would shadow any single-segment GET route registered after it (the same class of gotcha previously seen with "/validate"). - weed/admin/view/app/policies.templ: a shared, lazily-fetched-once policyPrincipalSuggestions datalist (flat list - unlike the progressive per-folder Resource ARN autocomplete, users/roles aren't hierarchical), wired into policyListRowHtml for field:"principal" and populated on input/focus, with "*" always offered first. Added tests for the new ARN-construction helper and route registration. Regenerated policies_templ.go with the already-stamped templ v0.3.1001. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe * admin: fix fieldset/legend styling in the structured policy editor Bootstrap's form reset stretches <legend> to the fieldset's full width (float: left; width: 100%), which loses the native "notch in the border" look and makes each section's label bar as wide as the card. Add two scoped classes: .policy-stmt-fieldset (border, rounded corners, spacing between sections) and .policy-stmt-legend (undoes the float/width so the legend hugs its content, with a little padding). Applied to the three per-statement sections (Actions, Resource/NotResource, Principal/NotPrincipal), replacing the ad hoc "border rounded" utility classes that were doubling up with the fieldset's own border. Also gave the "Advanced fields" <details> a small top margin to match the new spacing. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe * admin: suggest bucket/* alongside the bucket itself in Resource autocomplete At the bucket-name stage of the Resource field's progressive autocomplete, only "arn:aws:s3:::bucket" was offered. Add "arn:aws:s3:::bucket/*" right alongside it, since granting access to everything in a bucket is the more common case and previously required typing a "/" first to reach the folder-level "*" suggestion. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe * admin: keep an unparseable policy in the JSON tab instead of wiping it editPolicy() built the structured state inside the fetch .then, so a policy the editor cannot model threw into the sibling .catch, which alerted and called hide(). Showing the alert at that moment left the modal on screen with an empty editor and the document only in the JSON tab, and Save Changes then serialized the empty state over the policy. Reachable two ways, since neither create path rejects these: the admin API never validates on create, so "Effect":"allow" is stored as typed, and policy_engine.validateStatement lets Resource and NotResource sit in the same statement. Hand the document to the JSON tab instead, which is what that tab is for, and mark the state so nothing serializes the placeholder over it. * admin: validate a policy document before saving it Validation was wired only to the Validate button, so nothing stopped a document the server's own validator rejects from being stored. With the structured editor supplying the boilerplate and required dropped from the textarea, opening the modal, typing a name and clicking Create Policy was enough to save a statement-less policy. Share validatePolicyJSON with the two save paths and abort on failure. * admin: bound the folder autocomplete listing maxListFoldersEntries caps the folders collected, but nothing capped the entries paged through to find them, so a bucket holding only flat object keys - no subfolders to count - was walked to the end, 200 entries per round trip, behind one keystroke. Measured against an in-process filer: 6 entries 0.5ms, 3k entries 7.7ms, 30k entries 53ms, all of it linear in the directory rather than in the answer. Cap the scan as well, and let GetFileBrowser take a prefix so the segment the user is still typing is filtered by the filer instead of by paging. The same 30k directory now answers in 0.6ms once a prefix is typed. * admin: clean the path before scoping list-folders to /buckets util.CleanWindowsPath only rewrites backslashes, so "/buckets/../etc" walked straight past the prefix check the endpoint relies on for its scope. Nothing leaked - filer paths are literal keys, so the traversal resolved to nothing - but the check reads as a boundary and wasn't one, and the test asserting it didn't cover the one input that would try. validateAndCleanFilePath in the same file already does this. * admin: stringify policy values before escaping them escapeHtml calls text.replace directly, and the Sid, the per-item action and resource inputs, and the View modal's Resource/NotResource all pass values straight out of JSON.parse. A policy carrying "Sid": 5 or "Action": [1] threw "text.replace is not a function" and took the render with it. escapedJoin already coerced; use it everywhere and coerce the editor state at the point it's built. * admin: only show the NotResource hint in NotResource mode The hint rendered unconditionally, so it sat under a selector reading "Resource" telling the user the statement applies to everything except what they'd listed. Redraw the card when the selector changes so it follows the mode. --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris LuandGitHub
|
228500fe37 |
install.sh: install the Rust maintenance worker (#10882)
* install.sh: install the Rust maintenance worker The release publishes weed-worker but the installer only knew weed and the Rust volume server, so the one binary that cannot be built without a Rust toolchain was the one you had to download by hand. --component all skips it on a platform it has no build for rather than failing an install that already put two binaries in place; asking for it by name there still says so. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * install.sh: clean each component's temp directory as it finishes The EXIT trap is per-process, so installing more than one component left every extraction but the last behind. Cleaning at the end of the function keeps the trap for the paths that exit early. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm |
||
|
Chris LuandGitHub
|
7ebf2ebac3 |
Build the Rust worker against the protoc that ships with the build (#10881)
* worker: compile plugin.proto with the protoc that ships with the build seaweed-volume already does this: protoc-bin-vendored carries the binary, so the build needs no package manager and every build sees the same version. An explicit PROTOC still wins, which is what lets the lance crates - whose own build scripts read the same variable - share it. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * ci: point the worker builds at the vendored protoc The jobs installed protobuf-compiler for lance's build scripts. They read PROTOC, so pointing it at the binary protoc-bin-vendored already puts in the registry serves them without a system package - one less apt call on the way to a release, and the same protoc a developer's build uses. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * docs: say what the worker build needs from protoc The lance crates' build scripts are the ones that need it, not ours, and they take the same vendored binary. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm |
||
|
Chris LuandGitHub
|
cc8364a03e |
Ship the Rust maintenance worker with the release (#10879)
* worker: name the binary weed-worker It is the Rust side of `weed worker`, the way weed-volume is the Rust side of `weed volume`, and lance is the first family of jobs it carries rather than the only one it ever will. The crate keeps its own name: when a second family arrives the bin target moves to a crate of its own, under this name. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * docker: ship the Rust maintenance worker in the image Lance table buckets need a worker that can read the format, and until now the only way to get one was a Rust toolchain and a cargo build. It now sits at /usr/bin/weed-worker beside the Rust volume server, reached as `docker run chrislusf/seaweedfs worker-rust --admin host:23646` — the verb mirrors volume-rust, so plain `worker` still runs the Go one. Taken pre-built or not at all: the lance jobs pull in arrow and datafusion, far too large a tree to compile inside the image build, so an architecture CI did not build for gets the empty placeholder the entrypoint refuses to exec, the way the Rust volume server already does. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * ci: build the Rust worker for the container images The same native cross-compile the volume server uses, so the release, latest and dev images all carry it on amd64 and arm64. The artifact holds both binaries now, so it is named for that rather than for the volume server. Only the release directory each job builds is cached: with a debug profile beside it the worker's target/ reaches 24GB, against a 10GB cache budget. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * ci: publish Rust worker binaries with the release Linux amd64 and arm64 only: the worker runs beside the cluster it maintains, and its dependency tree makes every extra target an expensive build. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * ci: build and test the Rust workers on change Nothing built seaweed-worker in CI, so the release and the container images would have been the first place a break showed up. Tests run in release too, rather than compiling lance, arrow and datafusion again in another profile. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * docs: say how to get a released worker Neither the image nor the release tarballs were mentioned; a toolchain and a cargo build read as the only way in. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * ci: install protoc for the Rust worker builds lance's crates compile their own protos, and unlike seaweed-volume they do not vendor a protoc to do it with, so every job that builds the worker failed at lance-encoding's build script. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * ci: do not persist credentials in the worker release checkout The upload step is handed a token explicitly; a cargo build script should not find another one sitting in the checkout's git config. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * docker: keep the worker's argument boundaries Unquoted $@ splits on whitespace and expands globs, so an argument carrying either arrived as something else. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm |
||
|
Chris LuandGitHub
|
c3f4d799b5 |
helm: install chart CI against an image tag that exists (#10877)
* helm: install chart CI against an image tag that exists The release bumps appVersion on master well before the container build publishes that tag, and the chart CI runs on the bump commit, so every release turns it red with ImagePullBackOff. Resolve the tag first and fall back to latest while the new one is still building. * helm: run the chart CI when the workflow itself changes * helm: bound the registry lookup in the chart CI An unbounded curl can hold the job, and the log did not say why the tag was rejected. Cap it and print the status. |
||
 github-actions[bot]
|
3563738699 | 4.44 4.44 | ||
|
Chris LuandGitHub
|
c1a993bc3b |
filer: keep the TUS sub-chunks that already landed when a write fails (#10876)
* filer: keep the TUS sub-chunks that already landed when a write fails A PATCH is split into 4MB sub-chunks, and each one is recorded in the session as soon as it is stored. The session listing is what HEAD reports as Upload-Offset and what the final entry is assembled from, so a record is a promise that the data behind it exists. When a later sub-chunk failed - a read-only volume, or a client that hung up mid-body - the error path deleted the needles of every sub-chunk the same PATCH had written but left their records in place. The resuming client was then told to continue past bytes the filer had just queued for deletion, and the upload completed into a gapless manifest pointing at needles that were gone: HEAD returned the right size, GET died mid-body once a vacuum reclaimed them. Recorded sub-chunks now stay, which is what resumption expects: the client picks up at the offset the session reports, and an upload that is abandoned frees its chunks with the session. * filer: drop a TUS chunk's record before freeing its data filer.CreateEntry can return an error with the entry already inserted - the parent-directory pass runs after the insert and keeps the entry when it fails. A failed saveTusChunk therefore does not mean the record is absent, and deleting the needle outright left the same corruption the resume path used to cause: a session record pointing at data that is gone. Remove the record first and only free the needle once it is gone. A record lost with its data still stored merely leaks, which the vacuum and fsck paths already account for. * test: cover a TUS PATCH that is cut off mid-body Resets the connection after one 4MB sub-chunk has landed, resumes from the offset the session reports, and vacuums before reading the file back, so anything the filer deleted behind a kept record shows up as a short read. |
||
|
Chris LuandGitHub
|
34bb444f33 |
test: drive the Lance namespace with Spark (#10864)
* test: drive the Lance namespace with Spark
The counterpart of catalog_spark, which does this for the Iceberg REST
catalog. Spark is the engine most likely to be pointed at a lakehouse,
and it reaches the Lance catalog through the connector's DSV2 catalog -
org.lance.spark.LanceNamespaceSparkCatalog with impl=rest - over the same
routes every other client uses.
SHOW NAMESPACES -> ['`sparklance-lcephd80`.ml']
SHOW TABLES -> ['sparklance-lcephd80$ml$embeddings']
count -> 3
filtered -> [(2, 'two'), (3, 'three')]
count after a second commit -> 4
The second insert is there on purpose: a store that cannot order commits
fails on the second one, not the first.
Two things the run settled that were guesses beforehand. CREATE TABLE
works, because the connector declares through the namespace and writes the
data itself rather than pushing Arrow at the server. And SHOW TABLES
returns the namespace's own identifiers - bucket, namespace and name
joined by the delimiter - not bare Spark table names.
Credentials go under the catalog's storage.* prefix, which is handed to
lance as object_store options; a gateway without STS vends none, the same
trap the LanceDB suite documents.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: verify the Lance table bucket was actually created
weed shell prints a command's own failure and still exits 0, so the harness
would go on to blame Spark for a bucket that was never made.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: bound the Docker probe
An unhealthy daemon makes docker version hang, and the probe runs before the
test has a timeout of its own.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: keep the ivy cache under the user's cache directory
It is mounted into a container running as root, so a shared temp path lets
another local user pre-create it and choose what Spark loads.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: assert the vector column's type, not only its name
A column that came back as array<double> or array<string> would still be
called vector and still pass.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: read the dataset off its location for real
The catalog being optional is the property that lets duckdb and pandas read
these tables; it was asserted in a comment and printed, never exercised.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: do not persist credentials in the Spark Lance checkout
The job only uploads a log on failure; nothing in it pushes.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: say the hosts in the README are placeholders
The suite passes dynamically allocated host.docker.internal ports.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
|
||
|
Chris LuandGitHub
|
4af6798639 |
helm: render the mysql secret and env only for the mysql filer store (#10872)
The db credential secret and the filer's WEED_MYSQL_* env were gated on filer.enabled alone, so a filer on mongodb, redis, postgres or leveldb2 got a generated mysql secret it never reads - kept forever by resource-policy: keep - plus a mysql-db-host pointing nowhere. Gate all of it on WEED_MYSQL_ENABLED, which is how the store is selected, plain keys and secret-backed ones alike. An enable flag the chart cannot read - a valueFrom, or one in secretExtraEnvironmentVars - counts as selected, so nothing is dropped from a filer that is actually on mysql. |
||
|
df93d01c06 |
admin: add bucket lifecycle rule editing (#10860)
* admin: add bucket lifecycle rule editing * address greptile's comments * more small fixes * coderabbit's comments * more comment fixes * more fixes * more * maybe last * last ? * 14850 * 14851 * filer: stamp the content MD5 on every SaveInsideFiler write An entry's ETag falls back to Attributes.Md5, so conditional writers key IF_ETAG_MATCH off it. SaveInsideFiler carried the looked-up attributes forward without refreshing the hash, leaving it describing whatever the previous writer stored: a later conditional write matched the stale hash and overwrote content that had already changed. * s3api: give the bucket lifecycle constants and the write route key one definition each The extended-attribute keys, the XML size cap and the object-write ring key prefix were each spelled out in two places, so the admin dashboard's copies could drift from the gateway's. Move them to the packages both sides already import and alias them where the short local name reads better. * admin: patch the bucket entry's lifecycle keys instead of rewriting the entry The save read the bucket entry, edited its extended map and wrote the whole entry back, guarded by IF_UNMODIFIED_SINCE. Nothing that writes a bucket entry advances its mtime - not the S3 gateway's patchBucketEntry, not SetBucketOwner, not SetBucketQuota - so the guard never fired and the stale snapshot reverted whatever else had changed since the lookup. Send the PATCH_EXTENDED mutation the S3 gateway already uses for these keys: the filer re-reads and merges under the bucket path lock, so only the two lifecycle keys move. That removes the reason for the mtime snapshot, the verification retry loop and the compensating restore of the cleared day-TTL rules, which the migration now logs instead. * s3api: run the delete-lifecycle day-TTL migration through the shared helper DeleteBucketLifecycleHandler kept its own copy of the read-strip-write sequence the put handler now shares, including a missing return that let a ToText failure persist a truncated filer.conf and write a second response. It also wrote the whole file back unconditionally, reverting any concurrent edit; the shared helper writes conditionally. * admin: answer 404 when a lifecycle request names a bucket that does not exist Every SetBucketLifecycle failure came back as 500, including the lookup miss for an unknown bucket, so a client or monitor read a caller error as a server fault and retried it. * s3api: emit lifecycle XML a client would recognize Two changes to what MarshalCanonical writes, both visible through GetBucketLifecycleConfiguration, which replays the stored bytes verbatim: stamp the S3 namespace on the root, and put a size range under <And>. A <Filter> carries one predicate, so two size bounds side by side is a shape AWS does not document. Parsing still accepts either. * admin: fix the lifecycle editor's handling of stored status, deletes and empty saves Four things the editor got wrong: A stored <Status> the S3 API never validated, say 'enabled', left both radio buttons unchecked, so reading the form threw on a null querySelector result and Save did nothing. Collapse anything but an exact 'Enabled' to 'Disabled', which is what the engine already does with it. Deleting a rule re-rendered an open edit form from the snapshot taken when editing began, discarding what had been typed; every other transition folds the form in first. The Transition warning only matched a bare <Transition>, missing the form with attributes, self-closed or namespace-prefixed. Saving an emptied rule list clears the configuration through a path with no prompt, next to a Delete-all-rules button that asks. Also collapses the three divergent copies of formatBytes on this page to one. * filer: stop the day-TTL migration from deleting an operator's path rule The migration removed every rule under the bucket's path that carried a day TTL in the bucket's collection. The add path it is retiring used AddLocationConf, which merged its TTL onto whatever already sat at the prefix, so a rule can hold operator settings the lifecycle path never wrote - a disk type, WORM retention, a read-only flag, a placement pin. Deleting the whole rule to retire its TTL took those with it, leaving objects under that prefix on defaults nobody asked for. Delete only rules shaped like ones the add path created from scratch; anything else keeps its settings and loses just the TTL. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris LuandGitHub
|
301d83cc7a |
test: wait for the master to register the volume servers before failover tests run (#10871)
The failover harness treated an open volume server port as readiness, but the master only learns of a volume server from its heartbeat. A lone master refuses heartbeats until its bootstrap check elects it, and the servers back off and retry, so registration lands seconds after the ports answer. Tests that started writing in that window assigned against an empty topology, which fails with "no free volumes left" and reaches the mount as ENOSPC. |
||
|
Chris LuandGitHub
|
c0a9b110dd |
volume: stop reporting read-only volumes that are no longer here (#10867)
* volume: clear per-collection metrics when a collection leaves a server The read-only and disk size gauges are only ever set for collections the heartbeat still finds here, and nothing zeroes the rest. volume.balance marks a volume read-only to move it, so the last heartbeat that saw it counts it read-only - and if it was the collection's last volume on that server, that count stands until the process restarts. The dashboard then shows read-only volumes that volume.list -readonly cannot find anywhere. Remember what each heartbeat set, and drop what is gone on the next one. * volume: stop the read-only volume count from wrapping at 256 The per-collection counters were uint8, so a server holding 256 read-only volumes of one collection reported zero of them. * volume: read the read-only flags once when counting them The heartbeat asked IsReadOnly for the verdict and then read noWriteOrDelete and noWriteCanDelete straight off the volume, unlocked, so the reasons could disagree with the verdict they were explaining. Take them together, under one lock. The location is now nil-checked rather than skipped by short-circuit evaluation, so a volume that has not joined a disk location yet stays safe. * volume: let only a surviving volume keep its collection reported A volume being deleted for expiry still made an entry in the read-only counts, which is what the cleanup reads as "this collection is still here". The collection's last volume could go and its series would stand for one more heartbeat. Count the survivors only. * volume: size a collection from the volumes it still has The size totals are rebuilt from scratch every heartbeat, so subtracting a volume that is about to be deleted took the surviving volumes' sizes down with it: a collection keeping a small volume and losing a larger one reported the difference, or lost its entry and kept the previous heartbeat's number. * volume: cover the deleted bytes total in the surviving volume test Deleted bytes are totalled the same way as sizes and were going unchecked, so the test now leaves deleted needles on both volumes and pins that gauge too. |
||
|
Chris LuandGitHub
|
96304b6870 |
S3: source config credentials from the environment, and let the chart point at an existing secret (#10868)
* s3: resolve ${VAR} in static config credentials from the environment
A deployment that keeps its S3 keys in a secret store had no way to hand
them to the gateway: -config takes a file, so the keys had to be written
into that file. Let a key in the static config name an environment
variable instead, and drop any credential whose reference stays unset so
the placeholder never becomes a usable key.
* helm: source the generated s3 identities from an existing secret
The only way to reuse credentials that already live in a Secret was to
hand-author the whole seaweedfs_s3_config JSON, since the literal keys in
values.yaml end up in git and a lookup-based keyRef renders empty under
helm template and Argo CD. Let s3.credentials.admin/read name a Secret and
its keys instead: the generated config references them as ${VAR} and the
gateway resolves them from the environment, so nothing is read from the
cluster at render time.
* s3: treat an empty environment value as an unresolved credential reference
A secret store can hand over a key that exists but is blank. Resolving it
would leave an access key whose signing secret is empty, so count it as
unresolved and drop the credential.
* helm: render the s3 secret when only the all-in-one auth flag is set
The all-in-one deployment mounts the s3 secret whenever any of the three
enableAuth flags is set, but the secret itself only rendered for the s3 and
filer flags, so allInOne.s3.enableAuth on its own left the pod waiting on a
secret nothing creates.
* helm ci: check the credential wiring on every workload that mounts it
The render check only looked at the standalone s3 deployment and only at
one of the four variables, so a helper that bound a variable to the wrong
secret key would still pass.
* helm: create the all-in-one s3 secret for every flag that mounts it
The all-in-one pod mounts the secret on any of the three enableAuth flags,
so keying its creation off allInOne.s3.enableAuth alone still left
filer.s3.enableAuth without filer.s3.enabled pointing at a secret nothing
creates. Mirror the deployment's own condition instead, and check each
flag renders both the mount and the secret.
* s3: reject a malformed credential reference instead of keying on it
A typo such as ${MY-VAR} matches no substitution, so it survived expansion
and the placeholder itself became the access key the gateway accepted.
Require every ${ in a static credential to open a well-formed reference.
|
||
|
Chris LuandGitHub
|
480795d40d |
release: cut the whole release from the version bump workflow (#10870)
* release: cut the whole release from the version bump workflow The bump workflow stopped after pushing the version commit, and the rest was manual: create the release, then run "Prepare release" in the csi-driver and the operator. It now pushes the tag itself, which is what starts the binary, container and helm workflows, creates the release with generated notes, and dispatches the other two repositories, waiting for both. Pushing the tag and reaching the other repositories both need RELEASE_PAT; GITHUB_TOKEN raises no events that start workflows. * release: tighten the release workflow after review Check out master explicitly: a dispatch can select any branch, and the tag, the commit and the release would then come off that branch while the downstream job dispatches master. Scope contents:write to the job that pushes; the downstream job talks to the other repositories with RELEASE_PAT and needs nothing here. Wait for the module proxy to serve the release commit as the tip before dispatching, instead of priming it and hoping. The dispatched workflows pin seaweedfs with `go get -u ...@latest`, so a stale tip means they release against a pre-release commit, silently. Identify the dispatched run by diffing the run list against the snapshot taken before dispatching, rather than assuming the newest run is ours. * release: wait on the downstream release, not on the run that makes it A dispatched run cannot be told apart from a concurrent one: the API does not report the inputs a run was dispatched with, so watching "the run that appeared after mine" can watch someone else's and report their result as ours. Wait for a release to appear in the downstream repository instead. That is the thing being waited for, and it holds however many runs are in flight. |
||
|
Chris LuandGitHub
|
5e7ab43ddd |
test: read Lance tables from DuckDB (#10866)
* test: read Lance tables from DuckDB
The LanceDB and Spark suites go through the catalog. DuckDB does not: its
lance extension reaches the data over S3 with no namespace involved, which
exercises the other half of the design - a table bucket's layout is a
valid Lance dataset directory, so a table stays readable when the catalog
is not in the path.
scan_rows=128
scan_columns=id,title,vector
filtered_rows=5
nearest=1,0,2
It also pins the one place the layout costs us. DuckDB's replacement scan
recognises a dataset by a .lance path suffix, and tables created through
this catalog deliberately have none: the catalog entry is the dataset
directory, a table name may not contain a dot, and a suffix would leak
into ARNs and policies. So __lance_scan is the way in, and the bare
SELECT ... FROM 's3://...' form does not see these tables.
The test asserts both halves - a suffixed path is read, a suffix-less one
is not - so if the extension ever recognises a bare directory, it fails
and says to update the documentation rather than leaving it wrong.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: require the catalog error from the suffix-less read
Any failure satisfied the old check - a missing extension, bad credentials,
an unreachable endpoint - so the assertion could pass without the
replacement scan ever classifying the path.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: verify the Lance table bucket was actually created
weed shell prints a command's own failure and still exits 0, so the harness
would go on to blame DuckDB for a bucket that was never made.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: bound the Docker probe
An unhealthy daemon makes docker version hang, and the probe runs before the
test has a timeout of its own.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: order the aggregates the assertions read
string_agg over an unordered relation may return the names, and the vector
search's ids, in any order, so the expectations could fail on a run where
nothing changed.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: do not persist credentials in the DuckDB Lance checkout
The job only uploads a log on failure; nothing in it pushes.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
|
||
|
Chris LuandGitHub
|
35d53a20f6 |
master: let the leader admit a master that starts with no raft state (#10865)
* master: answer with the leader raft already knows Topo.Leader() backs off for up to 20 seconds waiting for an election. Callers that a health probe or a client is blocked on cannot afford that: /cluster/status, /cluster/healthz and /readyz all sit past the probe timeout of both the helm chart and the operator, so a master that is still joining looks dead rather than joining, and the kubelet restarts it. informNewLeader and SendHeartbeat hold the client on a master that cannot serve it, exactly when it should move on to find the one that can. Answer these from MaybeLeader instead, which reports what raft knows right now. MaybeLeader takes over the "am I the leader myself" fallback that Leader() used to apply on top of it, so one non-blocking call is still correct; Leader() keeps the backoff for callers that must wait. * master: let the leader admit a master that starts with no raft state Neither raft implementation lets a server outside the configuration campaign: goraft's promotable() requires a non-empty log, and hashicorp rejects vote requests from a candidate that is not in its configuration. A master that comes up with fresh state therefore cannot elect itself in — the leader has to pull it in. Nothing did. The peer list is static, rendered from the replica count, so scaling it up leaves the sitting leader running the old list with no idea the new masters exist. Under goraft they wait forever. Under hashicorp they are worse off: each bootstraps a cluster of its own from the new list, and two of them form a quorum next to the live leader, with their own TopologyId. That is the split brain SetTopologyId kills a master over. Admit the peer where it registers instead. Only the leader gets past the IsLeader check in KeepConnected, and a joining master's client lands there, so that is the moment it joins. The broadcast OnPeerUpdate rides on is not enough on its own: it only reaches masters already connected, which is why a leader that came up first missed both newcomers. RaftAddServer grew a goraft branch on the way, so cluster.raft.add stops silently doing nothing on the default raft, and RaftRemoveServer with it. Bootstrapping is now one call for both implementations, made only after the peers confirm nobody has a leader, and retried until this master is in rather than checked once and dropped. * master: do not evict a peer that is still in -peers The hashicorp leader drops a master from the raft configuration as soon as it stops answering pings. A master that is merely restarting answers nothing, so an ordinary bounce shrinks the quorum behind the operator's back — and then races its own return: the master comes back, registers, gets re-admitted, and the eviction lands after it. A randomized start/stop walk lands on it. Two of three masters running, the leader evicts the one that just went down, the restart re-adds it, the removal commits late and takes the leader's own leadership with it. What is left is a two-server configuration whose other half is down, and a running master that nobody will ask for a vote — no quorum, no way back until the third master returns. -peers is what declares membership. updatePeers already reconciles the configuration against it on every leadership change, and an operator who really means to drop a master can say so with cluster.raft.remove, so keep the eviction for masters that are no longer listed at all. * test: bounce masters at random and hold the election to it Twelve rounds of stopping or starting a random master, on both raft implementations, checking the two things an election must never get wrong: two masters claiming leadership at once, and a quorum that comes back without agreeing on one. The cluster's identity has to survive the whole walk, since a master that re-mints a TopologyId is the split brain SetTopologyId kills its peers over. The seed is random and logged, so a failure names the walk that reproduces it. Below a quorum the walk moves straight on. A master that has lost its quorum cannot commit anything, and goraft only checks whether it still has one on an election-timeout ticker, after its peers have been quiet for a full timeout — measured taking over 30 seconds to step down. That direction belongs to TestTwoMastersDownAndRestart, which was giving it ten seconds and would have started failing on a slower machine; it now waits on that behaviour explicitly rather than sleeping twice and hoping. WaitForTopologyId returns the id it waited for. Reading it separately raced the leader applying the raft entry that carries it, which shows up as an empty id right after an election rather than as a wrong one. |
||
|
Chris LuandGitHub
|
0c95137528 |
filer: stop aggregated metadata subscribers from spinning on a peer watermark hold (#10863)
* fix(filer): stop logging a held aggregated read as an error An aggregated subscriber may not read past the peers' low-watermark, and it stops at the first entry beyond it by returning a sentinel from the read callback. LoopProcessLogData logs every callback error, so on a cluster that keeps writing - where there is almost always an entry newer than the watermark - every read wrote an ERROR line naming the entry it stopped at, thousands per minute per filer. Mark the stop as control flow: an error wrapping StopReadingError is handed back to the caller unlogged, and the held-read sentinel wraps it. * fix(filer): release an aggregated watermark hold on peer progress A held read waited on the aggregated buffer's data channel, which the next write signalled - but a write cannot release a hold, only a peer reporting further progress can. On a cluster that keeps writing the loop therefore re-ran a whole pass per arriving event, log file listing and all, and held again on the same entry every time. Signal held readers from the meta aggregator instead, whenever a low-watermark rises: a peer reporting, or one dropped past its removal grace. The retry interval stays as the backstop for what no watermark covers. Count the holds so a parked subscriber stays visible. * fix(filer): floor how often an aggregated watermark hold releases Peers advance their delivery watermark on every event they stream, so releasing a hold on every advance is the same pass-per-event storm as releasing on every write, just without the log lines - and each pass lists a day of log files. Floor the release at 20ms. Advances inside the floor collapse into one release, which then delivers everything they covered. * fix(filer): pace a peer's delivery claim by what its subscribers hold at A filer's local metadata stream carries an idle heartbeat to its peer aggregators, and each peer turns it into that filer's delivery low-watermark. Aggregated subscribers hold at the minimum across peers, so a filer quiet enough to fall back on the heartbeat parked every subscriber in the cluster up to a keepalive interval - 5 seconds - behind live writes. With nine filers, most of them quiet at any moment, the minimum sat there permanently. Pace that heartbeat at 200ms once the filer has peers. It stays a keepalive, at the keepalive interval, for a filer with none. * fix(filer): wake each aggregated hold on its own watermark A persisted-log read is held by what the peers have flushed, an in-memory read by what they have delivered, but both parked on one channel closed whenever either minimum rose. Peers advance their delivery watermark on every event they stream, so a flush-held reader woke at the coalescing floor to re-list a day of log files and park again on the same entry - the storm this set out to fix, in the one place asymmetric peer progress still reached. Signal the two separately and park each read on the one that bounds it. |
||
|
Chris LuandGitHub
|
0dfaa103d0 |
test: take a table through its whole life, for Iceberg and Lance (#10862)
* lance worker: share the integration tests' scaffolding The recorder that keeps what a handler sent, the config builder and the storage-option fallback all lived inside compaction.rs, so a second test binary would have had to copy them. They move to tests/common. The fallback now reads AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_ENDPOINT_URL from the environment, defaulting to what it used before. A harness can then point these tests at a gateway that checks what it is given rather than one that accepts anything. * lance worker: maintain one named table, for a harness to drive Compacts and cleans up whatever WEED_LANCE_TABLE names, through the handlers' own detect-then-execute path: a proposal the worker would not have made is not one worth running. The existing tests seed the tables they check. This one deliberately does not, so a harness that has already written a table and knows what is in it can have the real handlers maintain it and then read it back. * test: take a table through its whole life, for Iceberg and Lance Created in the catalog, filled by a real client, maintained by the worker, read again, dropped. The step nothing was checking is the read after maintenance: compaction once rewrote every dictionary-encoded column onto a single value and shipped, because the maintenance tests were thorough about sequence numbers, manifest entries and metadata versions and none of them opened the parquet file the worker had just written. So the assertion is a tally - row count, the cardinality of each dictionary-encoded column, and an md5 over whole rows - taken before maintenance and again after, required to be equal. The cardinalities name the failure that happened; the digest catches a rewrite that keeps every column's cardinality and hands the values to the wrong rows. A compaction that merged nothing fails rather than passes, or the read afterwards is checking a file the worker never wrote. The Iceberg half runs two clients. DuckDB is the one the corruption was reported against and the only one here that writes the deprecated PLAIN_DICTIONARY encoding, which parquet-go normalizes away on write, so a Go writer cannot produce it. PyIceberg writes the modern spelling. Pinning parquet-go back to v0.30.1 fails the DuckDB half and passes the PyIceberg one, which is why both are here. Lance maintenance lives in the Rust worker, so it runs there where cargo is installed and through the two lance calls those handlers wrap where it is not. WEED_LANCE_MAINTENANCE picks one instead of letting the test guess. * ci: run the table lifecycle tests CI maintains the Lance table through the lance library rather than the worker: a cold build of the lance crate costs more than the glue it would be checking, and the worker's own tests cover its handlers. The suite drives the Iceberg maintenance worker, so a change to it now triggers this workflow too. * test: let the lifecycle harness fail instead of skipping Setup failures all exited zero, so a cluster that would not come up, or a port allocation that lost, reported a green run for code nothing had executed. That is the failure mode this whole directory exists to close, and it was in the harness itself. Only a checkout without a weed binary skips now, and it runs the tests so each one says so rather than the package quietly passing. Everything else fails. The filer existence probe gets a deadline while I am here: it ran without one, so an unresponsive filer would hang the suite past every timeout the clients have. * test: make the lifecycle checks check what they claim to Three of them could pass without having looked. The DuckDB skip matched "syntax error", "not implemented" and "Failed to load" anywhere in the output, in any phase. A parse error in the SQL this test generates, or a refusal from our own catalog, would have taken the only coverage of the PLAIN_DICTIONARY encoding out of CI and left it green. It now matches the extension failing to install, and only in the phase that installs it. Everything past LOAD is ours and fails. The digests covered id, category and value. Compaction rewrites the whole row, so a defect confined to ts, or to a Lance vector, changed nothing either side of maintenance. Every persisted column goes in now, ts as microseconds so no timezone sits between the two runs. The Lance drop check caught every exception as proof the dataset was gone. pylance turns credential and transport failures into the same ValueError, so it only accepts the message that means not found. * docs: say up front which maintenance path the Lance half takes The opening summary said the worker maintains both tables. It maintains the Iceberg one always and the Lance one only where cargo is installed, which is not what CI does. |
||
|
Chris LuandGitHub
|
3bd218e030 |
volume: cut idle memory at high volume counts (#10861)
* volume: start a volume's batch write worker on first use Mounting a volume started a goroutine parked on a 128-slot channel, plus the 128-entry batch slice it had already allocated. That is around 6.7KB per volume the server pays whether or not the volume ever takes a write: 7231 bytes per mounted volume, of which 4101 is goroutine stack. Only a write that asks for fsync ever reaches the worker, and a remote-tiered or read-only volume never can. Create the channel and its goroutine on the first such request instead, and let a write arriving after Destroy fall back to the inline path rather than queue onto a worker that has gone. Measured over 20000 mounted volumes: 7231 -> 1269 bytes each. * volume: update the heartbeat report state in place Every heartbeat built a second map of what it was about to tell the master, holding a freshly allocated short information message per volume, then swapped it in over the old one -- and computed departures through a third map of the live volume ids. A server holding 2M volumes rebuilt all three every VolumePulsePeriod for a report that usually says nothing. Number the heartbeats instead and mark the entry already held with the pass that found the copy, so a quiet volume costs a map lookup and no allocation. Departures are the entries a pass did not mark; the live-id map is now built only when there are some, sized to them. Measured over 10000 mounted volumes: 436 -> 196 bytes allocated per volume per heartbeat. * volume: fill one volume information message per heartbeat, not per volume The heartbeat built a message for every volume held so it could hash it, then dropped all but the few it had something to say about. At 2M volumes that is 2M messages allocated every VolumePulsePeriod to send almost none of them. Fill a message the caller supplies instead, and replace it only when the heartbeat keeps it, so a server with nothing to report fills the same one all the way through. Measured over 10000 mounted volumes: 196 -> 4 bytes allocated per volume per heartbeat, and a heartbeat runs a third faster. * volume: drop the per-volume trace from the heartbeat's status read glog.V(4).Infof evaluates its arguments whether or not the verbosity is on, so every volume boxed its id into a fresh interface slice on every heartbeat: 759 of the 773 allocations a 1000-volume heartbeat made, for a line that at this scale would print millions of unreadable rows. Measured over 1000 mounted volumes: 4776 -> 1792 bytes and 759 -> 14 allocations per heartbeat, which no longer grows with the volume count. * seaweed-volume: mirror the in-place heartbeat report state Same change as the Go volume server: number the heartbeats and mark the entry already held with the pass that found the copy, instead of building a second map of hashes and swapping it in. The volume snapshot must leave the reporting state as it found it, so it keeps asking through changed() while a real heartbeat marks through record(). * volume: refuse writes to a closed volume instead of dereferencing nil Close and Destroy leave the needle map and data backend nil, but a caller that already holds the volume can still reach the write path, where both are used unguarded: a write racing a volume deletion took the server down. syncDelete has always checked; syncWrite and the batch worker had not. Reachable before this series and now also from the inline fallback a durable write takes when the worker has gone. * seaweed-volume: guard the report state with one mutex, as Go does The full-list flag and the generation that answers it have to move together. Split across separate atomics they cannot: a request landing between begin's two reads returns full == false with the generation it just raised, and one landing between commit's read and its clear is marked answered by a heartbeat that carried no list. Either way the resend is dropped. Neither is reachable today -- every caller reaches this through the store's RwLock, the flag setters under a read lock and the heartbeat build under a write lock, so they cannot interleave. The type should not depend on that being true two files away, and Go holds a single mutex over exactly these fields. * test: build the servers under test to match the harness's offset size The mixed Go/Rust suites run both servers against one dataset, so both have to agree on the offset width. They did not: the harness built Go with no tags, 4-byte offsets, while the Rust crate defaults to its 5bytes feature, and the Rust server then refused the .vif the Go server had just written -- "bytes_offset mismatch: found 4, expected 5". Build each side to match the offset size the test binary itself was compiled with, so a plain `go test` and one with -tags 5BytesOffset both get a matched pair. |
||
|
930603eb74 |
S3: optionally serve remote-mounted objects from remote when the local read fails (#10837)
* feat(s3): serve from remote on local read failure When a locally-cached chunk of a remote-mounted object becomes unreadable (volume server down/restarting, or an evicted needle 404ing under retry-backoff), fall back to serving the object from its mounted remote instead of erroring. A bounded pre-flight probe makes a stuck volume trip the timeout rather than stalling the request. Gated by -localReadFallbackToRemote (default off) with -localReadFallbackTimeout (2s default), so existing deployments are unaffected until they opt in. * fix(s3): register local-read-fallback flags for mini/server/filer The mini, server and filer launchers build S3Options directly and only populate the flag pointers they register. Without registering the two new flags there, startS3Server dereferenced nil pointers and crashed at boot, failing every integration suite that runs `weed mini`. * fix(s3): treat a zero-byte probe read as unreadable A read that returns no byte -- whether it reports io.EOF or no error at all -- means the offset is not locally readable, so the probe must fall back to the remote rather than proceeding to stream a truncated response. Only a returned byte (including the object's final byte with a trailing io.EOF) counts as readable. * s3: finish a mid-stream local read failure from the remote mount The pre-flight probe only proves the byte at the requested offset readable. A multi-chunk object can still lose a later chunk after the 200/206 and its Content-Length are committed, which truncated the body with no fallback. Resume from the mounted remote at the byte the local copy stopped at, so the response still carries the declared length. A short local read that surfaces as a clean EOF is treated the same way instead of silently truncating. * s3: fall back to the remote mount without a CLI switch Serving a remote-mounted object from its authoritative remote is what the read should have done all along -- the alternative is a 500 on an object the cluster can still reach -- so make it the behavior instead of two new flags, with the probe bounded by a constant. * s3: trim the comments on the fallback path * filer: report only the contiguous prefix when a parallel chunk read fails The parallel branch of doReadAt fans the chunk reads straight into their own windows of the output buffer, then sums every task's bytesRead. A middle chunk failing while a later one succeeds therefore returned a length covering a hole the reader never filled, handing the caller zeros in the middle of otherwise valid data. * s3: only splice the remote onto a local prefix while it is the cached generation Eligibility establishes a size match, not byte identity: a remote key overwritten with same-size content between the cache fill and the fallback would have finished the response with bytes from a second generation, under the first one's ETag. Stat the remote before resuming and keep the local error when it no longer matches -- a truncated body is a visible failure, a spliced one is not. --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris LuandGitHub
|
6faa9d20e8 |
iceberg: stop compaction from corrupting dictionary-encoded columns (#10857)
* deps: upgrade parquet-go to v0.32.0 Iceberg compaction writes the merged file with the schema of its first input, encodings included. parquet-go before v0.31.0 took the deprecated PLAIN_DICTIONARY encoding that DuckDB writes at face value and encoded those pages as plain int32 indices, but the spec gives PLAIN_DICTIONARY the same bit-width-prefixed RLE layout as RLE_DICTIONARY. Every dictionary-encoded column in a compacted file then decoded onto a single dictionary entry, and anything past one page failed to decode at all. * iceberg: cover compaction of dictionary-encoded input The fixture is a DuckDB-written file, so it carries the PLAIN_DICTIONARY encoding a Go writer will not produce. * iceberg: tally whole rows in the dictionary merge test Counting each column on its own passes a merge that remaps names while leaving their cardinality intact. |
||
|
Chris LuandGitHub
|
813c439af6 |
admin: regenerate the gzipped static mirror (#10859)
The toast and modal changes edited static/js/ without rerunning gen_static_gz.go, so the embedded assets served to browsers still carry the old scripts and TestStaticGzMirror fails on master. |
||
|
Chris LuandGitHub
|
bd34565e56 |
admin: keep the copy confirmation in front of the access key modal (#10856)
* admin: raise nested modals above the ones already open Bootstrap gives every modal and every backdrop the same z-index, so a modal opened while another is showing paints behind it and its buttons cannot be clicked. Viewing an access key secret and then copying a field left the confirmation stuck behind the details modal with no way to dismiss it. Give each nested modal, and the backdrop Bootstrap creates for it, a z-index above what is already on screen, and put back the scroll lock that Bootstrap drops as soon as any one of them closes. * admin: confirm clipboard copies with a toast The access key details modal offers three copy buttons, and each one raised a modal that had to be dismissed before the next copy. Confirm with a toast instead, so the credentials stay in view and nothing has to be clicked away. |
||
|
github-actions[bot]
|
6c7f184381 | 4.43 4.43 | ||
|
Chris LuandGitHub
|
8a532cc0cf |
mini: state the format of a -tableBucket, do not infer it (#10851)
* mini: state the format of a -tableBucket, do not infer it
A table bucket holds one format and that format decides which catalog can
serve it, but the flag only took names. The format came from
miniTableBucketFormat(): Iceberg whenever its port was up, Lance only when
it was not. So -tableBucket=vectors on a default mini quietly made an
ICEBERG bucket that the Lance namespace then refused every table in, and
the only way to get a Lance one was -s3.port.iceberg=0, which buys it by
deleting the other catalog. One flag, two meanings, decided by an unrelated
port.
Each entry is now name[:FORMAT], unsuffixed meaning ICEBERG as before:
weed mini -tableBucket=warehouse,vectors:LANCE
Both catalogs stay up and both buckets are reachable. A name whose format
has no endpoint here is skipped with a warning rather than created out of
reach, and the Iceberg-only S3_TABLE_BUCKET default-routing hint gets the
Iceberg names alone, without their suffixes.
* mini: do not reuse a table bucket that holds another format
CreateTableBucket answers BucketAlreadyExists on the name alone, so
-tableBucket=vectors against a bucket created as LANCE logged "already
exists" and moved on, and the Iceberg default-warehouse hint then pointed
at it. Every table create against that catalog fails with "table bucket
vectors holds LANCE tables", far from the flag that chose it.
ensureMiniTableBuckets now reads the format of a bucket it did not create,
warns when it is not the one asked for, and returns only the buckets that
hold what was requested. S3_TABLE_BUCKET is seeded from that list, so an
unprefixed Iceberg request falls back to its own default rather than
committing into a Lance bucket. A bucket predating declared formats reports
an empty one and still accepts either.
* mini: normalize S3_TABLE_BUCKET whichever way the spec arrived
The rewrite that keeps Lance names out of the Iceberg default warehouse only
ran when the flag supplied the spec. Set the variable directly, as the docker
quickstart does, and it reached the catalog untouched: S3_TABLE_BUCKET=
vectors:LANCE,warehouse made the unprefixed default the literal string
"vectors:LANCE", a bucket no lookup finds, while warehouse sat behind it.
The variable is both mini's input and the catalog's routing hint, so it is
now always rewritten from the buckets that came back holding Iceberg tables,
and unset when there are none rather than left pointing somewhere stale.
* mini: reuse a table bucket only when its format reads back
An ordinary S3 bucket wearing the name answers CreateTableBucket with the
same BucketAlreadyExists as a table bucket does, and the format lookup that
follows returned "" for a failed read exactly as it does for a bucket
predating declared formats. So -bucket=data -tableBucket=data reported
nothing and published data as the Iceberg default warehouse, where every
unprefixed request 404s on a bucket that is not a catalog.
The lookup now returns its error, and only a bucket that reads back as the
format asked for is reused. Anything else is left alone with a warning
naming why, rather than routed to and discovered later.
|
||
|
Chris LuandGitHub
|
83753ccdad |
test: drive the Lance namespace with LanceDB (#10850)
* test: drive the Lance namespace with LanceDB
The Iceberg catalog is checked against Spark, Trino, ClickHouse, Doris,
Dremio and RisingWave. The Lance one had only its own reference client,
which is the same thing as checking it against ourselves.
LanceDB connects with connect_namespace("rest", ...), which speaks the
routes this catalog implements, so the suite exercises the protocol rather
than our idea of it: list the catalog, open a table through it, read the
schema, run a vector search and a filtered scan, create a table, and read
the same dataset straight off its URI with no catalog at all.
table_names -> ['lancedb-p0guidmm$ml$embeddings']
open_table -> 64 rows
search -> [1, 0, 2]
create_table -> 4 rows, listed by the catalog
direct read without the catalog -> 64 rows
Seeding is pylance, because the namespace records where a table lives and
does not carry its data. That split is the design rather than a limit of
the test.
One interop note the test encodes: a gateway without STS vends
storage_options carrying an endpoint and a region but no credentials, and
LanceDB uses what the namespace vends on some paths. The container gets
credentials in its environment as well, which is what a deployment without
STS would do.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: pin the LanceDB client, and index before searching
Three from review.
The client's dependencies were unpinned, so an unrelated upstream release
could change what an old commit reproduces. Pinned to the versions this
suite was verified against; the client is as much the thing under test as
the server.
The search was called ANN and was not: without an index LanceDB scans.
The test now builds an IVF_PQ index over 1024 rows first, which is worth
more than the wording fix - an index writes into a directory of the table
that the S3 door has to admit, and that guard has refused a Lance
directory before. It builds, covers all 1024 rows, and searches.
The assertion moved with it. Demanding the exact nearest neighbour was
right for a brute-force scan and wrong for a quantized index, which
answered 0 as readily as 1; both are correct, so the check is now the
neighbourhood.
And the pushdown check accepted any failure. It now requires the refusal
to be the catalog's Unsupported and requires that nothing was left behind,
or, when the client falls back, that the table is complete.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
|
||
|
Chris LuandGitHub
|
6f3b5a4f4d |
metrics: graph the plugin workers (#10849)
Nothing displayed the worker metrics, and the panels that look like they did are about something else: Workers Connected, Worker Slots and Worker Events in the Admin / Maintenance row read SeaweedFS_admin_*, which the older maintenance queue feeds. A cluster running plugin workers - Go or Rust - reads zero there while they are connected and busy. A Plugin Workers row graphs what the workers themselves publish: how many are connected, jobs and their failures, detection and proposal rates, job duration, slot usage, stream events, and what the Lance jobs reclaimed. The panel worth having is Objects Seen vs Skipped, since a sweep with nothing to do and a sweep that could read nothing report the same number of proposals. Also a commented scrape target in the sample Prometheus config. It is 9328 rather than 9327: the sample compose already gives 9327 to the S3 gateway, so the port the worker's own usage text suggests collides with it on a single host. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm |
||
|
Chris LuandGitHub
|
f56a7a1557 |
seaweed-worker: serve health, readiness and metrics (#10848)
* seaweed-worker: serve health, readiness and metrics A Rust worker had no surface of its own. If it wedged, the only signals were its stdout and whatever admin could infer from a stream that had gone quiet; nothing could be scraped and nothing could be alerted on. --metrics-port serves /health, /ready and /metrics, the same three the Go worker serves under -metricsPort, so one scrape config covers workers in either language. Off by default, loopback unless --metrics-ip says otherwise, since the endpoint is unauthenticated. Names follow the Go convention, SeaweedFS_worker_*. The counters live in core and are raised where the stream already knows what happened - connect, close, detection, execution, preview - so a worker for another format gets them without writing any of this. Slots are published from the heartbeat that already computes them, so a scrape and the admin UI cannot disagree. The pair worth having is objects_seen_total and objects_skipped_total. A sweep that proposed nothing because there was nothing to do and a sweep that proposed nothing because it could not read anything are the same number of proposals; they are not the same event, and until now only a log line told them apart. The Lance jobs add what they reclaimed - fragments, rows brought under an index, versions, bytes - on the same registry, so one endpoint serves both. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * seaweed-worker: fix the metrics address, the count, and a dead field Three from review. --metrics-ip ::1 failed at startup: the address was built by joining host and port with a colon, and "::1:9327" is not an address. It is parsed as a host and combined with SocketAddr::new now, so an IPv6 literal works, with or without the brackets an operator will reasonably type after seeing one in a URL. proposals_total counted before the send rather than after, so a stream that closed mid-sweep left the counter claiming proposals admin never received. And MeteredSender carried a Metrics clone and a job type it never read, kept alive by two statements that existed only to silence the warning about them. Everything is recorded by the caller, so both are gone. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm |
||
|
Chris LuandGitHub
|
fc97f8ea8f |
mount: index directory state by path (#10827)
Every directory-state lookup went through path2inode, the map that holds one full path per inode in the table, and then through dirStates. Directories now carry their own path and are indexed by it directly. There are orders of magnitude fewer directories than files, so this map stays small whatever the mount holds, and it is what a file needs before it can stop carrying a full path of its own: a child's path is its parent's plus its name. No behavior change - the two indexes are asserted to agree. |
||
 
|
f7c4636d22 |
topology: refresh oversized mark on every heartbeat (#10829)
* topology: refresh oversized mark on every heartbeat The oversized flag on a volume location was only set when the volume was registered (RegisterVolume). A volume that later grew past the size limit kept its stale "not oversized" mark, so the heartbeat path (ensureCorrectWritables) kept re-adding it to the writable list while RecordAssign removed it on every assign - a writable/unwritable flip loop that let writes continue past the limit and made vacuum race in-flight writes. Refresh the mark from each heartbeat's reported size in both heartbeat paths (ApplyVolumeChanges and SyncDataNodeRegistration), mirroring what RegisterVolume already did at registration time. A volume that grew past the limit now stays unwritable, and one that shrank back clears the mark and can recover. * topology: order heartbeat writable correction after decay and honor cooldown Review feedback (Greptile, CodeRabbit) on the oversized-mark refresh: 1. Greptile: clearing the oversized mark before EnsureCorrectWritables let the delay-unaware helper re-add a just-compacted volume to writables, bypassing capacityRecoveryDelay. ensureCorrectWritables now checks fullSince and skips the re-add while the cooldown is pending, so a volume removed for capacity only recovers through UpdateVolumeSize's heartbeat recovery path. 2. CodeRabbit: in the full-heartbeat path the mark was refreshed after the writable correction, so a newly oversized volume stayed writable for an extra heartbeat cycle. The standalone changedVolumes loop is merged into the volumeInfos loop and EnsureCorrectWritables now runs after UpdateOversizedState + UpdateVolumeSize in both heartbeat paths, using the freshly refreshed mark. 3. TestHandlingVolumeServerHeartbeat used a size (254320) that is past the test's volumeSizeLimit (32768); it only passed because the stale mark hid the oversized state. Sized down to 30000 to keep testing the add/remove flow, and added TestEnsureCorrectWritablesHonorsRecoveryCooldown covering the cooldown window and the recovery after it. * topology: do not restore a still-crowded volume after the cooldown Greptile review: after capacityRecoveryDelay elapses, ensureCorrectWritables could restore a volume whose effective size is still past the crowded threshold. UpdateVolumeSize refuses the recovery (effectiveSize > crowded threshold -> setVolumeCrowded + return false), but the cooldown check in ensureCorrectWritables only looked at fullSince, so once the delay passed it re-added the volume even though capacity tracking still considers it crowded. Check the crowded mark before re-adding: a volume UpdateVolumeSize just marked crowded must not be restored here, otherwise assignments resume while the volume is still flagged for growth. Adds TestEnsureCorrectWritablesDoesNotRestoreCrowdedVolume: effectiveSize decays to 10500 (past the 9000 crowded threshold) after a report of 8000, and ensureCorrectWritables keeps the volume unwritable past the cooldown. * ci: trigger re-run of flaky FUSE jobs * topology: gate the writable restore on the limit, not on crowded A crowded volume is above the growth threshold, not full, and is normally writable. Refusing to restore one locks it out for good: nothing writes to a volume that is not writable, so its size can never fall back under the threshold. Gate on the same size the assign path uses to remove it. * topology: let only the heartbeat refresh set the oversized mark Registration also set it, from whatever VolumeInfo it was handed. The incremental path builds that from a short heartbeat message, which carries no size, so every arrival announcement cleared the mark and handed the volume back to the writable list until the next full report. * topology: use the re-resolved layout after a dropped one is replaced A layout dropped with its collection makes RegisterVolume refuse, and the full heartbeat then re-registered against a fresh layout but kept applying the size, oversized and writable updates to the dropped one. --------- Co-authored-by: hzsunchao <hzsunchao@corp.netease.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris LuandGitHub
|
8c7d714d5e |
Lance catalog, and a Rust plugin worker to maintain it (#10841)
* iceberg: skip tables the maintenance worker does not own A Lance dataset registered through the Lance namespace's Iceberg REST adapter arrives as an Iceberg table with a placeholder schema and table_type=lance, and keeps its fragments under data/ - the same subdirectory the orphan cleaner walks. Every fragment is unreferenced by the Iceberg metadata, so a maintenance pass deletes the dataset. Views share the entry shape and were only skipped because parsing their metadata happened to fail first. Gate the scan and the execution path on the entry actually being an Iceberg table. Maintenance is off by default, so this was latent rather than live. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: let a table declare a format the catalog does not interpret CreateTable accepted ICEBERG and nothing else. A Lance table has no metadata file for the catalog to maintain - the entry records a name and the dataset root, and the client owns everything under it - so accept LANCE, and carry the declared format on the entry instead of hardcoding it back on the way out. ListTables now reports format and metadataLocation, so listing a catalog that holds both kinds takes one pass rather than a GetTable per row. AWS omits both fields; adding them is additive. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: move the in-memory filer into its own package The Lance namespace tests need the same harness, and copying it would leave two of them to keep in step. Extracted as it was, plus the two fidelity gaps that only surface once a paginating caller uses it: ListEntries ignored startFromFileName and limit, so a caller that paginates re-read the first page until it hit its own cap and reported the same entry over and over, and GetFilerConfiguration was missing, which CreateTableBucket needs to resolve the buckets directory. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: serve the Lance Namespace REST spec A second catalog surface beside the Iceberg one, over the same table buckets: the namespace and table metadata operations, the $-delimited identifier codec, the spec's numeric error model, the directory-catalog marker files, and storage_options vending through the STS path the Iceberg catalog already uses. Listens on -port.lance, 9101 by default, and inherits ARNs, policies and tags from the storage layer, so a Lance table needs no second permission model. Identifiers map bucket / namespace / table onto the three levels Lance clients already use, which is why there is no warehouse selector to invent. The data plane needs Lance format support that does not exist in Go and answers with the spec's Unsupported code rather than a bare 404. Two things it deliberately will not do: create a table bucket as a side effect of creating a namespace inside one, since a bucket carries its own policy and lifecycle, and resolve an Iceberg table's location for a Lance client, which would hand it a table another engine owns. The design note this follows is in design-lance-catalog.md, including the .lance directory suffix it proposed and this does not implement. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * mini: give the Lance port the same treatment as the Iceberg one The flag was registered but nothing else knew about it, so mini would start the server without reserving its port, waiting for it, or saying where it is. Adds it to the startup service list, the conflict resolver, the gRPC allocator's reserved set, the readiness wait, the stop reporting and the banner. The admin server still takes only the Iceberg port, because there is no Lance page for it to link to. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: stop deregister and repoint from deleting the dataset Deregistering preserves data by definition, and this did the opposite: the catalog entry is the dataset directory, so DeleteTable took the files with it. Registering over an existing name had the same shape, destroying the dataset the name used to hold. Found by driving the running server rather than the in-memory filer, where both looked like success because the table did stop being listed. Deregistering is now a state on the entry - the marker file hides it, and declaring or registering the name again brings it back. Repointing a name at another dataset is an UpdateTable against the version token, so neither dataset loses files. Drop is left alone; it is the operation that does remove data. The storage endpoint now falls back to the advertised -ip where the Iceberg derivation gives up. An Iceberg client brings its own s3.endpoint and advertising the wrong one hijacks it, but storage_options is the only place a Lance client learns where the store is, and without it object_store quietly talks to real AWS. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: refuse to create a table over one of another format Creating a table that already exists is idempotent, and that path returned the existing table without looking at its format. A Lance declare over an Iceberg table answered 200 and handed back a directory Iceberg owns, so the client would write its dataset on top. The view check immediately above it already guards the same class of collision. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: let a table bucket hold a format other than Iceberg The S3 door validated every object written into a table bucket against Iceberg's file layout, so a Lance client could not write its dataset at all: it got 403 on data/*.lance, on _versions/, and on the _transactions/ directory it turned out to write as well. Table buckets were only neutral containers by intention; in practice they were Iceberg-shaped and enforced as such. The allowed set is now the union of what the supported formats write, because the validator runs where the table's format is not in hand. Underscore-prefixed directories are treated as belonging to the format, since enumerating them means guessing at the next one - _transactions is exactly the one this missed - and their contents are checked only for traversal. Iceberg writes none of them, so it loses nothing. Marker files at the table root are admitted too, which the namespace/table/dir/file shape had rejected as too shallow. Describe also honours the request-body spellings of with_table_uri, load_detailed_metadata and check_declared. The spec puts them in the query string, but real clients send both. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record what the implementation found The table bucket being an Iceberg-shaped container, enforced at the S3 door, was the premise this design never questioned and the one that had to change before anything worked end to end. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * iceberg: prove the data loss the foreign-format guard prevents The guard landed with a unit test for the predicate and nothing showing what it saves. These seed what the Lance namespace's Iceberg REST adapter actually leaves behind - an Iceberg table with a placeholder schema and table_type=lance whose directory holds a Lance dataset - and assert both halves: orphan collection does flag the dataset's fragments, because the Iceberg metadata beside them references nothing, and the scan never reaches the table. An ordinary Iceberg table in the same shape is still scanned, so the guard is not just skipping everything. Confirmed against a running gateway first: our Iceberg catalog accepts the adapter's registration, and a real Lance client then writes a dataset into that table's location. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tablestest: make the in-memory filer safe to race against Two gaps that only matter once a test drives concurrent writers, which is what an exclusive create has to be tested with: the entry map had no lock, and CreateEntry ignored O_EXCL entirely, so both writers of the same name would have won and the test would have passed while proving nothing. The BeforeUpdate hook runs before the lock is taken. Its whole purpose is to land a competing write in a handler's read-to-write window, and that write needs the lock the hook would otherwise be holding. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: make the namespace an external manifest store Lance commits a version by writing _versions/{v}.manifest with put-if-not-exists. The S3 layer in front of this same filer evaluates If-None-Match by looking the entry up and then writing without a precondition, so two writers can both pass the check and one commit is lost. The filer itself has the primitive: CreateEntry with o_excl. Adds the four version operations a Lance client actually calls - create, list, describe and batch-delete - recording one entry per version under _lance_versions/, and advertises managed_versioning so the client routes its commits here. Reserving a version is the exclusive create, so exactly one of several racing writers wins and the rest rebase. Off by default, behind -lance.managedVersioning. Turning it on moves where a table's version history lives, and a reader that does not come through this namespace no longer sees all of it; that is the operator's call, not a default. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record what managed versioning does and does not reach The first commit through a namespace-backed store works and is recorded the way the protocol specifies. Later commits do not, because lance 4.0.0 refuses put_if_exists on that path in its own code, so the feature is capped upstream rather than here. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * test: integration tests for the Lance namespace Everything this surface got wrong so far - a deregister that deleted the dataset, an S3 door that refused every Lance file, a version reservation that could not actually be exclusive - passed against an in-memory filer first. So these run against a live gateway, and where the claim is about data they check storage rather than visibility. Five Go tests on the shared harness: namespace and table lifecycle including that deregister keeps the bytes and drop removes them, that a Lance client cannot resolve or declare over an Iceberg table, that a Lance dataset's files get past the table-bucket layout guard while junk still does not, and that eight writers racing for one version produce exactly one winner. One Docker-gated test drives the real Lance client, which is the only way to check that the location and storage_options the namespace vends are between them enough to write and read a dataset. It overrides the endpoint with the container's view of the same gateway, because the shared harness binds a wildcard address and so vends none. The harness gains a Lance port and turns managed versioning on; the flag touches nothing outside that surface. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: a directory with no namespace metadata is a missing namespace Three callers resolved a namespace by reading its metadata attribute and each tested only for a missing entry, so a directory that carried no metadata came back as an internal error saying "attribute not found". Creating a table under a namespace that does not exist answered 500. Collapses the three copies into one helper that reports both conditions as absent, which is what they are: a directory without namespace metadata is not a namespace. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * iceberg: stop reporting storage-layer refusals as server faults writeManagerError recognised a missing table bucket and sent everything else to 500, so a missing namespace, a duplicate name and a commit conflict all reached the client as InternalServerError with nothing to act on. Creating a table in a namespace that does not exist is the case that turned up: 500 where the spec wants 404 NoSuchNamespaceException. Maps the storage error types onto the exception names this package already uses, and keeps the existing bucket message, which explains how to select a table bucket. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * iceberg: skip a foreign-format table by name, not by failing to parse it A table the namespace created as LANCE carries no Iceberg metadata, so the worker skipped it only because the parse failed, and logged that as damaged metadata. The catalog records the format on the entry and this never read it. Reading it turns an accident into a decision, and separates a mixed catalog from a corrupt one in the logs. The property check beside it still covers the other shape: a real Iceberg table wearing table_type=lance, which is what the Lance namespace's Iceberg REST adapter writes. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: answer whether a Lance table needs maintenance It does, and index optimization has no Iceberg equivalent: rows written after an index was built are not covered by it, so a vector search quietly misses them. None of the three jobs can run in the Go worker, and there is no useful subset, because deciding what an old version still references means parsing Lance manifests. Version cleanup at least has an answer that needs nothing from us - Lance can enable it on the dataset itself. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: the Lance maintenance worker is a plugin worker, in Rust Framing it as a sidecar was wrong. plugin.proto already defines a language-agnostic gRPC contract for external maintenance workers, and "weed worker -admin=..." is the Go reference implementation of it from outside the admin process. seaweed-volume already compiles protos out of weed/pb with tonic_build, so a Lance worker is that build plus plugin.proto and the lance crate. Scheduling, retries, dedupe, progress and the admin settings page all come from the protocol: a worker that answers RequestConfigSchema with a descriptor gets its configuration form rendered without a line of Go. The data plane is the part that genuinely does need a process answering HTTP, and this had the two conflated. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * seaweed-worker: Rust plugin worker workspace, with Lance as the first one plugin.proto is language-agnostic and the Rust toolchain was already in the tree, so a Lance maintenance worker needs no new integration surface: core is the contract and nothing else, and a worker crate beside it supplies handlers and a binary. A second worker is a new member here rather than a fork of the protocol, which is why this is seaweed-worker and not seaweed-lance-worker. Verified against a running admin: it connects, is accepted, and admin prefetches descriptors for lance_compact, lance_optimize_indices and lance_cleanup_versions, so their settings pages render from the Rust side without a line of Go. The stream stays up across heartbeats. The job bodies are stubs that report failure. Doing the work means adding the lance crate and opening the dataset, and claiming success before that would be worse than saying so. Two things running it caught that reading the proto did not: the admin address has to be converted to the gRPC port the way pb.ServerToGrpcAddress does, or the dial fails as an h2 frame error; and the generated field names differ from the Go ones in several places, so JobCompleted carries success rather than a state enum. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: implement compaction Detection lists tables from the namespace, opens each one, and proposes a job for any with more fragments than the policy allows; opening a dataset reads its manifest and not its data, so a sweep stays cheap. Execution re-resolves the table rather than trusting what detection saw - it may have been repointed, and the vended credentials expire - then compacts and reports the fragment counts either side. Verified against a live gateway: a twelve-fragment dataset became one fragment with all twelve rows intact. The test drives the handler directly and skips unless WEED_LANCE_NAMESPACE names a namespace, the way the Go integration tests skip without Docker. Running it turned up a gap the design had not: a gateway without STS vends no credentials at all, so the worker could not open anything and detection quietly proposed nothing. --access-key/--secret-key are the fallback, and whatever the namespace vends still wins over them. Two API assumptions did not survive contact either. Datasets open through DatasetBuilder::with_storage_options, not ReadParams, and lance 10's ObjectStoreParams has no storage_options field at all. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: implement index optimization and version cleanup Index optimization is the job with no Iceberg equivalent: rows appended after an index was built are invisible to a search of it until this runs. Detection reads num_unindexed_rows from each index's statistics and proposes a table once more rows sit outside its indices than the budget allows; a table with no indices is skipped, which is different from one whose indices have fallen behind. Cleanup applies a retention window, refusing rather than silently dropping a tagged version, and leaving unverified files alone because they may belong to a commit still in flight. Both verified against a live gateway: 512 uncovered rows became 0, and a fourteen-version table lost its old ones. Each test now seeds what it needs, including building an IVF_PQ index and appending rows outside it. The first version of these depended on state a script had left, so the second run found the work already done and asserted nothing - a test that passes by doing nothing is worse than no test. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: answer an empty catalog with an empty list, not null ListAllTables built its result from a nil slice, so a namespace holding no tables answered {"tables":null} on a field the spec marks required. A generated client may decode that differently from an empty list. Found running the namespace on a dev box, where the catalog was empty. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: give Lance maintenance its own scheduler lane Lane assignment is a hardcoded map, so the three lance_* job types fell through to the default lane. That lane serialises its work under the cluster admin lock because volume management shares global state, which would queue a table's compaction behind volume balancing for no reason - Iceberg has its own lock-free lane for exactly this. Adds the lane, maps the three job types to it, and puts it in the sidebar beside Iceberg and Lifecycle. The lane routes were already generic, so only the nav was hand-written. The lane-coverage test spelled out the three known lanes, so a fourth failed it. It now checks against AllLanes(), which is the property it was reaching for and does not need editing next time. Found by connecting the Rust worker to a real admin: it registered fine and its job types were known, but they were filed under "default" and had no page. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: log what detection saw "Detection proposed nothing" and "the worker could not read the table" look identical from the admin side, and the second is what a missing credential produces. One line per table separates them. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: fix a leaked heartbeat and a silent reconnect loop spawn_heartbeat returned a handle to an empty task rather than the ticker it had just spawned, so aborting it aborted nothing and every reconnect left another heartbeat running against a dead channel. A stream that admin closes cleanly is not an error, but reconnecting in silence hides why. Two workers sharing an id evict each other forever and the log shows nothing but a login every five seconds - which is exactly how this presented on a dev box, and it took a look at the admin's own log to see it. The message now names the id to check. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: a namespace cannot be created without its parent Storage keeps a namespace's parts flattened, so creating "a.b" with no "a" was accepted and left an intermediate that only existed inside a name. Listing derives child names by slicing those parts, so it reported "a", while describe and exists on "a" both answered 404 - a client walking the tree got a 404 on something the listing had just handed it. The spec asks for NamespaceNotFound when the parent is missing, which is also what keeps listing and describe telling the same story. Namespaces created through the S3 Tables API still bypass this, so listing keeps deriving intermediates rather than hiding whatever is already there. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: say why a non-Iceberg table shows no schema The table pages read Iceberg metadata for schema and snapshots, and a Lance table has none, so both panels rendered "No schema available" - which reads as an empty table rather than a table this page cannot describe. The dataset behind the one that prompted this holds 1024 rows. The format is already on the entry and shown two rows above, so the empty states now use it: the catalog records where a LANCE table lives, not what is in it. Reading the schema for real needs Lance format code, which is the same wall as the data plane. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * seaweed-worker: run rustfmt over the workspace Committed the crates unformatted, so `cargo fmt --all --check` failed on files nothing had touched since. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * plugin: let a worker report what it saw about an object Admin cannot read a Lance table: it knows where the dataset lives and nothing else, so the details page had a location and two empty panels. The worker already opens every dataset during detection to decide whether it needs compacting, so it knows the schema, the row count and the fragment count at that moment. It just had no way to say so. Add a WorkerObservations body to the worker stream. Admin caches the last observation per object and serves it back, timestamped, for display; nothing schedules from it. The Lance compaction sweep reports what it opened, and the S3 Tables details page fills its schema panel from the cache when it has no metadata of its own, badged with when the worker looked and which worker it was. Nothing about this is Lance-specific past the reporting side, which is the point: any format admin cannot parse can describe itself the same way. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record the observation channel Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * plugin: ask a worker for sample rows of a table admin cannot read Browse Data reads an Iceberg table's Parquet files directly, so it shows real rows. For a Lance table it showed "Table has no Iceberg metadata" and an empty grid, because there is no Go Lance reader and never will be one worth maintaining. The worker has the reader. Add RequestObjectPreview / ObjectPreviewResponse to the stream, mirroring the config-schema round trip that already exists, and give the Rust worker a PreviewProvider that scans the dataset and formats the rows with Arrow's own formatter, so a vector column reads as a vector. Admin picks the worker from the observation store: whichever one last described this table is the one that can read it. Unlike an observation the rows are not cached. They are the table's data rather than a description of it, and a copy sitting in admin would be both stale and nobody's business. The page fetches on load, bounded at 200 rows and a 15 second round trip, and drops the snapshot and data-file panels that only mean something for Iceberg. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record the preview channel Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * test: disable the lance listener when two gateways share a host * test: keep AllocatePorts away from the lance default port * s3tables: let a table bucket declare the format it holds A bucket is a catalog, and a catalog serves one protocol. Format was recorded per table, so nothing could answer "where do I point a client at this bucket" without opening a table first, and an empty bucket had no answer at all. CreateTableBucket takes an optional format, stored with the rest of the bucket metadata and returned by Get and List. Empty means ICEBERG, which is what AWS S3 Tables serves and therefore what an SDK that has never heard of the field means. CreateTable refuses a table of another format, and CreateView refuses outright in a bucket that is not Iceberg, since a view is Iceberg metadata. Buckets that already exist carry no declaration and keep accepting anything, so nothing is migrated and nothing that worked stops working. The Lance namespace declares LANCE for the buckets it creates, which is what stops one of them being described to a client as an Iceberg catalog. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: take the Lance port the way it takes the Iceberg one The UI cannot name the endpoint that serves a Lance bucket without it, and every format-aware page below needs to. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: show which format a table bucket holds The bucket list printed an Iceberg endpoint for every bucket, including ones holding Lance datasets, where that endpoint serves nothing. It was the most visible place the UI assumed one format. The list gains a Format column and its endpoint column follows the bucket's declaration. The banner names both endpoints rather than asserting everything is Iceberg, and says so only for the servers that are actually running. Create Bucket picks a format with two cards rather than a dropdown, since what matters is not the name but which clients can read the result, and the endpoint under them updates as you choose so the operator leaves the modal knowing where to point one. A bucket from before the declaration existed shows "unset" in an outline badge, explained on hover. It is a fact about the bucket's age, not a fault, so nothing nags about it. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: carry the bucket's format into the pages inside it Namespaces and tables are reached through a bucket, so both now say which catalog they belong to rather than making you go back up to find out. The tables list gains a Format column and a Rows column filled from what a worker last observed, since for a format admin cannot read that is the only row count there is; a table nothing has looked at shows a dash, not a zero. Create Table stops offering a choice the bucket has already made: in a declared bucket the format is fixed and says why, and only an undeclared one still offers both. Before this the select had exactly one option, hardcoded, which made a Lance table impossible to create from the UI at all. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: let the table page speak the table's own format Partitions and Snapshot History are Iceberg's shape. Rendering them empty for a Lance table reads as a fault; a Lance table has neither, and says so by not showing them. In their place is a Versions panel, which is what that format calls its history, carrying the worker's timestamp so it is clear the numbers are a cached look rather than something read live. The breadcrumb carries the format badge, so the page names what it is looking at before you read a panel and wonder why it is empty. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: show how to connect to either catalog, and group the two format workers The client examples on the buckets page were Iceberg's alone, so the one thing an operator wants after creating a Lance bucket - what to type to reach it - was not written down anywhere in the UI. Both formats now get a pair of snippets, and only for a server that is running. In the Workers menu, Iceberg moves below Lifecycle so it sits next to Lance: the two table-format workers together, the two cluster-wide ones above them. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * shell: create a table bucket of either format s3tables.bucket -create takes -format, so a Lance bucket can be made without going through the UI. The integration harness passes it too: its Lance tests were creating Iceberg buckets and getting away with it only because nothing checked. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record that a bucket declares its format Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: drop managed versioning; the store already orders commits The namespace offered itself as an external manifest store, so that a commit could reserve a version through a real put-if-not-exists. That was designed around a gateway that no longer exists: If-None-Match: * is reduced to a filer WriteCondition and evaluated at the object's owner under its per-path lock, or under the object write lock on the fallback path. Sixteen writers racing one fresh key get a single 200 and fifteen 412s, every time. Lance needs nothing else. commit_handler_from_url hands every s3:// dataset a ConditionalPutCommitHandler, which puts with PutMode::Create, which object_store sends as If-None-Match: *. So the feature solved a problem this store does not have, while moving a table's version history out of the dataset and into the catalog - and lance could not use it past the first commit anyway, since its own namespace-backed store answers "put_if_not_exists is not supported" to the second. The version operations answer Unsupported with the rest, managed_versioning is false, and the flag is gone. In place of the reserve-once test there is one that races eight writers at the manifest key through S3, which is the path a commit actually takes. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: honour the version floor, the slot limits, and a shutdown Five findings from review, all of them things the worker claimed to do and did not. The version floor was checked when a cleanup job was proposed and ignored when it ran, so a table whose versions had aged past the retention window in between could be taken below the count the operator asked to keep. Execution now computes the floor itself and passes it as before_version; CleanupPolicy ANDs its clauses, so a version has to be both too old and below the floor to go. Both settings are clamped to the range the form offers, since Duration::hours panics on a large enough value and a negative min-versions wraps to a huge usize. Admin's shutdown was answered by returning from the stream, which the reconnect loop read as a healthy close and logged straight back in: the worker could not be stopped. serve_once now says which of the two happened. The advertised concurrency limits bounded nothing - every request spawned a task - and the heartbeat reported zero slots in use whatever was running. Both now go through semaphores sized from the limits, with the permits held for the life of the request and reported in the heartbeat. A namespace call had no timeout, so a gateway that accepted the connection and went quiet held a detection slot forever. And one table whose stats could not be read failed the whole sweep, losing the proposals for every table already scanned; it is now skipped and warned about, like a table that cannot be opened. The tests drove one shared catalog concurrently, which is why one of them asserted "no proposals at all" and passed by luck. They now take a lock and judge only their own tables. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: fix the review findings on the format-aware pages The endpoint hint in Create Bucket built its HTML by concatenating the bucket name the operator is typing, so a name like <img onerror=...> ran in the admin origin as they typed it. It is built from DOM nodes now. A preview reply looked its channel up under the lock and then sent outside it, which Shutdown can close in between: a Gosched in that gap panics with "send on closed channel" every time. The send now happens under the lock. Observations were looked up by path alone, so a table dropped and remade in another format at the same path was described by the observation left behind. Lookups now have to agree on the format. Also: the Lance namespace caps a request body rather than reading whatever arrives; the details action no longer says "Iceberg" over a Lance table; mini stops advertising a catalog port when it is not running S3; a format whose server this cluster does not run cannot be picked in the modal or accepted by the API, since a bucket nothing can reach is not worth creating; and the unused catalogPortFor helper is gone. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: let the control stream use mTLS The channel was hardcoded to http://, so off loopback the stream carried preview rows and execution commands in the clear - and a cluster with grpc TLS turned on would refuse the worker outright. --tls-ca, --tls-cert and --tls-key take the same certificates the Go worker reads from the [grpc.worker] section of security.toml, and must be given together: a CA on its own would quietly mean one-way TLS, which a mutual setup rejects anyway. Without them the stream stays plaintext, which is what the Go worker also does when nothing is configured. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: answer null properties rather than an empty map The catalog does not keep a table's properties. Declare echoed the request's back and describe answered {}, both of which claim they were stored and are empty. Null says the catalog does not keep them, which is what the spec distinguishes and what is true here. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: test the slot accounting The heartbeat reporting and the waiting are the two things the semaphores are for, and neither is observable from outside without catching a sweep mid-flight. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * test: fix the mixed-format catalog test, and name the binary it drives The integration suite passed locally and failed in CI on TestLanceRefusesIcebergTables. Both were right: CI builds the binary first, my tree had one from the day before, so locally the test drove a gateway with no format enforcement at all. The test itself no longer holds as written. It made a bucket, put an Iceberg table in it, and checked the Lance surface hid it - but a bucket that declares LANCE now refuses the Iceberg table outright. The invariant still matters from the other side, so it starts from an Iceberg bucket instead: Lance must not describe or list a table whose format it does not serve, and must refuse to declare one beside it. The harness now prints which weed binary it is about to run and when that was built. `make test` rebuilds first; a plain `go test` will happily drive a weeks-old binary and report a pass for code it never ran, which is exactly what happened here. Also make the row-limit conversion in the preview request explicitly bounded: CodeQL flagged the int-to-int32 conversion, and clamping by reassignment beforehand is not a form it recognises. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: prove concurrent commits are kept, and preselect the only format on offer Two more from review. The commit test asserted that exactly one writer wins the conditional PUT, which is the mechanism, not the claim. The claim is that nothing is lost: the losers see the conflict, rebase and commit again. So there is now a test that has eight writers append to one dataset at once and counts the rows afterwards - all eight batches survive. That is also the sequence managed versioning could not finish, since its store refuses the second commit outright. And when Iceberg's endpoint is not running, the format picker offered two options with neither selected, so Create Bucket submitted no format at all, fell back to ICEBERG, and was refused by the guard added last round. Lance is preselected when it is the only format this cluster serves. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * Clamp the remaining worker settings, and bootstrap buckets in a served format Compaction and index optimization read their thresholds and cast straight to usize and u64, so a negative arrives as an enormous number and turns the threshold into "never": compaction and reindexing both go quiet with nothing to say. The cleanup job was fixed last round; these are the same bug. Clamped to the values that stay meaningful rather than to what the form offers - zero uncovered rows is a real setting, meaning reindex as soon as anything is not covered, so the floor there is zero and not the form's thousand. mini pre-creates the buckets named by -tableBucket, and did so without a format, which now means Iceberg. Started with the Iceberg endpoint off and the Lance one on, that left buckets nothing could reach and which refused every Lance table. It takes the format from the endpoint that is actually running, and creates nothing when neither is. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3: allow-unordered is a listing parameter, not an unimplemented subresource The guard that stops a bucket GET with an unknown subresource from being answered with a listing does not know about allow-unordered, so it answers 501 NotImplemented - to a parameter the listing handlers already read and already validate against delimiter. This is why test_bucket_list_unordered and test_bucket_listv2_unordered fail in the Ceph s3-tests suite. They fail on master too; this is not a Lance change and can be taken on its own. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm |
||
|
Chris LuandGitHub
|
814ee75af4 |
s3: allow-unordered is a listing parameter, not an unimplemented subresource (#10846)
The guard that stops a bucket GET with an unknown subresource from being answered with a listing does not know about allow-unordered, so it answers 501 NotImplemented - to a parameter the listing handlers already read and already validate against delimiter. This is why test_bucket_list_unordered and test_bucket_listv2_unordered fail in the Ceph s3-tests suite. They fail on master too; this is not a Lance change and can be taken on its own. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm |
||
|
Chris LuandGitHub
|
e5edd8be3c |
s3: place multipart part chunks by the destination object's storage rule (#10845)
Multipart parts stage under /buckets/<bucket>/.uploads/<id>/, so the filer resolved filer.conf storage rules against that path when the gateway assigned volumes for them. A rule scoped to a key prefix - fs.configure -locationPrefix=/buckets/b/data/ -ttl=30d - then matched a small object but not the parts of a large one, so an object whose entry carried the rule's TTL had its bytes spread over TTL-less volumes. Assign part chunks against the destination object's filer path instead, the way the x-seaweedfs-destination header made the filer resolve it before the S3 write path moved off the filer proxy. Covers PutObjectPart and both UploadPartCopy paths. The part entry itself is still written under .uploads, so a read-only rule there still rejects it. The lifecycle XML Expiration.Days TTL keeps passing 0 for parts: that rule targets the user-visible object key and would start its clock before CompleteMultipartUpload. |
||
|
Chris LuandGitHub
|
abd61de52c |
S3: stamp the gateway's own uid/gid on PutObject and copy entries (#10844)
* fix(s3): stamp the gateway's own ids on single-shot PutObject entries putToFiler builds the entry in the gateway now instead of proxying a PUT to the filer, and it hardcoded Uid/Gid 0 while every sibling write path stamps filer_pb.OS_UID/OS_GID. On a non-root deployment that leaves single-shot PUTs and multipart parts owned by root while directories and completed multipart objects keep the real ids, so a mount reader can list the tree but gets EACCES on every open once objects are not world-readable. * fix(s3): stamp mode and ownership on copy destinations CopyObject and UploadPartCopy build the destination attributes themselves and then assign them over the entry filer_pb.MkFile just stamped, so the copy landed with mode 0000 and uid/gid 0 - unreadable on a mount even by the filer's own user. Build the destination with the same mode PutObject resolves for the request and the gateway's own ids. |