* s3: log the requester's principal ARN in the audit entry
An STS session authenticates as an opaque session subject, so requester
alone gave an operator no way back to the assumed role or the session
name. Record the principal ARN next to the identity name and emit it as
requester_arn.
* s3: record the caller identity in the STS handlers
AssumeRole, GetFederationToken and GetCallerIdentity verify the caller
themselves and are not wrapped by the auth middleware that records the
identity, so every audit entry for minting a session had an empty
requester.
* s3: resolve the audit principal ARN the way policy evaluation does
A JWT-authenticated identity carries no PrincipalArn — the auth layer
hands the principal over in a request header — so reading the field
directly left requester_arn empty for OIDC callers. buildPrincipalARN is
the resolver the policy path already uses: header first, then the
identity's own ARN, then a synthesized user ARN for legacy identities
that have none.
* s3: keep an admin's role session scoped to the role
AssumeRole copied the caller's admin standing into the minted session as
the is_admin claim, which short-circuits base policy evaluation. An admin
assuming a scoped-down role therefore kept full access and the role's
attached policies, explicit denies included, were never evaluated.
Only a session the caller assumed for itself carries the claim now — a
legacy static admin has no IAM policies for such a session to inherit.
* s3: name the caller when it assumes a session for itself
An identity that carries no principal ARN left the self-assumed session
with an empty role name in its assumed-role ARN. callerPrincipalArn
synthesizes the canonical user ARN for that case.
* test: cover listings spanning a run of retracted keys
A listing drops entries whose current version is a delete marker. When a run
of consecutive entries drops out, the page being filled can come back empty,
and an empty page is easily mistaken for the end of the listing — everything
after the run then never appears and the caller is told those objects do not
exist.
Backup repositories produce exactly this shape: a batch of keys under one
prefix is retracted while writing continues under the next.
Covers a retracted run before live keys and between live keys, walked with
page sizes smaller than the run so at least one page is filled entirely from
entries that get dropped, plus the version view of the same namespace where
every version and every delete marker must still be reported.
* test: sweep every page size in both walks and paginate the version listing
A listing on a versioned bucket is served from metadata cached on the
.versions directory entry so the whole listing is a single scan. The cache
carried size, mtime, ETag, owner and the delete-marker flag but not the
storage class, so newListEntry found none and fell back to STANDARD.
The result was that HEAD and the listings disagreed about the same object:
HEAD reported the class the object was stored with, while ListObjectsV2 and
ListObjectVersions reported STANDARD for every object. Clients that filter or
tier on storage class act on the listing.
Caches the class alongside the other listing fields, clears it with them, and
copies it in the routed RECOMPUTE_LATEST path so both finalize paths agree.
* test: compare ListObjects and ListObjectVersions over the same namespace
The two listings walk the same tree through separate code paths, so a client
navigating by versioned listings can see a different namespace than one
navigating by plain listings, and concludes keys are missing that are plainly
there. Testing each path on its own never catches that; only comparing them
does, and nothing compared them.
Asserts both report identical current keys and identical common prefixes
across a backup-shaped tree: nested prefixes, a prefix naming an object
exactly, a key that is simultaneously an object and the parent of other keys,
a partial key fragment, and a prefix matching nothing.
The version view is reduced to what a plain listing reports — latest versions
that are not delete markers — so the comparison is like for like.
* test: guard against truncated pages and cover the delete-marker path
* test: pin verb parity on lock-arbitration keys through acquire and release
Backup clients arbitrate repository ownership by writing and retracting small
keys under a fixed prefix and re-probing them, each probe using a different
verb. They trust those verbs to agree; a key reported present by one and
absent by another makes the client either spin or declare the repository
corrupt, and neither shows up as an error on the storage side because each
individual answer is locally correct.
The keys are written and immediately deleted by version id, which is the cycle
that empties a version container, so parity is asserted on both sides of the
delete and across repeated re-acquire cycles where residue accumulates.
Reports which verbs disagreed rather than just failing.
* test: run the reacquire cycle on every lock key, and drain probe bodies
* test: pin that an unusable version id is refused, never resolved
A version id containing a path separator, or "." / "..", can never name a
stored version. Resolving one to the null or latest version instead would let
a caller destroy a live version by asking for one that does not exist, on a
bucket configured for immutability.
The guard exists today and holds on every verb; it had no test. Pins two
properties: such a request is refused with a client error rather than a 5xx
(a 5xx invites endless retries of something that can never succeed), and the
version that does exist survives every refused request.
* test: require exactly 400 for an unusable version id
* test: cover delete idempotency on versioned object-locked buckets
Backup clients probe and retract lock keys continuously, so they routinely
delete keys and versions that are already gone, and they batch those deletes
alongside keys that do exist. S3 makes all of that succeed; returning an error
turns ordinary lock arbitration into a job failure.
The behaviour is correct today but had no coverage, and it runs through the
object-lock retention check, which is the most likely place for a missing
object to start being reported as an error.
Covers: deleting a key that never existed, deleting a well-formed version id
that names nothing (twice, and without disturbing the version that does
exist), and a batch whose middle key is missing — every requested key must
come back under its own name rather than silently taking another row's slot.
* test: verify the deletes actually took effect, not just that they returned
* s3: report a peer that went away as ClientDisconnected, not IncompleteBody
A streaming PUT whose body ends early is always reported as IncompleteBody
(400). That collapses two cases with opposite causes: the peer vanished
mid-upload, and the peer sent fewer bytes than it promised while still
connected. The first points at the network path, the second at the client,
and once merged they cannot be told apart from the logs.
Split out ClientDisconnected (499) and select it when the request context
shows the peer is gone. The upload itself keeps running on a background
context so chunks still finish, which means cancellation races the read
error; a missed signal degrades to IncompleteBody exactly as before.
* s3: note what request-context cancellation is taken to mean
* helm: give the bucket hook the credentials weed shell needs
The post-install bucket hook pipes s3.configure into `weed shell`. Since
#9442 the filer only serves its IAM gRPC service to callers presenting an
admin-signed JWT, and since #9536 `weed shell` mints that token itself -
but only if it can find the filer signing key. The hook job sees neither
a security.toml nor the WEED_* environment overrides: its env list is
hardcoded, and it is the only workload in the chart without a
security.toml mount.
So with jwtSigning.filerWrite=true the hook logs
error: failed to get user anonymous: rpc error: code = Unauthenticated
desc = missing authorization metadata
and `weed shell` exits 0 regardless, which leaves the Job green while
anonymousRead is never applied.
Render the merged extraEnvironmentVars into the job's env - keeping
non-string values as valueFrom, so keys held in a Secret stay a
reference - and mount security.toml the way every other workload does.
The hardcoded WEED_CLUSTER_* entries go away because those values are
part of the global extraEnvironmentVars defaults and would otherwise
render twice.
Signed-off-by: Sebastian Preisner <preisner@puzzle-itc.de>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm ci: report a missing bucket hook Job instead of crashing
Signed-off-by: Sebastian Preisner <preisner@puzzle-itc.de>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm ci: assert where the bucket hook mounts security.toml
A mount under the wrong path leaves weed shell without the key just as
surely as no mount at all, so check the target, not only the name.
Signed-off-by: Sebastian Preisner <preisner@puzzle-itc.de>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm: keep the bucket hook's cluster endpoints chart-computed
Dropping the hardcoded WEED_CLUSTER_* left the readiness waits
dereferencing $WEED_CLUSTER_SW_*, which exist only while the values keep
the default alias. Rename the cluster, or clear extraEnvironmentVars, and
the hook polls "http://" for five minutes and then fails the release.
Derive the env names from the alias and render the addresses from the
chart, as all-in-one already does.
* helm ci: assert the bucket hook follows a renamed cluster alias
Renames the cluster and drops the default addresses, then checks that
every env name the readiness waits dereference is set, and that the
address is the chart's rather than the one left in the values.
* helm ci: check the hook's endpoints and security.toml source
The bucket hook tests took two shortcuts a wrong render slips through.
The alias test only rejected the stale master address the values kept, so
any other wrong address passed and the filer address was never compared at
all. And the security.toml test matched the volume by name, which a
same-named emptyDir or a foreign ConfigMap satisfies while `weed shell`
still has no signing key.
Compare both cluster addresses against the Services the chart renders, in
the default and the renamed-alias case, and assert the volume is backed by
the chart's security-config ConfigMap. Checked by pointing the volume at
another ConfigMap and the filer env at a wrong but non-empty address:
both now fail, both passed before.
Signed-off-by: Sebastian Preisner <preisner@puzzle-itc.de>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Sebastian Preisner <preisner@puzzle-itc.de>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* wdclient: age vid map entries by generation instead of chaining snapshots
The vid map kept its history as a linked list of past snapshots, trimmed
in place by storing nil into a node's cache pointer. That cost up to six
full copies of the volume-location map, a recursive walk taking a
different lock per level, and deletes that had to cascade through every
generation. It also had to special-case explicitly-empty entries, or
fallback would resurrect locations a newer snapshot had cleared.
Keep one map instead, and stamp each entry with the generation it was
learned in. resetVidMap bumps the generation and drops entries that were
not relearned within the retained window, which is the same retention
the chain provided: an entry survives DefaultVidMapCacheSize resets.
The first write of a generation replaces an entry rather than merging
into it, so a volume that moved answers with where it is now — the
property a fresh map per reset used to give for free. Entries are
copy-on-write, so locations handed to a caller are no longer shifted
underneath it by a concurrent delete.
The map is never swapped now, so the client-side lock and its stable /
current accessors go away with it.
* wdclient: make vid map entries immutable and drop them once emptied
Review follow-up. Updating an entry in place left the copy-on-write
guarantee resting on callers never holding the entry pointer; install a
new entry instead, so the rule is simply that a stored entry never
changes.
Deleting a volume's last location now drops the entry rather than
keeping an empty one, which a client that never resets would otherwise
hold for every volume it ever saw deleted. Lookups already treat an
empty entry as a miss, so nothing observable changes.
* wdclient: let the newest generation decide between regular and EC locations
GetLocations checked the regular map first whatever its generation, so a
volume that was EC encoded kept answering with the regular copies the
previous master knew until they expired — for as long as the retained
window, since nothing relearns a copy that no longer exists.
The snapshot chain did not have this problem: the newest map was
consulted first and only a volume it knew nothing about fell through to
older ones. Restore that by comparing generations, with regular copies
winning a tie, since a tie means one generation reported both.
Sealing a logic chunk index that already held a sealed chunk dropped the
old chunk's only reference and freed its page chunk. That chunk's upload
may not have started reading it yet — Execute() returns as soon as the
job is handed to a goroutine — so mem.Free could hand a live 2 MiB mem
chunk back to the slot pool, the next NewMemChunk would overwrite it,
and the in-flight upload shipped whatever bytes were there. Under fio
randwrite the volume server rejected those needles with "Content-MD5 did
not match md5 of file data" and the FUSE write failed with EIO.
Give the sealed chunk a second reference for its upload, dropped only by
the upload itself, and let the upload unindex itself only while it still
owns the index — the unconditional delete could evict a newer sealed
chunk and hide its dirty pages from readers.
resetVidMap trims the cache chain by storing nil into a node's cache
pointer once it ages past vidMapCacheSize. A lookup that missed in its
own map and only then loaded that pointer could find the link already
severed, reporting "not found" for a volume that stayed resolvable the
whole time.
Load the link first, while it is still guaranteed live. A published
vidMap's cache pointer only ever goes from its ancestor to nil, so
reading it earlier can never yield staler history.
* filer: let a nested path rule turn worm off
mergePathConf ORs the booleans, so worm set on a bucket could never be
lifted on a directory under it, while every string field is overridden by
the more specific rule. Make worm tri-state instead: unset inherits, set
wins. readOnly, fsync and disableChunkDeletion keep the OR, so a nested
rule still cannot escape a lock the bucket set.
Configurations written before this carry an explicit "worm": false on
every rule, because they are marshalled with EmitUnpopulated. Reading
those back as an override would quietly drop worm from nested paths, so
filer.conf is now stamped with a version and the flag is dropped to unset
when the version predates it.
* filer: copy the worm value out of the matched rule
mergePathConf aliased the pointer into the merged result, so a caller that
wrote through it would reach into the stored rule.
* test: pin that a .vif replication outranks the superblock
Store.ConfigureVolume rewrites the .vif and never the replica-placement byte in
the .dat, so that byte keeps whatever the volume was created with for good.
readSuperBlock reads it and then overrides it from the .vif, which is what makes
a replication change take effect and survive a remount.
Invert that and every replication change silently reverts on the next mount,
while the .vif on disk still records what the operator asked for -- a durability
setting quietly going back to its old value, with nothing to indicate it.
Worth pinning rather than reading off the code, because the field beside it
resolves the other way: version takes the superblock over the .vif. Two fields,
one function, opposite precedence, each a line to invert wrongly.
Covers the empty case too, since a .vif that declares no replication has to
leave the superblock standing or a volume whose replication was never
configured would be forced to whatever the zero value parses as.
* test: drop the unreachable nil check on MaybeLoadVolumeInfo
It initialises the returned pointer before the existence check and every
return is naked, so it never yields nil. Guarding against it implied a
contract the callee does not have.
The master could die with a nil pointer dereference inside
raft.(*server).TakeSnapshot. The leader and follower loops ask for a
snapshot on every iteration and callers can ask at any time, so several
goroutines built one at once, all writing to the one pendingSnapshot
field. Whichever finished first saved it and set the field to nil, and
the rest either dereferenced nil while attaching peers and state, or
stored nil as the current snapshot, leaving the master with no snapshot
to send to a lagging peer.
v1.2.0 holds a mutex for the length of a snapshot and keeps the one being
built in a local. It also stops a snapshot recovery that cannot finish
from being reported as done, writes snapshots to one side and renames
them into place, loads the snapshot covering the most of the log rather
than the last one in filename order, and takes the lock that guards the
peer map on both sides, where a snapshot cloning the peers alongside a
membership change was a concurrent map iteration and write.
* s3: allow copying an object onto itself in a versioned bucket
The copy writes a new version instead of overwriting in place, which is how an
earlier version is restored. Buckets with versioning off or suspended keep
rejecting a self-copy that changes nothing.
* s3: cover the suspended-versioning self-copy rejection
Suspended versioning overwrites the null version in place, so a self-copy that
changes nothing stays rejected. Pin that alongside the never-versioned case.
* s3: keep the list marker exclusive for versioned objects
A versioned object lives in a "<key>.versions" directory, so the entry name
never matched the marker and start-after/marker returned the marker key itself.
* s3: match the list marker against the raw entry name too
A backend that echoes the marker it was given returns the ".versions" directory
name, which no longer matched once the comparison used the object name alone.
Cover both, and unit test each half.
TestProxyReadDropsCallerJwtQueryParam mints a read token up front and requires
the token the volume server would evaluate to equal it byte for byte. The expiry
claim has one-second resolution -- GenJwtForVolumeServer sets it from
jwt.NewNumericDate(time.Now().Add(...)) -- so two mints on either side of a tick
produce different strings for the same authority and the same file, and the
assertion fails for a reason the test is not about.
It surfaces on the 32-bit job, where the runner is slow enough that the HEAD
subtest (the second one, after a full proxy round trip) lands in a later second
than the mint at the top of the test. Confirmed directly: minting the same file
id with the same key either side of a boundary yields different tokens.
Assert what the test is actually about instead -- that the credential decodes
against the read key and authorizes this file id -- which holds whatever second
it is minted in, and is a closer statement of the property than string equality.
* helm: optional NetworkPolicy per component
In a namespace with a default-deny policy the chart cannot be installed:
the components never reach each other, and the post-install bucket hook
waits on the master and filer until it gives up.
networkPolicy.enabled renders one policy per component, selecting its
pods by the standard app.kubernetes.io labels and admitting the other
pods of the release on the ports that component listens on. The port
lists come from the same values as the containerPorts, and CI asserts
the two agree. Restricting egress is a second opt-in with extraEgress
for the filer store and notification sinks, which the chart cannot
know about.
Closes#10421
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm: say "changed" instead of "retuned" in the policy comments
codespell reads "retuned" as a misspelling of "returned" and fails the
spelling job.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm: refuse empty port and DNS peer lists instead of widening
In a NetworkPolicy an empty ports list means every port and a missing peer
selector means every pod, so `kubeApiServer.ports: []` silently opened the
API server CIDRs on all ports, and nulling a DNS selector rendered
`podSelector: null`, which is every pod in kube-system. Both now fail the
render, and the DNS rule emits only the selectors that are set.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm: gate the bucket hook Job and its policy on one helper
Both were deriving the same condition from the same values, kept in step by
a comment. seaweedfs.bucketHookEnabled makes it one definition, so adding an
S3 mode cannot leave the Job running without its policy - which under
default-deny means the hook hangs. CI pins the pairing across the eleven
modes that decide it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm: say that an egress default-deny needs both toggles
networkPolicy.enabled on its own only covers a default-deny that restricts
ingress. Where Egress is in its policyTypes as well, which is the usual
baseline, the components still cannot resolve DNS and egress.enabled is
required too. Both values and the README said the first half of that.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm: quote the label values the policies emit
The same int-coercion the hook templates were fixed for, in the file this PR
adds: unquoted, a release named 123 renders app.kubernetes.io/instance as a
YAML integer in the metadata, the podSelector and both peer selectors, and
the API server rejects the object - a policy that silently never applies.
CI now renders the chart as release "123" and fails if any policy label
comes out as a non-string.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm: gate the resize hook policy on the same lookup as its Job
The policy rendered whenever the hook was enabled, which is the default, so
every release carrying networkPolicy got one - as a pre-install hook that
Helm never collects, left in the namespace after uninstall. It also made
egress.kubeApiServer.cidrs mandatory for every release, since the policy
claims the API server, for a Job that only runs on an upgrade that grows a
PVC.
Move the command computation the Job is gated on into a helper and read it
from both.
* helm: drop the API server rule from the admin policy
No seaweedfs binary talks to the Kubernetes API - there is no client-go in
go.mod - so the rule granted admin an egress path it never uses, and forced
anyone running admin with egress on to name an API server address for it.
The pod-RW ClusterRole the comment was reasoning from is a leftover from a
migration and is not read by any component.
* helm: reject a networkPolicy.components key that names no component
The component names are not guessable - objectstorage-provisioner,
seaweedfs-all-in-one, volume-<name> - and a typo silently dropped the rules
it was carrying. Check against every component the chart can produce, not
the enabled ones, so a values file shared across releases can still hold
overrides for a component this one leaves off.
* helm: name the all-in-one policy after the workload
componentName prefixes the release fullname onto the suffix it is given, and
the component label already starts with seaweedfs-, so the policy came out as
<release>-seaweedfs-seaweedfs-all-in-one.
* helm ci: assert the denied probe failed rather than that it did not succeed
kubectl run's "pod/x created" was captured alongside the pod log, so an
empty log would still not match exit=0 and the denial would pass without
anything having been tested.
* helm: document what turning the network policies on costs
Three things the values did not say: a Prometheus outside the release stops
scraping and nothing reports it, the resize hook's policy is a hook resource
that uninstall leaves behind, and the DNS selectors are wrong on OpenShift.
* helm: spell out the managed distributions codespell reads as a typo
codespell has AKS in its dictionary as a misspelling of ASK, so the DNS
selector note failed the spelling job.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm ci: read the probe verdict from the marker line, not the whole log
The default-deny step decides both probes from the pod log, but kubectl logs
returns stderr as well, and busybox wget reports "download timed out" there
even under -q. The denied probe therefore produced two lines beginning with
wget:, which matches neither exit=0 nor exit=*, so a correct denial landed in
the arm meant for a probe that produced no result at all and failed the job.
The earlier form hid this behind a catch-all that treated anything without
exit=0 as a denial; tightening that assertion made the stray line fatal
without narrowing the input it reads.
Pick the marker line out instead. The trailing || true is required: the step
runs under bash -e, so a grep that matches nothing would abort it rather than
reach the arm that reports an empty result, which is the case that assertion
exists to catch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Sebastian Preisner <preisner@puzzle-itc.de>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* webdav: answer PROPFIND child stats from the listing
golang.org/x/net/webdav discards the FileInfo that Readdir returned and stats
every child again, five times over, so a PROPFIND on a directory costs five
sequential filer lookups per entry: 15006 lookups and 1.4s for 3000
subdirectories, 24s for 60000. Windows Explorer times out well before that.
Hold the entries a listing already fetched for the lifetime of the request and
serve those stats from them - 6 lookups and 0.03s for the same 3000 entries.
WebDavFile.Stat has to stop dropping its request context for the held entries
to be reachable.
* webdav: keep the request context on the lookups a listing drives
stat, Readdir and Seek reached the filer on context.Background(), so a client
that walked away left the listing streaming and the lookups running. Seek also
missed the entries the listing had already fetched.
Write and cleanup paths keep their own context - a cancelled request must not
abandon a flush half done.
* helm: label the hook Job pods so selectors can reach them
The bucket hook pod carries only managed-by/instance and the volume
resize hook pod carries no labels at all, so nothing keyed on the
standard app.kubernetes.io set can address either of them. Give both
the same name/chart/managed-by/instance/component labels the other
workloads use, with the component naming each hook.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm ci: assert the whole label set on the Job and its pod
The block checked three keys and only looked at the pod's component, so a
missing chart/managed-by label, a wrong component on the Job, or a Job and
pod that disagree would all have passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm ci: compare the hook's release labels against a real workload
Presence alone let a wrong value through. The name/instance/chart values now
have to match a workload that already renders them, which also keeps the
chart version out of the test. managed-by stays a presence check: the chart
puts it on workload metadata but not on pod templates.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm: quote the resize hook's managed-by and instance labels
Matches the bucket hook, and keeps a numeric release name a string instead
of an int the API server rejects.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Sebastian Preisner <preisner@puzzle-itc.de>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Short-lived clusters report once under a fresh raft topology id and never
again, so CI runs and demo stacks each become their own cluster. Held
forward for the whole active window they pile up, and the volume server
line climbs every day while capacity stays flat.
Sum the fleet series over confirmed clusters only, the same set the
version and OS charts already use.
The reports are what tell us which versions and cluster sizes are
actually in use, and almost nobody flips the flag on, so the numbers we
have are close to useless. Default it on for master, server and mini,
and say in the flag help and the startup log how to turn it off.
Nothing new is collected: still an in-memory cluster id that changes on
restart, version, os, server counts, volume count and disk bytes, sent
once a day by the leader master only.
Counting by node lost the per-node ShardsInfo, and with it the shard ids the
old summary printed -- the message an operator gets when the pre-delete check
refuses now says only how many shards each node holds.
That is the wrong half. A set holding shards 0-9 and one holding 4-13 are
both "10 shards", and only the ids say whether what survived can rebuild the
volume, or which node to go looking at. Keep the count and list the ids
beside it.
* ci: move the fusermount3 repair into a composite action
Three copies of the same block were already drifting apart, and the
target comes from PATH: only ever add setuid root to a root-owned,
non-symlink binary under the system bin paths, and say why otherwise.
* test: say that the process exited in the wait errors
"process exit status 1 before ... accepted connections" is missing its
verb. Also mark the SIGTERM return discarded - it fails with
os.ErrProcessDone exactly when the select below already handles it.
* ci: prefer the distro fusermount3 over escalating a shadow copy
The shadowing /usr/local/bin/fusermount3 is not root-owned either, so
setting its setuid bit would have handed root to a binary the runner
user owns - the repair now symlinks the distro one earlier in PATH and
touches nothing, keeping the in-place chmod for a root-owned binary with
no distro alternative. A setuid bit only grants root when root owns the
file, so accept an existing one only then.
* ci: run the FUSE workflows when the shared action changes
Their paths filters listed each workflow file but not the composite
action all three now call.
generateEcShards writes shards beside the source volume, so encoding a
volume that lives on a non-default medium puts them on that medium while
-diskType still says hdd. The pre-delete check counted only the -diskType
bucket, so it saw a complete set as zero shards, called it unrecoverable
and aborted -- leaving the volume as both a .dat and a full shard set,
which every later reader then disagrees about.
Count by node across disks, as waitForEcShardsToRegister in the same file
already does. The spread check is unaffected: it locates shards through
collectEcShardBitsByNode and only uses diskType to find free slots.
* ci: restore the setuid bit on a shadowed fusermount3
Newer ubuntu-22.04 runner images carry a source-built fusermount3 in
/usr/local/bin that shadows the distro one in PATH and is not setuid
root. go-fuse looks the helper up through PATH, so every unprivileged
mount fails with "mount failed: Operation not permitted".
* test: fail a fuse test as soon as its mount process dies
A mount that cannot mount at all exits within a second, but the harness
still waited out the 30s readiness timeout and then reported "mount
point not ready within timeout", leaving the real cause buried in the
log tail. Watch the child processes and report their exit instead.
* mount: report a failed mount without a goroutine dump
A mount failure is an environment problem - no /dev/fuse, fusermount not
setuid, stale mount point - and the all-goroutine stack dump Fatalf adds
buries the one line that says so.
* s3: register the advertised ip with the master
The cluster address came from the bind ip, falling back to the
auto-detected interface, so -ip never reached the S3 registration.
weed mini -ip=localhost binds the wildcard and ended up registering
whatever interface happened to sort first -- on a host with VPN
interfaces, an address that stops routing once the tunnel drops.
IAM changes are pushed to registered S3 servers over gRPC, so every
mutation then blocked the full 10s propagation deadline before logging
a failure, and cluster.ps and the admin UI listed a node nothing could
reach. Identities still arrived through the /etc/iam metadata
subscription, so this cost latency and visibility, not credentials.
Add an advertise ip to the gateway option, preferring it over the bind
address, and wire the parent -ip through server, filer and mini.
* s3: treat any unspecified bind address as a wildcard
net.ParseIP + IsUnspecified covers ::, [::] and the expanded IPv6 forms
instead of only the 0.0.0.0 literal, so an IPv6 wildcard bind no longer
registers an address peers cannot dial. Host names parse as nil and stay
addresses in their own right. Apply the same guard to the advertised ip.
* s3api: load document-style policies from the advanced IAM config
The advanced IAM file doubles as the S3 identity config when only
-s3.iam.config is given. protojson drops its "document" field, so every
policy landed with empty content and warned "skipping invalid policy" on
each reload. Worse, if the same file also declares identities the empty
content sticks in the policy map and fails the whole runtime policy sync
into the IAM manager, so policies created later never reach it.
* iam: skip an unparsable policy instead of failing the whole runtime sync
One policy the engine cannot parse aborted SyncRuntimePolicies before it
touched anything, so every other policy stayed unsynced and the engine
kept serving whatever it last held.
* s3api: reject a non-role RoleArn in AssumeRole as a bad request
arn:aws:iam:::user/name can never resolve to a role, but the handler ran
it through the trust-policy check and answered "not authorized to assume
role", pointing the caller at a permission problem they do not have.
* s3api: build the policy content before touching the entry
Deleting "document" up front meant a marshal failure left the policy with
neither field, so a later rewrite would emit it with no definition at all.
* iam: pin the fail-closed handling of an unparsable policy
Say in the comment that dropping it from the desired set deletes it from
the engine on purpose, and cover it with a test.
* s3api: widen the non-role RoleArn test to canonical ARN shapes
The reported ARN omits the account id; a user ARN that carries one, and a
non-principal ARN, must be rejected the same way.
* volume: recover .idx rows overwritten by tiered deletes
A delete on a read-only volume backed by a remote tier used to write its
tombstone row at .idx offset 0 rather than appending it, so each delete
overwrote one more row at the front and lost the Put rows indexing the
first needles in .dat. Those needles 404 even though .dat still holds
them, and rebuilding .idx with weed fix means stopping the server and
pulling the whole .dat back from the tier.
The damage has a fingerprint -- .idx opening with a run of offset-0
tombstones, which a healthy .idx never does -- and .idx and .dat grow in
lockstep, so the lost rows indexed exactly the first N .dat records.
Detect it at load and re-derive them from a header-only walk over the
head of .dat, cheap even against a remote tier, appending only the keys
the .idx no longer names.
* rust volume: mirror the .idx head tombstone recovery
Port the Go detection and repair: an .idx opening with a run of offset-0
tombstones lost the Put rows indexing the first needles in .dat, so
re-derive them at load from a header-only walk over the head of .dat and
append the keys the .idx no longer names.
* volume: put recovered .idx rows back in front instead of appending
Appending left the offset-0 tombstone run at the head, so every later
load re-walked .idx to the tail to notice the volume was already
recovered, and the rows for the head of .dat sat past the .dat-tail row
-- costing CheckVolumeDataIntegrity its O(1) path and breaking the
ascending append order BinarySearchByAppendAtNs assumes.
Rewrite .idx as the recovered rows followed by its current contents,
through a temp file and a rename. .idx is back in .dat append order, so
a later load stops after reading one row.
* volume: keep the .idx mode when the repair replaces it
The recovery renames a fresh temp file over .idx, so a fixed 0644 (Go)
or whatever the umask allows (Rust) would silently widen an index an
operator had locked down. Carry the mode off the file being replaced.
* s3tables: read Iceberg manifest lists that omit the Avro format version
The Iceberg spec pins the Avro header metadata of manifest files but says
nothing about manifest lists, so writers disagree. Java and PyIceberg record
"format-version"; DuckDB writes no header metadata at all. iceberg-go reads a
missing entry as v1, so every v2 manifest listed in a DuckDB-written list is
rejected with
manifest file's 'format-version' metadata indicates version 2,
but entry from manifest list indicates version 1
and, because v1 has no "content" field, delete manifests silently decode as
data manifests.
ReadManifestList derives the version from the record schema the writer
embedded - v2 added "content" and the sequence numbers, v3 added
"first_row_id" - and splices it into the header before handing the bytes to
iceberg-go. Lists that already carry the entry, and input that is not a
parseable Avro container, go through untouched.
* iceberg: parse DuckDB-written manifest lists in maintenance and data preview
Every manifest list read - the four maintenance operations and the admin
table data preview - went straight to iceberg-go, so tables written by DuckDB
failed detection and all of compact, remove_orphans, rewrite_manifests and
expire_snapshots before they touched anything. Route them through
s3tables.ReadManifestList, which recovers the format version the writer left
out of the Avro header.
This also restores the manifest content type on those tables: with the list
read as v1 every delete manifest looked like a data manifest, which hid
deletes from the compaction guard and made the preview report a table with
position deletes as having none.
* telemetry: total disk usage over time counted each cluster on one day
GetMetrics aggregated s.instances, which holds only each cluster's most
recent report. Every cluster therefore landed in a single date bucket --
the day it last reported on -- so the chart plotted the disk usage of
clusters that went silent that day, and piled the whole live fleet onto
today. Aggregate the daily histories instead, reusing the day alignment
that the per-cluster size series already does.
* telemetry: don't pad the charts with days the server has no history for
The dashboard asks for 30 days, but daily history only starts when a
server first collects it, so the charts opened on a run of zeros and then
jumped -- reading as a fleet that appeared overnight. Start the window at
the oldest sample on hand when it is younger than the requested range.
* s3api/iceberg: report the reason a table schema was rejected
newTableMetadata swallowed the iceberg-go error and returned nil, so every
schema the metadata builder refused came back as a bare 500 "Failed to build
table metadata". A v3-only column type is the common case: creating a table
with a variant field but no format-version 3 property leaves the client with
nothing, while "variant is not supported until v3" sits in the server log.
Return the error instead and classify it. Schema, spec and argument failures
are the caller's input, so they answer 400 with the underlying reason; the
rest stay 500. Paths that build placeholder metadata with no schema keep
their existing 500 via newEmptyTableMetadata.
* s3api/iceberg: fail LoadTable when placeholder metadata cannot be built
buildLoadTableResult dropped a nil from the placeholder path straight into
the response. That serializes as "metadata":null under HTTP 200, which no
Iceberg client can parse -- a worse outcome than the 500 the nil was meant
to signal.
Return an error instead and let the five callers answer 500. The nil-return
convention goes away with it, so the commit and transaction paths check an
error rather than a sentinel.
* s3api/iceberg: route rejected schemas through writeManagerError
The two helpers added here duplicated work the package already does.
writeManagerError is the canonical error-to-response mapper -- it already
downgrades client-input failures to 400 and defaults the rest to 500 -- so
teach it the iceberg-go schema and spec sentinels instead of standing up a
parallel classifier. The placeholder wrapper was a pure alias for
newTableMetadata with nil arguments; call that directly.
No behavior change beyond the 500 message, which now reads err.Error()
like every other manager error rather than carrying its own prefix.
curl (unlike wget --spider) exits 0 as soon as it gets any HTTP response, even 401/403 - required if the filer rejects unauthenticated requests when JWT auth is enabled
The staged-new-volume placement skipped a disk holding the vid's EC shards using only the in-memory ecVolumes map, missing a shard present on disk but not mounted. Also scan the candidate disk for <vid>.ecNN files, so the promise holds regardless of mount state.
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
volume: skip a shard-holding disk when staging a decoded volume
ReceiveFile staged-new-volume mode picked any free disk of the target
medium. Skip a disk that already holds the vid's EC shards (Go
DiskLocation.FindEcVolume / Rust ec_volumes), so a decoded .dat never
lands in the same directory as a shard. This lets a caller safely stage
onto a shard host that has a spare disk, instead of requiring a host with
no shard of the vid at all.
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
Decoding EC shards back to a normal volume in place reconstructs <vid>.dat
in the shards' own directory, so the vid is momentarily registered as both
an EC and a normal volume in one location — the load/scan path then sees it
as both, risking mount ambiguity and needle loss. VolumeEcShardsToVolume
still supports that in-place path; this adds the primitives to decode onto
a *clean* peer instead:
- ReceiveFile gains a staged-new-volume mode: when the volume does not
exist here and ReceiveFileInfo.disk_type is set, pick a free-slot disk
of that medium and write <base><ext>.copying (not a valid volume name,
so the scanner never half-loads a partial push).
- VolumeEcShardsToVolume gains from_staged: adopt the pushed .dat/.idx/
.vif — rename .copying into place under a .note in-progress marker,
then mount — so <vid> lands on the peer only as a normal volume.
The caller decodes the shards off-box and streams the finished volume to a
peer holding no shard of the vid on the target medium. Go and Rust volume
servers get identical handlers. Proto: ReceiveFileInfo.disk_type (12; 8-11
reserved for versioned-EC), VolumeEcShardsToVolumeRequest.from_staged (3) +
disk_type (4).
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
VolumeMarkReadonly mutates raft-replicated master topology, so it must
reach the leader. notify_master_volume_readonly targeted the static seed
(config.masters.first()), so after any master failover it hit a follower
and failed "not current leader". Prefer current_master_url (the live
leader the heartbeat tracks), fall back to the seed before the first
heartbeat, mirroring store_ec.rs and Go's vs.GetMaster().
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
* s3: list the buckets an attached IAM policy grants
ListBuckets served an identity authorized by an attached IAM policy only
the buckets it had created itself. A user granted s3:ListBucket on a
bucket someone else provisioned could GetObject and ListObjectsV2 against
it, but the bucket never showed up in the listing any S3 client uses to
build its bucket picker.
The owner-index fast path is only valid for an identity whose grants name
every bucket it can reach, and the routing check assumed a policy could
never be enumerated. Read the names out of the policy instead: statements
that allow s3:ListBucket on a concrete bucket ARN become candidates, and
the per-bucket permission re-check still decides what is listed. A policy
that can reach a bucket it does not name -- a wildcard resource, a policy
variable, a NotResource, an STS session policy -- falls back to the full
scan, which evaluates the policy per bucket.
* s3: share the attached policy name lookup
authorizeWithIAM and the ListBuckets enumeration both built an identity's
policy names the same way, its own plus the ones from its enabled groups.
Pull that into one helper so group eligibility is decided in a single place.
* s3: read policy actions the way the IAM authorizer matches them
The IAM authorizer matches action names case-insensitively, so a policy
granting "S3:LISTBUCKET" or "S3:*" authorizes a list. The ListBuckets
classifier read those actions with the local case-sensitive matcher and
found no grant, so once the owner index was ready the buckets that policy
allows dropped out of the listing.
Match the action the looser way in the classifier: case-insensitive, and
true for any pattern holding a policy variable. Over-matching only costs a
candidate the per-bucket permission check then rejects, while under-matching
hides a bucket the caller can read.
* s3: infer a multipart grant in any case
The classifier matches action patterns case-insensitively but looked the
requested action up in a canonical-cased set, so "S3:UPLOADPART" missed the
s3:PutObject inference that the authorizer makes. Key the set for lookup in
lower case, matching how the IAM authorizer holds it.
* azure: let the blob endpoint be configured
The service url was always derived as <account>.blob.core.windows.net,
which leaves out Azure Government, Azure China, and private endpoints.
Name the blob service url instead and those accounts become reachable.
The url has to be https, since the account key or the bearer token would
otherwise travel in the clear.
* azure: reject an endpoint that carries no hostname
A url like https://:443/ has a host of ":443", so the emptiness check on
Host let it through and the request only failed once it reached Azure.
The hostname is what has to be there.
df on a mount shows the space the cluster gives up to the data: every
replica of a regular volume, every shard of an ec one. That is the honest
answer for capacity planning, but it is not the question a user asks when
they want to know how much of their data is stored.
Add -df.logical. The master reports the logical sizes alongside the raw
ones: one replica per regular volume, the data shards of each ec volume
counted once. Free space is divided by the copies the requested
replication makes, so used plus available stays the amount of data the
mount can still write, and it comes off the cluster-wide usage rather
than one collection's, since capacity is cluster-wide too.
Statistics through a filer resolves an unset replication to the filer's
default rather than the master's, matching where the writes it is sizing
for actually land.
The flag governs the quota check too, so a mount has one notion of how
much it is using. A filer that predates the new fields sends zeros, and
the mount keeps reporting the raw sizes.
Statistics aggregates the volume layouts of a collection, but EC volumes
are tracked outside collectionMap, so they were reported as nothing. A
mount over a cluster whose volumes have mostly been encoded showed a df
used size of a few GiB against terabytes of EC data.
Walk the data nodes and add the EC volumes of the requested collection.
Every shard copy counts, parity included, the way a regular volume's used
size counts every replica, so used size stays the space the cluster
actually occupies.
File count comes from the volume-wide .ecx and .ecj counts, taking the
largest a holder reports rather than summing them: both files travel with
the shards on a move, so several nodes can report the same tombstones.
* azure: authenticate the blob sink with Entra ID
Shared account keys have to be distributed and rotated everywhere a sink
runs. Leaving account_key empty now falls back to the identity chain, so a
workload identity or managed identity carries the authorization instead.
* azure: authenticate remote storage with Entra ID
The remote storage client demanded an account key and refused to start
without one. Fall back to the identity chain when it is absent, and let
azure.client_id pin a user-assigned identity.
* azure: reject a malformed storage account name
The account name is interpolated into the service URL, so a name carrying
a "/", "?" or "@" moves the authority elsewhere and an authenticated
request follows it. Hold callers to Azure's own naming rule instead.
* azure: keep a leftover environment key off the identity path
A configured client id asks for Entra ID, but AZURE_STORAGE_ACCESS_KEY
still filled in the account key behind it. An old mounted secret would go
on authenticating until it rotated, and the failure then blamed the key.
* azure: say what the identity path reads from the environment
A pinned client id alone is not enough for workload identity: the tenant
and the projected token come from the environment, and missing them only
surfaces later, when a token is first requested.
* s3: apply filer identity changes despite a static config file
A -config file with inline identities disabled the metadata-subscription
reload entirely, leaving the best-effort filer->s3 push as the only way
s3.configure changes could reach a running gateway. Reload on IAM events
regardless: the merge keeps the file's identities protected, and a full
credential-manager snapshot now also drops dynamic identities the store
no longer has, so revocation works without a restart.
* s3: log identity propagation failures as warnings
* s3: retry failed IAM reloads and reconcile policies and groups
An event-driven reload that fails now hands off to a coalescing retry
loop, so a transient filer error cannot strand a revoked credential
until the next IAM event. Full-state merges also drop dynamic policies
the store no longer has, keeping the static file's, and treat the group
snapshot as authoritative even when empty.
* s3: serialize IAM configuration loads
The SIGHUP file reload, subscription reloads, the retry loop, and the
postgres poll run on different goroutines. Without an end-to-end lock a
load holding an older store snapshot can commit after a newer one and
revert it. Hold reloadMu from snapshot through commit in both load
entry points; partial merges from pushed updates stay lock-free and
self-heal through the next event-driven reload.
* s3: keep static-file groups through full-state reconciliation
Group names from the static config file are tracked like identities and
policies, and a full snapshot that does not carry them keeps the current
definition and its memberships instead of dropping them.
* credential: include groups in postgres configuration snapshots
Full-state reconciliation treats absent groups as deleted, so a
snapshot that never carries them would erase every dynamic group.
* s3: revoke static-file groups dropped from the config file
A file reload is authoritative for the file's group set while keeping
dynamic groups, mirroring how full snapshots are authoritative for
dynamic groups while keeping the file's.
* credential: fail filer snapshots on unreadable entries
A skipped identity or policy file made the load report success with an
incomplete snapshot, which reconciliation reads as deletion and the
retry loop never sees. Unparseable content is still skipped: it is
durable, matches boot behavior, and must not block reloads forever.
* s3: ignore groups in static config files
Groups are managed through the IAM API and the dynamic store; no
deployment defines them in a bootstrap config file. Ignoring them with
a warning removes the two-directional group merge: full snapshots are
plainly authoritative and file reloads never touch groups.
Chris LuandGitHub
Lukas KalliesandGitHub
Kobi HikriandGitHub
Lars LehtonenandGitHub
