Three things the values did not say: a Prometheus outside the release stops
scraping and nothing reports it, the resize hook's policy is a hook resource
that uninstall leaves behind, and the DNS selectors are wrong on OpenShift.
kubectl run's "pod/x created" was captured alongside the pod log, so an
empty log would still not match exit=0 and the denial would pass without
anything having been tested.
componentName prefixes the release fullname onto the suffix it is given, and
the component label already starts with seaweedfs-, so the policy came out as
<release>-seaweedfs-seaweedfs-all-in-one.
The component names are not guessable - objectstorage-provisioner,
seaweedfs-all-in-one, volume-<name> - and a typo silently dropped the rules
it was carrying. Check against every component the chart can produce, not
the enabled ones, so a values file shared across releases can still hold
overrides for a component this one leaves off.
No seaweedfs binary talks to the Kubernetes API - there is no client-go in
go.mod - so the rule granted admin an egress path it never uses, and forced
anyone running admin with egress on to name an API server address for it.
The pod-RW ClusterRole the comment was reasoning from is a leftover from a
migration and is not read by any component.
The policy rendered whenever the hook was enabled, which is the default, so
every release carrying networkPolicy got one - as a pre-install hook that
Helm never collects, left in the namespace after uninstall. It also made
egress.kubeApiServer.cidrs mandatory for every release, since the policy
claims the API server, for a Job that only runs on an upgrade that grows a
PVC.
Move the command computation the Job is gated on into a helper and read it
from both.
The same int-coercion the hook templates were fixed for, in the file this PR
adds: unquoted, a release named 123 renders app.kubernetes.io/instance as a
YAML integer in the metadata, the podSelector and both peer selectors, and
the API server rejects the object - a policy that silently never applies.
CI now renders the chart as release "123" and fails if any policy label
comes out as a non-string.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
networkPolicy.enabled on its own only covers a default-deny that restricts
ingress. Where Egress is in its policyTypes as well, which is the usual
baseline, the components still cannot resolve DNS and egress.enabled is
required too. Both values and the README said the first half of that.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both were deriving the same condition from the same values, kept in step by
a comment. seaweedfs.bucketHookEnabled makes it one definition, so adding an
S3 mode cannot leave the Job running without its policy - which under
default-deny means the hook hangs. CI pins the pairing across the eleven
modes that decide it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
In a NetworkPolicy an empty ports list means every port and a missing peer
selector means every pod, so `kubeApiServer.ports: []` silently opened the
API server CIDRs on all ports, and nulling a DNS selector rendered
`podSelector: null`, which is every pod in kube-system. Both now fail the
render, and the DNS rule emits only the selectors that are set.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
codespell reads "retuned" as a misspelling of "returned" and fails the
spelling job.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
In a namespace with a default-deny policy the chart cannot be installed:
the components never reach each other, and the post-install bucket hook
waits on the master and filer until it gives up.
networkPolicy.enabled renders one policy per component, selecting its
pods by the standard app.kubernetes.io labels and admitting the other
pods of the release on the ports that component listens on. The port
lists come from the same values as the containerPorts, and CI asserts
the two agree. Restricting egress is a second opt-in with extraEgress
for the filer store and notification sinks, which the chart cannot
know about.
Closes#10421
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Matches the bucket hook, and keeps a numeric release name a string instead
of an int the API server rejects.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Presence alone let a wrong value through. The name/instance/chart values now
have to match a workload that already renders them, which also keeps the
chart version out of the test. managed-by stays a presence check: the chart
puts it on workload metadata but not on pod templates.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The block checked three keys and only looked at the pod's component, so a
missing chart/managed-by label, a wrong component on the Job, or a Job and
pod that disagree would all have passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The bucket hook pod carries only managed-by/instance and the volume
resize hook pod carries no labels at all, so nothing keyed on the
standard app.kubernetes.io set can address either of them. Give both
the same name/chart/managed-by/instance/component labels the other
workloads use, with the component naming each hook.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* volume: recover .idx rows overwritten by tiered deletes
A delete on a read-only volume backed by a remote tier used to write its
tombstone row at .idx offset 0 rather than appending it, so each delete
overwrote one more row at the front and lost the Put rows indexing the
first needles in .dat. Those needles 404 even though .dat still holds
them, and rebuilding .idx with weed fix means stopping the server and
pulling the whole .dat back from the tier.
The damage has a fingerprint -- .idx opening with a run of offset-0
tombstones, which a healthy .idx never does -- and .idx and .dat grow in
lockstep, so the lost rows indexed exactly the first N .dat records.
Detect it at load and re-derive them from a header-only walk over the
head of .dat, cheap even against a remote tier, appending only the keys
the .idx no longer names.
* rust volume: mirror the .idx head tombstone recovery
Port the Go detection and repair: an .idx opening with a run of offset-0
tombstones lost the Put rows indexing the first needles in .dat, so
re-derive them at load from a header-only walk over the head of .dat and
append the keys the .idx no longer names.
* volume: put recovered .idx rows back in front instead of appending
Appending left the offset-0 tombstone run at the head, so every later
load re-walked .idx to the tail to notice the volume was already
recovered, and the rows for the head of .dat sat past the .dat-tail row
-- costing CheckVolumeDataIntegrity its O(1) path and breaking the
ascending append order BinarySearchByAppendAtNs assumes.
Rewrite .idx as the recovered rows followed by its current contents,
through a temp file and a rename. .idx is back in .dat append order, so
a later load stops after reading one row.
* volume: keep the .idx mode when the repair replaces it
The recovery renames a fresh temp file over .idx, so a fixed 0644 (Go)
or whatever the umask allows (Rust) would silently widen an index an
operator had locked down. Carry the mode off the file being replaced.
* s3tables: read Iceberg manifest lists that omit the Avro format version
The Iceberg spec pins the Avro header metadata of manifest files but says
nothing about manifest lists, so writers disagree. Java and PyIceberg record
"format-version"; DuckDB writes no header metadata at all. iceberg-go reads a
missing entry as v1, so every v2 manifest listed in a DuckDB-written list is
rejected with
manifest file's 'format-version' metadata indicates version 2,
but entry from manifest list indicates version 1
and, because v1 has no "content" field, delete manifests silently decode as
data manifests.
ReadManifestList derives the version from the record schema the writer
embedded - v2 added "content" and the sequence numbers, v3 added
"first_row_id" - and splices it into the header before handing the bytes to
iceberg-go. Lists that already carry the entry, and input that is not a
parseable Avro container, go through untouched.
* iceberg: parse DuckDB-written manifest lists in maintenance and data preview
Every manifest list read - the four maintenance operations and the admin
table data preview - went straight to iceberg-go, so tables written by DuckDB
failed detection and all of compact, remove_orphans, rewrite_manifests and
expire_snapshots before they touched anything. Route them through
s3tables.ReadManifestList, which recovers the format version the writer left
out of the Avro header.
This also restores the manifest content type on those tables: with the list
read as v1 every delete manifest looked like a data manifest, which hid
deletes from the compaction guard and made the preview report a table with
position deletes as having none.
* telemetry: total disk usage over time counted each cluster on one day
GetMetrics aggregated s.instances, which holds only each cluster's most
recent report. Every cluster therefore landed in a single date bucket --
the day it last reported on -- so the chart plotted the disk usage of
clusters that went silent that day, and piled the whole live fleet onto
today. Aggregate the daily histories instead, reusing the day alignment
that the per-cluster size series already does.
* telemetry: don't pad the charts with days the server has no history for
The dashboard asks for 30 days, but daily history only starts when a
server first collects it, so the charts opened on a run of zeros and then
jumped -- reading as a fleet that appeared overnight. Start the window at
the oldest sample on hand when it is younger than the requested range.
* s3api/iceberg: report the reason a table schema was rejected
newTableMetadata swallowed the iceberg-go error and returned nil, so every
schema the metadata builder refused came back as a bare 500 "Failed to build
table metadata". A v3-only column type is the common case: creating a table
with a variant field but no format-version 3 property leaves the client with
nothing, while "variant is not supported until v3" sits in the server log.
Return the error instead and classify it. Schema, spec and argument failures
are the caller's input, so they answer 400 with the underlying reason; the
rest stay 500. Paths that build placeholder metadata with no schema keep
their existing 500 via newEmptyTableMetadata.
* s3api/iceberg: fail LoadTable when placeholder metadata cannot be built
buildLoadTableResult dropped a nil from the placeholder path straight into
the response. That serializes as "metadata":null under HTTP 200, which no
Iceberg client can parse -- a worse outcome than the 500 the nil was meant
to signal.
Return an error instead and let the five callers answer 500. The nil-return
convention goes away with it, so the commit and transaction paths check an
error rather than a sentinel.
* s3api/iceberg: route rejected schemas through writeManagerError
The two helpers added here duplicated work the package already does.
writeManagerError is the canonical error-to-response mapper -- it already
downgrades client-input failures to 400 and defaults the rest to 500 -- so
teach it the iceberg-go schema and spec sentinels instead of standing up a
parallel classifier. The placeholder wrapper was a pure alias for
newTableMetadata with nil arguments; call that directly.
No behavior change beyond the 500 message, which now reads err.Error()
like every other manager error rather than carrying its own prefix.
curl (unlike wget --spider) exits 0 as soon as it gets any HTTP response, even 401/403 - required if the filer rejects unauthenticated requests when JWT auth is enabled
The staged-new-volume placement skipped a disk holding the vid's EC shards using only the in-memory ecVolumes map, missing a shard present on disk but not mounted. Also scan the candidate disk for <vid>.ecNN files, so the promise holds regardless of mount state.
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
volume: skip a shard-holding disk when staging a decoded volume
ReceiveFile staged-new-volume mode picked any free disk of the target
medium. Skip a disk that already holds the vid's EC shards (Go
DiskLocation.FindEcVolume / Rust ec_volumes), so a decoded .dat never
lands in the same directory as a shard. This lets a caller safely stage
onto a shard host that has a spare disk, instead of requiring a host with
no shard of the vid at all.
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
Decoding EC shards back to a normal volume in place reconstructs <vid>.dat
in the shards' own directory, so the vid is momentarily registered as both
an EC and a normal volume in one location — the load/scan path then sees it
as both, risking mount ambiguity and needle loss. VolumeEcShardsToVolume
still supports that in-place path; this adds the primitives to decode onto
a *clean* peer instead:
- ReceiveFile gains a staged-new-volume mode: when the volume does not
exist here and ReceiveFileInfo.disk_type is set, pick a free-slot disk
of that medium and write <base><ext>.copying (not a valid volume name,
so the scanner never half-loads a partial push).
- VolumeEcShardsToVolume gains from_staged: adopt the pushed .dat/.idx/
.vif — rename .copying into place under a .note in-progress marker,
then mount — so <vid> lands on the peer only as a normal volume.
The caller decodes the shards off-box and streams the finished volume to a
peer holding no shard of the vid on the target medium. Go and Rust volume
servers get identical handlers. Proto: ReceiveFileInfo.disk_type (12; 8-11
reserved for versioned-EC), VolumeEcShardsToVolumeRequest.from_staged (3) +
disk_type (4).
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
VolumeMarkReadonly mutates raft-replicated master topology, so it must
reach the leader. notify_master_volume_readonly targeted the static seed
(config.masters.first()), so after any master failover it hit a follower
and failed "not current leader". Prefer current_master_url (the live
leader the heartbeat tracks), fall back to the seed before the first
heartbeat, mirroring store_ec.rs and Go's vs.GetMaster().
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
* s3: list the buckets an attached IAM policy grants
ListBuckets served an identity authorized by an attached IAM policy only
the buckets it had created itself. A user granted s3:ListBucket on a
bucket someone else provisioned could GetObject and ListObjectsV2 against
it, but the bucket never showed up in the listing any S3 client uses to
build its bucket picker.
The owner-index fast path is only valid for an identity whose grants name
every bucket it can reach, and the routing check assumed a policy could
never be enumerated. Read the names out of the policy instead: statements
that allow s3:ListBucket on a concrete bucket ARN become candidates, and
the per-bucket permission re-check still decides what is listed. A policy
that can reach a bucket it does not name -- a wildcard resource, a policy
variable, a NotResource, an STS session policy -- falls back to the full
scan, which evaluates the policy per bucket.
* s3: share the attached policy name lookup
authorizeWithIAM and the ListBuckets enumeration both built an identity's
policy names the same way, its own plus the ones from its enabled groups.
Pull that into one helper so group eligibility is decided in a single place.
* s3: read policy actions the way the IAM authorizer matches them
The IAM authorizer matches action names case-insensitively, so a policy
granting "S3:LISTBUCKET" or "S3:*" authorizes a list. The ListBuckets
classifier read those actions with the local case-sensitive matcher and
found no grant, so once the owner index was ready the buckets that policy
allows dropped out of the listing.
Match the action the looser way in the classifier: case-insensitive, and
true for any pattern holding a policy variable. Over-matching only costs a
candidate the per-bucket permission check then rejects, while under-matching
hides a bucket the caller can read.
* s3: infer a multipart grant in any case
The classifier matches action patterns case-insensitively but looked the
requested action up in a canonical-cased set, so "S3:UPLOADPART" missed the
s3:PutObject inference that the authorizer makes. Key the set for lookup in
lower case, matching how the IAM authorizer holds it.
* azure: let the blob endpoint be configured
The service url was always derived as <account>.blob.core.windows.net,
which leaves out Azure Government, Azure China, and private endpoints.
Name the blob service url instead and those accounts become reachable.
The url has to be https, since the account key or the bearer token would
otherwise travel in the clear.
* azure: reject an endpoint that carries no hostname
A url like https://:443/ has a host of ":443", so the emptiness check on
Host let it through and the request only failed once it reached Azure.
The hostname is what has to be there.
df on a mount shows the space the cluster gives up to the data: every
replica of a regular volume, every shard of an ec one. That is the honest
answer for capacity planning, but it is not the question a user asks when
they want to know how much of their data is stored.
Add -df.logical. The master reports the logical sizes alongside the raw
ones: one replica per regular volume, the data shards of each ec volume
counted once. Free space is divided by the copies the requested
replication makes, so used plus available stays the amount of data the
mount can still write, and it comes off the cluster-wide usage rather
than one collection's, since capacity is cluster-wide too.
Statistics through a filer resolves an unset replication to the filer's
default rather than the master's, matching where the writes it is sizing
for actually land.
The flag governs the quota check too, so a mount has one notion of how
much it is using. A filer that predates the new fields sends zeros, and
the mount keeps reporting the raw sizes.
Statistics aggregates the volume layouts of a collection, but EC volumes
are tracked outside collectionMap, so they were reported as nothing. A
mount over a cluster whose volumes have mostly been encoded showed a df
used size of a few GiB against terabytes of EC data.
Walk the data nodes and add the EC volumes of the requested collection.
Every shard copy counts, parity included, the way a regular volume's used
size counts every replica, so used size stays the space the cluster
actually occupies.
File count comes from the volume-wide .ecx and .ecj counts, taking the
largest a holder reports rather than summing them: both files travel with
the shards on a move, so several nodes can report the same tombstones.
* azure: authenticate the blob sink with Entra ID
Shared account keys have to be distributed and rotated everywhere a sink
runs. Leaving account_key empty now falls back to the identity chain, so a
workload identity or managed identity carries the authorization instead.
* azure: authenticate remote storage with Entra ID
The remote storage client demanded an account key and refused to start
without one. Fall back to the identity chain when it is absent, and let
azure.client_id pin a user-assigned identity.
* azure: reject a malformed storage account name
The account name is interpolated into the service URL, so a name carrying
a "/", "?" or "@" moves the authority elsewhere and an authenticated
request follows it. Hold callers to Azure's own naming rule instead.
* azure: keep a leftover environment key off the identity path
A configured client id asks for Entra ID, but AZURE_STORAGE_ACCESS_KEY
still filled in the account key behind it. An old mounted secret would go
on authenticating until it rotated, and the failure then blamed the key.
* azure: say what the identity path reads from the environment
A pinned client id alone is not enough for workload identity: the tenant
and the projected token come from the environment, and missing them only
surfaces later, when a token is first requested.
* s3: apply filer identity changes despite a static config file
A -config file with inline identities disabled the metadata-subscription
reload entirely, leaving the best-effort filer->s3 push as the only way
s3.configure changes could reach a running gateway. Reload on IAM events
regardless: the merge keeps the file's identities protected, and a full
credential-manager snapshot now also drops dynamic identities the store
no longer has, so revocation works without a restart.
* s3: log identity propagation failures as warnings
* s3: retry failed IAM reloads and reconcile policies and groups
An event-driven reload that fails now hands off to a coalescing retry
loop, so a transient filer error cannot strand a revoked credential
until the next IAM event. Full-state merges also drop dynamic policies
the store no longer has, keeping the static file's, and treat the group
snapshot as authoritative even when empty.
* s3: serialize IAM configuration loads
The SIGHUP file reload, subscription reloads, the retry loop, and the
postgres poll run on different goroutines. Without an end-to-end lock a
load holding an older store snapshot can commit after a newer one and
revert it. Hold reloadMu from snapshot through commit in both load
entry points; partial merges from pushed updates stay lock-free and
self-heal through the next event-driven reload.
* s3: keep static-file groups through full-state reconciliation
Group names from the static config file are tracked like identities and
policies, and a full snapshot that does not carry them keeps the current
definition and its memberships instead of dropping them.
* credential: include groups in postgres configuration snapshots
Full-state reconciliation treats absent groups as deleted, so a
snapshot that never carries them would erase every dynamic group.
* s3: revoke static-file groups dropped from the config file
A file reload is authoritative for the file's group set while keeping
dynamic groups, mirroring how full snapshots are authoritative for
dynamic groups while keeping the file's.
* credential: fail filer snapshots on unreadable entries
A skipped identity or policy file made the load report success with an
incomplete snapshot, which reconciliation reads as deletion and the
retry loop never sees. Unparseable content is still skipped: it is
durable, matches boot behavior, and must not block reloads forever.
* s3: ignore groups in static config files
Groups are managed through the IAM API and the dynamic store; no
deployment defines them in a bootstrap config file. Ignoring them with
a warning removes the two-directional group merge: full snapshots are
plainly authoritative and file reloads never touch groups.
* s3: require a bucket-policy action to write a bucket policy
PutBucketPolicy and DeleteBucketPolicy were gated on ACTION_WRITE, the same
action that grants object writes. An explicit Allow in a bucket policy
short-circuits IAM entirely -- authRequestWithAuthType sets policyAllows and
skips VerifyActionPermission -- so anyone who could write an object could
author a policy granting itself, or anonymous, anything on the bucket.
That is what separates a bucket policy from the sibling bucket controls also
gated on ACTION_WRITE: rewriting cors or lifecycle can destroy data, but only
a policy hands out access.
Give the two verbs their own actions, mapped to the AWS names that were
already defined but unrouted. ACTION_ADMIN would also have closed it, but it
resolves to s3:* for IAM identities, forcing a blanket grant on a user holding
a precise s3:PutBucketPolicy. Admins are unaffected, since isAdmin
short-circuits CanDo, and an operator can delegate with PutBucketPolicy:bucket.
The route binding is asserted from the router source: checking the action
constants alone still passes when the route says ACTION_WRITE.
* s3: also read the action from a direct iam.Auth call in the route test
Routes read iam.Auth(cb.Limit(handler, ACTION)), a multi-value pass-through:
Limit returns (http.HandlerFunc, Action) and those become Auth's parameters, so
the action Auth authorizes on is Limit's second argument and the two cannot
disagree -- Auth(Limit(h, X), Y) does not compile.
A route that skipped Limit and called Auth with its own action would compile,
though, and the test reported that as a missing route rather than as the wrong
action. Recognise the two-argument Auth form so it names the action instead.
* s3: make the bucket-policy actions grantable through an IAM policy
The new actions close the escalation only if an operator can grant them, and
they were not reachable: MapToStatementAction had no entry for PutBucketPolicy,
so an IAM policy naming s3:PutBucketPolicy was rejected outright with "not a
valid action". GetBucketPolicy was unmapped the same way.
DeleteBucketPolicy was mapped, but to ACTION_ADMIN -- granting an identity
permission to delete a bucket policy handed it full administrative access.
Map all three to the actions the router now uses, and add the reverse
direction so an identity holding them renders back as a policy statement
instead of a bare "s3:".
* admin: offer the bucket-policy permissions in the user editor
The two new actions are otherwise only grantable by hand-editing identity JSON
or by calling the IAM API, so an operator using the UI cannot delegate bucket
policy management without granting Admin.
Regenerating this file also picks up codegen the repo has not taken yet: the
checked-in _templ.go files were produced by templ v0.3.1001 while go.mod pins
v0.3.1020, so the generator rewrites the attribute-value calls. That churn is
confined to this one file; running `make generate` in weed/admin reproduces it
across all 36.
* volume: gate the admin RPCs that only shell and workers call
checkGrpcAdminAuth covered 19 of the 48 VolumeServer RPCs, so an operator who
sets -whiteList expecting it to cover the gRPC surface gets partial coverage.
Extend it to ten that mutate state and are only ever called by the shell or a
worker: SetState, VolumeCopy, the EC generate/rebuild/copy/unmount/to-volume
pair, both tier moves, and VolumeTailReceiver. That is safe because the same
callers already reach gated RPCs today -- VolumeMarkReadonly, VacuumVolume*,
VolumeEcShardsDelete, VolumeDelete -- so a whitelist deployment already lists
those hosts. Nothing here is on a master or peer path, which is what made the
earlier fail-closed gate break multi-host clusters.
The split is by caller rather than by blast radius: the guard matches a peer IP
against the whitelist, and a whitelist holds masters, shell hosts and workers,
not every peer volume server. Gating a call one volume server makes to another
would break replication, EC and tiering, so those stay open.
Two test fakes embedded a nil grpc.ServerStream and only implemented Send;
they now implement Context, which the streaming RPCs read to authorize.
* volume: fail the build when a gRPC method skips the admin gate
The admin gate is an opt-in list in a 48-method service, which is how it
drifted down to covering 19 of them: nothing tied adding an RPC to deciding
whether it needed the gate.
Parse volume_server.proto, walk the AST of every *VolumeServer method, and
require each RPC to either call checkGrpcAdminAuth or appear in
ungatedVolumeServerRPCs with the reason it stays open. A stale entry naming an
RPC that no longer exists fails too, so the list can't quietly stop exempting
anything.
The exemptions are the cluster-internal calls -- replica sync, EC shard
distribution, vacuum reads, backup, tailing -- plus the read-only and liveness
RPCs. Closing the cluster-internal ones needs a peer identity rather than an
IP whitelist; recording them here makes that a visible decision instead of an
omission.
The AST walk also corrects the count: a line-window scan credits
VacuumVolumeCheck and VolumeServerStatus with a neighbouring function's guard.
* volume: fix EC decode/reconstruct index locality under -dir.idx
EC->replicated decode failed under -dir.idx and on multi-disk with "volume not
found on disk". The reconstruct rebuilds the .dat on the data disk but the
on-demand VolumeMount scans only the data directory, matching on .idx/.vif;
with the rebuilt .idx off in the index directory it matched the volume's
leftover EC .vif and skipped the volume as EC metadata.
- Resolve the EC .ecx local-first: prefer the copy co-located with the shards
over the shared -dir.idx copy, with a non-empty preference so a 0-byte local
stub still yields to a valid sibling (the cross-disk fallback).
- Co-locate the rebuilt .idx with the .dat at the end of the reconstruct so the
mount finds it; sweep .ecx/.ecj from both the data and index directories on
Destroy so a stale copy cannot re-mount as a phantom EC volume.
- Add VolumeConsolidateIndex: once the EC shards are deleted, unmount, move the
.idx/.sdx from the data disk back to the -dir.idx directory (copy fallback
across filesystems), and remount. A no-op without -dir.idx.
* volume: tests for EC index locality (local-first .ecx, sweep, consolidate)
- NewEcVolume prefers a non-empty local .ecx over the shared index dir, and a
0-byte local stub yields to a non-empty shared copy (the #9212 fallback).
- Destroy sweeps .ecx/.ecj from both the data and index directories.
- ConsolidateVolumeIndex moves a co-located index back to the -dir.idx dir and
keeps the volume mounted; no-op without a separate index dir.
- RenameOrCopyFile moves a file and drops the source.
* volume: relocate the decoded index in place, without a read gap
ConsolidateVolumeIndex previously unmounted the volume, moved the index, and
remounted it. Between the EC-shard delete and the remount the volume had neither
a normal nor an EC form mounted, so a read landing in that window got a
not-found (or was proxied away).
Move the index in place instead: RelocateIndexTo takes the data-file write lock,
closes the needle map and data backend, moves the .idx (and derived .sdx), then
retargets dirIdx and reloads — the same close-swap-load CommitCompact uses. The
volume never leaves the mounted set, so a concurrent read blocks briefly on the
lock rather than failing. The test now writes a needle before consolidating and
reads it back after, proving the in-place reload keeps the volume serving.
* volume: address review — maintenance guard, no orphan on copy failure
- VolumeConsolidateIndex now rejects the request under maintenance mode, like
VolumeConfigure and the other mutating volume RPCs.
- RenameOrCopyFile rolls the cross-device copy back when the source cannot be
removed, so a failed move never leaves two divergent copies (the loader would
keep the data-dir one while the idx-dir orphan goes stale).
- RelocateIndexTo logs a failed reopen-after-failed-move instead of swallowing
it, since that leaves the volume unusable until the next load.
The sentinel stores hardcoded a 30s read timeout and a 1m retry backoff.
After a sentinel failover every request that picked a pooled connection to
the old master sat there for 30s before the connection was retired, and the
pool timeout derived from it (read timeout + 1s) queued the rest behind
them. The other redis stores took the go-redis defaults with no way to tune
anything.
Read the dial, timeout and pool knobs from each redis store section instead,
keeping the go-redis default for every key left unset.
security.GetJwt reads the "jwt" query parameter before the Authorization
header, and the proxy forwarded the caller's whole query apart from
proxyChunkId. So on a read, where the filer mints a volume token and sets the
header itself, a caller-supplied ?jwt= silently outranked it: the volume
server validated a credential the caller chose rather than the one the filer
attached, and the read failed with a 401 the filer could not explain.
Drop it on the read path, where the filer owns the credential. Writes keep
theirs -- the proxy forwards a writer's own AssignVolume token either way, so
the query parameter is just a second channel for the same credential and
stripping it would break a caller that presents it that way.
Nothing in the tree passes a jwt by query; maybeAddAuth always sets the
header.
The S3 write path cut fixed 8MB chunks, so an object stored through S3
chunked differently from the same bytes stored through the filer, WebDAV
or a mount, and -maxMB had no effect on it. Read maxMB from the filer
configuration at startup and use it, falling back to 8MB when the filer
reports none.
* filer: add a placement overlay seam for the write path
New volumes take their disk type, replication, and data center from the
explicit request or the matched filer.conf rule. That leaves no way for a
feature to steer a whole collection onto a medium without an operator
writing an fs.configure rule by hand.
Add a generic PlacementOverlay hook on the filer: a func that maps a
collection to a placement override, installed by a factory the way the
plugin-worker handlers register. detectStorageOption consults it between
the explicit request value and the filer.conf rule, so it overrides the
rule but yields to a value the caller asked for.
The seam names no feature concepts, so it stays generic; a downstream
build registers the overlay it wants (e.g. a storage-class Landing tier).
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
* filer: address review on the placement overlay seam
Honor ResolvePlacement's ok flag explicitly rather than relying on empty
values falling through the util.Nvl chain, and log at V(4) when the
overlay steers a collection. Document that RegisterPlacementOverlay is
init-only, so the unsynchronized read in NewFiler cannot race the write.
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
* filer: claim the base fid when minting a volume read token
GenJwtForVolumeServer stamped the fid verbatim, but the volume server strips a
trailing _N delta suffix before comparing the claim, so a token minted for a
batch-assigned fid like 3,01637037d6_1 was checked against 3,01637037d6 and
never matched. Reading such a chunk through the filer returned 401 wherever
jwt.signing.read.key was configured.
Strip the suffix before minting, via a helper shared with the proxyChunkId
validation that was already doing the same thing inline.
* filer: don't mint a volume write token for an anonymous proxy caller
The ?proxyChunkId= branch dispatches and returns before the JWT gate, so
whatever credential the proxy attaches is reachable without authentication.
It attached a token from maybeGetVolumeReadJwtAuthorizationToken, which fell
back to the write signing key when jwt.signing.read.key was unset -- the
configuration scaffold/security.toml recommends for a filer, since read JWTs
are only supported in a master+volume setup. An anonymous
DELETE /?proxyChunkId=<fid> therefore arrived at the volume server holding a
write-key token scoped to that fid, and the volume server honored it.
Sign read tokens with the read key only. The fallback bought nothing on a
read anyway: a volume server enforces read JWTs solely when that same key is
set, so when the fallback fired the read was unchecked regardless.
Mint only for reads. Writers proxied through the filer carry their own volume
JWT from AssignVolume, forwarded with the rest of the caller's headers, so
weed mount -filerProxy uploads are unaffected. Moving the dispatch below the
JWT gate instead would have broken them, since that token is signed with
jwt.signing rather than jwt.filer_signing.
On a read with nothing to mint, drop the caller's Authorization rather than
relaying it: there it is a filer credential, and forwarding it would hand a
volume server a token it never used to see.
* filer: keep proxied writes out of the read concurrency semaphore
The semaphore is named and documented for reads -- it exists so replication
bursts can't open hundreds of connections to one volume server -- but it was
applied to every proxied method. A write queued behind sixteen in-flight reads
can wait past the 10s default expiry of the AssignVolume token it carries, and
the volume server then answers 401. shouldReassignUpload treats a 4xx as final,
so the uploader replays the same expired token instead of re-assigning and the
write fails up to the caller.
This only became reachable once the filer stopped re-minting a fresh token
after the wait.
LookupFileId only requires the fid to contain a single comma, and the value
is pasted straight into the volume server URL path, so ?proxyChunkId=3,x/../../status
resolves to volume 3 and then addresses an endpoint the caller never named:
Go sends the dot segments verbatim, the volume server's mux cleans the path
and redirects to /status, and the filer follows the redirect and relays the
body. That reaches any handler on the volume server -- status, stats, the UI --
past a filer that operators expect to be the only exposed surface.
Parse the fid before the lookup and answer 400 when it doesn't parse.
A trailing _N delta suffix from batch assigns is legal, so it is stripped
first, but only when it is a non-empty run of digits. The volume server strips
at the last "_" unconditionally, which is safe there because its fid came out
of a path the mux parsed and so cannot hold a "/"; here the value is raw query
input, and an unguarded strip would reduce "3,01637037d6_1/../../status" to a
valid fid and wave the traversal through.
The queue holds sixteen sealed windows, which is a memory bound only
while a window is BufferSize. An entry larger than that grows its window
to fit, and the depth then multiplies straight through: sixteen queued
copies of a 100 MB window is 1.6 GB of flush data alone.
Account the queued bytes and make producers wait once they pass the
ceiling the depth was chosen for. What is charged is the pooled slab
rather than the window length, since mem.Allocate rounds up to a size
class and the queue holds the whole slab. A window larger than the whole
budget still goes through on its own, so an oversized entry is never
stuck.
Windows are admitted in the order they were sealed. A producer can now
park here for seconds, and letting a later window overtake an earlier one
would persist them out of order and walk lastFlushedOffset and
lastFlushTsNs backwards.
A window is copied into its slab under the write lock, before the
reservation is taken, so a burst of concurrent oversized writers would
each hold a full copy in hand while queueing up -- memory the budget
never sees. Large writers wait for queue headroom before they take the
lock, which throttles the burst; it does not bound it, since a writer
that passes the check still seals unconditionally.
Take any room in the queue before the shutdown escape, too: the window is
already sealed by then, so dropping it loses records the caller was told
were accepted. A shutdown that races a full queue can still drop one --
that predates this change and needs the flush loop's lifetime reworked.
Size a grown window to the entry rather than to twice it: the extra room
only bought space for a second oversized record in the same window, which
doubles the flush copy and the snapshot taken of it. The overflow guard
halved its bound for that doubled allocation, so raise it to match what
is now allocated and what maxBufferSize documents.
* volume: reject needle blob writes to read-only volumes
WriteNeedleBlob appends the blob to .dat and only then calls nm.Put. On a
read-only volume the needle map is a SortedFileNeedleMap whose Put always
fails, so the append is never indexed and never rolled back.
Nothing upstream stops this: volume.check.disk picks its targets from the
master's cached topology, which goes stale the moment a volume server marks
a replica read-only itself — a failed data integrity check at load, or an
EIO quarantine. Each sync attempt then grows the .dat of a replica that is
supposed to be frozen by one unindexed needle, and reports it as "invalid
argument", the bare os.ErrInvalid the needle map returns.
Check IsReadOnly before touching .dat, same as the upload path does.
* volume: say which needle and volume failed to index
An index write that fails surfaced as a bare errno with no volume, no needle
and no file — "invalid argument" for a read-only needle map, or a plain
ENOSPC when .idx lives on its own filesystem via -dir.idx. Both were logged
at V(4), so by default the operator saw only the errno the client got back.
* filer: write the metadata log in pieces a volume server will accept
A single oversized metadata event grows the log buffer past the volume
server's fileSizeLimitMB, and the flush of that buffer is then rejected
forever: the retry loop has no exit, so the blob at the head of the queue
blocks every later flush and the metadata feed stalls until restart.
Split the flushed buffer into BufferSize pieces, on record boundaries
where possible so each piece still decodes on its own, and retry each
piece separately so a partial success is not replayed. Log files are
already read as a chunk stream, with a whole-file fallback when a chunk
does not decode standalone, so a record may cross a piece boundary.
* log_buffer: let go of a window array grown for an oversized entry
An entry larger than BufferSize grows the window array to 2*size+4, and
window arrays cycle through SealBuffer rather than being freed. One such
entry therefore leaves every later window carrying its size, and
currentSnapshotView allocates a snapshot as wide as the array on each
window, so a few KB of metadata keeps paying for it.
Drop the array when SealBuffer hands it back. Growth is on demand, so
the next oversized entry just reallocates.
* iceberg maintenance: store merged data files as chunks, not inline
saveFilerFile had no size threshold, so compaction wrote whole merged
parquet files -- hundreds of MB -- as Entry.Content. That puts the
parquet bytes verbatim in the filer store and sends them through the
metadata change log again as one event.
Keep manifests and metadata JSON inline, upload anything larger to
volume servers in chunks, assigning through the filer so the path's
storage rules apply.
* filer: follow the file size limit the volume servers report
The starting piece size is a constant, so a cluster whose
-fileSizeLimitMB is set below it would reject every piece and wedge just
the same. The rejection names the limit, so take it from there and
re-cut the rest of the flush to fit.
Piece the buffer one at a time rather than up front, since the size can
change partway through a flush.
plugin: stop the scheduler lock test racing its own background loops
TestRunLaneSchedulerIterationLockBehavior constructed the plugin with a
cluster-context provider, which makes New start a background scheduler
loop per lane. Those loops call runLaneSchedulerIteration on the same
lane the test then drives by hand, so a loop could consume the due job —
running detection and pushing the next-detection time forward — before
the manual call observed the lock. The Default case then saw the lock
acquired zero times and failed intermittently.
Construct without the provider so no loops start, and set the provider
afterward so the manual iteration can still detect. This is the pattern
scheduler_status_test.go already uses for the same reason.
Reproduced under -race -count=100 -p 4 before, green after.
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
* util: match transient error messages case-insensitively, expose the message form
The same condition reaches different layers capitalized differently -- a
volume server relays its idle timeout as "I/O timeout" inside a JSON string --
and the callers that grew their own substring lists all lower-case first.
Also split out IsTransientErrorMessage for the paths that carry only the text,
such as the per-file status strings in a batch delete response, and pick up
"no route to host" and "network is unreachable" from the gRPC classifier.
* filersink: classify transient network errors through util.IsTransientError
The local list caught i/o timeout, connection reset, and broken pipe but not
connection refused, no such host, unexpected EOF, the syscall errnos, or the
gRPC and S3 overload codes. Keep only the bare io.EOF case, which is transient
here -- a truncated chunk read -- but a clean stream end elsewhere.
* filer deletion: reuse util.IsTransientErrorMessage for the network patterns
Six of the sixteen patterns were already covered. Keep the ones specific to
this pipeline -- read-only volumes, lookup failures, backpressure -- and note
why context cancellation stays retryable here: it decides whether to requeue
the deletion, not whether to retry a call.
* wdclient: fold the shared classifier into the volume lookup retry check
The string tail duplicated the shared list and missed the syscall errnos and
net.Error timeouts. Keep "connection" and "timeout", which are broader than
the shared classifier on purpose: a volume lookup is a cheap read-only call.
* filer.replicate: commit the kafka offset after replicating, not on receipt
The partition consumer committed the offset as soon as it handed the message
to the channel, so a sink write that failed was logged and the message was
already behind the committed offset -- never redelivered, permanently missing
from the sink.
Commit in onSuccessFn instead, and hold the committed offset behind the
oldest offset that failed to replicate so a restart redelivers from there.
* filer.replicate: delete the sqs message after replicating, not on receipt
ReceiveMessage deleted the message before the replicator had a chance to run,
so a failed sink write dropped it for good. Move the delete into onSuccessFn
and leave the message in the queue otherwise, letting the visibility timeout
redeliver it.
* util: retry transient errors, not just the ones containing "transport"
util.Retry only retried when the error string contained "transport", so a
plain "read: connection reset by peer" from S3 got zero retries. Classify
the error instead: net timeouts, connection resets, and the throttling and
overload replies S3 and gRPC return are all worth another attempt, while a
cancelled or expired context is not.
* filer sync: hold the sync offset behind a failed event
A sync job that returned an error was logged and forgotten, and the
watermark advanced past it anyway. The offset is the durable resume point,
so the event was never replayed: for filer.remote.sync that left the file
present locally, absent on the remote, with no RemoteEntry and nothing to
retry it.
Pin the watermark at the oldest failed event. Later events keep flowing,
but the persisted offset stays behind the failure, so a restart replays it.
The scaffold advertises enable_tls, ca_cert_path, client_cert_path and
client_key_path under redis2, redis2_sentinel and redis_cluster2, but only the
plain redis2 and redis3 stores ever read them, and under a different name,
enable_mtls. Sentinel and cluster setups quietly connected in plaintext.
Build the TLS config in one place and use it from all six stores. enable_mtls
still works. The CA and the client key pair are optional now, so enable_tls
alone verifies against the system roots, and ServerName is left unset so
go-redis validates each address the sentinel and cluster clients dial.
redis2_sentinel and redis3_sentinel stores only passed Username/Password
into redis.FailoverOptions, which authenticates against the Redis
master/replica servers. When Sentinel itself requires auth (requirepass
set in sentinel.conf), go-redis had no credentials to send to it,
causing a NOAUTH error before ever reaching the master.
Add sentinel_username/sentinel_password config options that map to
go-redis's SentinelUsername/SentinelPassword fields, distinct from the
existing master auth credentials.
Deleting the only object under a prefix in a versioned bucket writes a
delete marker and keeps the version history, so the filer directory
survives with nothing a current-version listing would return. A delimited
ListObjects kept reporting that path in CommonPrefixes, because the
prefixes come from the directory tree rather than from the keys, while a
listing scoped inside the prefix correctly came back empty.
Probe a directory before reporting it: one that holds entries but no key
the listing returns is neither a CommonPrefix nor a path the
trailing-slash probe answers for. Empty directories keep the meaning they
have today, and the probe only runs for buckets with versioning
configured, the only ones that can reach this state.
A maintenance snapshot carried only its own labels — merged-files,
delete-groups and friends — and no summary counters, so every engine that
reads a table's size out of the current snapshot summary reported nothing
for it: PyIceberg's inspect.snapshots, Trino's $snapshots and Spark's
DESCRIBE all read total-records, total-data-files and total-files-size
verbatim, and a table lost them the moment compaction touched it.
Accumulate the files each operation adds and removes, and render them the
way the spec defines: the added-*/removed-* counters from the files
themselves, then the totals carried over from the parent snapshot.
Carry a total only when the parent recorded it. Iceberg treats a missing
total as zero, which turns a compaction replacing two files with one into a
negative total-data-files, or a table with millions of rows into
total-records: 0. Leaving the field out lets a reader fall back to the
manifests instead of believing a made-up number.
Compaction also accounts for the delete files it consumes, so a run that
folds every delete into the rewritten data reports them as removed.
The worker assumed every file of a table sits under its catalog path, so
loadFileByIcebergPath stripped the scheme off a recorded location and joined
the remainder onto /buckets/<bucket>/<ns>/<table>. A table the REST catalog
placed elsewhere in the bucket — which is what a client gets whenever the
catalog path is already occupied — then resolves to a doubled path:
lookup /buckets/lake/source/t/lake/source/t-0cd81bca-.../metadata/snap-.avro
so the very first manifest list read fails and the job fails again on every
scan interval, indefinitely.
Resolve absolute references (s3:// URIs and /buckets paths) from the bucket
root and keep relative ones under the table's own directory; the
bucket-relative form is now the canonical key everywhere references are
compared. That directory comes from the metadata location the catalog stores,
so reads, writes and deletes all land where the table's other files are
instead of splitting it across two trees. References outside the table's
bucket are rejected rather than silently misresolved.
Rewritten position-delete files now name their data file by absolute URI,
the way the table itself names it, instead of a path relative to the table.
The dashboard charted one summed disk-usage line, so a step in the total
gave no hint which cluster moved. A new panel stacks each cluster's daily
size as its own band: the top of the stack is the fleet total, each band
is one cluster, and the clusters past the twentieth are summed into an
"other" band so the stack still adds up to the total.
The series is built from the per-cluster daily histories and served by
/api/cluster-sizes. Clusters report roughly once a day at no fixed hour,
so a day with no report carries the previous value forward — dropping it
to zero would sag the total every day as the clusters that have not
reported yet fall out from under it. A cluster that stops reporting past
the active window ends at its last sample instead of holding capacity
forever. Ranking is by the most recent day, tie-broken on cluster id so
the colors do not shuffle between refreshes.
Hover and click resolve to the band under the pointer: Chart.js's builtin
interaction modes match the nearest line, which on a stack of thin bands
is rarely the band being pointed at. Clicking one fills the per-cluster
history lookup below it.
LiveMoveVolume and the copy, tail, delete, mark, replicate, and
configure helpers around it issued every RPC on context.Background(), so
a caller had no way to bound or abort a move once it started. They now
take a context, which the exported LiveMoveVolume in particular needs:
callers outside the shell drive long moves and want to stop them.
The deferred restore in copyVolume runs on a detached, bounded context
rather than the caller's. Marking the source writable again is cleanup,
and cancelling the copy must not skip it and leave the volume readonly —
the same guard balance_task.go already applies for the same reason.
Shell commands pass context.Background(): their Do signature carries no
context, and changing it would touch every command in the package.
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
* filer: stamp a log position on lookup and remote-cache responses
Metadata events are logged after their store write and stamped with the
filer clock. Reading that clock before serving an entry therefore gives
a timestamp with a causal guarantee: every event at or below it is
reflected in the returned entry. Clients caching filer state can use it
as the entry's version to order the response against subscription
events, including events committed before the call but delivered after
it.
* mount: version open file handles by filer log position
A subscription event refreshing an open handle did a second lookup; a
transient failure left the handle pinned to its old entry with no
retry, since the subscription cursor had already advanced. The deeper
problem is ordering: the handle is a cache written by three unordered
channels — the async invalidation worker, local mutation acks, and
open-time lookups — and overwriting cached state safely requires
knowing which write is newer.
The filer log timestamp is that order, and it now travels with every
value instead of being derived out of band. Events carry it natively;
lookup and remote-cache responses carry the log position stamped before
the serving read; mutation acks carry it in their returned event; and
the local store pairs each read with a version cursor advanced under
the same lock as the store write. Each handle records the version its
entry reflects, and one rule replaces the per-site reasoning: state at
or below the handle's version is old news and must not be installed.
The invalidation itself applies the event's own entry — no lookup, so
no transient-failure window — except under a cached parent, where the
store entry is the ordered merge of the event and anything applied
since, and its version outranks the event's. An uncached parent
receives no store writes, so a hit there would be a stale leftover
masking the event. A vacated path (delete, rename away) keeps the last
entry so unlinked-but-open reads still work. Directory builds version
the completed directory at the listing snapshot and re-invalidate
buffered events at that version, since their mid-build refresh ran
against an incomplete store.
The tests replay every race this replaces machinery for: rollback of a
newer local flush (queued, cached, and read-through), stale leftovers
under uncached parents, the build window including abort, handles
opened after an event was queued, events landing mid-lookup, and
undelivered events at remote-cache time across a filer failover.
* filer: serialize the log position fence with mutations, stamp mutation acks
The fence stamped before an unlocked entry read could precede state the
read returned: a mutation writes storage first and assigns its event
timestamp only at notify time, so a lookup racing that window handed
the mount an entry newer than its fence, and the event's later delivery
looked like fresh news — destroying dirty pages for a change the handle
already had. The mutation handlers already hold an exclusive per-path
lock across read, write, and notify; the lookup and remote-cache reads
now take it shared around the stamp and the read, making the fence
exact: everything at or below it is in the entry, nothing above it is.
A no-change update returns success without an event, leaving the mount
nothing to fence with even though the response confirms current state.
Create and update acks now carry a log position stamped under the same
lock, and the mount falls back to it whenever the ack has no event.
Also regenerate the VT marshalers, which the earlier generation missed:
without them a VT round-trip silently zeroed every log position.
* java: sync filer.proto
* mount: scope store versions to what they vouch for; atomic handle install
The store's version cursor claimed too much. Advanced by local mutation
acks and directory listing snapshots, it inflated the version of store
reads for unrelated paths whose events the subscription still owed, and
those events were then fenced out permanently. The cursor now tracks
subscription progress only — events arrive in log order, so everything
at or below it has been delivered for every path — and a completed
listing records its snapshot as a per-directory floor instead of a
global claim. Local acks never touch it: they version their own handle
directly. Buffered build events advance the cursor at delivery, since
their store write may never happen (abort) while their invalidation is
already queued; their read-through directory pairs no store read with
it, and rename fragments are applied first.
Concurrent first opens raced: a slower opener's older lookup could
overwrite the newer entry a faster opener had installed, while the
monotonic version kept the newer timestamp — an old entry fenced at a
new version, immune to every correcting event. Entry and version are
now installed as one decision under the handle map lock, and an install
that does not outrank the handle's version is dropped.
The remote-cache commit also escaped the fence: it wrote storage and
notified without the path lock, so a lookup's shared-locked fence and
read could land between the two and hand out the cached state
under-versioned. The commit now re-reads and writes under the exclusive
path lock, and backs off entirely when the entry changed during the
download — the concurrent writer supersedes the cached content.
* mount: floors gate store applies; installs respect handle users; renames join the fence
A directory floor certifies the listing state as of its snapshot, but a
delayed event at or below the floor was still applied to the store —
rolling the content back to pre-snapshot state while the floor kept
claiming the snapshot version, so the correcting events were fenced out
of every future read. Events are now gated against the affected
directory's floor, each half of a rename independently.
Fences are lower bounds: a listing or lookup can include a mutation
whose event has not been delivered yet, and that event later passes
every gate carrying state the handle already holds. Such a re-delivery
now advances the version without destroying dirty pages or reinstalling
the entry — invalidating local writes over a no-op was the real damage
in every remaining under-fence window, including the unlocked listing
snapshot, which no per-path lock can serialize.
The concurrent-open install moved from the map lock to the handle lock
every reader, writer, and invalidation synchronizes on, and rejects
what cannot improve the handle: dirty state (local writes would be
lost), unversioned lookup responses (they cannot outrank anything, and
two zero-version opens must not overwrite each other), and anything not
strictly newer. New handles are still fully initialized before the map
exposes them.
Renames committed metadata and emitted events with no path lock, so a
lookup could read the renamed state under a fence preceding its events.
Both rename handlers now hold the source and destination locks, ordered
by path, across commit and notification; descendants of a renamed
directory are not individually locked and rely on the no-op re-delivery
handling above.
* mount: per-entry store versions replace the cursor and directory floors
The store's aggregate versions — a global subscription cursor and
per-directory listing floors — were versions at coarser granularity
than the values they described, and every over-claiming bug in this
series traced to that gap: an aggregate vouching for state its source
never saw. Each store entry now carries the filer log position of the
write that produced it — the event that applied it, or the listing
snapshot that inserted it, recorded in the store's key-value space
under the same lock as the entry write. The store becomes what the
handle already is: a last-writer-wins register with one rule, install
only what outranks the current claim.
The cursor, the floors, their advancement rules, the pairing ordering
constraint, and the floor gating all collapse into that rule. Applies
are gated per entry, each half of a rename independently; an
unversioned local write clears the claim its content no longer proves;
version records lingering after a bulk folder wipe cannot fence a
recreate, since a claim only blocks while its entry exists. Listing
inserts are stamped at build completion, before the buffered replay so
newer replayed events override the stamp.
Filer side, the fence dance every versioned read must perform is now a
single choke point, fencedFindEntry, so a future read RPC gets the
lock-serialized stamp by construction rather than by convention.
* mount: judge no-op re-deliveries against an immutable base, not the live entry
The equal-state skip compared the incoming event to the live handle
entry, but local writes mutate the live entry — size, timestamps,
chunks — so a delayed event re-delivering the base the handle was
opened with no longer matched, and the installer destroyed the dirty
pages and rolled the entry back over nothing new. The handle now keeps
an immutable snapshot of the filer state it last installed or
acknowledged, refreshed at every install and mutation ack (flush acks
snapshot the request entry before the id mapping mutates it), and the
no-op judgment runs against that base: an event carrying the base
brings nothing, whatever the live entry has diverged to since.
* mount: tombstones for versioned deletes, absence floors, copy enrollment
Four gaps in the per-entry version protocol, all the same shape: a
versioned fact with nothing carrying its version.
A deletion is a fact about a path with no entry left to hold it —
clearing the record let a delayed older event resurrect the deleted
path, permanently, since the deletion's own redelivery is
dedup-suppressed. Versioned deletes now leave a tombstone record that
fences without an entry; renames tombstone their source the same way.
Plain records still only block while their entry exists, so records
lingering after a bulk folder wipe cannot fence a recreate.
A completed listing proves absences as well as presences: a name it
omitted was deleted as of the snapshot, and a delayed create below the
snapshot re-creates it. The snapshot is kept per directory strictly as
an absence fence, consulted only when a path has neither an entry nor
a version record — present entries carry their own versions and never
touch it, which is what separates this from the over-claiming floor it
replaces.
A rebuild against a pre-upgrade filer returns no snapshot; stamping
now clears the children's records in that case, so a reinserted entry
cannot reactivate the stale claim its previous incarnation left
behind and reject valid events below it.
Server-side copies installed the copied entry without enrolling in the
base protocol, so the copy's own event differed from the stale
pre-copy base and destroyed writes made to the destination after the
copy. The install now refreshes the base and takes its version from
the fenced readback.
* mount: deletion facts outlive the cache's knowledge of the entry
A versioned delete of a path the store held no entry for recorded
nothing, so a delayed older event recreated the path — permanently,
with the deletion's redelivery dedup-suppressed. The tombstone is now
written whenever a versioned event vacates a path: the deletion is a
fact about the path, not about what this cache happened to hold.
For an absent entry, the listing's absence floor now speaks whatever
older record remains: a tombstone at one position does not exhaust
what is known about the path when a newer snapshot has confirmed the
name still absent, and an event between the two was slipping past
both.
A committed copy whose readback failed installed a synthesized base
with local timestamps; the copy's real event legitimately differs from
it, and was read as foreign state — destroying writes made to the
destination after the copy. The handle now marks that its own event is
en route and adopts that event's state as the base without touching
the live entry or the dirty pages; the adoption is one-shot, so a
genuinely foreign event still invalidates.
* mount: authoritative acks cancel pending event adoption; tombstones scoped and pruned
The copy-event adoption flag could outlive its purpose: a flush after
the failed readback installs a newer base and advances the version, the
copy's own event is then version gated without consuming the flag, and
the next genuinely foreign event was silently adopted — base advanced,
live entry and dirty pages untouched — leaving the mount to later
overwrite that remote change. Every local acknowledgment now installs
its base through one helper that also cancels any pending adoption: the
ack supersedes the mutation the adoption was waiting for.
Tombstones were written for every versioned delete under the mount and
survived directory eviction by design, growing LevelDB with historical
deletions on delete-heavy mounts. They are now scoped to directories
whose cached state the fence actually protects — an uncached parent
never serves from the store nor applies the resurrecting insert — and a
completed listing prunes the direct-child tombstones its absence floor
supersedes, leaving only those above the snapshot. The store gains a
key-prefix visitor for the sweep.
* mount: acked saves install their value; trailer snapshots; direct-child prune range
A version must never advance without its value. saveEntry stamped any
open handle with the acknowledgment's version, but a handle opened
while the save was in flight holds the pre-mutation entry — stamping it
fenced out the events carrying the state it lacked, permanently, with
the local apply performing no invalidation and the redelivery
deduplicated. The acknowledged entry is now installed together with its
version, through the same guarded install the racing-open path uses:
under the handle lock, only when it outranks the handle, never over
dirty local writes.
Empty listings return no in-band snapshot — a snapshot-only response
would be read as an entry by older consumers — so directories that end
empty gained no absence floor and their tombstones were never pruned.
The filer now sends the snapshot in the stream trailer, which older
clients ignore, and the client reads it when no in-band snapshot
arrived. Empty directories get real floors, their tombstones prune,
and their buffered replays gain the snapshot filter instead of the
replay-all fallback.
Version records now encode the parent directory and name separated by
a NUL, making a directory's direct children one contiguous key range:
the tombstone prune scans exactly them under the cache lock, instead
of walking every descendant record — the whole store, for root.
* mount: fix dirty-page loss, uid/gid base, download race, copy adopt, leak; dedup
Correctness fixes from the versioned-invalidation review:
- A foreign delete/rename-away of a file held open with unflushed local
writes destroyed the dirty pages unconditionally. A process may keep
writing to an unlinked-but-open file and those writes were already
acknowledged; preserve the pages when the handle is dirty.
- downloadRemoteEntry stored the handle's base with filer-side uid/gid
while every candidate it is later compared against is in local form,
so under a non-identity UidGidMapper an unchanged re-delivery looked
foreign and force-destroyed dirty pages. Map the base to local.
- downloadRemoteEntry wrote the entry/base/version triple under only the
handle's shared lock, so two concurrent reads of the same remote-only
file could tear it. Serialize the install with a dedicated mutex
(invalidation is already excluded by the exclusive handle lock).
- A committed server-side copy whose readback failed adopted the FIRST
event past the version gate as its base; a foreign write delivered
first was silently swallowed. Adopt only an event whose content
matches the synthesized base — the copy's own event — and install any
other normally.
- The deferred-create path relied on AcquireFileHandle installing the
passed entry on a pre-existing handle, which the version rework
dropped. Restore that install in the compat wrapper; the versioned
open path keeps its gated install.
Growth and hot-path cost:
- Per-entry version records and tombstones leaked when a directory was
evicted or read-through without a rebuild. An uncached directory
gates its own inserts, so its records fence nothing; clear a
directory's child version records when it is wiped for eviction.
- FindEntry paid for the version KvGet on every lookup/getattr cache hit
and threw it away. FindEntry now reads only the entry; the hot
lookupEntry cache-hit path skips the version entirely.
Cleanups:
- Extract ackVersionTsNs over the shared response interface, replacing
the metadata-event-else-log-ts snippet copy-pasted at four ack sites.
- Extract acquireRenamePathLocks, replacing the verbatim sorted
two-path lock fence in both rename handlers.
* mount: no resurrection on foreign delete, version no-event acks, gate downloads, tighten copy adopt
Follow-ups to the review patches:
- Preserving dirty pages on a foreign delete let the next flush pass the
isDeleted guard and CreateEntry, resurrecting the remotely-unlinked
name. Mark the handle deleted in the vacate branch: the open fd can
still read its buffered writes, but a flush no longer recreates the
file.
- A no-event acknowledgment (log fence only) synthesized a metadata
event with TsNs 0, so the cache stored the entry unversioned and an
older subscriber event rolled it back. Stamp the synthesized event
with the ack's log position at all four ack sites.
- downloadRemoteEntry serialized its install but did not check the
version, so an older response arriving last overwrote the entry/base
while the monotonic version kept the newer value, fencing corrections
out. Install only when the response is at least as new as the handle.
- sameEntryContent compared only size and chunks, so a foreign chmod
with unchanged content was adopted as the copy's own event. Compare
everything except server-assigned timestamps, so a metadata-only
foreign change installs instead.
* mount: trim comments to the non-obvious why
The versioning work accumulated multi-line comment blocks restating what
the code says. Keep the constraint a reader cannot derive — why a fence
is exact, why a version must not advance without its value, why an
uncached parent's records fence nothing — and drop the rest.
* mount: distinguish rename from delete, tighten the download and adopt gates
- A rename emits a nil old-path invalidation just like an unlink, so the
vacate branch marked the handle deleted and later writes through the
already-open descriptor were skipped instead of persisted. Carry the
delete/rename distinction on the invalidation and mark only an actual
delete.
- The remote-download install accepted an unversioned response
regardless of the handle's version, so during a rolling upgrade a
delayed response could install stale content under a newer version.
Require the response to be at least as new, with one exception: a
handle still lacking local chunks takes the content anyway — it cannot
read without it — but does not claim the response's log position.
- Copy-event adoption returned without installing, so a foreign touch
arriving before the copy's own event lost its timestamps. Content is
unchanged either way, so the dirty pages stay valid; a clean handle now
takes the entry, while a dirty one keeps its diverged version.
* mount: one directory floor instead of a record per child; agree on TTL
Review feedback:
- Build completion wrote one KV record per direct child inside the cache
write lock, so a large directory stalled every other cache operation
for O(children) store writes. The directory's listing snapshot already
covers every child it saw; make that floor the version for any child
without a record of its own, and a child earns a record only when a
later event touches it. One map write per build replaces the per-child
writes, with the same fencing.
- The presence probe read the store directly and so counted a
TTL-expired entry as present, judging the path by a record describing
content that has logically vanished. It now applies the same expiry
the read path does, and an expired path falls back to its directory
floor.
- Preserve ErrNotFound identity when the commit-time re-read finds the
object deleted, so callers still surface a 404.
- Assert the rename-away source fence timestamp in the invalidation test.
Also record the tombstone ceiling: distinct deleted names in a cached
directory accumulate until it is rebuilt or evicted, which prunes
everything at or below the new snapshot.
* mount: pin the fence's clock domain instead of letting skew decide
A log-position fence is stamped by one filer's clock under that filer's
in-process lock, so comparing it to an event another filer logged is
comparing two unrelated clocks. The two error directions are not equally
costly: applying an event the fence already covered is a re-apply the
base-equality check absorbs, while skipping one it does not cover leaves
the handle holding exactly the state the event was meant to correct,
with the subscription cursor already past it — the unhealable staleness
this whole PR exists to remove.
So refuse to guess. Fences now carry the signature of the filer that
stamped them, and a handle records it alongside the position. An event
is only fenced out when the filer that logged it is the one that stamped
the fence — the logging filer appends its own signature, so its presence
identifies the clock domain. Events from any other filer are applied.
Positions taken from events keep comparing as before; the subscription
already delivers those in order.
The invalidation callback takes a struct now: it carries the path,
entry, position, delete/rename distinction, and signatures, and was
about to need a fifth positional parameter.
* mount: follow a foreign rename; key page invalidation on content, not equality
- A rename's old-path invalidation now carries the destination, and the
handle follows the file there: an open fd tracks the inode, and leaving
it on the old path made its next flush recreate that name instead of
updating the renamed file.
- Dirty pages overlay content, so only a content change invalidates them.
Keying that on exact equality meant any timestamp-only event destroyed
them, which the copy-adoption marker existed to paper over — a foreign
touch could consume the marker and leave the copy's own event to drop
the post-copy writes. Comparing content instead makes the marker
unnecessary, so it is gone: a metadata-only event keeps the overlay,
and a dirty handle keeps its diverged entry unless foreign content
supersedes it.
- A remote download response that is merely older is now refused even
when the handle still lacks chunks; only an unversioned one is taken
(and claims no position), since an older response's content predates
what the handle reflects.
- A refused or unversioned download no longer publishes to the metadata
cache, where a zero-position event would clear the entry's version and
let an older subscriber event roll the cache back.
* mount: page invalidation keys on content alone; unversioned writes claim no position
- sameEntryContent compared everything but timestamps, so a foreign
chmod, chown, or xattr change counted as a content change and
destroyed the dirty-page overlay. It was strict only to serve the
copy-adoption marker, which is gone; its one caller now asks the
question it actually needs — did the bytes change — so metadata-only
events leave the overlay alone.
- A rename over an existing file destroys that file, but its open handle
was left live and still pointed at the name the renamed source now
occupies, so its flush could overwrite it. MovePath already reports the
displaced inode; mark that handle deleted.
- An acknowledgment was refused whenever its position was numerically
lower, even when a different filer stamped the fence it lost to. Two
known, differing signatures mean unrelated clocks, so the comparison no
longer applies there; unknown signatures still compare as before.
- A local write with no log position behind it now records that
explicitly instead of deleting its version record. Absence means the
directory listing covers the path, which is why the snapshot floor
applies; local content the listing never saw must not inherit it, or
the events that would correct it are fenced out.
* mount: widen the existing lookup functions instead of forking WithVersion twins
The versioning work grew a parallel function for every accessor that
needed to return a log position — lookupEntryWithVersion beside
lookupEntry, maybeLoadEntryWithVersion beside maybeLoadEntry,
FindEntryWithVersion beside FindEntry, AcquireFileHandleWithVersion
beside AcquireFileHandle, advanceEntryVersion beside
advanceEntryVersionTsNs, plus a getPbEntryWithVersion wrapper and an
InsertListedEntriesForTest hook. Two names for one operation is two
places to keep in step, and the split let callers pick the one that
happened to compile.
Each pair is now the single original name carrying the position, with
callers that do not want it discarding it. filer_pb.GetEntry returns the
fence its response already carried rather than a mount-side wrapper
re-issuing the lookup, and InsertEntry takes the position its content
reflects rather than a test-only twin that inserted without one.
The one behavioural knot the merge exposed: AcquireFileHandle had been
installing the entry on a pre-existing handle only in its unversioned
form, which conflated 'the caller is authoritative' with 'the lookup had
no version'. Deferred create is the only caller that means the former,
so it now installs explicitly and the map function just acquires.
On renewal failure the renew goroutine stored isLocked=false before its
deferred renewGoroutineRunning.Store(false) ran. A concurrent RequestLock
interleaving there reacquires the lease (sees isLocked=false), sets
isLocked=true, then its CompareAndSwap on renewGoroutineRunning fails because
the old goroutine's flag is still set — so no replacement renewer starts. The
lock is then held locally with nothing renewing it, and silently expires on
the master after the lease TTL, admitting a second holder.
Clear renewGoroutineRunning before isLocked on the failure path so the
reacquire path always starts a fresh renewer. Builds and vets clean; no
behavior change on the success path.
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
The copying and tagging tests force-drop each bucket's collection at the
master so volume slots are freed deterministically between tests. But the
copy-tests CI job runs its master on 9336 and the tagging Makefile on 9338,
while the tests default to 9333 — the cleanup dialed a dead port and quietly
no-oped. Each test bucket then grows 7 volumes against -volume.max=100, and
whenever async deletion lagged the data node ran out of slots and PutObject
500ed with "No writable volumes and no free volumes left".
Set MASTER_ENDPOINT where the master port is non-default: the copy-tests
workflow step, and the copying/tagging Makefiles (derived from MASTER_PORT).
* telemetry: validate reports on the collect endpoint
/api/collect is anonymous, so reports can't be authenticated, but a
real master can't produce a non-UUID topology_id, a version outside
N.NN(-enterprise), an unknown GOOS/GOARCH, or absurd counts — reject
those to keep casual junk out of the collected data, and cap the
request body at 4 KB.
* telemetry: integration test fixtures pass collect validation
The test's topology id and version were exactly the junk shapes the
new validation rejects; use a UUID and a plain version number.
telemetry: confirmed-cluster stats
Count a cluster as confirmed once it has reported on >=2 distinct UTC
days (per-cluster history makes this a length check). Version/OS
distributions in /api/stats are computed over confirmed clusters, so a
one-shot injected report can't appear in them; falls back to all active
clusters while no confirmed ones exist (fresh server). Adds the
seaweedfs_telemetry_confirmed_clusters gauge and a dashboard card.
telemetry: per-cluster usage history
Keep one compact sample per cluster per UTC day (disk bytes, volume
count, volume servers), retained for -max-age and persisted in the
state file. Serve it at /api/history?cluster_id=...&days=90 and add a
per-cluster lookup with disk/volume charts to the built-in dashboard.
* telemetry: persist server state across restarts
The telemetry server kept the instance map and Prometheus gauges only
in process memory, so every deploy or restart reset all collected
metrics until clusters re-reported over the next 24h.
Snapshot the instance map to a JSON state file (atomic tmp+rename) on
a debounced interval and on SIGTERM, and restore it on startup,
preserving received_at so the cleanup and active-cluster windows stay
correct. Defaults to data/telemetry-state.json, which the deployed
systemd unit's WorkingDirectory already provides; -state-file=''
disables.
* telemetry: keep instances for 90 days by default
With state now persisted across restarts, a longer retention default is
meaningful; raise -max-age from 30 to 90 days so per-cluster data
survives long enough for quarterly views.
ci: build telemetry server from its own module in deploy workflow
telemetry/server has had its own go.mod since #9924, so building
./telemetry/server/main.go from the repo root fails with 'no required
module provides package'. Build from within the module instead.
telemetry: key value gauges by cluster_id only
The value gauges were labeled {cluster_id, version, os}, so a cluster
reporting back after an upgrade started a new series while the old one
kept its last value forever: sum() double-counted every upgraded
cluster, and per-cluster history broke at every version change.
Key the five value gauges by cluster_id alone so each cluster keeps one
continuous series across upgrades; version/os metadata stays on
cluster_info (deleted and re-set on change), available to value queries
via 'on(cluster_id) group_left' joins. Update README accordingly.
telemetry: aggregate /api/metrics per day so dashboard charts render
The dashboard expects {dates, server_counts, disk_usage} as parallel
arrays, but GetMetrics returned per-instance {date,value} lists under
different keys, so the two over-time charts never rendered.
* Add GitHub Actions workflow for codespell on master
* Add rudimentary codespell config
* Tune codespell config: skip generated code, ignore camelCase, whitelist domain terms
Add camelCase/PascalCase regex to ignore common Go/Rust/JS identifiers
like allLocations, publishErr, ReadInside, FlushInterval. Also skip
templ-generated *_templ.go files, and whitelist a handful of
short/domain-specific words (visibles, fo, te, ser, bject, unparseable,
keep-alives, tread, anc, ue) that show up as false positives across the
tree.
Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Fix ambiguous typos and protect false positives
Fixes typos that codespell reports with multiple candidate suggestions
(so `codespell -w` cannot auto-apply them), plus one inline pragma and
one config entry to protect legitimate identifiers.
Manual fixes (single correct answer chosen from context):
- pattens -> patterns (5x) in filer/upload/shell flag help strings
- finded -> found (2x) in tarantool storage.lua comment
- spacify -> specify (2x) in helm chart values.yaml comment
- wether -> whether in skiplist.go docstring
- simpe -> simple in mq schema test case name
False-positive protection:
- Add `//codespell:ignore` next to `source GET's` (possessive of HTTP
verb) in s3api_object_handlers_copy_stream.go
- Whitelist `auther` in .codespellrc — it's a local variable meaning
"authenticator" in weed/security/tls.go, not a typo of "author".
Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Extend codespell ignore list: .git-meta path and thirdparty groupId
Also skip `.git-meta` (scratch dir for commit messages that may contain
typo words verbatim) and whitelist `thirdparty` — it appears as the
literal Maven groupId `org.apache.hadoop.thirdparty` in hdfs3 poms
and cannot be renamed.
Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* [DATALAD RUNCMD] Fix non-ambiguous typos with codespell -w
Auto-applied fixes to the 44 remaining single-suggestion typos across
docs, comments, log messages, tests, config, and one Java pom.
=== Do not change lines below ===
{
"chain": [],
"cmd": "uvx codespell -w",
"exit": 0,
"extra_inputs": [],
"inputs": [],
"outputs": [],
"pwd": "."
}
^^^ Do not change lines above ^^^
* Revert breaking codespell fixes; whitelist unknwon and atleast
Two of the auto-applied `codespell -w` fixes were false positives that
would break the build/tests:
- go.mod: `github.com/unknwon/goconfig` is a real Go module path — the
upstream author's GitHub handle is literally `unknwon`. Renaming to
`unknown` would fail dependency resolution.
- test/benchmark/fuse_db/bin/{sqlite_verify.py,run_mysql.sh,run_sqlite.sh}:
`atleast` is a literal CLI mode value (a string constant compared and
passed as a positional argument). Rewriting to `at least` splits it
into two arguments and breaks the mode check.
Reverted those files and whitelisted both words in .codespellrc so
future runs won't re-suggest the same broken fixes.
Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(admin): implement ApplyPluginConfigFromToml to propagate settings to plugin config store
* Update weed/admin/dash/config_toml.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* admin: overlay admin.toml onto plugin configs through bootstrap defaults
Creating a config from scratch at startup skipped the descriptor-defaults
bootstrap, so a job type with only worker keys in admin.toml persisted
Enabled=false and RetryLimit=0 and silently stopped running. Overlay
existing configs at startup, and apply the same overlay in
enrichConfigDefaults when the plugin bootstraps a fresh config from
descriptor defaults.
Also place collection_filter in the admin values where workers read it,
map preferred_tags as a string list, and stamp UpdatedAt.
* admin: trim the admin.toml help text and call-site comment
* admin: clamp toml retry values to the int32 range
* admin: fail startup when admin.toml cannot reach the plugin config
The legacy overlay already aborts startup when declared settings cannot
persist; continuing here would let workers bootstrap with stale values.
* admin: fix the retry clamp test on 32-bit
A 32-bit int cannot hold the oversized toml value, so viper returns 0
before the clamp runs.
---------
Co-authored-by: baracudaz <baracudaz@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* volume.fix.replication: add a well-placed replica before deleting a misplaced one
A misplaced volume with no surplus replica (replica count == copy count) used to have its misplaced replica deleted first, and the replacement copied only on the next pass. That drops the volume below its intended durability, and permanently so when no destination can accept the replacement copy. Now such volumes get a well-placed replica first, and the misplaced one is trimmed as surplus on a later pass, following the same fulfill-before-delete principle as volume.tier.move (#8950).
The classification switch is extracted into classifyReplicaSet so the add-wins-over-trim priority is unit-testable.
Also stop the fix loop when an -apply pass makes no progress (nothing copied, nothing deleted); previously an unplaceable under-replicated volume made the loop spin forever, re-collecting the topology every 15 seconds. And propagate ewg.Wait() errors instead of swallowing them.
* volume.fix.replication: drop unused allLocations from deleteOneVolume
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* helm: generate the SFTP host key per install
The SFTP secret template shipped one fixed ed25519 host key, so every
install that did not override it presented the same host identity.
Generate the key at install time instead, following the
getOrGeneratePassword pattern: an existing secret keeps its key across
upgrades, except the previously bundled one, which is replaced with a
freshly generated key on the next upgrade.
* helm: create the SFTP host-keys secret the deployments mount
Both the sftp and all-in-one deployments mount /etc/sw/ssh from
<fullname>-sftp-ssh-secret, but no template created it, so a default
install could not start its pod and host keys only reached the server
when enableAuth happened to mount them elsewhere. Create the secret
with a generated ed25519 key, keeping whatever keys an existing secret
already holds. The sshPrivateKey default becomes empty: the file it
pointed at only exists when enableAuth mounts /etc/sw, and a configured
but missing key file is fatal to the server, while hostKeysFolder now
always has a key.
* helm: test SFTP host key generation and secret lifecycle
Template checks: keys render into the secret the deployments mount,
parse as PKCS#8 ed25519, differ between installs, and the render
carries no key material from the chart itself; existingSshConfigSecret
and all-in-one wiring covered. On the kind cluster, exercise the
secret lifecycle: a generated key survives upgrades, the key earlier
chart versions bundled is replaced, and operator-managed keys are kept
untouched. chart-testing now also installs with sftp enabled, where
the pod only becomes ready if the server loads the generated host
key.
* helm: treat a whitespace-only stored SFTP host key as missing
A whitespace-only secret value skipped regeneration and then rendered
an empty key file.
* helm: mount the SFTP host keys secret at the configured hostKeysFolder
The secret was mounted at a fixed /etc/sw/ssh, so a custom
sftp.hostKeysFolder pointed the server at an empty directory. Mount at
the configured path in both the sftp and all-in-one deployments, and
pin flag/mount agreement in the rendering tests.
* s3: track manifest blob ownership through multipart completion
Manifest blobs made three orphan paths. A partial fold that failed midway
kept its earlier batches on volume servers while the write fell back to
flat chunks; the fold now records each saved blob and deletes them on
error. A completion that failed after preparing left its fresh manifests
behind on every retry; the completion state now owns them and deletes
them unless a failed rollback left the version entry still holding them.
And a completed upload removes its parts metadata-only, which stranded
the part-manifest blobs superseded by flattening; those are collected
during flattening and deleted once the completion commits.
Two shared-chunk hazards nearby: the version-file rollback deleted its
data, destroying the still-registered parts (worse once manifests
resolve to inner chunks), and the idempotent-replay cleanup data-deleted
leftover parts whose chunks the live object references. Both are
metadata-only now.
* s3: trim chunk manifest comments
* s3: test manifest fold rollback and part range selection
The fold-with-rollback and the boundary-to-byte-range logic were only
exercised by hand against a live server; give both an injectable seam
and cover the fold, the below-threshold and SSE no-ops, the midway
failure deleting the blobs it saved, and offset-vs-legacy-index range
selection including indexes that no longer address the chunk list.
* erasure_coding: WriteDatFile takes the encode-time dat size for the shard block layout
* volume server: derive EC decode layout from the encode-time dat size, not the live extent
* erasure_coding: test decode after tail deletions shrink the live extent below a large-block row
* seaweed-volume: write_dat_file_from_shards takes the encode-time dat size for the shard block layout
* seaweed-volume: derive EC decode layout from the encode-time dat size, not the live extent
* seaweed-volume: test decode after tail deletions shrink the live extent below a large-block row
* erasure_coding: reject decoding with no data shards
* worker: record the encode-time dat size in the .vif
* erasure_coding: fall back to the shard-derived layout only when the encode-time dat size is missing
* erasure_coding: reject an ambiguous shard-derived block layout
* seaweed-volume: fall back to the shard-derived layout only when the encode-time dat size is missing
* seaweed-volume: reject an ambiguous shard-derived block layout
* filer: accept a WriteCondition on UpdateEntry, under the per-path lock
UpdateEntry was a bare read-modify-write: the precondition check, the
chunk garbage diff, and the store write could interleave with a
concurrent update to the same path. Take the per-path lock CreateEntry
already holds, and evaluate an optional CreateEntry-style WriteCondition
under it, failing with FailedPrecondition like expected_extended.
* filer: IF_CHUNKS_EQUAL write condition compares the stored chunk fid set
A chunk-preserving read-modify-write (tagging, setattr, copy-in-place)
races UpdateEntry's garbage diff: if a concurrent update empties the
chunk list first, the stale writer's commit resurrects fids that are
already queued for deletion, stranding the entry on a dead needle once
vacuum reclaims it. The reverse also holds: a writer that read an empty
chunk list can wipe chunks a concurrent update just added.
IF_CHUNKS_EQUAL guards both: the stored chunk fid multiset must still
equal what the caller read, order-independent, with an empty fids list
expecting no chunks. Absent entry counts as no chunks for CreateEntry
overwrites and transactions.
* filer: delete and append serialize on the entry path lock
DeleteEntry queues the entry's chunks for deletion and AppendToEntry
rewrites the chunk list, but neither held the per-path lock, so either
could interleave with a conditional update between its precondition
check and its write — a passed IF_CHUNKS_EQUAL would then resurrect
fids already on the deletion queue, or clobber a freshly appended
chunk. AppendToEntry keeps the cluster lock for cross-filer append
serialization; the path lock covers the local read-modify-write.
* filer: reuse lockPath in UpdateEntry lookup
s3: fold large chunk lists into manifest chunks on the direct write path
The S3 gateway uploads chunks itself and hands the filer a fully prepared
entry. On the routed write path (ObjectTransaction) the filer stores that
entry as-is, so a large PutObject or CompleteMultipartUpload persisted its
whole flat chunk list - a 900GB object carries 120k chunk references in one
entry. Manifestize on the gateway before the entry is written, the same way
mount, WebDAV, and filer.copy prepare theirs.
Multipart part boundaries also record byte offsets now: the stored chunk
indexes stop matching the entry once the list is folded, and partNumber
reads plus GetObjectAttributes prefer the offsets. Legacy index-only
records still work, with bounds checks instead of a possible panic.
Copy paths resolve a manifested source into data chunks before their
per-chunk copy loops - copying a manifest chunk raw would store its blob
as object data still pointing at the source - and a large copied list is
folded again on the destination. Completion likewise resolves manifest
chunks a part entry may carry (the filer folds an oversized UploadPartCopy
range) before rebasing part offsets.
The maintenance operations built manifest, manifest-list, data-file and
metadata-log paths with a bare path.Join("metadata", ...), so a single
maintenance run wrote scheme-less relative paths into the new snapshot,
the metadata-log and the table xattr. SeaweedFS reads those back fine
because normalizeIcebergPath accepts both forms, but strict readers
resolve every location through S3FileIO and fail with "Invalid S3 URI,
cannot determine scheme", leaving the whole table unreadable after any
maintenance commit.
Add absoluteIcebergPath, the inverse of normalizeIcebergPath, and apply
it at every site that authors locations: rewrite_manifests, compact,
rewrite_position_delete_files, and the shared commit path. The base is
derived from the bucket and table path since that is where the worker
physically writes, matching the locations the REST catalog generates.
The pre-move stale check compared the master's location urls (host:port)
literally against the proposal's node addresses, which carry the grpc
suffix (host:port.grpcPort). Every move failed as "stale move: volume no
longer on source" even though the volume was still there. Normalize both
sides through pb.ServerAddress before comparing.
* regenerate master_grpc.pb.go with protoc-gen-go-grpc v1.6.2
The other generated pb files are already on v1.6.2; this one was stale.
* shell: keep unlock from racing the lease renewal
A renewal RPC in flight while ReleaseLock runs re-creates the lock on the
master after the release deletes it, and can blank the client name if the
renewal reads it mid-release. The stale-token release is then ignored, so
the lock stays held (sometimes anonymously) until it expires. Serialize
the renew and release RPCs, and set the client name before flipping
isLocked so the renewal never sends a partial acquisition.
* shell: restart lease renewal after a failed renewal
The renewal goroutine exits on error but never cleared its running flag,
so later locks in the same process were never renewed and silently
expired after ten seconds.
* shell: show who holds the cluster lock
A blocked lock command gave no hint that another client holds the lock
(the refusals only surfaced at -v=2), and cluster.status reported the
shell's own lock state as if it were the cluster's. Add a
GetAdminLockStatus RPC to the master so lock prints the holder before
blocking and cluster.status shows the actual cluster-wide holder. Both
degrade silently against masters without the RPC.
* shell: bound admin lock RPC attempts with timeouts
The lease, renew, release, and holder-status calls all ran without a
deadline, so an unresponsive master could hang the renewal goroutine,
an unlock (which now waits on the renewal mutex), or the shell prompt.
Give each attempt its own short context; the retry loops still resolve
a fresh leader on the next try.
* master: reject admin token release on non-leaders
A follower holds no lock state, so it answered a release with success
while the leader kept the lock until expiry. Refuse like LeaseAdminToken
does so the client can try the leader instead.
* shell: leave the lock release call unbounded
A release cut short by a deadline leaves the lock held on the master
until it expires, so a slow master would turn every unlock into a
ten-second ghost lock. Restore the single fire-and-forget attempt;
the timeouts stay on the lease and renew paths, where a stalled call
forfeits the lease anyway.
* shell: release only the token unlock started with
A RequestLock racing a slow release (the admin presence lock does this
on shutdown) could have its freshly acquired token sent in the release
request or zeroed by the trailing stores. Capture the token once under
the mutex and compare on clear so a concurrent acquisition survives an
in-flight unlock.
Reads of remote-backed entries now record hit or miss in
SeaweedFS_remote_cache_read_total{source,bucket,result} on the filer HTTP
path and the S3 gateway, so cache effectiveness of mounted buckets can be
graphed. Inline-content entries count as hits since they are served
locally without chunks. The filer purges the per-bucket series when the
bucket directory is deleted, so a standalone filer does not accumulate
series across bucket delete/recreate churn.
Only the heartbeat path read the .dat mtime; collectStatForOneVolume left
the field at 0, so /status consumers could not tell how long a volume had
been idle.
Commit 8bff3b32 changed BatchDelete to keep processing after a cookie
mismatch but left the integration test asserting the old early-break
behavior, breaking Volume Server Integration Tests (grpc - Shard 1) on
master. Align the test with the new semantics and port the same
break->continue to the Rust volume server, which runs the same suite
via VOLUME_SERVER_IMPL=rust.
The initiator's shed check (initiatedGrow != HasGrowRequest) compares
against an err from a PickForWrite that may predate the growth
concluding: the grower registers its volumes before clearing the flag,
so when growth lands between the failed pick and the check, the assign
shed ResourceExhausted even though a writable volume was already
registered. Re-pick once after observing the conclusion and shed only
if the volume layout still has nothing writable. Applies to both the
gRPC Assign and the HTTP dirAssign paths, which share the shed logic.
Flaked in CI as TestAssignInitiatorWaitsForItsOwnGrowth; reproduced
deterministically by widening the enqueue-to-check window.
* fix: report short S3 ReaderAt reads
Problem: S3BackendStorageFile.ReadAt returned a short buffer with a nil error, hiding truncated remote data.
Root cause: Every terminal io.EOF was cleared regardless of how many bytes were read.
Fix: Clear io.EOF only when the requested buffer was completely filled.
Validation: go test ./weed/storage/backend/...
Co-authored-by: Codex <noreply@openai.com>
* fix: validate S3 ReaderAt requests
Co-authored-by: Codex <noreply@openai.com>
* s3 ReadAt: simplify negative offset error
* s3 ReadAt: stop reading once the buffer is full
Skips the extra Read that only collected the terminal EOF, and bounds
the loop if the server ignores the Range header and returns more data
than requested, where Read on an empty slice can spin forever.
* s3 ReadAt: cover full reads that arrive with io.EOF
---------
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix: reject overflowing needle ID deltas
Problem: Parsing a file ID with a delta can wrap a valid maximum needle ID back to zero without returning an error.
Root cause: Needle.ParsePath added the parsed uint64 delta without checking whether the sum exceeded the needle ID range.
Fix: Compare the delta with the remaining uint64 capacity before addition and return a contextual overflow error when it does not fit.
Validation: go test ./weed/storage/needle -run ^TestNeedleParsePathRejectsDeltaOverflow$ -count=1; go test ./weed/storage/needle -count=1; git diff --check 10cdaf381875492a2c752d1038797e96ff18208f..HEAD
Co-authored-by: Codex <noreply@openai.com>
* fix: propagate needle ID delta parse errors
Co-authored-by: Codex <noreply@openai.com>
* print the needle id in hex in the delta overflow error
* batch delete: keep processing after a cookie mismatch
* rust volume: reject overflowing needle id deltas
---------
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
The dev container workflow prebuilds the Rust volume server from
seaweed-volume/ but did not watch that directory, so Rust build fixes
landed without this workflow validating them.
pass --experimental_allow_proto3_optional to protoc in the Rust build
filer.proto now carries a proto3 optional field, which the protoc 3.12
shipped in ubuntu-22.04 apt rejects unless this flag is set. Newer
protoc versions still accept the flag, so it is safe everywhere.
When every volume is full, a chunk upload fails only after the assign
retry budget, the failed chunk's data is dropped, and the mount kept
accepting writes anyway. cp would crawl for hours pushing the rest of
the file through a pipeline that could not persist it, and only close()
reported an error - a generic EIO.
Poison the file handle on the first failed chunk upload so subsequent
writes fail immediately, and map "no writable volumes" / "no free
volumes" upload errors to ENOSPC so the writing process aborts with
"No space left on device". Also guard lastErr with a mutex: it was
written concurrently by uploader goroutines, and keep the first error
so later failures do not mask the root cause.
* remote storage: carry Content-Encoding into mounted entries
A RemoteEntry now records the remote object's Content-Encoding, and every
path that materializes a local entry from remote metadata (lazy fetch, lazy
listing, remote.mount, remote.meta.sync, remote.cache) stamps it into the
entry extended attributes, so HTTP and S3 HeadObject/GetObject return the
header. GCS and Azure populate it on listing and stat; S3 only exposes it
via HeadObject, so listings leave it empty.
* remote storage: set Content-Encoding when uploading to the remote
An entry carrying Content-Encoding in its extended attributes (a native S3
upload, or a value pulled from the remote) now keeps it when
filer.remote.sync or remote.copy.local writes the object to GCS, S3, or
Azure, instead of silently dropping it.
* gcs: read remote objects without decompressive transcoding
GCS transparently decompresses gzip-encoded objects on download, which
ignores range requests and returns byte counts that disagree with the
tracked RemoteSize. Request the stored bytes instead; chunked reads of
gzip-encoded objects then behave like any other object.
* remote storage: track Content-Encoding presence so removals propagate
A listing that does not report encodings (S3) leaves the field unset and
the local header untouched, while an authoritative report of no encoding
(GCS, Azure, any stat) now clears a previously stamped header instead of
leaving it stale. remote.cache also schedules a metadata update when only
the reported encoding changes.
* remote storage: propagate Content-Encoding on metadata-only updates
filer.remote.sync routes same-content changes through UpdateFileMetadata,
which only touched custom metadata (GCS, Azure) or tags (S3), so a
Content-Encoding change in the extended attributes never reached the
remote object's real header. GCS now patches contentEncoding alongside
the metadata, and Azure reissues the blob's HTTP headers with the new
value, carrying the others over since the call replaces the full set.
S3 stays tags-only: changing the header there means rewriting the
object, which the sync already does whenever content changes.
* remote.meta.sync: optional per-file stat for listing-omitted metadata
S3 listings carry no Content-Encoding, so entries synced from them never
learn it and the lazy-stat path never runs once an entry exists. With
-statFiles, each new or changed file whose listing left the encoding
unreported is stat-ed before reconciling, and the stat-derived value is
persisted so the next run only stats files that changed. Off by default:
it costs one remote request per file, and GCS and Azure listings already
carry the encoding.
* s3: apply metadata-only Content-Encoding changes with an in-place copy
Content-Encoding is S3 system metadata, so the tags-only metadata update
silently left the object's real header untouched. When the encoding
differs, reissue the object as a self-copy with replaced metadata,
carrying the content type and configured storage class like a fresh
write does. CopyObject caps at 5 GiB; beyond that the change is logged
and applies on the next content write.
* azure: skip the metadata call when user metadata is unchanged
An encoding-only change reissues the blob's HTTP headers; sending the
unchanged user metadata alongside it wastes a round trip and bumps the
blob's ETag once more than needed.
* s3: carry existing object metadata through the encoding copy
The replace directive drops everything not resent, and a mounted entry
usually has no local mime or user metadata, so the in-place copy wiped
the object's Content-Type, Cache-Control, user metadata, encryption
settings, and storage class. Read them back with a HeadObject first and
carry them over, overriding only what SeaweedFS manages: the encoding,
a locally set mime, and the configured storage class. S3 reports
Expires as a string while the copy input wants a time, so it is parsed
and skipped when malformed.
* generate vtproto marshalers for filer_pb and use them on the metadata log path
Reflection-based proto.Unmarshal allocates a fresh message tree through
reflect.New on every call. On the metadata subscription fan-out the same
event is decoded once per subscriber, so reflect.New tops the decode
churn under many mounts.
Generate MarshalVT/UnmarshalVT/SizeVT for filer.proto (a separate
filer_vtproto.pb.go, filer.pb.go untouched) and call them on the log
entry marshal and the subscribe/replay decode paths. UnmarshalVT
allocates message structs directly and copies byte and string fields, so
it stays wire-compatible with proto.Unmarshal and preserves the
non-aliasing the persisted-log cache depends on.
For SubscribeMetadataResponse this cuts decode allocations 69 -> 50 and
~4.5us -> ~2.1us per event; the win scales with subscriber overlap.
* marshal log entries directly into the buffer
SizeVT is allocation-free and MarshalToSizedBufferVT writes into a
pre-sized slice, so the log entry can be marshaled straight into
logBuffer.buf. This drops the per-entry MarshalVT allocation and the
follow-up copy on the write path.
* expand vtproto benchmarks: marshal, decode, and marshal-into-buffer by chunk count
Parametrize by nested-message count (chunks per event) and add encode +
zero-alloc marshal-into-buffer benchmarks alongside the decode one, so
the write-path win from MarshalToSizedBufferVT is measurable too.
* keep proto.Unmarshal for metadata events to preserve UTF-8 validation
UnmarshalVT skips proto3's UTF-8 validation of string fields, so a
SubscribeMetadataResponse with an invalid-UTF-8 string (e.g. Directory
"\xff") that proto.Unmarshal rejects would decode and reach path
filtering and subscribers. Decode events with proto.Unmarshal again;
UnmarshalVT stays on the log entry paths, whose only variable-length
fields are bytes and so carry no UTF-8 constraint.
Tests cover the codec difference and that a malformed event is skipped
before delivery.
Put/DeleteObjectTagging set the update directory to the bucket root for a
null version, so a tag change on allowed/protected.txt landed on
protected.txt at the bucket root instead. A principal scoped to one
nested key could overwrite a different object sharing the basename, the
same class of scope bypass fixed for PutObjectAcl. Share the object's
parent-directory resolver across both paths.
* route log buffer flush copies through the shared slab pool
Each flush copied the sealed window into a bytes.Buffer drawn from a
package-local sync.Pool. GC drains that pool, so once collections run
often (e.g. under GOMEMLIMIT) most flushes miss and Write grows a fresh
window-sized array, the dominant bytes.growSlice source under write load.
Copy into a size-classed slab from weed/util/mem instead. Slabs are
reused process-wide and returned to their exact size class after the
flush, so variable-sized flushes across many partitions stop churning
mismatched buffers.
* guard nil slab in flush releaseMemory
mem.Free(nil) resolves to the smallest slot pool and stores a zero-cap
slice, so a later mem.Allocate could return nil and panic. The current
call sites never pass nil, but the guard keeps a double release harmless.
The FUSE harness picks the filer HTTP port and lets "weed mount" derive the
filer gRPC port as HTTP+10000. freePort kept the HTTP port under the Linux
ephemeral floor (32768) but not the derived gRPC port, which ranged up to
42000. When the gRPC port landed above the floor an outbound connection could
transiently hold it, so mini relocated its filer gRPC port while mount kept
dialing HTTP+10000, timing the mount out.
Cap the HTTP port at 22000 so the gRPC sibling stays at or below 32000, and
verify both ports are bindable before returning.
* Fix PutObjectAcl writing back to the wrong object for nested keys
PutObjectAcl set the update directory to the bucket root, so an ACL
change on allowed/protected.txt landed on protected.txt at the bucket
root instead. A principal scoped to one nested key could overwrite a
different object sharing the basename. Target the object's own parent
directory.
* Test object ACL update directory resolution for nested keys
* Introduce weed shell command `ec.check.replication`.
This command performs a quick check of EC volume shard replication, reporting
volumes whose shards are over- or under-replicated. Each volume is checked
against its own data+parity ratio, obtained via
erasure_coding.EcShardsVolume{Data,Parity}Shards, so builds that derive the EC
ratio per volume report custom ratios correctly.
The name follows the shell's convention (cluster.check, volume.check.disk); the
closest normal-volume counterpart is volume.fix.replication.
* shell: ec.check.replication reports mixed under+over-replication in both lists
* rust volume: verify the .dat ends at the last indexed needle
The Go loader quarantines a volume whose .dat extends past the last
indexed needle - the leftover of a torn shutdown - but the Rust check
only verified header fields of the trailing index entries and never
compared file sizes, so a torn tail loaded clean and writable. Appends
land at the raw file end, and past a misaligned tail the next needle
sits at an offset the 8-byte-unit .idx encoding rounds down, pointing
the index a few bytes before the needle.
Replace the last-10-entries walk with the current Go shape: find the
entry physically last in the .dat (append-ordered fast path, max-offset
scan for key-sorted rebuilds), verify that needle - tombstones with
their on-disk Size=0 - and require the file to end exactly at it,
marking the volume read-only otherwise.
Claude-Session: https://claude.ai/code/session_01XgGXMLknzaNgQzHMyo2Vhb
* rust volume: buffer the .idx max-offset scan
The slow path read one 16-byte entry per syscall; a BufReader batches
the sequential scan like Go's WalkIndexFile does.
Claude-Session: https://claude.ai/code/session_01XgGXMLknzaNgQzHMyo2Vhb
StatFs on a mount reported cluster-wide total capacity but used size from
a single volume layout keyed by collection, replication, ttl, and disk
type. A mount without -collection therefore showed only the default
collection's usage, hiding data in named collections, and even a
collection-scoped mount missed volumes with a different replication,
ttl, or disk type.
Aggregate used size and file count across all layouts of the requested
collection, and across every collection when the collection is empty,
matching how Topology.Lookup treats an empty collection. Looking up
stats no longer creates a phantom collection as a side effect.
An absent needle is normally indistinguishable from a vacuumed delete,
so check.disk skips it and replicas diverged by replication failures
cannot be reunited. But a replica with compaction revision 0 has never
been vacuumed: every delete it processed still holds its tombstone, so
a needle absent there is provably a missing write.
The new flag resurrects absent needles in exactly that case. The
receiving replica's live compaction revision is read after the index
snapshot and must be 0; vacuumed replicas keep the skip, now with the
revision in the message. The decision is passed per direction since
pass 2 runs pairs concurrently. Resurrected needles count toward
-nonRepairThreshold as before.
* filer: authorize TUS existing-session verbs against the validated stored target
Scope-check on TUS HEAD/PATCH/DELETE only populated a resource path for POST,
so a prefix-restricted token that learned another tenant's session id could act
on that session and land content at a TargetPath its own AllowedPrefixes forbid.
Split the filer JWT check into authenticateFilerJwt (signature and method) and
authorizeFilerJwtPaths (resource scope), and make the scope check fail closed:
a prefix-restricted token with no resolved resource path is denied instead of
authorized on signature alone. The TUS handler now authenticates first, reads
and validates the session once, authorizes the stored TargetPath, then operates
on that single pinned snapshot. readTusSessionInfo rejects a session whose id,
target or size is unusable, and getTusSession is split so the authorization
lookup no longer lists chunks.
* filer: reject non-canonical TUS upload ids
The uploads route took the first path component as the session id, so a trailing
path or other non-canonical spelling aliased one session under several URLs.
Require the id to be a canonical UUID, the only form the server mints, both at
routing and when reading a session's metadata, so one URL maps to one resource.
* filer: revalidate the pinned TUS session before completing an upload
Completion re-read chunks but not the session identity, so a PATCH finishing
after a concurrent DELETE or metadata replacement could still land at the id's
stored path. Before completing, confirm the session still exists and its target,
size and creation time are unchanged from the authorized snapshot; otherwise the
completion fails instead of writing to a path the request never authorized.
* filer: log TUS session lookup failures before returning not-found
readTusSessionInfo and loadTusSessionChunks failures answered "not found" with
no log line, so a transient filer or listing error was indistinguishable from a
genuinely missing session. Log the lookup at V(1) (a missing session is common
and benign) and the chunk-load error at Errorf (the session already resolved).
* volume: copying a remote-backed volume only needs space for the index
VolumeCopy sized its target-location check by the source .dat even when
that .dat lives in a cloud tier and only .idx/.vif land locally, so
re-replicating a tiered volume demanded the full remote size in free
disk. Require the index size instead.
* shell: volume.tier.upload keeps volume replicas
Tiering a replicated volume deleted every replica but the upload
source, leaving one server holding the only .idx and the only .vif
that knows the remote object key — losing that server orphaned the
volume even though its data sat intact in the cloud.
Replicate the uploaded .idx/.vif onto the other replica servers
instead (VolumeCopy skips the .dat for remote-backed volumes), so all
replicas serve reads from the same remote object and the volume keeps
its replica count. An already-tiered replica is preferred as the
upload source, so a rerun after a partial failure reuses the existing
remote object instead of uploading a second copy under a new key.
* shell: group tier upload locations instead of re-prepending
* rust volume: copying a remote-backed volume only needs space for the index
Mirror the Go VolumeCopy change: size the free-location check by the
source .idx when the .dat lives in a cloud tier, since only .idx/.vif
land locally.
* admin: surface lifecycle rule counts in bucket listing
* admin: add bucket lifecycle JSON endpoint
* admin: show lifecycle rules on the buckets page
* admin: make the lifecycle count badge keyboard-accessible
* admin: match buckets empty-state colspan to the column count
* admin: drop stale lifecycle modal responses
* make
* sftp: match permission paths on path-component boundaries
Permission checks compared paths with raw string prefixes, so a
permission entry for /tenants/alice also matched sibling paths such as
/tenants/alice-archive, letting a scoped user read and overwrite
another tenant's files. The home directory containment check had the
same flaw. Route both through pathWithin, which cleans the paths and
requires exact equality or a separator-delimited descendant.
* sftp: clean permission path into a local when ranking matches
* filer: allow clearing a bucket read-only flag stuck after quota removal
Once s3.bucket.quota.enforce marks a bucket read-only, removing or
disabling the quota orphans the flag: enforcement skips buckets with a
non-positive quota, and fs.configure merges booleans with OR so
-readOnly=false could never turn it off. The only way out was deleting
the whole path rule.
- s3.bucket.quota -op=remove/disable and the admin server quota update
now lift the read-only flag on the bucket's path rule
- fs.configure now honors explicitly passed false boolean flags
(-readOnly, -fsync, -worm) instead of OR-merging them away
* filer: ClearBucketReadOnly reports unchanged when the save fails
* fix(rust-volume): remove .ecsum sidecars on EC destroy / shard delete
Rust EcVolume::destroy removed shards and .ecx/.ecj/.vif but left bitrot
checksum sidecars (.ecsum / .ecsum.v*). On clusters that run weed-volume
(not Go weed volume), collection.delete therefore orphans every sidecar
while correctly wiping shards — observed live on 4.39 (14/14 .ecsum
survived after collection.delete on a freshly encoded EC volume).
Go Destroy already calls RemoveBitrotSidecars; this brings Rust to parity:
- hoist remove_bitrot_sidecars into ec_bitrot (shared helper)
- call it from EcVolume::destroy for dir / dir_idx / ecx_actual_dir
- call it from Store::delete_ec_shards when a disk has no remaining shards
- unit test: test_destroy_removes_bitrot_sidecar
* rust volume: gate the shard-delete sidecar sweep on a local shard removal
Only sweep a disk's .ecsum when this delete actually removed a shard file
there, matching Go's found gate: a delete that never touched a disk must not
strip a sidecar it does not own — a shared -dir.idx sibling with surviving
shards, or an ec.rebuild index-prep copy that lands .ecx/.ecsum before any
shard. The shard-presence probe now treats unexpected stat errors as
"exists" so a transient failure cannot orphan-classify live shards, and
check_all_ec_shards_deleted reuses it.
* rust volume: destroy() sidecar sweep needs only the data and idx bases
ecx_actual_dir is always one of the two, so the third branch could never
run; this is now exactly Go Destroy()'s two-base sweep.
* rust volume: call the shared sidecar removal helper directly
* rust volume: unit-test remove_bitrot_sidecars
Mirrors Go's TestRemoveBitrotSidecars: legacy and versioned sidecars are
removed, a shard file and a longer-vid sidecar survive, absent is success.
* rust volume: keep the shared idx-base sidecar while a sibling disk has shards
One -dir.idx serves every location, so emptying one disk must not sweep
<idx>/<vol>.ecsum out from under a sibling that still holds shards. Nothing
reads the idx-base sidecar today, but .ecx shows index-dir files are real;
this keeps the defensive sweep safe if a writer ever lands one there.
* ec shard delete: keep the shared idx-base sidecar while a sibling disk has shards
One -dir.idx serves every disk, so emptying one disk must not sweep
<idx>/<vol>.ecsum out from under a sibling that still holds shards of the
volume — the same gate the Rust volume server applies. A status error counts
as in-use so a transient failure never strips it early.
* rust volume: drop a shard-only disk's stale .vif with the node's last shard
Go's removeEcSharedIndexFiles also clears the data-base .vif in the
all-shards-gone pass, gated on .idx absence so a disk still hosting the
source volume keeps its live .vif; the Rust delete path left it behind.
Unexpected stat errors count as .idx-present so a transient failure never
strips a live volume's .vif.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* filer: scope TUS HEAD/PATCH/DELETE against the session target path
checkTusJwtAuthorization only populated the scoped-path list for POST, so
a prefix-restricted token was scope-checked at session creation but not on
HEAD/PATCH/DELETE, which act on an existing session addressed by id. A
low-privilege tenant holding a valid write token who learns another
upload's session id could PATCH attacker bytes into that session, or
DELETE/HEAD it, landing content at a target path its own AllowedPrefixes
forbids. The session id is unguessable to an unauthenticated attacker but
is not an authorization boundary against a legitimate tenant.
Resolve the effective target for HEAD/PATCH/DELETE from the session's
stored TargetPath and scope the prefix check against it, the same way POST
scopes the create target. The scoped paths are resolved lazily so the
session read runs only for a prefix-restricted token; unrestricted tokens
and deployments without a signing key touch no extra state. An unknown
session stays unscoped so the handler still answers 404.
* filer: fail closed when a TUS session target cannot be resolved
The lazy scope resolver swallowed every readTusSessionInfo error and left
the request unscoped, so a filer read error or a corrupt session .info
would authorize a prefix-restricted token against a target the server
never resolved. Only a genuinely missing session should stay unscoped (so
the handler answers 404); any other failure now propagates and denies the
request. checkJwtAuthorizationScoped's resolver returns an error and a
failed resolution is treated as not-authorized.
* filer: fold the scoped-path resolver into checkJwtAuthorization
checkJwtAuthorization now takes the lazy resolver directly instead of a
separate checkJwtAuthorizationScoped wrapper, and the surrounding comments
are trimmed to the load-bearing why.
* shell: ec.encode batches 10 volumes by default
Encoding a whole collection as one batch means a late failure leaves
everything half-converted. Default -batchSize to 10 so each batch is
encoded, rebalanced, verified, and its originals deleted before the
next starts. -batchSize=0 keeps the all-at-once behavior. Batch
progress messages now print only when there is more than one batch,
so small runs read as before.
* shell: ec.decode decodes in batches, refreshing topology in between
ec.decode walked every volume off the single topology snapshot taken
at startup, which goes stale as earlier decodes move shards around and
create volumes. Decode 10 volumes per batch by default, re-collecting
the topology and rebuilding the free-space accounting between batches.
-batchSize=0 keeps the single-snapshot behavior.
Mounting freshly generated shards notifies the master asynchronously, while the post-encode rebalance plans from a fresh VolumeList snapshot; a snapshot that predates the delta shows zero shards, so the balance plans no moves and silently succeeds, and the original .dat is deleted with all shards on the generation host. Poll until every encoded volume reports a full shard set before balancing — across all disk types, since fresh shards register under the source volume's disk type, and scoped to the newest encode generation so an orphaned older generation neither satisfies the wait nor defeats the clump check. As defense in depth, refuse to delete originals when a volume's shards still sit on one node while another node has free EC slots.
* s3: accept raw semicolons in query strings
Go's url.ParseQuery drops any key=value pair containing a raw ';'. A
presigned PUT that signs content-type carries
X-Amz-SignedHeaders=content-type%3Bhost; when a client or proxy decodes
the %3B, the parameter vanished and the upload failed with
MissingFields, while AWS accepts the raw ';' as query data. Re-encode
it before routing so the pair survives parsing and signature
verification.
* iam, iceberg: recover raw-semicolon query pairs on the other listeners
The standalone IAM API verifies SigV4 with a canonical query recomputed
from the parsed query, and Iceberg REST warehouse/parent values may
legally contain ';'. Move the normalization middleware to util/http and
attach it to both routers.
A raw OIDC token sent as Authorization: Bearer was validated and mapped
to a role through the provider's roleMapping, then authorized against
the role's attached policies without ever consulting the role's trust
policy. A federated user could act as a role that
AssumeRoleWithWebIdentity would refuse to issue a session for with the
same token. Run the same trust-policy validation before returning the
principal on the bearer path.
* fix(filer.backup): stop silently dropping events on transient not-found
Under a write burst, filer.backup could consume a metadata event (advancing
the persisted offset) without replicating the file, with no error logged:
1. filersink CreateEntry/UpdateEntry swallowed replicateChunks errors
(glog.Warningf + return nil), so the offset advanced past entries that
were never written.
2. The manifest-chunk branch of replicateChunks resolved via LookupFileId
with no retry, unlike the data-chunk branch — transient lookup races
dropped exactly the large manifest-backed files while small
inline-content siblings landed.
3. isIgnorable404 matched "LookupFileId" / "volume id ... not found",
misclassifying those races as genuine source 404s at the backup layer.
Fix: on a replicateChunks failure the filer sink now skips only when the
live source has moved past the replayed version (deleted or strictly-newer
mtime) — lossless, a later event carries the current content — and
propagates otherwise so the event is retried. The manifest resolve retries
transient errors like the data-chunk path. isIgnorable404 is narrowed to
genuine 404s; non-filer sinks and the initial-snapshot walk, which relied
on the broad match as their only lossless-skip valve, now make the same
live-source decision (filersink.SourceSupersedes) instead of retrying
forever on a permanently gone volume.
Tests cover propagation of unconfirmed lookup failures, the narrowed 404
classification, and the supersession guards.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(filer.backup): derive supersession path with directory fallback
eventSourceSuperseded built the source path from NewParentPath alone.
Legacy metadata events (persisted by older filers) carry an empty
NewParentPath, so the probe looked up "/<name>", read the miss as
"source gone", and skipped a live file on a transient lookup error —
the silent drop this change is meant to eliminate.
Derive the path via MetadataEventTargetFullPath (the same directory
fallback genProcessFunction uses) and cover both event shapes with
TestEventSupersessionProbe_PathDerivation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(filersink): retry manifest resolve only for transient errors, bounded when unverifiable
The manifest-resolve retry stopped only when hasSourceNewerVersion proved
the source moved past the replayed version, which wedged the sink in two
cases: incremental sinks use dated target keys that cannot map back to a
source path (supersession never provable), and permanent resolve errors
(corrupt manifest data, bad file ids) fail forever while the source entry
stays live.
Gate the retry instead: keep retrying only transient errors (volume-lookup
races, network interruptions), stop after a few attempts when supersession
cannot be checked, and propagate everything else immediately so the
configured metadata error policy applies (-disableErrorRetry included).
Propagation is lossless: filer.backup's fallback decides with the event's
real source key, and both filer.backup and filer.sync re-deliver the event
(RetryForeverOnError) without advancing the offset.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(filer.backup): make error classifiers nil-safe
isIgnorable404, isSourceLookupError, and isTransientResolveError called
err.Error() without a nil guard. All current call sites pass a non-nil
error, but the guard is free and matches isRetryableNetworkError.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* admin: let waiting shell clients win the admin lock
The admin lock manager re-acquired the cluster shell lock immediately
after releasing it, while a waiting weed shell client only polls the
master once a second, so an operator could lose the race indefinitely.
Leave the lock free for slightly more than one poll interval before
re-acquiring, and once overlapping reference-counted holds have kept
the lock continuously held for over a minute, make new admin-side
acquires wait for a full release before piggybacking.
* admin/plugin: take the shell lock per detection and per job, not per batch
The default-lane scheduler acquired the cluster shell lock once and
held it across the whole pass: every due job type's detection plus all
of its dispatched jobs, each drained to completion. A manual weed
shell operation sharing that lock could stall behind the batch for up
to the extended execution window.
Hold the lock only while it protects something: around each detection
scan and around each dispatched job. Manual shell operations now wait
for at most one in-flight job. The admin UI detect+execute path stops
wrapping dispatch in an outer hold, since a nested acquire would
deadlock against the lock manager's fairness window; detection and
per-job dispatch take the lock themselves.
* admin/plugin: stop extending the dispatch window to the largest job estimate
When any proposal's estimated runtime exceeded the remaining
JobTypeMaxRuntime, the whole dispatch context was replaced with a
fresh one capped at eight hours, so a single balance backlog could
hold the default lane (and with it erasure_coding and vacuum
detection) for that long.
Keep the dispatch window at JobTypeMaxRuntime and instead detach each
started attempt onto its own estimated-runtime deadline. Large jobs
still get their full time once started; jobs not yet started when the
window closes are canceled and re-proposed by a later detection, so
sibling job types get a turn every window.
* worker/balance: re-check each planned move against the master before executing
A balance plan is computed at detection time, but the admin lock is
released between detection and execution, so a manual shell operation
can rearrange the volume in the gap. The task's own guards catch a
vanished source, but a target that gained a replica in the meantime
would be silently overwritten by VolumeCopy and the source delete
would then reduce the volume to a single copy.
Before executing each move, ask the master for the volume's current
locations (uncached) and skip the move if the volume has left the
source or the target already holds a replica. Skipped moves fail with
a stale-move error and the next detection replans them. Without master
addresses in the cluster context the check is skipped, preserving the
old behavior with older admins.
* admin: block re-acquire while the final lock release is in flight
Release dropped the manager mutex before calling ReleaseLock, so a
concurrent Acquire could see hold count zero and call RequestLock
while the locker still considered itself locked. That request no-ops,
leaving a hold with no live master lease. Track the in-flight release
and make Acquire wait for it.
Also normalize a nil release function from lock manager
implementations, and make the yield and fairness windows per-instance
fields so tests stop mutating globals.
* rust volume: accept odd-length needle id hex in file ids
Go formats the needle id with strconv.FormatUint and parses it back with
strconv.ParseUint, neither of which pads to an even number of hex digits.
hex::decode rejected such file ids with "Odd number of digits", so
volume.fsck could not purge orphans from a rust volume server. Parse the
needle id and cookie with from_str_radix, matching Go's ParseNeedleId
and ParseCookie.
* storage: emit even-length needle id hex in NeedleId.FileId
volume.fsck and volume.check_disk build purge file ids here with unpadded
FormatUint hex, while every other fid formatter strips whole leading zero
bytes. Pad to even length so the output matches the canonical fid format
and strict hex parsers accept it.
* s3: verify SigV4 against each plausible reverse-proxy host
A portless X-Forwarded-Host leaves the client's true port ambiguous: a
proxy that kept the Host header makes the backend Host port right, one
that rewrote it makes X-Forwarded-Port right, and a client on the
scheme's default port signed no port at all. The verifier bet on the
backend Host port whenever the hostnames matched, so nginx-style
$host/$server_port forwarding got SignatureDoesNotMatch whenever the
proxy and backend share a hostname. Try each plausible host value in
likelihood order instead of guessing one.
* s3: unbracket IPv6 X-Forwarded-Host before matching the request host
net.SplitHostPort strips brackets from the request host, so a bracketed
portless X-Forwarded-Host like [::1] never matched and lost its port
candidate.
* s3: cover unbracketed IPv6 forwarded-host candidates; compare with slices.Equal
TestSharedSnapshotConcurrentIntegrity writes as fast as one core can
marshal for two seconds, so its transient heap scales with host speed.
On linux/386 a fast runner walks the heap into the 4GB address-space
ceiling and the run dies with out of memory. Set a 1GB memory limit for
the duration of the test so the GC paces the writer instead of the
address space; the seal-and-recycle race being tested is unaffected.
* s3: keep listing when empty directories fill the listing window
doListFilerEntries issued a single ListEntries request per directory with
Limit = maxKeys+2. Entries that emit nothing - empty directories, the
.uploads folder, the marker echo - consume that window without consuming
maxKeys, so a bucket whose first window held only empty directories was
reported empty and not truncated, and a subdirectory whose window filled
up silently dropped the entries behind it. Keep requesting from the last
received entry until the quota is filled or a short window shows the
directory is exhausted.
* s3: test listing across windows of empty directories
The test filer client now honors StartFromFileName so repeated windows
advance like the real filer.
* s3: propagate the request context into directory listing RPCs
A disconnected client now cancels the ListEntries streams instead of
letting the listing keep issuing requests against the filer.
* s3: group per-directory listing parameters into a request struct
doListFilerEntries took ten positional parameters; call sites read as
string and bool soup. Wrap the per-directory arguments in
listDirectoryRequest so recursions and tests name what they pass.
* s3: group list request parameters into a struct
listFilerEntries took eight positional parameters ending in two bare
booleans. Wrap them in listObjectsRequest so the V1 and V2 handlers
name what they pass.
Move pickOneReplicaToCopyFrom, pickOneReplicaToDelete,
pickOneMisplacedVolume, and isMisplaced into weed/topology/balancer,
following satisfyReplicaPlacement. Selection functions take the shared
balancer.Replica shape and return indices, so the shell keeps its own
replica type; behavior is unchanged and covered by the existing shell
tests.
* volume.fix.replication: parallelize under-replicated volume copies
Fan out fixOneUnderReplicatedVolume up to -maxParallelization at a time.
Destination selection and free-slot accounting stay atomic behind a
scheduler mutex, so concurrent fixes see each other's reservations; a
failed copy returns its reserved slot. A per-server in-flight cap
(-maxParallelizationPerServer, default 1) keeps many simultaneous copies
from swamping a single destination server; when every eligible
destination is at the cap the fix waits for a slot instead of failing.
* volume.fix.replication: isolate the scheduler's location ordering
Clone allLocations before the per-volume fan-out so the scheduler's
re-sorting cannot alias the slice shared with the concurrent delete
phases, and make the test's per-iteration location copy explicit.
* s3: routed last-version delete removes the emptied .versions directory
The routed versioned delete (routedDeleteSpecificVersion) repoints the
latest pointer and deletes the version file, but unlike the lock-path
fallback (updateLatestVersionAfterDeletion) it never tears down the
.versions/ directory it just emptied. The residue keeps the key's read
path in the self-heal rescan loop: every GET of the deleted key logs
event=surfaced plus a GetObject error until the background
EmptyFolderCleaner gets to the directory — at least two minutes away on
its delay queue, and possibly never (the queue is in-memory, bounded,
and gated on the bucket's allow-empty-folders policy). Veeam's lock
arbitration probes deleted lock keys continuously, so those windows are
always open and the log spam is chronic.
ObjectMutation DELETE gains remove_empty_parent: after the child delete,
the filer best-effort removes the parent directory in the same locked
transaction. Non-recursive on purpose — a concurrent write that lands a
new version fails the removal instead of being lost with it. The routed
last-version delete sets it on the version-file DELETE, matching the
lock-path fallback's contract.
Claude-Session: https://claude.ai/code/session_014mMYAHXZySkCCUfpRFNtSv
* s3: drain empty .versions residue on read heal and in s3.versions.audit
Directories already stranded by pre-teardown deletes (or dropped from
the EmptyFolderCleaner's bounded in-memory queue) previously re-entered
the self-heal rescan on every GET forever: the heal only cleared the
pointer and nothing ever removed the directory, and s3.versions.audit
counted the state as clean.
When the heal rescan finds no remaining version, remove the directory
outright (non-recursive, so orphan children still block and fall back to
the pointer clear) and log event=healed mode=empty_dir_removed; the next
GET takes the clean not-found path. The audit gains an empty category so
the residue is visible, and -heal removes such directories in bulk.
Claude-Session: https://claude.ai/code/session_014mMYAHXZySkCCUfpRFNtSv
2026-07-08 18:50:54 -07:00
523 changed files with 50664 additions and 4334 deletions
In a namespace with a default-deny policy the install hangs: the components cannot resolve each other, and the post-install bucket hook waits on the master and filer until it gives up. `networkPolicy.enabled` renders one `NetworkPolicy` per component, selecting its pods by the standard `app.kubernetes.io/{name,instance,component}` labels and admitting traffic from the other pods of the release on the ports that component listens on.
That alone leaves outbound traffic untouched, which is enough when the namespace's default-deny only restricts ingress. If it lists `Egress` in its `policyTypes` too - the usual baseline - the components still cannot resolve DNS, and you need the second opt-in below as well.
Egress is separate because the chart knows where its own components live but not where your filer store, notification sink or remote tier does, and because in a namespace with no default-deny at all, adding egress rules would narrow the components from "may reach anything" to "may reach these peers":
```yaml
networkPolicy:
enabled: true
egress:
enabled: true
kubeApiServer:
# the endpoint behind the kubernetes service, not its ClusterIP
cidrs: ["172.18.0.2/32"]
extraEgress:
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: postgresql
ports:
- protocol: TCP
port: 5432
```
`kubeApiServer.cidrs` is only demanded when something in the release actually needs the API server, which is the COSI sidecar and, on an upgrade that grows a volume PVC, the resize hook. No seaweedfs component itself speaks to it.
Anything reaching the release from outside - an ingress controller, a Prometheus in another namespace - goes into `networkPolicy.extraIngress`, or into `networkPolicy.components.<component>.extraIngress` for a single component. See the `networkPolicy` block in `values.yaml` for the full set.
Two things worth knowing before you turn this on:
- **Monitoring stops.** The metrics ports are admitted from release pods like every other port, so with `global.seaweedfs.monitoring.enabled` the ServiceMonitors keep scraping targets a Prometheus in another namespace can no longer reach. Nothing reports it; add the scraper's namespace to `extraIngress`.
- **The resize hook's policy is a Helm hook.** Its Job runs before the release manifest is applied, so the policy has to be a `pre-install` hook too. Helm does not garbage-collect hook resources, so on an upgrade that grows a volume PVC the policy is created and then left behind on uninstall - delete `<release>-seaweedfs-volume-resize-hook` by hand if it bothers you.
The DNS selectors default to CoreDNS as kubeadm, kind, EKS, GKE and AKS install it. On OpenShift, override `egress.dnsNamespaceSelector` and `egress.dnsPodSelector` to match `openshift-dns`; see the comment in `values.yaml`.
## OpenShift Support
SeaweedFS can be deployed on OpenShift or any cluster enforcing the Kubernetes "restricted" Pod Security Standard. By default, OpenShift blocks containers that run as root or use `hostPath` volumes.
{{- fail (printf "networkPolicy.components: %q is not a component of this chart, so its rules would never be applied. Known components: %s." $name (join ", " (sortAlpha $known))) }}
{{- end }}
{{- end }}
{{- range $target := $targets }}
{{- $component := $target.component }}
{{- $override := get $overrides $component | default dict }}
{{- /* Emit only the selectors that are set. An empty one has to be left out
rather than rendered as null:a missing podSelector means every pod in
the namespace, and a missing namespaceSelector means this namespace
instead of the one DNS runs in. */}}
{{- if not (or $egressCfg.dnsNamespaceSelector $egressCfg.dnsPodSelector) }}
{{- fail "networkPolicy: egress.allowDNS is on but both egress.dnsNamespaceSelector and egress.dnsPodSelector are empty, which would open port 53 to every pod in the release namespace. Set at least one of them, or turn allowDNS off and name the resolver in egress.extraEgress." }}
{{- end }}
- to:
{{- if and $egressCfg.dnsNamespaceSelector $egressCfg.dnsPodSelector }}
{{- if and $target.apiserver ($egressCfg.kubeApiServer).enabled }}
{{- if not $egressCfg.kubeApiServer.cidrs }}
{{- fail (printf "networkPolicy: %s needs the Kubernetes API server, but networkPolicy.egress.kubeApiServer.cidrs is empty. Set it to your API server address(es), or disable networkPolicy.egress.kubeApiServer and allow it through networkPolicy.egress.extraEgress." $component) }}
{{- end }}
{{- /* An empty port list is not "no ports" in a NetworkPolicy, it is every
port, so refuse it rather than quietly opening the CIDR up. */}}
{{- if not $egressCfg.kubeApiServer.ports }}
{{- fail "networkPolicy: egress.kubeApiServer.ports is empty, which allows every port on the API server CIDR(s) rather than none. Set it (the default is [443, 6443]), or turn egress.kubeApiServer off and describe the access in egress.extraEgress." }}
{{- end }}
{{- range $cidr := $egressCfg.kubeApiServer.cidrs }}
- to:
- ipBlock:
cidr:{{$cidr }}
ports:
{{- range $port := $egressCfg.kubeApiServer.ports }}
# maxVolumes: 0 # If set to zero on non-windows OS, the limit will be auto configured. (default "7")
#
# You may also spacify an existing claim:
# You may also specify an existing claim:
# - name: data
# type: "existingClaim"
# claimName: "my-pvc"
@@ -1142,6 +1142,12 @@ s3:
# Service settings
service:
type:ClusterIP
# fixed nodePorts, used only when type is NodePort or LoadBalancer
nodePorts:
http:null
https:null
iceberg:null
metrics:null
icebergIngress:
enabled:false
@@ -1164,7 +1170,7 @@ sftp:
loggingOverrideLevel:null
# SSH server configuration
sshPrivateKey:"/etc/sw/seaweedfs_sftp_ssh_private_key"# Path to the SSH private key file for host authentication
sshPrivateKey:""# Optional path to a single SSH host key file; the server fails to start if set but missing. Host keys come from hostKeysFolder by default.
hostKeysFolder:"/etc/sw/ssh"# path to folder containing SSH private key files for host authentication
authMethods:"password,publickey"# Comma-separated list of allowed auth methods: password, publickey, certificate
maxAuthTries:6# Maximum number of authentication attempts per connection
@@ -1239,6 +1245,10 @@ sftp:
# Service settings
service:
type:ClusterIP
# fixed nodePorts, used only when type is NodePort or LoadBalancer
nodePorts:
sftp:null
metrics:null
admin:
enabled:false
@@ -1376,6 +1386,10 @@ admin:
service:
type:ClusterIP
annotations:{}
# fixed nodePorts, used only when type is NodePort or LoadBalancer
nodePorts:
http:null
grpc:null
# ServiceMonitor annotations (separate from pod/deployment annotations)
serviceMonitor:
@@ -1595,6 +1609,18 @@ allInOne:
annotations:{}# Annotations for the service
type:ClusterIP # Service type (ClusterIP, NodePort, LoadBalancer)
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Chris Lu
Sebastian PreisnerandClaude Opus 5
Chris LuandGitHub
Lukas KalliesandGitHub
Kobi HikriandGitHub
Lars LehtonenandGitHub
TJDawson10andGitHub
github-actions[bot]
qzhelloandGitHub