* iam: bind CreateServiceAccount target to caller in AuthorizeIamAction
A non-admin holding iam:CreateServiceAccount could pass an arbitrary
ParentUser and mint a service account for any identity, inheriting that
identity permissions. Add a self-target category so a granted non-admin
may only target their own identity; admins remain unrestricted.
* iam: authorize CreateServiceAccount against its ParentUser target
AuthIamManagement passed UserName as the authorization target for every
action, so CreateServiceAccount was authorized with an empty target and
the self-target binding never saw the caller-supplied ParentUser. Pass
ParentUser for that action so the binding takes effect on the live path.
* iam: test CreateServiceAccount binds target to caller
Regression test: a non-admin holding iam:CreateServiceAccount may target
itself but is denied targeting another identity; admins remain
unrestricted.
* iam: authorize CreateServiceAccount against ParentUser on the S3 port
UnifiedPostHandler passed UserName as the authorization target for every
IAM action, so CreateServiceAccount was authorized with an empty target
on the S3-port route and the self-target binding never saw the caller
ParentUser. Extract iamTargetUserName (ParentUser for CreateServiceAccount,
UserName otherwise) and use it from both IAM dispatch surfaces so the
binding applies on the live S3-port path as well as the standalone iam
server.
* iam: test CreateServiceAccount ParentUser binding on the S3 port
End-to-end regression test through UnifiedPostHandler: a non-admin
holding iam:CreateServiceAccount is denied (403) when targeting another
identity and passes authorization when targeting itself.
* fix issue-11214
* fix(filer): cancel sibling manifest reads on failure
* fix(filer): scope manifest cancellation to read batch and propagate context to encrypted reads
Address PR review comments on #11215:
- Scope cancellation to each parallel read batch instead of the resolver-wide
context, so a later manifest failure does not cancel recursive work for an
earlier successful manifest (CodeRabbit #3952446554).
- Propagate the resolver context through GetAuthenticatedWithContext so
encrypted sibling reads observe cancellation and stop promptly when another
manifest fails (Greptile #3952422343).
- Use net.ListenConfig.Listen with an explicit context in the test fixture to
satisfy the noctx linter (CodeRabbit #3952111696).
- Add regression tests for encrypted sibling cancellation and for preserving
earlier manifest children on later failure.
* fix(filer): return known manifest errors without blocking on earlier children
Address Greptile review comment on #11215:
After all parallel reads complete, pre-scan slots for the first real
(non-internal-cancel) error before recursing into earlier manifests'
children. If a later manifest already failed, return its error promptly
with data chunks already in hand, instead of blocking on recursive
network reads of earlier manifests' children.
Updated the regression test to verify the error returns within 1 second
when an earlier manifest's child has a 2-second delay, and that the
child is never loaded.
* fix(filer): filter partial child chunks by requested range on error path
Address CodeRabbit review comment on #11215:
The pre-scan error path appended non-manifest child chunks from earlier
manifests without applying the [startOffset, stopOffset) overlap check
used for top-level chunks. A child outside the requested range could be
returned in dataChunks alongside the later manifest's error.
Apply the same range predicate before appending. Add regression test
with an out-of-range child chunk.
* fix(filer): buffer job channel and abort submission on batch cancellation
Address Greptile review comment on #11215:
- Use a buffered job channel (capacity 128) so submission does not block
when all workers are busy. This ensures a promptly-failing manifest is
always queued and can cancel stalled sibling reads once a worker picks
it up, instead of blocking the caller on the unbuffered channel send.
- Add batchCtx.Done() to the submit select so submission aborts promptly
when the batch is already cancelled by a sibling failure.
- Add regression test with 5 manifests (4 stalled + 1 failing) verifying
the failing job is queued and picked up after a stalled worker is freed.
* fix(filer): avoid double WaitGroup decrement on batch cancellation in submit
Address Devin review comment on #11215:
When batchCtx.Done() fired in submit, it called job.done.Done() and
returned false. The caller in resolve also called reads.Done() on the
same WaitGroup, causing a double decrement that would panic with a
negative counter.
Fix: submit sets the result error but does not decrement the WaitGroup.
The caller always owns the decrement and skips overwriting the result
when submit already set it.
* fix(filer): overflow execution when job queue buffer is full
Address Greptile follow-up review comment on #11215:
With a 128-entry buffer, if more than 132 in-range manifests (4 workers +
128 buffer) stall at one level, a promptly-failing manifest beyond the
buffer cannot be submitted and cannot cancel the stalled reads.
Fix: when the buffer is full, run the job directly in a goroutine instead
of blocking on the channel send. This only triggers for >132 manifests at
one level (exceedingly rare), so the bounded concurrency guarantee (4
workers) holds for all normal workloads. Extracted executeJob method
shared by both workers and overflow goroutines.
* fix(filer): bound overflow execution with a semaphore
Address Devin review comment on #11215:
The unbounded overflow goroutines could create thousands of concurrent
reads for large files, defeating the four-worker resource bound.
Fix: add a semaphore (capacity = maxChunkManifestResolveWorkers) that
overflow goroutines must acquire before doing the read. While waiting for
the semaphore, they also watch batchCtx and r.ctx so they exit promptly
on cancellation. Total concurrency is now bounded to 2 * workers (4
workers + 4 overflow) in the degenerate case.
* refactor(http): add ctx to GetAuthenticated signature instead of new function
Reuse the existing GetAuthenticated name by adding ctx as the first
parameter, matching the pattern of ReadUrl, ReadUrlAsStream, and
RetriedFetchChunkData. Removes the GetAuthenticatedWithContext wrapper.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* admin: support setting volume read-only and read/write modes
* admin: address PR review on volume access-mode persistence
Reject trailing JSON values in the SetVolumeReadOnly handler so
requests like {"read_only":true}{} no longer pass validation, and add
a trailing-value case to the invalid-request test.
Propagate .vif persistence failures through the access-mode chain.
PersistReadOnly now returns the SaveVolumeInfo error and rolls back
the in-memory volumeInfo on failure; Store.MarkVolumeReadonly and
Store.MarkVolumeWritable propagate that error and roll back their
noWrite flags, so the API reports failure instead of success while
restart would revert the mode.
* admin: make .vif persistence atomic and preserve error chain
SaveVolumeInfo now writes to a .vif.tmp file, syncs it, renames it
over the target, and fsyncs the directory. A write/sync/close failure
leaves the existing .vif intact, so the PersistReadOnly in-memory
rollback matches the durable state instead of diverging from a
partially written file that restart would apply.
Switch the error wrappers in PersistReadOnly, MarkVolumeReadonly, and
MarkVolumeWritable from %v to %w so callers can use errors.Is and
errors.As to classify persistence failures.
* admin: treat post-rename dir fsync failure as a warning
After os.Rename commits the new .vif, the on-disk file already holds
the requested mode. A directory fsync failure only risks losing the
rename across a crash; returning an error here would make
PersistReadOnly roll back in-memory state while the durable file keeps
the new mode, splitting the replica. Log the failure as a warning
instead, matching the best-effort nature of FsyncDir (already skipped
on Windows).
* admin: distinguish post-rename durability failures and use unique temp files
SaveVolumeInfo now uses os.CreateTemp for the staging file, preventing
concurrent saves for the same volume from colliding on a shared .tmp
path.
A directory fsync failure after os.Rename returns a
NotCrashDurableError instead of being silently swallowed. The rename
already committed the new metadata to disk, so PersistReadOnly,
MarkVolumeReadonly, and MarkVolumeWritable skip the in-memory rollback
for this error type (keeping state aligned with the durable file) while
still propagating the failure to the API. Pre-commit failures continue
to roll back as before.
* admin: continue post-commit work after NotCrashDurableError
MarkVolumeWritable now clears the EIO quarantine and the gRPC handlers
(makeVolumeReadonly step 3, makeVolumeWritable master notification)
proceed with their post-commit work when SaveVolumeInfo returns a
NotCrashDurableError, instead of aborting and leaving the volume
unavailable or the master unaware of the mode change. The durability
warning is still propagated to the API caller. Pre-commit failures
continue to abort early as before.
* admin: handle NotCrashDurableError in tier and EC callers
VolumeTierMoveDatFromRemote and VolumeEcShardsGenerate now check for
NotCrashDurableError from SaveVolumeInfo. When the rename has already
committed the new .vif, they continue with their post-commit work
(backend switch, remote deletion, keeping generated EC shards) instead
of aborting and leaving the on-disk metadata inconsistent with the
file layout. The durability warning is logged for the operator.
* build(deps): bump github.com/twmb/avro from 1.7.2 to 1.8.0
Bumps [github.com/twmb/avro](https://github.com/twmb/avro) from 1.7.2 to 1.8.0.
- [Commits](https://github.com/twmb/avro/compare/v1.7.2...v1.8.0)
---
updated-dependencies:
- dependency-name: github.com/twmb/avro
dependency-version: 1.8.0
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* iceberg: adapt to twmb/avro v1.8.0 and iceberg-go defensive copies
avro v1.8.0 changes Schema.Root() to return *SchemaNode, which breaks
iceberg-go v0.6.0's internal avro_schemas.go. The fix (apache/iceberg-go#1843)
is only on iceberg-go's main branch, unreleased, so bump iceberg-go to
that commit (c210509) alongside the avro bump.
That iceberg-go revision also changes two behaviors seaweedfs worked
around:
- It now infers a manifest list's format version from the embedded
writer schema, so a list missing the "format-version" header entry
(DuckDB's shape is read as v2, not v1. ReadManifestList's header
patching is now a redundant safety net; tests updated to expect v2.
- It returns defensive copies from DataFile.Partition(), so the
ReadManifest shim's in-place partition normalization was silently
discarded. Rebuild the entry through NewDataFileBuilder when any
partition value is normalized, copying every other DataFile field so
manifest round-trips are preserved.
- It converts day-transform partitions to iceberg.Date on read
(applyDayTransformDates), so the day-partition cases the shim and
tests guarded now convert without help; tests updated to expect
iceberg.Date from the raw read.
EOF
)
* iceberg: accept assert-ref-snapshot-id without snapshot-id
iceberg-go's new nullableInt64 parser rejects an assert-ref-snapshot-id
requirement whose "snapshot-id" field is absent from the JSON, even
though the Iceberg REST spec makes it optional (null means the ref must
not already exist). v0.6.0 used a plain *int64, so absent was nil and
accepted. ClickHouse sends the requirement without snapshot-id when
asserting a branch does not yet exist, so its writes fail with
"missing required field \"snapshot-id\"".
normalizeRequirements splices an explicit null into any
assert-ref-snapshot-id requirement missing the field before handing
the JSON to iceberg-go's parser, restoring the v0.6.0 behavior across
both iceberg-go versions.
* iceberg: fix v1 block_size_in_bytes default in rebuilt manifest entries
rebuildManifestEntry set block_size_in_bytes to 0, but the v1 manifest
schema requires the default of 64 MiB ("Always write default in v1").
The original value is not exposed on the DataFile interface, so use the
spec default. Also clarify the fallback comment to note that empty
(zero-record / zero-byte) files also trigger it, not just a nil spec.
Add a round-trip test that writes a rebuilt entry as v1 and verifies
block_size_in_bytes is 64 MiB via Avro decoding.
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* rust volume: insert redb rebuilds in needle-id order
Unlink the .rdb before create (create does not truncate). Collapse
last-write-wins, then insert live keys sorted so 4.2.0 packs leaves.
* rust volume: rebuild redb from a BTreeMap and clear leftover keys
Peak rebuild memory is one ordered map instead of HashMap + Vec +
stable-sort scratch. Unlink stays best-effort: if it fails, retain
clears the leftover table before sorted insert. Compute idx metrics
before the write so a read error does not unlink a committed .rdb.
* rust volume: drop the extra redb read transaction on put/delete
put uses insert()'s previous value. delete gets then inserts the
tombstone in the same write transaction. Truncate the .idx row on
any failed redb write after the append.
* rust volume: unpack redb blobs through packed_to_needle_value
save_to_idx, ascending_visit, and collect_entries used the same
length-check copy as get. Route them through the helper so a
wrong-length value is absent everywhere, not a panic.
* rust volume: quick-repair redb on durable checkpoints
set_quick_repair(true) on the durable checkpoint transaction so an
OOM-killed volume server opens without a full-file repair scan.
* rust volume: reopen redb from .idx on non-poisoned commit error
redb 4.2.0 can make a Durability::None commit visible before returning
Err(CommitError::Storage(..)). In that state the database refuses further
write transactions, so truncating the .idx row (the old behavior) would
leave a redb-only put or tombstone that the stored idx_size makes the
reload skip.
Distinguish CommitError::TransactionPoisoned (txn rolled back, db still
usable -- truncate the orphan .idx row as before) from other commit errors
(change may be visible, db refuses writes -- keep the .idx row, close the
database, and reopen from .idx to repair redb's internal state).
db becomes Option<Database> so reopen_from_idx can drop the old file lock
before load_from_idx opens the same path. rdb_path, version, and
cache_bytes are stored so the reopen uses the same configuration.
* rust volume: truncate .idx row when redb is closed in put
put appends the .idx entry before acquiring the write transaction.
When db_or_err() fails (db is None after a failed reopen), the ?
returned without calling truncate_idx_to_offset, so a write reported
failed remained in the authoritative .idx and was replayed on restart.
Handle db_or_err() explicitly and truncate the orphan .idx row before
returning the error, matching the existing handling for begin_write,
open_table, and insert failures.
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
SetVolumeCrowded mutated the crowded map under accessLock.RLock(), while
GetWritableVolumeCount reads the same map under RLock() on the Assign hot
path. Two concurrent RLock holders with one writing and one reading the
map triggers a fatal "concurrent map read and map write" that kills the
master process (unrecoverable, bypasses recover).
Take the write lock in SetVolumeCrowded instead. This event path is a
low-frequency single consumer driven by the crowded-volume event loop,
and every other mutation of crowded already holds Lock(); setVolumeCrowded
takes no nested locks, so there is no deadlock path. The hot readers
(GetWritableVolumeCount, CloneWritableVolumes) keep using RLock.
Adds a -race regression test that fails (race detected) on the old RLock
and passes with the write lock.
Fixes#11211
* fix(log_buffer): stop closing a notification channel another reader still holds - #10810
The report blames the polling loop for the busy spin, but that loop is
not what burns the core. LogBuffer keeps one notification channel per
subscriberID, and UnregisterSubscriber closes it. Two registrations that
share a subscriberID share that channel, which happens whenever a client
opens a second stream or an old stream has not yet noticed it was
replaced, so the first unregister closes a channel the other reader is
parked on. A closed channel makes every receive in
awaitNotificationOrTimeoutFor return instantly, and that reader then
spins at full speed for the rest of its life.
Subscriptions are now reference counted. Registering an existing
subscriberID hands back the same channel and raises the count; the
channel is only closed when the last holder unregisters.
* test: fail if the surviving reader stops instead of keeps reading
Review caught that the iteration count alone proves nothing: had
LoopProcessLogData returned when the duplicate reader unregistered, the
counter would sit at 0 and the assertion would pass without a reader
ever having been there to spin. Check the reader is still running before
trusting its low count.
---------
Co-authored-by: Junker der Provinz <jdp@braethoria.com>
* rust volume: quick-repair redb on durable checkpoints
set_quick_repair(true) on the durable checkpoint transaction so an
OOM-killed volume server opens without a full-file repair scan.
* rust volume: optional redb insert_before bulk load
Behind redb-experimental-cursor (default off). Production binary
stays on sorted insert(). CI unit tests run both feature settings.
* rust volume: exercise insert_before across leaf splits
Replace the 5-key cfg clone with a 4000-key reverse-order rebuild
so CursorMut::insert_before hits page splits. CI runs the feature
only on storage::needle_map unit tests.
* rust volume: insert redb rebuilds in needle-id order
Unlink the .rdb before create (create does not truncate). Collapse
last-write-wins, then insert live keys sorted so 4.2.0 packs leaves.
* rust volume: rebuild redb from a BTreeMap and clear leftover keys
Peak rebuild memory is one ordered map instead of HashMap + Vec +
stable-sort scratch. Unlink stays best-effort: if it fails, retain
clears the leftover table before sorted insert. Compute idx metrics
before the write so a read error does not unlink a committed .rdb.
* rust volume: drop the extra redb read transaction on put/delete
put uses insert()'s previous value. delete gets then inserts the
tombstone in the same write transaction. Truncate the .idx row on
any failed redb write after the append.
* rust volume: unpack redb blobs through packed_to_needle_value
save_to_idx, ascending_visit, and collect_entries used the same
length-check copy as get. Route them through the helper so a
wrong-length value is absent everywhere, not a panic.
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* rust volume: insert redb rebuilds in needle-id order
Unlink the .rdb before create (create does not truncate). Collapse
last-write-wins, then insert live keys sorted so 4.2.0 packs leaves.
* rust volume: rebuild redb from a BTreeMap and clear leftover keys
Peak rebuild memory is one ordered map instead of HashMap + Vec +
stable-sort scratch. Unlink stays best-effort: if it fails, retain
clears the leftover table before sorted insert. Compute idx metrics
before the write so a read error does not unlink a committed .rdb.
* shell: volume.check.disk — actionable verdict for diverged vacuumed replicas
When -resurrectMissingNeedles is gated off because both replicas have been
vacuumed (compaction revision > 0) — the normal state of any production
cluster — check.disk previously stopped at 'cannot prove they are missing
writes vs vacuumed deletes' and did nothing, leaving a diverged replica with
no repair path. volume.fix.replication does not catch it either: it only
acts when the replica COUNT is below the expected replication, never when
two replicas are both present but hold different live data.
Classify the divergence instead of dead-ending:
liveDivergence() counts live (non-deleted) needles present on one replica's
index but entirely absent from the other, in both directions. Tombstones
are excluded, so vacuum asymmetry (a compacted replica that dropped deleted
entries) is not mistaken for divergence.
reportDivergenceVerdict() turns the count into an operator action:
- one-sided (one replica has all the live data, the other has no unique
live needles) -> print the exact safe repair:
volume.copy -source <complete> -target <lagging> -volumeId <id>
Re-copying the complete replica is safe precisely because the lagging
side holds no unique live data; VolumeCopy's verify-before-destroy gate
independently confirms the source holds the volume before deleting the
target.
- two-sided (split-brain, both sides have unique live data) -> warn and
do NOT emit an auto repair; point to volume.fsck -findMissingChunksInFiler
to confirm the 'missing' needles are orphans before converging.
Report-only: no data is modified and the resurrection safety gate is
untouched. This is what lets a 13-volume diverged cluster be diagnosed and
repaired in minutes instead of by hand-diffing every index.
Observed motivating case: home SeaweedFS 4.45 cluster, 13 010 cross-rack
volumes diverged after failed replicate writes (ReplicatedWrite MaxAttempts=1
fire-and-forget), 11 one-sided + 2 two-sided, all repaired via volume.copy.
* shell: volume.check.disk — address review nits on divergence verdict
- Use pb.NewServerAddressFromDataNode (dialable ip:port, Address with Id
fallback) for the advertised volume.copy -source/-target instead of the
logical node Id, which may not be dialable.
- Make the one-sided verdict tombstone-aware: when the lagging replica has
been vacuumed, absent live needles may be valid deletions whose tombstones
were dropped, so a whole-volume re-copy would resurrect them. The command
is only advertised as safe when the lagging side is proven never-vacuumed
(compaction revision 0 read under -resurrectMissingNeedles); otherwise a
caveat is printed pointing at fsck/needle-level repair.
- Fix reversed copy direction when the source replica is the lagging one
(must copy complete -> lagging in both cases).
- Test: real tombstone (negative size) with the correct 0/0 expectation and
|| assertion; verdict test now covers dialable address, corrected
direction, and caveat on/off.
* shell: volume.check.disk — per-replica revision knowledge, no copy command for vacuumed lagging side
- Track srcRevKnown/tgtRevKnown separately: in unidirectional mode the
target revision IS read, so a proven never-vacuumed target no longer
gets a false resurrection warning (regression: bidi=false, target rev 0).
- A one-sided verdict now only emits the volume.copy command when the
lagging side is proven never-vacuumed; when it is vacuumed (or unproven)
the verdict refuses to print the destructive command and points at
fsck/needle-level repair instead — an appended caveat next to a ready-to-
paste copy command was still inviting the resurrection.
- deletionCaveat now returns the boolean safety decision.
* shell: volume.check.disk — preserve gRPC port, proven two-sided is not split-brain
- Emit the raw ServerAddress string (host:port.grpcPort) instead of
String()/ToHttpAddress(), which drops the custom gRPC port and would
make the suggested volume.copy dial the default port and fail.
- Two-sided divergence with both replicas proven never-vacuumed under
-resurrectMissingNeedles is mutually missed writes, not split-brain:
recommend re-running with -apply (in-place resurrection both
directions) instead of the split-brain no-auto-repair warning.
- Tests: case F (proven two-sided -> -apply, no split-brain warning),
case G (custom gRPC ports preserved in emitted addresses).
---------
Co-authored-by: timolow <tim@timolow.dev>
* rust worker: add install_default_crypto_provider helper
lance's aws backend pulls aws-lc-rs and reqwest's rustls-tls pulls ring,
so rustls 0.23 cannot auto-select a CryptoProvider and tonic's client TLS
panics on first use. Add install_default_crypto_provider, pinning the
default to aws-lc-rs, mirroring the Rust volume server's helper of the
same name. Includes a regression test that builds a TLS channel and
panics without the install in this crate, where both providers link.
* rust worker: install the crypto provider at startup
Call install_default_crypto_provider before any TLS use, the way the
Rust volume server does in its main. Without this a worker started with
--tls-ca/--tls-cert/--tls-key panics on the first admin dial (#11194).
* s3: gate IAM-cache gRPC RPCs behind admin Bearer auth
The SeaweedS3IamCacheServer registered on the S3 gateway's internal gRPC
port (default 0.0.0.0:18333) accepted PutIdentity/RemoveIdentity/PutPolicy/
DeletePolicy/GetPolicy/ListPolicies/PutGroup/RemoveGroup with no per-RPC
authentication. An unauthenticated network peer could call PutIdentity with
Actions:[Admin] and write straight into the live accessKeyIdent map that
the SigV4 path reads, bypassing S3 authentication entirely.
Mirror the filer's IamGrpcServer.checkAdminAuth: require a Bearer token
signed with jwt.filer_signing.key (read from the existing s3a.filerGuard)
at the top of every IAM-cache RPC. With no key configured the check is a
no-op, matching the rest of SeaweedFS's gRPC surface.
* credential: attach admin Bearer token to S3 IAM-cache propagation
The filer's PropagatingCredentialStore fans IAM mutations out to peer S3
servers over the SeaweedS3IamCache gRPC service. Now that the S3 handlers
require a Bearer token signed with jwt.filer_signing.key, attach one to the
outgoing propagation context (mirroring shell/iamAdminAuthContext). With no
key configured it is a no-op, so deployments that run without the signing
key keep working.
* credential: mint IAM-cache admin token after master discovery
propagateChange attached the admin Bearer token before ListClusterNodes,
so master-client retries could run down the (default 10s) token lifetime
before the peer S3 fan-out began, leaving peers to reject an expired token
and IAM caches stale. Move withIamCacheAdminAuth to after discovery
succeeds, immediately before the propagation timeout is derived.
* credential: cap IAM-cache propagation timeout below JWT lifetime
The propagation fan-out used a fixed 10s timeout. If an operator
configures jwt.filer_signing.expires_after_seconds below 10, the admin
token can expire while slower S3 peers are still being contacted, leaving
their IAM caches stale. Derive the propagation deadline as
min(10s, tokenTTL) so it never outlives the token. withIamCacheAdminAuth
now returns the token's lifetime (0 = no expiry) for this purpose.
Both tier-move handlers run in a detached tokio::spawn and report
progress through a closure that returns (), with the send result
discarded. Nothing observes the caller leaving, so an abandoned move
uploads or downloads the whole .dat anyway and then commits the
transition.
Go aborts both. Its progress callback returns `stream.Send`'s error,
which surfaces out of the reader in s3_upload.go:99 and the writer in
s3_download.go:84 and fails the transfer, so the volume info is never
rewritten. The Rust port dropped that by typing the callback as
FnMut(i64, f32) with no result.
Give the callback Go's signature -- FnMut(i64, f32) -> Result<(), String>
-- and abort when the caller's channel is closed. Checked on every part
rather than only where progress is reported, since the report is
rate-limited to one a second and would miss a caller that left in
between. A merely full channel is a slow reader, not a departed one, so
only TrySendError::Closed counts as cancellation.
Two consequences of aborting mid-transfer that the old code never had to
handle:
- upload_file now aborts the multipart upload when the transfer fails.
An abandoned multipart upload does not show up in an ordinary object
listing but still accrues storage charges until a lifecycle rule reaps
it, and cancellation makes that a routine path rather than a rare one.
- The tier-down handler removes the partial .dat. download_file
pre-allocates the destination to the object's full size, so an aborted
download leaves a file of the right length and the wrong content --
and this handler refuses to run at all when a local .dat exists, so
leaving one wedges every retry on "already on local disk" and a
restart would load the sparse file as the volume's data.
There is deliberately no check between a finished transfer and the
bookkeeping that follows. Once the object is in S3, or the .dat is on
disk, that bookkeeping is what makes the state consistent; stopping
there would leave an object paid for and referenced by nothing, or a
complete local .dat the volume still calls remote. Go does not gate
there either -- its callback only runs during the transfer.
Claude-Session: https://claude.ai/code/session_0122W3eqt6gmLUMxmRoZdPAb
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* rust volume: stop a VolumeCopy whose caller has gone
VolumeCopy runs its copy in a detached tokio::spawn and reports progress
with the send error discarded, so nothing observes the client leaving.
When the caller cancels the RPC -- which weed-admin's batch balance does
routinely, starting far more copies than it finishes -- the server
streamed the whole volume from the source, wrote it to disk, and mounted
it. The destination is then left holding a volume nobody took delivery
of: its index cache is never reclaimed, and under replication=000 one
volume id ends up on two servers, both writable, which concurrent writes
can diverge.
Three checks now reach the task:
- Every chunk in copy_file_from_source, via the sender's is_closed().
This is the one that matters in practice. The first progress report is
128MB in, so for a smaller volume -- the ordinary balance move -- no
send ever happens and its result says nothing; only the closed channel
does. The sender is passed for the .idx and .vif copies too, with
reporting gated separately, so those phases notice as well.
- The throttle sleep, which for a throttled copy runs for seconds at a
time, now races the sender's closed() instead of being slept through.
- Immediately before mount_volume, and once at the top of the task.
Cancellation surfaces as an ordinary Err(Status::cancelled), so it lands
in the existing error branch that already removes the partial .dat/.idx/
.vif and the .note. That branch also logs now: the error otherwise went
to a channel nobody was reading, leaving the operator with the
balancer's "delete that copy, then re-run the move" and no cause.
This also clears the stranded read-only sources reported on the issue.
They are downstream of the orphan mount, not a separate defect:
LiveMoveVolume's cleanup probes the target before undoing the freeze
(volume_move.go:95, "the server can finish the copy and mount the target
even when the client loses the stream"), and when it finds a mounted
copy it cannot attribute, or cannot delete, it deliberately keeps the
source readonly rather than risk two writable replicas -- the messages
at volume_move.go:110 and :123. With nothing mounted on the target the
probe reports clean and the freeze is undone.
On Go parity: the progress send result is honoured here too, matching
`return false` in volume_grpc_copy.go. But that report is Go's only
abort signal, and measured against a 120MiB volume -- above the
throttler's activation threshold, below the 128MiB report interval -- a
Go destination mounts an abandoned copy as well. The issue's premise
that Go aborts holds only above the report interval. The Rust side now
stops in both cases; the Go behaviour is worth its own issue.
Tests: the integration test runs against both implementations and is
green on Go, red on Rust before this change. Its 192MiB fixture is sized
for two separate constraints, documented at the fixture: IoBytePerSecond
is a no-op below ~100ms of wall clock (64MiB copies in ~110ms on a tmpfs
loopback cluster), and the payload must exceed the 128MiB report
interval for the Go leg to pass at all. The two Rust unit tests cover
what the integration test cannot reach: cancellation detected with no
progress report at all, and the cleanup of the partial files plus the
.note.
Fixes#11186
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0122W3eqt6gmLUMxmRoZdPAb
* rust volume: surface VolumeCopy cancellation as Status::cancelled
copy_file_from_source returned Result<_, String>, so the per-chunk
cancellation path -- the one that matters in practice for volumes
below the 128MB report interval -- was wrapped to Status::internal at
the call sites. The spawn logging branch then classified it as a
generic failure instead of the intended "abandoned by caller",
defeating the logging change in the same PR for the case that occurs
most often.
Return Result<_, Status> from copy_file_from_source: Status::cancelled
for caller-gone, Status::internal for the existing errors. Drop the
.map_err(|e| Status::internal(e)) at the three call sites.
The unit test now asserts the code is Cancelled, not just the message
text, so the classification is locked in.
* rust volume: close cancellation gaps in VolumeCopy
Address two review findings on the same PR:
1. Roll back a mount that races a departing caller. The pre-mount
is_closed() check cannot close the window between the check and
mount_volume: if the receiver drops in that gap, the volume mounts
and the final tx.send(Ok(...)) fails, but its error was discarded
(let _ =), so the task returned Ok(()) and the error branch never
ran. The destination then held an orphaned mounted replica — the
exact defect this PR prevents.
Fix: track a mounted flag. The final send now checks its result;
on failure it returns Status::cancelled, and the error branch
calls store.delete_volume (which unmounts AND removes the files)
when mounted is true, instead of only unlinking.
2. Observe cancellation while awaiting the source stream. The
per-chunk is_closed() check only runs after stream.message().await
returns. A stalled source (slow disk, partition, GC pause) never
delivers a chunk, so a caller that has already left cannot preempt
the read: the task, the source connection, and the partial files
(including the .note) all outlive the caller indefinitely.
Fix: race stream.message() against progress_tx.closed() in a
tokio::select!, so a departing caller preempts a stalled source.
Adds test_volume_copy_after_mount_cancellation_rolls_back_mount to
cover the after-mount rollback path. cargo test --release green
(497 + 5 + 1 + 28).
* rust volume: keep remote data on after-mount rollback, race RPC startup
Two review findings on the after-mount rollback added in ff51f6a:
1. The rollback called delete_volume(vid, false, false), i.e.
keep_remote_data=false. A remote-tier copy .vif points at the same
cloud object the source replica references, so destroying the
abandoned destination with keep_remote_data=false deletes the
source remote data via Volume::destroy backend.delete_file_blocking.
Use keep_remote_data=true, matching the pre-spawn delete_volume at
the top of volume_copy.
2. client.copy_file(copy_req).await (the initial RPC establishment)
was not raced against progress_tx.closed(). If the source stalls
before sending response headers, the per-message select! added in
ff51f6a is never reached, and the task, source connection, and
preallocated files outlive a departed caller. Race the RPC
establishment against progress_tx.closed() the same way.
cargo test --release green (497 + 5 + 1 + 28).
* rust volume: race master-configuration wait against caller cancellation
try_get_master_configuration().await was the last un-raced await in
the VolumeCopy task before copy_file_from_source. A stalled master
(or slow leader election) would hold the task and its .note past a
departing caller, since the per-chunk cancellation checks are never
reached. Race it against tx.closed() the same way the source stream
reads already are.
cargo test --release green (497 + 5 + 1 + 28).
* rust volume: fix after-mount rollback test to actually reach that path
The previous version of
test_volume_copy_after_mount_cancellation_rolls_back_mount dropped the
response immediately after volume_copy returned, so tx.is_closed() was
already true at the spawn first check and the task returned before
mount_volume. The test was green for the wrong reason: the mounted
flag, the rollback, and the keep_remote_data=true line were all
uncovered.
Use the store write lock as a seam: take it immediately after
volume_copy returns so the task runs the copy to completion with the
caller still attached, passes the pre-mount is_closed() check, then
parks entering the mount block. Drop the response (caller gone) and
release the guard: the task mounts, fails the final tx.send, and must
roll back via delete_volume.
Verified by setting the rollback guard to if false: the test fails
with "destination still holds a mounted volume". With the rollback
enabled, probe eprintlns confirmed the full path: about to mount ->
mounted = true -> final send failed -> rolling back mount -> rollback
done.
cargo test --release green (497 + 5 + 1 + 28).
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* rust volume: checkpoint the redb index durably every 1000 writes
Every put and delete on a redb-backed volume committed with
Durability::None and nothing ever committed durably, on the theory that
the .idx file is the source of truth. redb, however, keeps an entry in
its transaction tracker for every non-durable commit and cannot recycle
pages that were on disk at the last durable commit until a durable one
happens. With no durable commit for the life of the process, both grew
with every write, and .rdb files could bloat toward double size after a
restart (#11179, the hash-table rehash stacks in the memleak output).
The needle map now counts non-durable commits and reports when a
checkpoint is due; the volume takes it, data first: flush the .dat, then
the map fsyncs the .idx and commits redb durably, recording in the same
transaction how much of the .idx the table reflects. A checkpoint makes
the index durable, so the bytes it points at must be down before it, or
after a power loss the index would reference past the end of the .dat
and the volume would load read-only. A failed .dat flush skips the
checkpoint; it is retried on the next write.
Volume::close() now closes the needle map instead of only syncing it,
and the redb map's close() takes the same checkpoint. Before, a clean
shutdown left the table durable (redb flushes on drop) but the recorded
.idx size stale at its load-time value, so the next load replayed every
entry written since load on top of the counters.
On load, the redb map's counters now come from the whole .idx history,
the way Go's LevelDB map rebuilds them (newest entry first, with a bloom
filter of seen keys), instead of from the table's final state. Both the
reuse and the full-rebuild path use it, so overwritten and deleted bytes
keep counting as garbage across restarts, and the incremental replay of
the .idx tail only touches the table, which makes it idempotent whether
or not the table is ahead of the recorded .idx size.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019x36FiSeyePh77YXao15kK
* rust volume: skip redundant .idx fsync on checkpoint after flush_idx
On the fsync=true write path, flush_idx() already fsyncs the .idx before
maybe_checkpoint_index() runs, so the checkpoint's own sync() fsyncs the
same file a second time for nothing. Thread an idx_already_synced flag
from the volume through maybe_checkpoint_index into checkpoint(sync_idx):
when it is true the checkpoint skips its .idx fsync and only does the
durable redb commit. The delete path and close() still sync (they have
not flushed the .idx beforehand).
Co-Authored-By: Chris Lu <chris.lu@gmail.com>
* rust volume: saturate writes_since_checkpoint to prevent u32 overflow
If checkpoints keep failing (e.g. a persistent .dat flush failure whose
error is not EIO and so does not mark the volume read-only), the counter
increments on every write with no upper bound and wraps at ~4.3 billion.
Use saturating_add so it pins at u32::MAX instead, which keeps
checkpoint_due() true and retries on every subsequent write.
Co-Authored-By: Chris Lu <chris.lu@gmail.com>
* rust volume: only update max_file_key on live entries in idx metric rebuild
metrics_from_idx called maybe_set_max_file_key on every entry including
tombstones, but the live on_put path only calls it for puts and on_delete
never does. A tombstone always has a preceding put for the same key that
already set max_file_key, so the result is the same today; restricting it
to live entries makes the parity with the live path exact and self-evident.
Co-Authored-By: Chris Lu <chris.lu@gmail.com>
* rust volume: advance idx_file_offset only after redb commit succeeds
put() and delete() appended to the .idx file and advanced idx_file_offset
before committing to redb. If the redb commit failed, the offset included
the orphan row that redb doesn't reflect. A later checkpoint would record
that offset as "the table reflects up to here," and the reload would skip
the orphan row entirely — the entry becomes permanently unindexed.
Move the idx_file_offset increment to after the successful redb commit.
The .idx file still has the orphan row (append-only), but idx_file_offset
stays behind it, so the next checkpoint records the smaller offset and
the reload replays the orphan row back into redb.
Co-Authored-By: Chris Lu <chris.lu@gmail.com>
* rust volume: skip index checkpoint on close when .dat sync fails
Volume::close() discarded the .dat sync_all() result and always
checkpointed the redb index. If the .dat sync failed, the checkpoint
made the index durable with entries that may point past the unflushed
.dat tail, and a power loss would leave the volume read-only on reload
(the max_needle_end check fires).
Check the .dat sync result: on success, checkpoint as before; on
failure, call close_without_checkpoint() — sync the .idx and drop the
writer without a durable redb commit. META_IDX_SIZE stays at the last
successful checkpoint, so the reload replays the uncheckpointed tail
(redb still flushes on drop, but without recording idx_size).
Co-Authored-By: Chris Lu <chris.lu@gmail.com>
* rust volume: schedule checkpoints on every index mutation path
maybe_checkpoint_index was only called from do_write_request and
do_delete_request. put_needle_index and write_needle_blob_and_index
also call NeedleMap::put, which increments writes_since_checkpoint,
but neither triggered the checkpoint. Through those paths the counter
could grow past the interval without ever being satisfied, leaving
non-durable redb transaction state until close().
Add maybe_checkpoint_index(false) after the successful nm.put in both
methods. The .dat flush inside maybe_checkpoint_index covers the blob
write in write_needle_blob_and_index; put_needle_index pairs with a
prior write_needle_blob, so the flush covers that too.
Co-Authored-By: Chris Lu <chris.lu@gmail.com>
* rust volume: truncate orphan .idx row on failed redb commit
Commit 947ee28 moved the idx_file_offset increment after the redb commit
so a failed commit doesn't advance the watermark. But the .idx file is
append-only: the orphan row stays in the file, and the next successful
write appends after it. That write's idx_file_offset += entry_size
advances past the orphan, so a later checkpoint records an offset that
makes the reload skip the orphan row — hiding a persisted put or
restoring a deleted needle.
On a failed redb commit, truncate the .idx file back to idx_file_offset
before returning the error. This removes the orphan row, so the next
write appends at the correct position and idx_file_offset stays a
contiguous replay watermark. Add a truncate_to method to IdxFileWriter
(set_len for std::fs::File) and a truncate_idx_to_offset helper.
Co-Authored-By: Chris Lu <chris.lu@gmail.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: chrislusf <chris@chrislusf.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
The star_history workflow referenced actions/setup-python@v8, which
does not exist, causing the workflow to fail at the "Set up job" step.
Pin to v7, matching the version used across the other workflows.
The Admin UI documents its read-only account as view-only and blocks its
write requests, but the authenticated read routes returned object-store
users with plaintext access and secret keys. A read-only admin user could
retrieve another user's live S3 credential pair from GET /api/users and
GET /api/users/{username} and use it directly against the S3 endpoint,
converting view-only access into the victim identity's object-store
authority.
Redact the reusable secret_key in GetUsers, GetUserDetails, and the
rendered users page whenever the requesting session has the read-only
role. The public access_key identifier is retained so identities remain
browsable; only the reusable secret is stripped. Admin and no-auth
sessions are unaffected.
* fix(master): honour -volume.fileSizeLimitMB on the master's /submit - #6748
`weed server -volume.fileSizeLimitMB=2048` still refused anything over
256MB, and the reason is not the one the report assumes: the option does
reach the volume server. The master does not use it. Uploads through the
master's /submit are buffered by submitForClientHandler, which passed a
hardcoded 256MB to needle.ParseUpload, so the master rejected what the
volume server it started would have accepted.
The limit is now passed in. `weed master` gains its own -fileSizeLimitMB
with the same 256 default, so a standalone master behaves exactly as
before, and `weed server` and `weed mini` hand it the value their volume
server already got.
* master.follower: take the same upload limit, and say which flag to match
Review found the follower left behind. It serves /submit like the leader
and buffers uploads under the same limit, but kept the fixed 256MB, so a
cluster raised above that would accept an upload through the leader and
refuse the identical one through a follower.
Two smaller points from the same review: the master's flag description
named only the standalone volume server's spelling, and now names the
weed server and weed mini form too; and the under-limit test asserted on
the error message alone, so it would have passed had the limit rejected
that payload with different wording. It now requires the request to get
past parsing.
* master: delete replica_placement_mismatch labels when volumes leave topology
Fixes#10804. Setting the gauge to 0 left stale Prometheus time series
that grew unbounded with volume churn; remove the label set on unregister
instead.
* master: delete replica_placement_mismatch only after last placement leaves
Unconditional DeleteLabelValues on UnRegisterVolumeLayout dropped the series
while other data nodes still held the volume, hiding under-replication until
the next collect cycle. Delete only when Lookup is empty, and cover the
two-copy case in a regression test.
admin: bind to loopback by default, refuse public unauthenticated bind
The admin HTTP server (port 23646) defaulted to binding 0.0.0.0 with
authentication disabled when -adminPassword was not supplied, exposing
the full admin REST API (user creation, credential issuance, bucket
deletion, filer deletion) unauthenticated on the network. This is the
footgun described in GHSA-m3m8-mrgq-hf9h.
Keep the no-auth mode for local dev, but remove the network exposure:
- Add -ip flag (default 127.0.0.1) so the server binds loopback only
unless the operator explicitly chooses a public address.
- Refuse to start when binding a non-loopback address with no
-adminPassword and no [https.admin] mTLS. The operator must enable
auth or use loopback.
- weed mini sets -ip from its existing -ip.bind; the guard does not
apply because mini calls startAdminServer directly, not runAdmin.
Addresses GHSA-m3m8-mrgq-hf9h.
* s3: warn when a lifecycle change leaves fast-path-stamped objects on their old TTL
The per-write TTL fast path (opt-in via s3.bucket.lifecycle.fastpath)
stamps a volume TTL at PutObject time that can't be taken back. When an
operator lengthens or removes an Expiration.Days rule (or deletes the
bucket lifecycle) on a fast-path-enabled bucket, objects already written
keep their baked-in TTL and won't be rescued by the change — unlike the
default worker-driven path, which re-evaluates the current rules each
pass. This is the data-loss direction described in #11183.
Surface it: Put/DeleteBucketLifecycle now emit a glog warning and set
X-Seaweed-Lifecycle-Fastpath-Warning on the response when the change
removes, disables, lengthens, or re-scopes a fast-path-eligible rule.
Shortening a rule does not warn (old objects simply expire later, not
data loss). Tag-only and overflow-day rules are never on the fast path
and never warn.
Addresses the warning half of option 2 in #11183.
* s3: address review — emit warning after mutation succeeds, fix ID-rename false positive
Two issues raised by CodeRabbit, Greptile, and Devin reviews:
1. Failed mutations retained the warning header. The warning was set on
the ResponseWriter before storeBucketLifecycleConfiguration /
clearStoredBucketLifecycleConfiguration was called; if that failed,
the error response carried a warning for a change that was never
applied. Now the reason is computed before the mutation but the log
and header are emitted only after it succeeds.
2. Rule renames produced false "removed" warnings. fastpathRuleKey used
Rule.ID as the sole identity when present, so renaming a rule (same
prefix/size/days, different ID) treated the old rule as removed.
Replaced with two-pass matching: first by ID, then by fast-path
predicates (prefix + size). An ID-only rename with unchanged
predicates and days no longer warns. Greedy matching ensures each
new rule is consumed by at most one old rule.
Added regression tests: ID-only rename (no warn), rename + lengthen
(warn), rename + shorten (no warn).
* feat(s3): make RenameObject idempotent for a retried request - #10661
A rename that succeeds but whose response is lost leaves the client with
no safe move: retrying returned 404, because the source is already gone,
so a retry was indistinguishable from a rename that never happened.
The destination now carries what the rename that created it was, under
x-seaweedfs-rename-token: the client's token, the source key and the
time. A retry that names the same token and the same source and
destination is answered 200 without touching anything. The same token
sent for a different rename is refused with 409 rather than silently
answered, and a token older than 24 hours is treated as unrelated so a
key cannot answer for a request indefinitely.
Requests without the header behave exactly as before.
* s3: answer a reused rename token with 409, not 400
The PR promised Conflict and the code returned Bad Request. 400 tells a
client its request was malformed and invites it to give up; this request
is well formed and resending it unchanged will not help, because what it
collides with is a rename the same token already stands for.
The status code is now asserted in a test, since it is the part of this
behaviour a client actually acts on.
* Update weed/s3api/s3err/s3api_errors.go
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: fix rename token review notes
- ErrIdempotentParameterMismatch returns 409 Conflict, not 400. The
comment and TestRenameTokenReuseAnswersConflict both expect 409; the
code regressed to 400 in a later commit.
- stampRenameToken: clarify that markRenameToken mutates srcEntry in
place, so the token reaches the destination via the move regardless
of whether the UpdateEntry succeeds. The precondition only guards the
pre-move write, not the move itself.
- Extract the handler retry branch into retryRenameDecision and add
TestRetryRenameDecision, covering the source-still-exists fallthrough
that was previously reasoned about but not tested.
* s3: IdempotentParameterMismatch returns 400, matching AWS docs
The AWS S3 RenameObject API documentation specifies HTTP Status Code: 400
for IdempotencyParameterMismatch. Revert the previous 409 change and align
the comment and test with the documented behavior.
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
The Rust volume server opens one redb database per volume and built
each with redb's defaults, which give every database a 1 GiB page
cache (0.9 GiB read cache + 0.1 GiB write buffer). With hundreds of
volumes behind one disk the process-wide ceiling was volumes x 1 GiB:
memory grew in proportion to the pages traffic touched, never shrank
when traffic stopped, and hosts running many instances were OOM-killed
under bulk ingest. redb, redbMedium and redbLarge were also treated
identically, so the "memory~performance" tiers did nothing.
Size the cache per tier instead: 4, 8 and 16 MiB per volume, mirroring
the Go server's 3/6/12 MiB LevelDB block cache + write buffer. Thread
the budget through RedbNeedleMap::new/load_from_idx so every open path
(create, reuse, full rebuild) uses Database::builder().set_cache_size.
Fixes#11179
Claude-Session: https://claude.ai/code/session_019x36FiSeyePh77YXao15kK
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
No source changes: the API surface the needle map uses (Database
create/open/builder, set_cache_size, set_durability, tables, iterators)
is unchanged and the on-disk format is still v3, so existing .rdb files
open as-is. The 4.0.0 breaking changes (Drop on AccessGuardMut, removal
of the Legacy type) do not touch this crate.
Relevant to the redb-backed index (#11179):
- 4.1.0: optimizes cache usage and memory usage; ~1.5x faster writes.
- 4.2.0: Durability::None commits ~2x faster; pages freed by a durable
transaction are reused by the very next one; a crash-recovery fix for
a crash during repair of an earlier crash.
Claude-Session: https://claude.ai/code/session_019x36FiSeyePh77YXao15kK
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Change the matplotlib figure size from (10, 4) to (10, 6) so the
star history chart renders vertically longer in the README. The
regenerated note/star_history.svg reflects the new 5:3 aspect ratio
(720x432pt) instead of the previous flat 2.5:1 (720x288pt).
The "Mount and exercise" step left the weed.exe mini server and the final
WinFsp mount running when it exited. The next step's pwsh.exe then failed
with STATUS_DLL_INIT_FAILED (0xC0000142), failing a job whose actual test
step had passed. The same code passed on both the PR branch and the next
master run, so this was a transient launch failure — but it was caused by
an unclean environment and made fatal by a diagnostic step.
Tear down all weed.exe processes at the end of the test step so subsequent
steps launch into a clean environment, and mark the Logs step
continue-on-error so a diagnostic step can never fail the job on its own.
2026-09-05 00:10:40 -07:00
Chris LuGitHubdevin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* add a per-mount cache_wait_ms to the remote storage mount mapping
A read of an uncached remote-only object waits on a hardcoded size tier
before it can fall back to the origin, so every ranged read of a large
remote-only object pays that wait. Carry the wait in the mount mapping so
it can be tuned, or set to zero, per mount.
* resolve the cache wait of an uncached remote-only read from its mount
The wait came only from the object size, so an operator could not trade
cache hits for time to first byte. Both read paths now resolve the mount
covering the object and let its cache_wait_ms replace the size tiers.
* read straight from the remote when a mount waits zero for its cache
A mount used as a streaming source pays the cache wait on every ranged
read of an object too large to finish caching, and the caching itself is
wasted work. A zero wait now skips the cache call, so both read paths go
to the origin immediately.
* let remote.mount set the cache wait of a mount
remote.mount -cacheWait=0 turns a mount into a streaming source, and any
other duration trades cache hits against time to first byte.
* keep the size based wait for a version-specific read
A read pinned to a version cannot fall back to the origin, since the
mounted remote only holds the current key, so a mount that opts out of
caching would leave it on the 503 retry loop forever.
* let the operator allow a remote-only read to dial an internal endpoint
The remote-mount read paths in the filer and the S3 gateway always refused
an endpoint resolving to a loopback or private host, so a mount backed by
an internal S3 could never be read from its origin, only through the local
cache. Both now take the allowance the volume server already has, still
off by default.
* skip the background cache of a mount that waits zero for its cache
GetObjectHandler kicks off caching for every remote-only read, so a mount
serving as a streaming source kept downloading whole objects even though no
read ever waited for them.
* cover a zero cache wait end to end
The read has to reach a real origin, so the harness also opts the filer and
the S3 gateway into dialing the loopback remote it already allows for the
volume server.
* resolve the S3 cache wait once so the background cache follows it too
The background cache that GetObjectHandler starts read the mount on its
own, so it skipped a version-specific read that the foreground path still
waits for. Both now ask the same resolver.
* answer 404 when the origin of a zero-wait read is gone
Metadata can outlive the object it points at, and with no cache to fill
the read would sit on the 503 retry path forever. The remote backends
already report a missing object as ErrRemoteObjectNotFound.
* open the origin at write time for a multipart range
Every part of a multipart Range is prepared before any is written, so
opening eagerly would hold one origin connection per part and leak the
ones already opened when a later part fails to open.
* reject a cache wait shorter than a millisecond
The mapping stores milliseconds, so -cacheWait=500us truncated to zero
and silently turned caching off instead of waiting.
* restore the doc comment of cacheRemoteObjectForStreamingWithShortTimeout
Extracting the wait resolver left its comment on the new function.
* stat the origin before committing a multipart range
Opening at write time keeps no connection through the preparation, but it
also moved a failure past the point where the multipart body picks the
response status, so a gone origin truncated a 206 instead of answering
404. One stat up front puts the status back.
* stat the origin once per request
Every part of a multipart Range is prepared on its own, so the preflight
ran once per range instead of once per read.
* map Azure and GCS stream not-found to ErrRemoteObjectNotFound
ReadFileAsStream on Azure and GCS returned provider-specific not-found
errors instead of ErrRemoteObjectNotFound, so a zero-wait read of a
deleted object was misclassified as a transient cache failure and
retried indefinitely. Map BlobNotFound and ErrObjectNotExist the same
way StatFile already does.
* Update weed/remote_storage/gcs/gcs_storage_client.go
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: count a TTL volume's age from its last write, not the .dat mtime
A delete appends a tombstone needle and vacuum rewrites the .dat wholesale,
so the file's mtime moves without any write ever landing. The loader read
lastModifiedTsSeconds back from that mtime, so every restart of a volume
taking delete traffic re-armed expired() for another full TTL: an
overwrite-heavy collection kept growing until it hit the max-volume cap.
Recover the clock from the newest .idx entry that is not a tombstone and
read that needle's append timestamp, falling back to the mtime when no
write is recoverable. Only TTL volumes pay for the scan.
Fixes#11160
* volume: count the .vif destroy time from the last write too
ExpireAtSec is what an EC volume is reclaimed on, and it was recomputed as
now+TTL every time the .vif was written. A read-only mark, a tier upload or
an EC encode therefore handed an already expiring volume another full TTL,
the same way the .dat mtime did.
Derive it from the volume's last write, falling back to now for a volume
that has not taken one yet so a fresh volume is not born expired.
* volume: mirror the last-write TTL clock in the Rust volume server
Same recovery as the Go loader: scan the .idx backwards for the newest
entry that is not a tombstone and take that needle's append timestamp,
leaving the clock on the .dat mtime when no write is recoverable.
* volume: mirror the last-write destroy time in the Rust volume server
Both .vif writers and the EC encode computed ExpireAtSec as now+TTL, the
same way Go did, so the destroy time moved every time the sidecar was
rewritten. Route all three through the volume's last write.
* volume: report the .dat mtime in the Rust heartbeat, like Go does
The Rust server reported its TTL clock as ModifiedAtSecond while Go
reports the .dat mtime. The shell's quiet-period gates (volume.tier.move,
volume.delete_empty) read that field as "last touched", which a delete
has to count towards even though the TTL clock deliberately ignores it --
and with the clock now recovered from the last write, the two drift
further apart.
* volume: take the newest write by timestamp on a vacuumed volume
The reverse .idx scan trusted position, which holds only while the .dat is
append ordered. Vacuum rewrites it in key order, and since an overwrite
keeps its original key, the highest-key survivor is not necessarily the
newest write -- the recovered clock could land up to a TTL early and take
the volume with data still inside its TTL.
A volume that has been vacuumed (CompactionRevision > 0) now takes the
maximum append timestamp over a bounded window of write entries instead.
An append-ordered volume still answers in one read.
* volume: never guess a vacuumed volume's last write, and resolve wrapped offsets
Two holes in the reverse scan, both from review:
A vacuumed volume's writes are ordered by key, so any of them can hold the
newest timestamp. Reading a capped window sampled the highest keys, which
could still miss a recently overwritten low-key needle and expire data
inside its TTL. The scan now covers every write a vacuumed volume indexes,
and a volume too large to scan keeps the .dat mtime rather than report a
partial maximum -- late is recoverable, early is not.
A .dat past MaxPossibleVolumeSize wraps the offsets in its .idx, so reading
a timestamp at the unwrapped offset picks up an unrelated needle. Resolve
the entry against the needle header first and retry one volume size in,
the way doCheckAndFixVolumeData already does.
* volume: drop GitHub issue references from TTL comments
* docs(readme): replace star-history.com with self-generated chart
The star-history.com SVG is a third-party dependency that can rate
limit or go down. Replace it with a GitHub Action that fetches
stargazers via the REST API and renders an SVG with matplotlib,
committing note/star_history.svg weekly. The README references the
committed file directly, so the chart has no runtime dependency on
any external service.
* ci(star-history): run daily instead of weekly
starchart.cc is rate-limiting the SVG endpoint, so the Stargazers
chart renders blank. Switch to star-history.com, which serves a live
SVG for this repo and links to the interactive chart.
The Patreon CTA and Gold Sponsors logos sat between the logo and the
project intro, pushing the actual description below the fold. Move the
whole block to a dedicated `# Sponsors #` section after `# License #`,
add it to the TOC, and give it a real markdown heading so the anchor
works on GitHub.
* mount: let volumeName take an explicit override
volumeName only ever derived the disk's label from -filer.path, -dir,
or the filer address, so a name that happened to collide with
something else - e.g. a UNC share's own name - could not be changed
without moving what was mounted. Give it an override parameter that
wins over all three; nothing passes one yet.
* mount: add -volumeName to name the disk explicitly
Windows has no equivalent of the "weed fuse" -o passthrough that lets
a Linux or macOS mount override its derived volname, so a name picked
up from -dir - e.g. a UNC share's own name - could not be changed
short of moving what was mounted. -volumeName overrides it on every
platform.
* mount: document -volumeName
* mount: scope -volumeName's help text to macOS and Windows
Linux has no volume-label mount option for -volumeName to feed, so
the flag's own description says where it applies instead of leaving
that unstated.
* mount: forward -volumeName through the weed fuse option parser
weed fuse (the /etc/fstab helper) turns -o key=value into the same
MountOptions weed mount takes, but volumeName had no case, so it fell
through to being forwarded as a literal, unrecognized FUSE option
instead of ever reaching mountOptions.volumeName.
* mount: apply -volumeName to FsName on Linux and FreeBSD
FsName only ever took the filer address and -filer.path, so
-volumeName had nothing to override there and silently did nothing;
the skipAutofs case still forces "fuse", since that name is what
util-linux/mount requires to recognize the pseudo filesystem.
* filer: guard the FoundationDB value size limit, not the transaction limit
An entry's whole chunk list is one FoundationDB value, and FDB caps a value at
100,000 bytes while a transaction may reach 10MB. UpdateEntry checked the
transaction limit, so every entry between the two limits passed the guard and
was rejected by FDB itself with error 2103 (Value length exceeds limit). The
failure surfaced inside the store rather than at the guard, so the S3 layer
dropped the connection and clients saw a network fault instead of an error.
Check the value limit in UpdateEntry and KvPut instead, after gzip and before
the transaction, with an error that names the limit it hit. The removed
transaction-size constant guarded nothing else: DeleteFolderChildren batches by
entry count.
Refs #11158
* filer: fold at 500 chunks in the foundationdb build
Manifest packing is what keeps a large file's entry small, but it only ran once
a flat chunk list reached 10000 chunks. A FoundationDB value stops at 100,000
bytes and an entry's whole chunk list is one value, which at ~100 bytes per
chunk record is about 1000 chunks -- so on FDB the write always failed before
packing could help: a 3.3 GiB PutObject at the default -maxMB=4 was already
past the limit.
FoundationDB support is its own build (`go build -tags foundationdb`, shipped
as its own image), so the batch is a build-time choice and needs no negotiation
at run time. The tagged build folds at 500, every other build keeps 10000 and
is untouched.
500 is not arbitrary: a single fold level leaves (chunks/batch) manifest
pointers plus up to (batch-1) unfolded chunks in the entry, so the reachable
chunk count is highest when the two terms are near equal. For a 100,000-byte
budget that optimum is 500, which holds an entry inside the limit up to
~250,000 chunks -- ~1 TB at -maxMB=4, against ~4 GB before. Larger files need
nested packing, which no batch size substitutes for.
One binary serves every role in that image, so the filer and each client that
folds -- S3, mount, WebDAV, weed shell, filer.copy -- agree on the batch by
construction. A binary built with the tag but pointed at another store folds
earlier than that store requires, costing one manifest blob per 500 chunks and
one read to resolve it.
Fixes#11158
* filer: fold with rollback inside MaybeManifestize, not beside it
A fold that fails midway has already uploaded manifest blobs for its earlier
batches, and returns only the data chunks -- dropping the manifests it had
separated out of the caller's list. Both were wrong in ways that mattered:
- AppendToEntry assigned that shortened list straight to entry.Chunks and
created the entry, so an append to an already-folded file whose fold
failed lost every previously folded chunk. weed mount had the same shape.
- cleanupChunks logged the error as "not good, but should be ok" and then
returned it through a named result, failing the whole CreateEntry or
UpdateEntry, while the blobs it had written stayed behind referenced by
nothing.
The S3 path was alone in handling this, through a private helper beside
MaybeManifestize. A second entry point next to the one everything else calls
just means the wrong one gets used, so the behaviour moves inside
MaybeManifestize: on failure it returns inputChunks as it received them, and
hands the blobs it saved to a deleteChunks callback. The filer, S3 and
filer.copy pass their existing deleters -- filer.copy already cleans up this
way after a failed upload -- and mount, WebDAV and weed shell pass nil, which
reports the blobs rather than collecting them, as before. Each caller keeps its
own error policy: the filer HTTP PUT path and filer.copy still fail the request,
the rest still continue with the flat list, which is a correct entry.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3: HEAD with partNumber reports the part's size and range
HeadObject set its headers from the total object size and then only
validated the partNumber, so a client probing part 1 with HEAD got the
whole object's Content-Length and a 200 while the same GET returned the
part's size, a Content-Range and a 206.
Resolve the part's byte range before the headers are written, through the
range logic GetObject already used, and answer a partNumber HEAD as the
ranged HEAD that AWS documents.
* s3: answer an unsatisfiable partNumber with 416 InvalidPartNumber
GET and HEAD rejected a partNumber past the number of parts with 400
InvalidPart, the code for a missing part in CompleteMultipartUpload. AWS
answers a read of a part that does not exist with 416 InvalidPartNumber,
which lets a client probing for the part count tell the two apart.
The ceph suite pins RGW's 400 InvalidPart here, so the s3tests jobs patch
that expectation the way they already patch prefix ordering.
* s3: keep the whole-object checksum off a partNumber response
The stored checksum covers the whole object, so it is already withheld
from a ranged read. A partNumber HEAD now describes one part while the
request carries no Range header, so exclude it there too rather than
handing a client a checksum that does not match the bytes described.
* s3: resolve a partNumber against the parts the object records
Completion accepts ascending, not consecutive, part numbers, so the part
count is not the highest part number. Comparing the two rejected an
uploaded part 3 of a two-part object, and let a request for the absent
part 2 fall through to the positional chunk lookup and serve part 3's
bytes. Ask the recorded boundaries for the part instead, and keep the
count comparison for objects written before boundaries were stored.
* s3: apply a client Range within the part on HEAD too
GET narrowed the part by a Range sent alongside partNumber; HEAD reported
the whole part, so the two disagreed again for a request that carries
both. Move the narrowing into the shared range lookup so either verb
describes the same bytes.
* fix(chart): serve S3 internal gRPC with mTLS when security enabled
The security.toml generated by the chart has no [grpc.s3] section, so
security.LoadServerTLS(viper, "grpc.s3") returns nil in weed/command/s3.go
and the S3 server listens plaintext on its gRPC port (httpPort+10000 = 18333
by default). Workers dial that port with mTLS credentials (grpc.worker),
producing:
walker dispatch ...: rpc error: code = Unavailable desc = connection
error: desc = "transport: authentication handshake failed: tls: first
record does not look like a TLS handshake"
This breaks the s3_lifecycle worker's LifecycleDelete RPC path (recovery
walk, daily replay) and any S3->S3 IAM cache propagation would fail the
same way if clients enforced TLS.
Add [grpc.s3] reusing the client cert already mounted on s3 pods (or
s3.tlsSecret when set, mirroring the seaweedfs.s3.tlsArgs helper for the
HTTPS listener).
Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf)
* fix(chart): always use internal client cert for grpc.s3 identity
s3.tlsSecret is the public HTTPS listener certificate (possibly issued by
a public CA); internal gRPC peers only trust grpc.ca, so presenting it on
the internal gRPC port would break lifecycle/IAM RPC verification. Keep
the two trust domains separate.
Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.8-Flash-Next-ROCmFP4)
golang.org/x/image v0.44.0 is affected by CVE-2026-46603 (GO-2026-6222):
a denial of service via excessive memory allocation when decoding
malformed VP8L (lossless WebP) data. It is fixed in v0.45.0, released
2026-08-11.
The decoder is reachable from SeaweedFS: weed/images/resizing.go
blank-imports golang.org/x/image/webp, which registers the VP8L decoder
with image.Decode, so the filer image resizing path decodes attacker
supplied WebP data with the affected version.
This is a go.mod/go.sum only change produced by
`go get golang.org/x/image@v0.45.0 && go mod tidy`; no other dependency
moved. `go build ./weed/`, `go vet ./weed/images/...`,
`go test ./weed/images/...` and `go mod verify` all pass.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* filer: require a read token for the root listing
maybeCheckJwtAuthorization waved through every GET/HEAD on "/", so a filer
with jwt.filer_signing.read.key set still served its root directory listing --
entry names, sizes and chunks[].file_id -- to a caller holding no token at
all, and served the same listing to a token restricted by allowed_prefixes.
The exemption was added for health checks before the filer had /healthz and
/readyz. Both are registered on the default and read-only muxes ahead of the
"/" handler and answer without a token, so drop it.
Point the mTLS harness at /healthz, which is what it was probing for.
* filer: keep the jwt query parameter out of a proxied chunk request
The proxy stripped "jwt" from the forwarded query on reads only, on the
grounds that a writer's own credential travels there. It does not: an
uploader carries its AssignVolume token in the Authorization header, and the
query parameter on this path holds a filer credential.
Strip it for every method. A volume server has no business seeing a filer
token, and because security.GetJwt reads the query before the header,
relaying one would hide the writer's own token behind it.
* filer: dispatch the chunk proxy after the JWT gate
The ?proxyChunkId= branch returned before maybeCheckJwtAuthorization ran, so
GET, PUT, POST and DELETE against any needle in the cluster were reachable on
the filer's HTTP port with no filer credential, on a filer where every other
request answered 401. An anonymous caller read a stored object, replaced its
bytes, or deleted the needle, which the master's next vacuum makes permanent.
#10434 stopped the filer from minting a volume write token for that caller,
which closes the write half only where the volume server has a jwt.signing.key
of its own -- not the shipped default, and not what scaffold/security.toml
recommends for a filer deployment. The read half stayed open in every
configuration, because the filer mints the read token itself.
Move the dispatch below the gate. A file id carries no path, so a token
restricted by allowed_prefixes cannot be scoped against one and is refused
here; every consumer of this endpoint holds an unrestricted token.
* filer: mint the volume credential for a proxied write too
The proxy minted a volume token on reads and forwarded whatever the caller
sent on writes. #10434 made it that way because the branch ran ahead of the
JWT gate, so a token minted here would have been signed for an unauthenticated
caller; the branch now runs behind the gate, and the credential the caller
presents there is a filer one, which a volume server cannot validate and has
no business seeing.
Mint at the access level the request needs, and drop the caller's
Authorization when there is no key to mint from. A proxied uploader then needs
only the filer credential, instead of holding one for each hop with a single
header to put them in.
* mount, mq, filer.sync: send the filer credential for a proxied chunk
Every in-tree consumer of ?proxyChunkId= reached the filer anonymously: mount
and the broker put the AssignVolume token in the Authorization header, which
is a volume credential, and filer.sync sent nothing at all. That was enough
only while the branch ran ahead of the filer's JWT gate.
Build the URL through one helper, and pick the credential from the URL it
returns: a chunk proxied through a filer is a request to the filer, which
authorizes it and attaches the volume credential itself, so the token there is
a filer one at the access level the request needs.
* filer: honor -exposeDirectoryData
The flag was declared on all three commands that start a filer and read by
none of them: FilerOption.ExposeDirectoryData was only ever assigned from
filer.expose_directory_metadata in security.toml, so -exposeDirectoryData=false
silently left the listing exposed. Only the TOML key had any effect.
Plumb the flag through and let either switch turn the listing off.
* filer: count a proxied chunk request once
Moving the dispatch below the gate put it after the deferred request
observation, so every proxied chunk now landed in FilerRequestHistogram twice,
once under its HTTP method and once under chunkProxy. Name the deferred one
after the proxy instead, the way the unsupported-method branch already does,
which also gives the endpoint the status codes FilerRequestCounter records.
* filer: do not 404 a TUS session on a transient chunk-load failure
readTusSessionInfo already proved the session exists before
loadTusSessionChunks is called, so a failure there is a read failure,
not evidence the session is gone: a volume-server timeout or a
canceled request context surfaces through ListDirectoryEntries the
same way a missing session would.
Every such error was mapped to writeTusSessionNotFound, answering 404
to HEAD/PATCH and 204 to DELETE. A spec-compliant TUS client trusts
that and discards the session, orphaning every chunk it had committed
until the 24h expiry sweep, or forever if it never issues a DELETE.
Only an error matching filer_pb.ErrNotFound is now reported as not
found; anything else answers 500 so the client retries against the
same session instead of abandoning it.
* test: cover a TUS session's transient chunk-load failure
Adds a listErr hook to the in-memory test store, alongside the
existing commitErr/deleteErr, to simulate a store or RPC failure from
ListDirectoryEntries.
HEAD, PATCH and DELETE against a live session all answer with a
server error instead of a not-found status when the chunk listing
fails transiently, and the session is left on disk untouched. A
listing failure that genuinely means not found, filer_pb.ErrNotFound,
still answers 404 (204 for DELETE).
* filer.remote.sync: skip an upload whose source entry was deleted or rewritten
A replay from an earlier offset (-timeAgo) re-emits create and update
events for entries the filer has since deleted or rewritten. Their chunks
are gone from the volume servers, so the upload can never succeed, and
failing the event holds the sync offset before it: every restart of the
subscription replays it into the same dead chunks, and progress on
everything after it in the log is never persisted. One such entry stops
replication for the whole mount.
When the upload fails, look the entry up on the filer. Gone, or holding
other content than the event described, the event is superseded and is
skipped with an error log; the event that superseded it follows in the
log and brings the remote to the current state. Otherwise the failure
stands and the event is retried as before.
Fixes#11148
* filer.remote.sync: compare chunks by file id when deciding an event is superseded
filer.IsSameData compares chunk ETags, so a delete-and-recreate of
identical bytes, which stores the same content under new file ids and
drops the old ones, looked still as described and kept failing the event
on its dead chunks. Compare by file id with DoMinusChunks, the way the
filer itself decides which chunks an update leaves for deletion: the
event is superseded when the current entry no longer references every
chunk it named, and still as described when it does, including when more
chunks were appended after it.
* filer.remote.sync: ask the filer on the first failed upload attempt, not after the backoff
The superseded check ran after util.Retry had given up, so every dead
entry still cost the full retry cycle, about 13s, before it was skipped:
the SDK reports a missing chunk as "RequestError", which
IsTransientError takes as worth retrying. Move the check into the retry
loop with util.RetryOnError. Any failed attempt asks the filer, and the
loop stops at once when the entry is gone, surfacing errSuperseded for
the caller to skip. An entry the filer still holds keeps the retry policy
it had.
filer.remote.gateway shares retriedWriteFile and the same offset-pinning
processor, so its three call sites skip a superseded event the same way.
* filer: serve "//" paths at the cleaned path instead of redirecting
http.ServeMux redirects a non-canonical path ("//", "..") to its cleaned
form, but since Go 1.22 it builds the Location from the already-escaped
path, so it is percent-encoded twice (golang/go#79897). A client that
follows the redirect re-posts "/负极全景" as "/%25E8%25B4%259F...", and
the filer stores a directory literally named "%E8%B4%9F...".
Wrap the filer muxes in CleanPathHandler, which rewrites the request to
the same cleaned path ServeMux would have redirected to and dispatches
directly. The decoded name reaches the handler, the round trip goes
away, and clients that do not follow redirects work too.
Fixes#11125
* filer: keep RequestURI in step with the cleaned path
PostHandler derives storage rules, the bucket and the read-only check from
r.RequestURI while writing the entry at r.URL.Path. After CleanPathHandler
rewrote only the URL, a "//" or ".." request would be placed by the raw
path and written to the cleaned one. Rewrite RequestURI too, as the
redirect-following client used to.
* filer: match storage rules on the decoded write path
PostHandler resolved the storage rule from r.RequestURI, the raw
request-target. Clients percent-encode non-ASCII segments on the wire, so
a read-only or TTL rule configured on "/data/只读/" never matched a POST
to "/data/%E5%8F%AA%E8%AF%BB/" and the write went through. Use r.URL.Path,
the decoded path the entry is actually written to, as the header-based
destination check already does. The query string no longer reaches the
rule lookup, so the "?" trimming in the read-only error is gone.
2026-09-04 00:02:33 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: keep new volumes and writes off servers in maintenance mode
The master recorded a volume server's maintenance flag from the heartbeat
but never consulted it. A server in maintenance (#7977) is being drained,
yet the master kept creating volumes on it whenever it had free slots and
kept handing out its volumes for writes. Nothing on the volume server
blocks plain HTTP uploads either, so "read-only mode" was only a name.
Volume growth: a data node in maintenance mode reports zero free slots
through AvailableSpaceFor, which takes it out of every candidate list,
feasibility count and capacity reservation. Its slots still roll up into
its rack and data center, so the random offset drawn from those totals for
an other-rack or other-DC replica could land in space the walk then skips
and fail with "No free volume slot found!" while siblings had room; the
walk now folds the offset into the space that is actually eligible. This
also covers the pre-existing case of an over-committed sibling.
Assignment: a replica on a server in maintenance mode is treated like a
read-only replica in isAllWritable, so its volume leaves the writable
list and returns when the flag clears. Topology.SetDataNodeMaintenanceMode
re-evaluates the node's volumes on every change, since heartbeats are
digest-based and a full volume list may not follow for a long time. Reads
and lookups are untouched. The flag moves to an atomic so the assign and
growth paths can read it without the node lock.
Heartbeat: the Go volume server sent its state only when it changed, so a
master elected while a server sat in maintenance never learned about it.
The state now rides along on every heartbeat, as the Rust server already
does; the master's compare is an atomic swap, and only a change does work.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: hold maintenance mode through vacuum commit and mark-writable
SetVolumeAvailable and SetVolumeWritable put a volume back on the writable
list on the replica count alone. A vacuum that started before the server
entered maintenance, or a vacuum worker's mark-writable arriving after it,
handed the volume back to assignment with a replica on the draining server.
Heartbeats carry only changed volumes, so nothing re-evaluated it until the
volume itself changed.
Apply isAllWritable on both paths, the same test EnsureCorrectWritables
uses. Also pin that re-evaluating a volume a concurrent disconnect already
removed from its layout is a no-op.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: record a server's read-only notification on its node before judging the volume
A volume server notifies the master the moment it flips a volume between
read-only and writable, ahead of the heartbeat that repeats the flag. The
layout only set its per-location flag, so isAllWritable, which reads the
node's heartbeat copy, still saw the old value: a mark-writable was
withheld until the next heartbeat, and a re-evaluation landing between a
mark-readonly and its heartbeat put the volume back on the writable list.
Record the flag on the node's volume first. AddOrUpdateVolume keeps the
digest and the active volume count in step, so the heartbeat that follows
finds nothing to change.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: a read-only mark does not confirm a provisional volume
DataNode.SetVolumeReadOnly went through Disk.AddOrUpdateVolume, which
treats its input as a server report and so ended the grace period that
keeps a just-grown volume safe from a full report collected before the
grow. A volume marked read-only before its first report could then be
removed by that stale report.
Give Disk a SetVolumeReadOnly that flips the flag and keeps the digest and
active volume count in step without touching volumeAddedAt.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-03 23:50:12 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.remote.sync: confirm a missing RemoteEntry against the filer before re-uploading
The event is the entry as it was when the update was logged. A chmod or
utimes right after a write is logged while the sync is still uploading the
write, so it carries no RemoteEntry even though the object is on the remote
by the time it is processed. Gating on the event alone turned every such
update into a delete and a second upload of the same bytes; cp -p, rsync
and Django's FileSystemStorage all write that way.
Look up the filer's current entry when the event has no RemoteEntry: the
upload stamps it as soon as it completes, so the stamp is there for the
race and absent for a file that was never replicated. Skip the update when
the entry has since been deleted rather than upload from chunks that may be
gone; the delete event that follows removes the remote object.
Tests build entries from chunks, which is what IsSameData compares in
production, and cover both no-RemoteEntry cases through a stub filer.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.remote.sync: do not delete the remote object before overwriting it in place
The update write path deleted the old object and then wrote the new one,
even when both are the same key. S3, GCS and Azure all overwrite on write,
so the delete bought nothing and left the remote with no object between
the two calls, or at all if the write then failed and pinned the offset.
On a versioned remote bucket it also left a delete marker per rewrite.
Delete only when the key changes, which is what the delete was for.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.remote.sync: trim comments
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-03 19:14:34 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Maintenance mode exists to fence a volume server so it can be evacuated
without taking new writes (#7977), but the gate added in #8115 also
rejected the RPCs evacuation issues against the source: VolumeMarkReadonly
(the first step of every move, and the failure reported in #11066),
VolumeDelete (the last step), and VolumeEcShardsDelete (the last step for
EC shards). volumeServer.evacuate, volume.move and ec.balance therefore
all failed on exactly the server they were meant to drain.
Those three RPCs only remove data or restrict the server further, the same
class as DeleteCollection and the unmount RPCs that were never gated, so
they are exempted from the maintenance check in both the Go and Rust
volume servers. Everything that adds data or reopens the server for
writes (AllocateVolume, WriteNeedleBlob, BatchDelete, VolumeCopy,
ReceiveFile, EC generate/copy/rebuild, vacuum, tiering, VolumeMarkWritable)
stays blocked. A side effect is that scrub can now fence broken volumes
readonly on a server already in maintenance.
Fixes#11066
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
An entry whose content is rewritten unchanged before it first reached the
remote took the metadata-only branch, and UpdateFileMetadata returns early
when the extended attributes match without checking that the object is
there. shouldSendToRemote had already reported the entry as needing to be
sent, so the effect was that it stayed local for as long as its content did
not change, with the sync reporting healthy progress over it.
Require RemoteEntry to be set before treating an update as metadata-only.
Gating at the caller covers the S3, GCS and Azure clients, which share the
same early return.
Fixes#11139
2026-09-03 17:42:40 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Every image-signing job has failed since signing was added (#11129):
must provide --new-bundle-format or --bundle where applicable with
--signing-config or --use-signing-config
Cosign 3 turned on two defaults, not one. The action only disabled
--new-bundle-format to keep the .sig tag layout, but --use-signing-config
is still on, and cosign refuses that pairing because the signing-config
path has nowhere to write its verification material without a bundle.
Disabling it too falls back to the default Fulcio and Rekor URLs, the
same services the .sig layout always used. The verify step needs no
change: cosign verify looks for a referrer bundle first and falls back
to the .sig tag when there is none.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* storage: make DeleteVolume errors inspectable with errors.Is
An absent volume wraps ErrVolumeNotFound and an only-empty refusal now
wraps ErrVolumeNotEmpty with %w instead of %v, so callers no longer have
to match on the message.
Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm
* volume server: return NotFound and FailedPrecondition from VolumeDelete
An absent volume maps to codes.NotFound and a non-empty volume under
only_empty to codes.FailedPrecondition, so a caller retiring a volume can
treat NotFound as already done. The store message is kept in the status
description because the EC empty-replica sweep still matches on it.
Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm
* wdclient: add LookupVolumeIdsAuthoritative
Bypasses the vid map and asks the provider directly, for callers where a
stale positive location is unsafe.
Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm
* filer: add LookupDirectoryEntries batch lookup RPC
Up to 4096 exact-path lookups in one call, resolved concurrently with
results in request order, plus one deduplicated location lookup for every
volume the returned entries reference and per-fid read tokens when the
filer signs reads. unavailable_volume_is_miss lets cache-style callers
take an entry whose volume has no live location as a miss, resolved
against the master rather than the filer's location cache.
Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm
* filer: test that an expired file entry is deleted on read
Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm
* filer: test that AssignVolume and CreateEntry resolve the same TTL rule
Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm
* master: refuse partial lookups while warming up
LookupVolume returned Unavailable during warm-up only when every requested
volume was missing. A batch mixing a reported volume with one whose server
has not reconnected yet came back as a partial answer with a per-volume
not-found, which a caller treating the master as authoritative reads as
gone. Any not-found during warm-up is now Unavailable, which callers
already retry.
Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm
* filer: build batch test requests instead of copying a proto message
Copying a generated message copies its internal mutex, which go vet's
copylocks check rejects.
Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm
* filer: match ErrNotFound with errors.Is and state the miss rule's contract
A wrapped not-found from the store would otherwise be reported as an
error rather than a miss. The comments now say why a nil location map is
the only sign of an unanswered lookup: the provider returns nil when it
got no answer and a populated map, with unserved volumes reported as
errors, when the master did answer.
Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm
* volume server: map absent and non-empty VolumeDelete errors in the Rust server
Matches the Go server: an absent volume is NotFound and an only_empty
refusal is FailedPrecondition instead of Internal, with the messages the
EC empty-replica sweep matches on.
Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm
* filer: test that a malformed entry keeps its error outside cache mode
Same test file as the enterprise tree, so the next sync sees one version.
Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm
As an untyped constant it became int when passed to Infof, which
overflows on linux/386 and failed the 32-bit vet job. Every field it is
compared against is already uint64.
Claude-Session: https://claude.ai/code/session_015rYAmF8hV9yypb9yvy4A1z
* telemetry: tidy the server module after the protobuf bump
Claude-Session: https://claude.ai/code/session_01VGiphDxpsKMwu9XpUFhybQ
* telemetry: keep only clusters that store at least 10 GiB
Fresh weed server runs, CI jobs and throwaway containers each mint their
own cluster id. They came in at tens of thousands a day, were most of
the counted clusters and held almost none of the bytes, and the state
file and the metrics page grew with every one of them. Reports under
the floor are counted and dropped, and a state file written before the
floor sheds them on the first restart.
Claude-Session: https://claude.ai/code/session_01VGiphDxpsKMwu9XpUFhybQ
* master: report telemetry only once the cluster stores 10 GiB
A throwaway cluster no longer registers itself with its first report a
minute after start; a real one begins reporting at the first daily tick
after it crosses the floor.
Claude-Session: https://claude.ai/code/session_01VGiphDxpsKMwu9XpUFhybQ
* ci: composite action that signs and verifies an image with cosign
Keyless, by digest, with a verification pass against the calling workflow's
own identity right after signing. Signatures use the .sig tag layout rather
than the OCI-referrer bundle cosign 3 writes by default, since that is what
the verifiers people run today read. Dependabot is pointed at the action so
the cosign-installer pin keeps moving.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker release: sign every variant on both registries
The merge job signs each variant's multi-arch index on GHCR and Docker Hub
once the tag exists, recursively so the platform images are covered too.
latest re-tags the same manifest and inherits the signature.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker dev: sign the dev image
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker latest: sign a latest rebuilt by hand
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker release: sign the foundationdb image
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker: sign the per-version foundationdb and rocksdb builds
They push to the same repository as the releases, so an admission policy
that verifies chrislusf/seaweedfs would otherwise reject them.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker: document image signature verification
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* ci: pin the actions the signing jobs newly run by commit
These run with registry credentials and the OIDC token that signs under
the repository's identity, so a retargeted tag upstream must not be able
to reach them.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker latest: pass the dispatch tag through env, not the script
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker: complete Kyverno policy, digest note, identity scope
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker latest: keep the dispatch tag out of the manifest script too
The step predates signing, but the job now holds the OIDC identity.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* ci: pin every action in the jobs that sign
The jobs that hold the OIDC identity run these with registry credentials,
so a retargeted tag upstream must not reach them.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker release: copy and sign the digest the run created, pin the rest
crane copy and the signature both resolved the tag, which another
publisher could move between the two steps. The index digest is read once,
right after it is created, and the Docker Hub copy and both signatures use
it. The manual latest rebuild gets the same treatment. The actions in these
jobs are pinned to commits, crane to v0.22.0 by checksum, and the sparse
checkout no longer keeps the token.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker latest: the signing job checks out the workflow's own commit
The job only assembles and signs manifests, so nothing there needs the
source_ref checkout; the local signing action now comes from the same
revision as the workflow file that calls it.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker release: take the index digest from the create result
imagetools create writes the descriptor it pushed with --metadata-file
(buildx 0.32+, the runners ship 0.36), so the digest no longer comes from
re-resolving the tag even within the same step.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* http: try a volume server that failed to answer last
A cached location list is shuffled on every read, so once a replica dies
half the reads keep dialing it first and pay a connect failure or timeout
before the healthy replica answers. Remember, per host, when a request got
no answer at all and order such hosts last for the next half minute. Once
that passes, one read probes the host in its usual place while the others
keep it last until the probe settles, so a black-holed server costs one
stalled read per interval instead of one per read.
Nothing is ever skipped: a host that failed is still tried when the others
fail too. Any response, including an error status, counts as reachable.
Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs
* filer: refresh a chunk's locations after one of them fails
A mount's location cache is only relearned when every cached location
fails. When one replica dies and the other still answers, every read
succeeds and the dead replica stays in the cache, and in the shuffled
order it keeps being dialed first long after the master has dropped it.
When a read fails on one location and a later one answers, call the
refresh hook so the cached entry is dropped and looked up again. The read
that already paid for the failure returns its data; the reads after it
start from the locations the master knows now.
Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs
* http: claim the probe for every expired host, and try it first
The claim was only checked for the first url, so with two replicas whose
marks expired together the second was probed by every read at once. Claim
each expired host on its own and put the reads that won a claim ahead of
the reachable hosts, so a probe is always a real attempt and a lost claim
always means the host is tried last.
Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs
* filer: refresh a chunk's locations in the streaming read path too
The streaming loop had no refresh hook, so a manifest or streamed chunk
that failed on one cached location and was served by another kept the
stale entry until every location failed. Give it the same hook as the
buffered loop, built by one refreshUrls function shared by the reader
cache and the stream callers.
Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs
* http: probe at most one expired host per read
Claiming every expired host in one ordering left all but the first claim
without an attempt, since a read stops at its first answer, and a host that
had come back waited another interval for nothing. Claim only the first
expired host a read sees and leave the rest last and unclaimed, so each
following read probes one of them.
Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs
* test: start the live server before releasing the dead server's port
Closing the dead server first let the live server come up on the same
port, in which case the dead location answers and the partial failure
under test never happens.
Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs
The cached-location test needs two files on one volume, but it wrote six
files into the six volumes a 001 layout starts with, and every so often
each file landed on its own volume and the test had nothing to probe.
Seven files leave no way to spread them out.
Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs
* helm: values-driven labels on every ingress
Each ingress already takes annotations from values, but its labels were
a fixed block, so tools that select ingresses by label (ExternalDNS
label filters, for one) had nothing to key on. Every ingress block now
has a labels map rendered after the standard app.kubernetes.io labels,
including the Traefik IngressRouteTCP that shares the filer gRPC values.
Claude-Session: https://claude.ai/code/session_01L6eJGXtYkwe1W9QjGeUgr1
* helm ci: render check for ingress labels
Claude-Session: https://claude.ai/code/session_01L6eJGXtYkwe1W9QjGeUgr1
weedfs_stream_mutate_error_test.go names its *streamMutateError local
sme, which codespell reads as a misspelling of same/some. It is an
identifier, so exempt it beside the other variable-name entries.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: rebuild a missing .idx from the .dat
Pointing -dir.idx at a directory that holds no index aborted the whole
volume server: checkIdxFile found no .idx and load() called glog.Fatalf.
Every row of the index is derivable from the .dat, so walk it in append
order and write the index back, which reproduces byte for byte what the
server's own writes had left in the old directory.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: keep the index co-located with the data in the Rust server
Go's load() drops back to the data directory when an .idx already sits
beside the .dat, so naming a --dir.idx does not strand a pre-existing
index. Rust had no such adjustment: it opened the new directory with
create, and the volume came up on an empty index with every needle
invisible.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: rebuild a missing .idx from the .dat in the Rust server
Mirrors the Go side. Rust did not abort on a missing index the way
checkIdxFile did; it opened the new directory with create and mounted the
volume on an empty index, so every needle read as missing while the .dat
still held the data. Walk the .dat in append order and write the index
back, byte for byte what the server's own writes had left behind.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: stop the idx rebuild at a zero-padded .dat tail
An all-zero needle header is unwritten space, not a record. Go's .dat walk
keeps reading past it and would index a truncated data file's tail as
millions of needle 0 rows; the Rust walk already stops there. Stop the Go
rebuild at the same place.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: create the -dir.idx directory when it does not exist
Rust's DiskLocation creates the index directory as it takes it; Go only
resolved the path, so naming a directory that does not exist yet left every
volume unable to open or rebuild its index and took the server down.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: stop the idx rebuild at a torn .dat record
A crash between writing a needle's header and its body leaves a record
whose declared size runs past the end of .dat. Indexing it puts a row in
the .idx that points at bytes that do not exist, which fails every read of
that needle and trips the past-EOF check on the next load. Stop at the
first record that does not fit, in both servers.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: stop the idx rebuild at a negative-size header
A corrupt header whose size field is negative makes the .dat walk advance
backwards: NeedleBodyLength adds the negative size, so the next offset is
lower than the current one. The Go walk then reads at a negative offset and
the rebuild fails, which puts the volume server right back to exiting at
startup; the Rust walk seeks past EOF and truncates the index instead.
A negative size is never a record, so stop there.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: skip a volume whose index cannot be rebuilt, do not exit
glog.Fatalf calls os.Exit(255), so a rebuild that could not write -- a full
or read-only index directory -- put the server right back to dying at
startup for one bad volume. Return the error instead: loadExistingVolume
logs it and skips that volume, which is what the remote-volume branch just
above already does and what the Rust loader has always done.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: create the index directory from the rebuild too
The rebuild is the first thing to write into a fresh -dir.idx, and it runs
before the loaders that create the directory on their way to opening .idx.
Create it in both rebuilds so the ordering does not matter.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* ci: let codespell past the sme variable in the mount tests
weedfs_stream_mutate_error_test.go names its *streamMutateError local
sme, which codespell reads as a misspelling of same/some. It is an
identifier, so exempt it beside the other variable-name entries.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
-dir=/data fails on macOS, where the root filesystem is read-only, and
on any Linux box without root; -dir=./data is created on the spot.
go install of the weed package is refused because go.mod carries
replace directives, so the install script is the shortcut instead.
Claude-Session: https://claude.ai/code/session_014apMEkkquAtAYp89paTkAT
The Helm values put filer metadata on a claim through filer.data, which
is what the chart reads; enablePVC was rendering a hostPath. Claims use
the cluster default storage class instead of local-path. The AWS CLI
test carries its own credentials, the compose download includes the
Prometheus config the compose file mounts, and the disk-read claim is
per blob, since large files are chunked.
Claude-Session: https://claude.ai/code/session_014apMEkkquAtAYp89paTkAT
* filer: give the SQL stores' key-value reads their own connections
A listing holds the connection its rows are on for the whole iteration, and
FilerStoreWrapper calls maybeReadHardLink -> KvGet from inside that iteration,
so a hard-linked entry needs a second connection while the first is still busy.
Out of one bounded pool that is a deadlock: the listings fill the pool and then
wait for a connection none of them will release, and the wrapper's
context.WithoutCancel leaves the waiters without a deadline, so the filer stays
wedged rather than erroring.
The sqlite store shows it at its sharpest -- it allows a single connection, so
one listing over one hard-linked entry never returns. On postgres with
connection_max_open = 50, 60 concurrent listings over hard-linked entries made
no progress at all.
Key-value reads now run on their own pool, carved out of connection_max_open
rather than added to it, so the operator's cap still bounds what the store opens
against the database. An unbounded pool keeps a single pool: nothing can wait
there. sqlite's single connection becomes two, one per pool, and its writes get
a busy timeout so a write that meets the reader waits instead of failing.
Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t
* sqlite: keep both pools on one database, whatever the dbFile spells
A dbFile that already carries URI options got a second "?" appended, which the
driver reads as part of the preceding option value, and a bare :memory: is
private to each connection, so the key-value pool would open its own empty
database and every key-value operation would fail on a missing filemeta.
Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t
* sqlite: assert the busy timeout on the in-memory DSN too
Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t
* rdma: drop the sidecar prototype
The Rust engine under it never touched a wire: rdma.rs fabricates pattern
bytes and the crate's default feature is mock-ucx, with real-ucx unimplemented
since the directory landed. Nothing builds it, no CI runs it, and its only
consumer is weed mount's RDMA client, removed next. Two 22MB binaries were
committed along with it.
Claude-Session: https://claude.ai/code/session_01X3zhqLYwQwEQCrRbuzQKvy
* mount: remove the RDMA client that spoke to the deleted sidecar
Its only server was the sidecar's HTTP API, and the path could never have
worked in production anyway: it served a single chunk per call, ignored the
buffer's chunk boundaries, and had no test. Removing it also removes the
per-handle cumulative-offset cache, which nothing else used.
The -rdma.* mount flags go with it. They defaulted to off and pointed at an
address no released build ever listened on.
Claude-Session: https://claude.ai/code/session_01X3zhqLYwQwEQCrRbuzQKvy
Lead with what SeaweedFS is and how to start it: one command, Docker,
Docker Compose, a production-shaped Helm values file, build from
source, scale out. Then why: fast, scalable, the S3 API surface with
operation counts, the lakehouse with S3 Tables and the engines that
share it, the cloud cache, and cross-cluster replication and the rest
of the feature list, each pointing at its wiki page.
The blob store walkthrough and the master and volume server internals
move to the Blob Store Architecture wiki page. The comparisons,
benchmark, enterprise and license sections stay. The dev plan is gone,
it was done.
Claude-Session: https://claude.ai/code/session_014apMEkkquAtAYp89paTkAT
* shell: non-interactive mode exits non-zero when a command fails
A failed command in a piped weed shell run printed 'error: ...' but the process
still exited 0, so a CronJob wrapping e.g.
echo 's3.lifecycle.run-shard -shards 0-15' | weed shell -master=...
reported green while the run aborted partway (shards N+1..15 unwalked). An
unknown command likewise exited 0.
RunShell now returns the last command failure from the non-interactive stdin
path (unknown commands included), and the shell command exits 2 on it.
Interactive sessions are unchanged: errors are shown to the operator and the
session continues, exiting 0 as before.
* shell: route the piped-failure exit through main's shutdown path
Review follow-up: os.Exit(2) inside the shell command skipped main's shutdown
work. The command now records the status (SetCommandExitStatus) and returns
normally; main applies it via setExitStatus before exit(). exit() itself now
flushes sentry before os.Exit -- main's deferred sentry.Flush never ran on this
path (os.Exit skips defers), so the existing 'flush buffered events before the
program terminates' intent only worked for the autocomplete early-return.
Exit status 2 on a failed piped run is preserved (verified: piped success
exits 0, piped failing command exits 2).
* shell: test the registered-command failure path
Review follow-up: the error-propagation test only covered unknown commands.
A fake registered command now drives processEachCmd's real dispatch path:
a failing Do surfaces its exact error (errors.Is) and a succeeding one
returns nil. The non-interactive exit status itself is main-level plumbing,
verified end to end against the reproduction (piped failure exits 2).
* shell: trim the comments added with the exit status
Keep the non-obvious why -- why a piped run has to fail its wrapper, why the
status is recorded instead of os.Exit'ed -- and drop the narration.
Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t
* shell: fail a piped run with the status weed already uses for that
weed.go spends 1 on a command that failed and 2 on a usage or syntax error, and
runShell returns true precisely so the usage dump is skipped. Exiting 2 there
told a wrapper the command line was wrong.
Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t
---------
Co-authored-by: Carlos Leyva <carlos.leyva@idener.es>
* filer: SQL store pool defaults survive concurrent walks (idle == open == 50, lifetime 300s)
The code defaults for the four SQL stores were connection_max_idle=2 with NO
default for connection_max_open (unlimited) or lifetime, while the scaffold
filer.toml documents 10/50/300 -- so an env-configured or minimal-toml filer got
the worst possible pool. Under a concurrent listing burst (s3.lifecycle.run-shard
walks 16 shards in parallel) every operation released above the 2 idle slots
closes its TCP connection, so the walk opens a fresh connection per operation
until the filer exhausts its ephemeral ports:
list /buckets/... : failed to connect ... dial tcp ...:5432:
connect: cannot assign requested address
Measured on a production filer: 0 -> 28k TIME_WAIT with only ~1.3k concurrent,
and in the minimal docker-compose reproduction (2000-dir bucket, port range
narrowed to 400): the whole range in TIME_WAIT with only ~12 ESTABLISHED.
Default all three knobs, with idle == open so released connections are kept and
reused: idle connections only accumulate up to the actual peak concurrency and
connection_max_lifetime_seconds recycles them, so a quiet deployment holds
nothing extra. An explicit 0 still disables the caps as before. The scaffold's
connection_max_idle moves 10 -> 50 to match.
With this change the same reproduction completes all 16 shards with the default
configuration (TIME_WAIT peak 19 vs the whole port range).
* filer: trim the SQL pool default comments
One line of the non-obvious why is enough; the rest narrated the code.
Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t
* filer: leave the SQL stores' connection_max_open unset
A listing holds its connection for the whole row iteration while its callback
runs another query -- FilerStoreWrapper.maybeReadHardLink does a KvGet per
hard-linked entry -- so every concurrent listing needs two connections from the
same pool. With a default cap, listings past the cap wedge: 60 concurrent
listings over hard-linked entries made no progress at all against a 50
connection pool, and the wrapper's context.WithoutCancel leaves the waiters
without a deadline.
The idle pool is what fixes the connection churn: idle 50 with an unbounded
max_open holds the same 14 postgres sessions across a 16-way listing burst that
opened 455 with idle 2.
Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
mount: name the disk after the mount point when the whole tree is mounted
The mounted path was the only thing that named the disk, so a mount of the
whole tree was labelled with the filer address and the only way to give it
a name was to mount a subtree under that name — which hides everything
outside it. Fall back to the mount point's own name first, so
-dir=\\seaweedfs\Images labels the disk while -filer.path stays "/".
Claude-Session: https://claude.ai/code/session_01Q9f8pWBXu1ceJvcQfYRQ7x
* fix(mount): make Mkdir exclusive so a concurrent duplicate fails with EEXIST
Mkdir sent CreateEntryRequest without OExcl, so the filer treated a
concurrent duplicate as an update and reported success to both callers;
the kernel's pre-mkdir lookup only masks this when the winner's create
is already visible. Set OExcl, map the entry-already-exists sentinel to
EEXIST instead of EIO, and drop the parent's children cache on the
losing side so the next lookup fetches the winner's entry.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(mount): route exclusive creates to the path's owner filer
The filer's per-path lock is filer-local and the store insert has
upsert semantics, so two mounts streaming to different filers can both
create the same path even with OExcl (measured 18/30 both-success on a
3-filer cluster). Hash the path over the sorted filer list so every
mount sends the same path's exclusive create to the same owner filer:
keep the mutation stream when it already targets the owner, fall back
to it when the owner is unreachable. Also let doUnary hand failed
creates to CreateEntry so the structured error code survives as EEXIST
instead of collapsing into the stream's generic EIO. Same race after
the change: 30/30 exactly one winner, every loser fails with EEXIST.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(mount): review fixes — pin exclusive creates to the owner filer
An OExcl create now goes only to the path's owner filer: retrying on a
different filer would race the owner's possibly still-in-flight create
through a separate per-path lock, the very hole this routing closes. A
broken mutation stream retries the same owner over unary, and an
unreachable owner fails the create instead of degrading.
Pick the owner by rendezvous hashing so the choice is independent of the
configured filer order, and mounts configured with different but
overlapping lists still agree wherever the winning filer appears in
both.
Reject a stream create wrapper whose nested response is nil instead of
handing it to CreateEntry, which would dereference it.
Add ownerFilerAddress unit tests: order independence, subset agreement,
spread.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* mount: drop the client-side owner ring, the filer routes exclusive creates now
Exclusive creates are arbitrated cluster-wide on the server: the filer resolves
an OExcl create's ring owner and forwards one hop, so one filer's per-path lock
binds every creator — mount, S3, the HTTP surface and the Java client alike, not
only the ones that opted into a client-side ring.
That makes exclusiveCreateEntry redundant. It hashed the mount's configured
-filer list, which names a different owner than the master-maintained ring, and
failed the mkdir outright when its chosen owner was unreachable rather than
letting the ring reassign. Mkdir goes back to streamCreateEntry; OExcl and the
EEXIST mapping stay, and now mean what they say.
Claude-Session: https://claude.ai/code/session_01Fx1Hx8RqsJqHpbfbgTf4WJ
* mount: cover the create error plumbing that turns a lost race into EEXIST
Letting a failed create's structured code survive doUnary is what makes a lost
mkdir race report EEXIST instead of EIO, and it had no test. Pull the two steps
out so they can be exercised without a live stream: hasCreateResponse decides
whether a response still carries a code to unwrap, createEntryFromResponse does
the unwrapping.
Reading the guard the other way round also says what it means — consume the
response only when there is no nested code left to recover — rather than
negating a type assertion inline.
Claude-Session: https://claude.ai/code/session_01Fx1Hx8RqsJqHpbfbgTf4WJ
* mount: do not trust a create reply's shape before reading it
createEntryFromResponse read cr.ErrorCode without checking the nested response
was there. Nothing our filer sends is shaped that way, but the mount reads this
off the wire and a nil there panics the whole mount, so report it instead.
A top-level failure whose nested response carries no code was also returned as
success, silently losing the error. Fall back to the top-level errno when the
nested response explains nothing.
Claude-Session: https://claude.ai/code/session_01Fx1Hx8RqsJqHpbfbgTf4WJ
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* lance: a maintenance job that sorts a table by its declared fields
Lance appends fragments in write order and has no notion of a sorted table, so
nothing but a rewrite establishes one, and nothing but another rewrite restores
it once rows have been appended. lance_sort reads the order from the dataset's
own configuration, falls back to the worker's, and rewrites the table in it.
The spec and the marker live in crates/sort rather than in the job, because
weed/worker/tasks/iceberg sorts too: two jobs that disagreed about what
"id desc nulls-first" means would be two features wearing one name.
The sort spills. lance builds its DataFusion runtime with a FairSpillPool and a
disk manager, but only when LanceExecutionOptions::use_spilling is set, and that
struct derives Default over a plain bool — so Scanner::try_into_stream, which
fills its options with ..Default::default(), is precisely the path that does not
spill. The job builds the plan with create_plan and executes it with spilling on
and the operator's memory budget.
The marker rides in the same commit as the data: Operation::Overwrite is the one
operation carrying config values alongside fragments, so a sorted table and the
record of its sorting cannot disagree. It records the version the sort read, not
the one it wrote, which is not knowable while the marker is being assembled.
Detection treats anything committed after the sort's own commit as data the sort
did not produce — row counts alone cannot see a rewrite that leaves the count
where it was, and such a table would look sorted forever.
Claude-Session: https://claude.ai/code/session_015SZkLTUvd1svDu4xdr6Q3y
* lance: identify a sorted table by the files it wrote, not its version
Review found three ways the version-based marker misjudges a table, and they
share a cause: the version a sort produces is not knowable while the marker is
being assembled, so the marker recorded the version it read and detection
inferred the rest. A commit that rebases past a conflict lands on a different
number, and the inference then reads a rewrite into an ordinary table — a full
re-sort, and its indices, for nothing.
Data file names do not have that problem. They are chosen before the commit, so
the commit can carry them, and they do not change with the version it lands on.
The marker now records how many files the sort wrote and a digest of their
names, and detection asks whether the table still holds them: the same files
means untouched, the same files followed by more means appended, anything else
means the data was replaced.
That also closes the hole the row threshold left. A replacement that grew the
table by fewer rows than min_unsorted_rows read as sorted, however many rows had
actually moved; the threshold now applies only where the sorted files are still
in place, which is what it was for. A marker without a row count is stale rather
than a zero to compare against, and deletes stop forcing a re-sort — they write
a deletion file beside the data rather than rewriting it, and removing rows does
not unsort the ones that remain.
Sort fields are also compared exactly rather than case-folded. Arrow schemas are
case-sensitive, so `id` and `ID` are two columns, and folding them together
rejected a valid order.
Claude-Session: https://claude.ai/code/session_015SZkLTUvd1svDu4xdr6Q3y
* lance: count the rows appended after a sort, not the table's net growth
Review found that rows deleted from the sorted fragments hide appended rows one
for one: the threshold compared the live row count against the count recorded at
sort time, so 800 deletions and 300 appends read as a table that shrank, and a
table where deletions keep pace with appends stays "sorted" with an unsorted
tail forever.
The fragments say it directly. The marker already records how many fragments the
sort wrote, so the ones after that prefix are exactly what arrived since, and
the manifest carries each fragment's live row count — physical rows less its
deletions. Counting those is the arithmetic the threshold was always meant to
do, and it needs no row count from the marker at all.
A fragment whose length the manifest does not record cannot be counted, and a
table that cannot be judged is one to sort rather than one to leave alone
forever, so an uncountable appended fragment reads as stale.
Claude-Session: https://claude.ai/code/session_015SZkLTUvd1svDu4xdr6Q3y
A sorted rewrite collected every row of a bin into one slice and sorted it
there, so a bin larger than the worker's heap could not be sorted at all.
sort_max_input_mb existed for that reason and skipped the bins it capped.
parquet-go's SortingWriter buffers sort_buffer_rows rows, encodes each buffer
as a sorted run, and merges the runs at close; backing those runs with a
FileBufferPool keeps them in files rather than on the heap. sort_spill_dir says
where, defaulting to the system temp directory — NewFileBufferPool resolves an
empty path to the working directory, which is not what an unset setting means.
The output now also declares its sorting columns, which the plain writer the
sorted path used never did.
Claude-Session: https://claude.ai/code/session_015SZkLTUvd1svDu4xdr6Q3y
* proto: resync the java copy of filer.proto
The Makefile keeps other/java/client/src/main/proto/filer.proto a verbatim copy,
but AssignVolumeResponse.fsync and SubscribeMetadataResponse.flushed_ts_ns
landed without it. Copy them over; no behaviour change.
Claude-Session: https://claude.ai/code/session_01Fx1Hx8RqsJqHpbfbgTf4WJ
* filer: route exclusive and conditional creates to the entry's ring owner
CreateEntry with o_excl is a FindEntry-then-Insert. The per-path lock added for
it makes that atomic only on the filer running it, and the store's insert is an
upsert on every backend, so two filers both pass the existence check and both
report success. mkdir(2) then succeeds twice for the same path. The same hole
sits under the condition precondition, whose comment already told callers to
route the key's writes to the owner filer themselves.
Do it on the server instead, with the mechanism ObjectTransaction already uses:
resolve the entry's ring owner and forward one hop, bounded by is_moved. The
ring's membership comes from the master, so it tolerates a stale view and
reassigns when a filer dies, neither of which a client's configured filer list
can do. Every creator gets this — mount, S3, the filer's own HTTP surface, the
Java client — not only the ones that opted in.
Plain creates are upserts whoever applies them, so they stay local and pay
nothing. The route key shares the S3 gateway's namespace so an object's
ObjectTransaction and its CreateEntry land on the same filer's per-path lock.
Claude-Session: https://claude.ai/code/session_01Fx1Hx8RqsJqHpbfbgTf4WJ
ObjectTransaction forwards to the ring owner so one filer's per-path lock
arbitrates every writer of a key. But a ring change hands the key over before
the new owner has rebuilt the locks the prior owner still holds, so for the
cooling-off window both can grant it. LockRing.PriorOwner exists for exactly
this and nothing consulted it.
Route to the prior owner while that window is open. LockRing.WriteOwner
resolves prior-else-current under one read lock, so the pair cannot come from
different rings and name the same filer twice.
An unreachable owner fails the request rather than falling back to the current
one. gRPC reports a response lost in transit as Unavailable, indistinguishable
from a request the owner never saw, so re-sending elsewhere could re-apply what
the owner already committed; and an owner unreachable from here may be
partitioned rather than down, still serving the key to everyone else — which is
the split brain the routing exists to prevent. The window is bounded: once it
closes the ring hands the key to its new owner.
The owner resolution and the forward move into writeOwner/forwardToWriteOwner
so the next routed RPC reuses them rather than copying the block.
Claude-Session: https://claude.ai/code/session_01Fx1Hx8RqsJqHpbfbgTf4WJ
* filer: self-heal fetchWholeChunk on stale volume locations
Upstream #10156/#10800 wired cache invalidation into the buffer-based
read paths, but manifest resolution still goes through fetchWholeChunk,
which returns the raw error on failure. When cached volume locations
are stale (volume tiered to remote storage, server rolled), resolving
a large multipart file fails permanently even though other locations
are healthy.
Thread the ChunkGroup's cacheInvalidator through ResolveChunkManifest /
ResolveOneChunkManifest / fetchWholeChunk, and on failure invalidate,
re-lookup and retry once via the existing retryFetchWithFreshLocations
helper. The streaming bytesBuffer is reset before the retry so partial
bytes from the failed attempt cannot corrupt the manifest
proto.Unmarshal. Non-mount callers pass nil and keep their semantics.
* filer: move the manifest self-heal tests in with the other manifest tests
Also make the stale server stream a prefix and then abort mid-body, which is
what actually leaves partial bytes in the buffer: an HTTP error status returns
before ReadUrlAsStream ever calls the writer, so a 500 never exercised the
Reset the tests claimed to cover.
Claude-Session: https://claude.ai/code/session_01FK3oGC5ZVeJYvNBWgb9JUD
* filer: keep the cached volume locations when a manifest read is cancelled
A cancelled or timed-out read says nothing about where the volume lives, so
dropping the location and going back to the master only costs the next reader
a round trip. PrepareStreamContentWithThrottler already guards its self-heal
this way. The guard also goes inside retryFetchWithFreshLocations, since the
caller can be cancelled between its own check and the invalidation, and that
covers the reader cache and prefetch paths too.
fetchWholeChunk returns the context error rather than the stream failure it
provoked, and ResolveOneChunkManifest wraps with %w so errors.Is still sees it.
That matters even where no invalidator is passed: volume.fsck resolves
manifests with nil and tells its own abort from a corrupt manifest that way,
so the cancellation check sits ahead of the nil-invalidator return.
Claude-Session: https://claude.ai/code/session_01FK3oGC5ZVeJYvNBWgb9JUD
* filer: self-heal manifest reads on the filer and s3 paths too
Every caller that already holds the location cache backing its lookup function
can hand it over: the filer's read, copy and deletion paths and the log cache
have the MasterClient right there, and s3api has the FilerClient. MinusChunks
takes one for the same reason, since the deletion path resolves manifests
through it. Only the shell tools and the replication sinks, whose lookup
functions cache privately with nothing to invalidate, keep passing nil.
Claude-Session: https://claude.ai/code/session_01FK3oGC5ZVeJYvNBWgb9JUD
---------
Co-authored-by: bruce-zzz <bruce.zou@hhy-data.com>
* shell: clean up the target copy when a merge upload fails
A replicated write commits the needle to the local volume before it fans out
to the other replicas, so an upload that reports failure can still have left a
copy on the target. fs.mergeVolumes printed "failed to move" and carried on,
so that copy stayed behind forever: the filer is never re-pointed at it, and
nothing else knows it exists.
One sick replica orphans roughly half the chunks of a merge, two thirds with
three copies, since the entry node is picked at random from the replicas and
the replica upload uses MaxAttempts 1. A volume with a single copy has no such
window: the write is one local append that either succeeds or leaves nothing.
Delete the needle we may have written before continuing. The source side
already did exactly this, so deleteMovedSourceNeedles is renamed to
deleteOrphanedNeedles and reused for both ends.
Claude-Session: https://claude.ai/code/session_01XjiMGK72F4Gs3yWNJhnVjy
* shell: verify the cookie before deleting a merge target needle
The target cleanup deletes a needle an upload may or may not have written, and
BatchDelete matches on the needle id alone. Needle ids come from one global
sequence, so normally nothing else can hold that id — but a volume restored
from elsewhere, or one written either side of a master sequence reset, can, and
then a failed move deletes a live needle out from under its filer entry.
Have the volume server verify the cookie for those. Source needles keep
deleting by id: they are the ones the filer just pointed at, matching every
other filer-driven delete.
Claude-Session: https://claude.ai/code/session_01XjiMGK72F4Gs3yWNJhnVjy
* shell: delete the target copies an abandoned manifest rewrite leaves
rewriteManifestChunk moves sub-chunks one at a time and only then uploads the
rewritten manifest. Every error after the first successful move — a nested
rewrite failing, the marshal, the manifest upload — returned without touching
the copies already written to the target volumes. The filer keeps pointing at
the old manifest, so those copies orphan, one per sub-chunk moved so far.
Track them alongside the sources and delete them on the way out. Nested
rewrites hand theirs up so an outer failure clears the whole subtree.
A failed UpdateEntry deliberately still leaks its copies: that error can also
mean the filer applied the update and lost the response, and deleting there
would turn a leak into data loss.
Claude-Session: https://claude.ai/code/session_01XjiMGK72F4Gs3yWNJhnVjy
* shell: give the plan its room back when a manifest rewrite is abandoned
allocate reserves plannedSize against the chosen target for every move a
multi-target source makes, and release hands it back when the move fails.
Abandoning a manifest rewrite now deletes the copies that did land, so those
reservations stopped matching anything on disk: the plan kept counting bytes
that are gone and refused later chunks with "no target volume has room".
Release them alongside the delete. Nested rewrites hand theirs up so an outer
failure unwinds the whole subtree's accounting.
Claude-Session: https://claude.ai/code/session_01XjiMGK72F4Gs3yWNJhnVjy
* iceberg: normalize foreign day partitions during manifest rewrite
* test: cover manifest rewrite with foreign day partitions
* iceberg: restore every foreign partition value, not just day transforms
iceberg-go takes a partition field's logical type from the last branch of
its Avro union, so a writer that spells an optional partition [<type>, null]
rather than [null, <type>] leaves the value as whatever the Avro decoder
produced. A day or date partition then arrives as a time.Time the manifest
writer cannot encode, and a time partition is worse: time.Duration converts
to int64 nanoseconds and silently records the wrong value.
ReadManifest sits next to ReadManifestList, the other shim for what foreign
writers put on the wire, and converts each partition value back to the
Iceberg representation for its field type.
Claude-Session: https://claude.ai/code/session_01FdQyRuWF9SuCnPn21iH9yR
* iceberg: read manifests that carry partition values through the shim
Compaction, delete rewrite and their detection passes read entries and write
the same partition values back into new manifests, so they fail on a foreign
day partition exactly as manifest rewrite does. Where filters see it too:
literalMatchesActual falls through to fmt.Sprint, so a time.Time renders as a
timestamp and never matches the day the user asked for.
The two remaining readers, orphan collection and the admin preview, only look
at file paths and stay on iceberg.ReadManifest.
Claude-Session: https://claude.ai/code/session_01FdQyRuWF9SuCnPn21iH9yR
* iceberg: convert partition values before the writer rebinds logical types
Dimonyga checked the manifests of a live Doris table: every input spells the
partition union null-first, with the date logical type present, so the union
ordering is not what breaks the merge.
The conversion is lazy. iceberg-go converts what the Avro decoder returned on
the first Partition() call, using the logical types read from the manifest
being parsed, and ManifestWriter.addEntry rebinds them to the manifest it is
about to write before it makes that call. A day partition is where the two
disagree -- iceberg-go's day transform reports an int32 result type, so the
manifest it writes carries no date logical type at all -- and an entry nobody
looked at in between converts against that and keeps its time.Time.
That is why only rewrite_manifests failed: compaction and delete rewrite group
entries by partitionKey(df.Partition()) first, which converts them, and a where
filter does the same. Reading every entry's partition here converts them all
while the manifest's own logical types are still in place.
Claude-Session: https://claude.ai/code/session_01FdQyRuWF9SuCnPn21iH9yR
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* expose whether a volume replica is backed by remote storage
Volume locations returned by lookups do not indicate whether a replica
has been tiered to remote storage. Readers cannot distinguish a local
replica from a remote-backed one, so they may hit a remote-backed
replica first even when a local replica is available.
Add DataInRemote to the lookup location message, populate it from the
master's volume info, and carry it through the wdclient vid map so
clients can prefer local replicas when resolving chunk locations.
* wdclient: prefer local volume replicas over remote-tier replicas on lookup
LookupFileIdWithFallback (and the publicUrl variant in FilerClient)
didn't honor the DataInRemote flag when shuffling URLs, so the
DataInRemote patch only took effect in LookupVolumeServerUrl. Apply
the same ReorderToFront(localUrls) to sameDcUrls/otherDcUrls so
non-remote replicas stay at the front, matching the existing vidMap
convention.
* wdclient: propagate DataInRemote across tier transitions on existing replicas
When a volume is tiered to remote storage or a remote-backed replica is
restored locally, the cached DataInRemote on the same volume-server URL
stayed at its old value because two pieces of state never updated:
* master_grpc_server.go only split newVolumes and (already-tracked) volumes
into NewVids vs RemoteVids. ChangedVolumes went straight to NewVids, so
the broadcast announced the re-classified volume as a fresh arrival and
the client had no way to tell whether its existing cache was stale.
* vid_map.addLocationToMap early-returned when an entry already had the
same URL. A tier transition reports the same URL with DataInRemote
flipped, so the cached entry stayed at the old classification.
Wire both sides together: ChangedVolumes now go through the same IsRemote
split as newVolumes, and addLocationToMap replaces the existing entry in
place when the URL matches but DataInRemote has changed. The server
reference key only depends on URL/grpc port, so the refcount does not
move across the flip.
Adds vid_map_remote_transition_test.go covering the local->remote and
remote->local paths so the in-place update and the cache-key stability
are pinned by tests.
* wdclient: prefer local replicas across data-center boundaries
The previous local-first ordering hoisted local URLs to the front of each
data-center bucket separately, then concatenated same-DC before other-DC.
That meant a same-DC remote replica could still be tried before an
other-DC local replica even though the local one would answer cheaply.
Reorder once across the full candidate list: concatenate same-DC and
other-DC first, then ReorderToFront pulls every local replica to the very
front while preserving the DC preference inside each tier. Apply the same
ordering in all four lookup paths so the cached vidMap, the
LookupFileIdWithFallback provider path, FilerClient.GetLookupFileIdFunction
(PublicUrl-preferred variant), and the deprecated filer.LookupFn all agree:
- weed/wdclient/vid_map.go (LookupVolumeServerUrl)
- weed/wdclient/vidmap_client.go (LookupFileIdWithFallback)
- weed/wdclient/filer_client.go (LookupFileId)
- weed/filer/reader_at.go (LookupFn)
Strengthen the existing local-first tests: vidmap_client_localfirst_test
now asserts both endpoints are present (not just the local one is first),
and slice_test asserts an exact match instead of accepting two orderings.
Add TestLookupFileIdWithFallbackGlobalLocalFirst to pin the cross-DC
ordering invariant: any local replica (same or other DC) precedes every
remote-tier replica; within each tier DC1 precedes DC2.
Add docstrings to ToVolumeLocations, ReorderToFront, LookupVolumeServerUrl,
LookupFileId, GetVidLocations, GetLocations, LookupFileIdWithFallback, and
updateVidMap so the touched lookup paths are described in one place.
* topology: broadcast tier transitions on existing replicas
When a volume replica is tiered to remote storage or restored locally, the
wdclient's cached DataInRemote went stale: every connected client kept
preferring a remote-backed replica over a freshly restored local one, or
demoted a freshly tiered remote replica. The fix in commit 116982595 routed
ChangedVolumes to NewVids/RemoteVids on the master, but ApplyVolumeChanges
returned only fresh arrivals and previously servable replicas. An existing
replica whose IsRemote() classification flipped was neither, so it never
reached the broadcast loop and the wdclient never learned.
Make Disk.doAddOrUpdateVolume return a third signal -- tierTransition --
true exactly when an existing replica's IsRemote() flips. ApplyVolumeChanges
treats that as an arrival so the existing SendHeartbeat routing loop now
sees it. Add a master-side end-to-end test covering local->remote,
remote->local, no-op re-reports, and a mixed heartbeat that only announces
the tier transition.
Also add docstrings to LookupFileId, wdclientLocationsToPb, and
LookupVolume where the prior change touched their bodies.
* topology: broadcast tier transitions received through full reconciliation
The previous commit added tier-transition routing on the ChangedVolumes
delta path, but that is not the only way a re-tiered replica reaches the
master. After a digest mismatch the volume server resends a full Volumes
list, and SyncDataNodeRegistration applies the new IsRemote() classification
silently -- the changedVolumes return value was being thrown away. The
master therefore never broadcast NewVids/RemoteVids, and a wdclient connected
during the recovery kept the stale DataInRemote until it lost contact with
the master.
Surface the changed set through UpdateVolumes.changedVolumes (now covering
both ReadOnly flips and tier flips) and SyncDataNodeRegistration, then route
it through NewVids/RemoteVids in SendHeartbeat the same way the delta path
already does. Add an end-to-end test for the full reconciliation path.
* master: keep an EC volume's locations in the volume lookup
The nodes that answer for an EC volume hold shards, not a volume record,
so asking them for one fails. Dropping the location on that failure
emptied the result and turned every EC read through the master's HTTP
lookup and fid redirect into a 404.
Treat an absent volume record as a local read and keep the node in the
answer. The per-node conversion moves into topologyLocation so the EC
case is covered by a test.
Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW
* wdclient: replace a tier-flipped location without writing under a reader
GetLocations hands back the entry's own slice and the caller walks it
after the read lock is dropped, which is why every other mutation here
builds a new slice. Writing the flipped replica into the array in place
raced LookupVolumeServerUrl, reported by -race.
Copy the slice, swap the one element, and publish it.
Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW
* master: keep a remote volume on NewVids for older clients
Moving remote-tier volumes out of NewVids and into RemoteVids alone is a
wire break in the wrong direction. A master upgraded ahead of its filers
and mounts -- the usual order -- announces a tiered volume only on a
field the older client ignores, so the volume drops out of that client's
vid map entirely and reads for it fail.
Announce every volume on NewVids and repeat the remote-tier subset on
RemoteVids, so a new client still learns the tier and an old one keeps
the location. The routing moves into announceVolume, which the heartbeat
paths and their tests now share instead of each restating it.
On the client, RemoteVids no longer needs a second write per volume: the
tier is settled before anything is added.
Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW
* topology: split the volume snapshot by tier without copying the records
ToVolumeLocations runs on every KeepConnected, so a filer or mount
connecting made the master allocate a full VolumeInfo per volume per node
just to read four bytes of id off each one. AppendVolumeIds exists to
avoid exactly that.
Extend it to fill the remote-tier list alongside the full one, and use it
again in the snapshot.
Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW
* wdclient: keep the data-center preference ahead of the local-first ordering
Hoisting every local replica to the very front puts an other-DC local
read ahead of a same-DC remote one. When the remote tier sits in the same
region as the replicas -- the common arrangement -- that trades an
in-region GET for a WAN round trip and costs more than the remote read it
avoids.
Reorder inside each data-center bucket instead, so local still wins among
equals and the data-center preference still wins overall.
Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW
* operation: pick the read replica from one list
The local-preferring lookup built a list of local URLs and then branched
on whether it was empty, duplicating the random pick. Fall back by
filling the same list with every replica instead.
Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW
---------
Co-authored-by: Bruce Zou <gift_secondst@msn.com>
Co-authored-by: bruce-zzz <bruce.zou@hhy-data.com>
The comment said enableReplication overrides "master & filer
defaultReplicaPlacement", but the two components take different keys:
master.defaultReplication and filer.defaultReplicaPlacement.
Claude-Session: https://claude.ai/code/session_014Yr1Asxq3qTjJo2r9G4cLX
* s3: ListParts on a completed or unknown upload answers NoSuchUpload
complete/abort delete the .uploads/<id> directory, but most filer stores list
a missing directory as empty rather than erroring, so listObjectParts answered
200 with an empty Parts list for an upload that no longer exists -- the same
response an open upload with no parts yet gets. AWS (and Ceph/RGW, MinIO)
answer NoSuchUpload, and clients lean on that: tusd derives the resumable
upload offset from the ListParts part sizes, so every completed upload read
back as zero bytes received.
Probe the upload record before listing, the way completeMultipartUpload
already does: not found, or a directory a late part write resurrected without
the destination key, answers NoSuchUpload. An open upload with no parts keeps
answering 200 with an empty list.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GGQEfVUvoATtwRCR8oC2jG
* s3: have the ListParts test filer refuse a directory it was not asked about
The fake answered the upload lookup on the name alone and the part listing
regardless of directory, so a wrong genUploadsFolder or upload-id suffix
would still have passed. Both calls now refuse any other directory with an
Internal error, which surfaces as ErrInternalError rather than the
NoSuchUpload the tests expect.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01StsRz9wbu5dUCMGbFgPoRM
---------
Co-authored-by: tomislavcivcija <9787657+tomislavcivcija@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* s3: return the multipart object checksum in the CompleteMultipartUpload body
S3 carries the flexible-checksum members of CompleteMultipartUploadResult in
the XML body, not in response headers, so every SDK read back an empty
checksum from an upload that asked for one.
Claude-Session: https://claude.ai/code/session_01Huux1uh7JxAbf8yypMYrMk
* s3: echo the checksum algorithm and type from CreateMultipartUpload
The upload directory already records both, but the response dropped them, so a
client could not confirm which checksum its parts had to carry.
Claude-Session: https://claude.ai/code/session_01Huux1uh7JxAbf8yypMYrMk
* test: multipart upload reports the object checksum it was asked for
Covers every algorithm end to end: the create response echoes the algorithm and
type, the complete response carries the checksum, and it matches what a later
HEAD reports.
Claude-Session: https://claude.ai/code/session_01Huux1uh7JxAbf8yypMYrMk
* master: survive a layout deleted twice
Two volume servers dropping the last replica of volumes that share a layout
both find it empty and both delete it. The loser's lookup misses, and the
single-value type assertion on the result crashed the master before the
caller could look at the found flag.
Claude-Session: https://claude.ai/code/session_01WmX6Rchx298NQksHDXg7sk
* master: remove a layout and read it back in one step
DeleteVolumeLayout looked the layout up and then deleted it, so two deleters
could each release the lookup ownership of the same layout, or one could find
nothing to release at all. Have the map hand back what it removed.
Claude-Session: https://claude.ai/code/session_01WmX6Rchx298NQksHDXg7sk
* s3tables: build the catalog attribute keys from one shared prefix
Every attribute the catalog stores on a bucket, namespace, table or view
entry is spelled out with the same literal prefix. Name it once in
s3_constants so code outside the package can recognize a catalog entry
without repeating the string.
Claude-Session: https://claude.ai/code/session_01GfZsc4cyNB2yr6KYLRv9q1
* filer: keep empty folders that are s3tables catalog entries
A namespace, table or view is a directory whose extended attributes are
the catalog record. Its files can live elsewhere - a rename moves only
the catalog pointer and leaves the data at the old path, and a view has
no files at all - so an empty one is still a live entry.
Drop a table, then rename another table onto that name: the drop queues
the old table's folders, the rename recreates the name path, and two
minutes later the cleaner deletes it and cascades into the namespace,
losing a table the catalog still lists.
Claude-Session: https://claude.ai/code/session_01GfZsc4cyNB2yr6KYLRv9q1
* filer: drop a queued cleanup when the folder is created again
A cleanup is queued against the folder that was found empty. If that
folder is deleted and a new one takes its name, the queue entry outlives
the folder it was about and the next pass deletes the replacement. A
drop followed by a rename onto the dropped name does exactly this: the
name path comes back as a live catalog entry two minutes before the
queue is read.
Claude-Session: https://claude.ai/code/session_01GfZsc4cyNB2yr6KYLRv9q1
* filer_pb: build BFS child paths with FullPath.Child
A start path with a trailing slash produced "/dir//sub" for every
subdirectory, and the filer only trims a trailing slash, so those
listings came back empty and the walk stopped after the first level.
Claude-Session: https://claude.ai/code/session_01Jp9tXRpBv9gvh8fkaVFqxQ
* filer_pb: normalize the BFS start path
Entries directly under the start path were reported with the caller's
trailing slash, so filer.meta.backup wrote them under a directory the
incremental stream never names again.
Claude-Session: https://claude.ai/code/session_01Jp9tXRpBv9gvh8fkaVFqxQ
* s3: reject a version-id header that is not a valid path segment
putToFiler stored the client-supplied Seaweed-X-Amz-Version-Id header
verbatim into object metadata. That value is later read back and used
as a filer path component when building the .versions/v_<id> path, so a
value containing "/", "\" or ".." could steer retention/legal-hold
writes and remote-cache reads outside the object's own bucket tree.
Validate the header with isValidVersionID before storing it, the same
check the versioned read paths already apply, and reject the request
otherwise. Server-set version ids ("null" and generated hex) pass.
Claude-Session: https://claude.ai/code/session_011QqNaxZwnpHgMoAZNp3RkY
* s3: validate a stored version-id before using it as a path
The retention and legal-hold sinks build a .versions/v_<id> path from a
version id read back out of object metadata, and the remote-cache path
builder does the same from either the request or the stored id, without
the isValidVersionID check the other version-id consumers apply. Guard
these so a value that is not a valid path segment falls back to the
regular / unversioned path instead of steering the write or read out of
the bucket tree.
Claude-Session: https://claude.ai/code/session_011QqNaxZwnpHgMoAZNp3RkY
recursivelyCheckLocksWithClient tested EntryHasActiveLock only on
non-directory entries, so a directory-marker object (an S3 key ending
in "/") that carries retention or a legal hold was recursed into but
never lock-checked. DeleteBucket then saw no locks and removed the
bucket, destroying an object under active Object Lock along with the
rest of the bucket. DeleteObject already enforces the lock on the same
key, so the two paths disagreed.
Check the directory entry for an active lock before recursing.
Claude-Session: https://claude.ai/code/session_011QqNaxZwnpHgMoAZNp3RkY
ValidatePassword compared the stored and supplied passwords with
subtle.ConstantTimeCompare, which returns 1 for two zero-length slices.
A user provisioned for public-key-only auth has an empty stored password,
so an empty supplied password authenticated as that user whenever
"password" was among the enabled auth methods (the default).
Treat an empty stored or supplied password as a non-match.
Claude-Session: https://claude.ai/code/session_011QqNaxZwnpHgMoAZNp3RkY
mini hardcoded the per-directory volume limit to 0, so the volume server
always auto-sized it as free disk space divided by the volume size. That
sizing reserves a whole volume size for every writable volume, so a
workload spreading small objects over many buckets runs out of slots long
before the disk fills and assign starts failing with "no free volumes
left". Same name and semantics as the flag weed server already carries,
and still 0 (auto) by default.
Claude-Session: https://claude.ai/code/session_01BiQLeBvZLzG8XitjiypDKu
The replace pinned thrift to a post-v0.23.0 master commit so 32-bit
GOARCHes would compile. That fix shipped in v0.24.0, so the replace was
only overriding the v0.24.0 require back to the pseudo-version and
holding the build below the CVE-2026-43871 fix.
Claude-Session: https://claude.ai/code/session_01C5BpSeYD3yULWmfVXwPRmB
Membership was a linear scan over a slice, and ensureCorrectWritables runs it
for every volume on every heartbeat, so the master's steady-state cost per
volume server is quadratic in that server's volume count.
BenchmarkSyncDataNodeRegistration, median of 3:
1000 volumes 565.7us -> 535.0us -5.4%
100000 volumes 1.665s -> 55.3ms -96.7%
Allocations are unchanged at both sizes, so the difference is the scan.
Claude-Session: https://claude.ai/code/session_01P3pE6J2UPFp6G3ksfMV4s1
The POSIX suites skirt this: pjdfstest's unlink/14.t covers only fstat and
pread on an unlinked descriptor — its driver has no fchmod at all and never
opens a directory. Pin the full rule in the FUSE integration suite:
ftruncate, fchmod, futimes, fstat, and the f*xattr calls keep working
between the removal of the last name and the final close, for a file after
unlink and a directory after rmdir, with nlink 0 and the changes visible to
a following fstat.
Claude-Session: https://claude.ai/code/session_01GYqLENjZzbV5hgt4L8cSAK
* mount: remember the entry of a directory removed while still referenced
A directory removed while a descriptor is open on it keeps its inode until
the kernel's final forget, but unlike a file it has no handle to live on
through: OpenDir hands out only a listing cursor. Keep the last-known entry
in memory, keyed by inode, from rmdir until that forget.
Claude-Session: https://claude.ai/code/session_01GYqLENjZzbV5hgt4L8cSAK
* mount: serve metadata ops on a removed open directory from its remembered entry
fchmod, futimens, and the f*xattr calls on a descriptor whose directory was
removed failed with ENOENT: maybeReadEntry resolved the inode to a path, and
rmdir had already dropped it. Fall back to the remembered entry the same way
an unlinked file falls back to its open handle. Mutations publish a changed
copy back rather than editing in place, so a concurrent reader never sees a
half-applied change, and the empty path keeps nlink 0 in every reply.
Claude-Session: https://claude.ai/code/session_01GYqLENjZzbV5hgt4L8cSAK
* mount: stash the entry the delete itself returned, not an earlier snapshot
A chmod landing between Rmdir's entry load and the delete RPC would be
resurrected pre-change: the remembered entry was the earlier local snapshot.
The filer serializes the delete against updates under the path lock and hands
the entry back in the delete event, so prefer that, keeping the local load
for the sticky-bit check and as fallback when no event comes back.
Claude-Session: https://claude.ai/code/session_01GYqLENjZzbV5hgt4L8cSAK
* mount: drop a remembered entry whose insert lost to the final forget
The forget's cleanup runs between RemovePath and the insert when the kernel
evicts the inode concurrently, finds nothing, and the entry would sit in the
map for the life of the mount. Re-check the inode after inserting and take
the entry back out; every interleaving now ends with the map empty.
Claude-Session: https://claude.ai/code/session_01GYqLENjZzbV5hgt4L8cSAK
* mount: insert the remembered entry under the inode table lock
The post-insert HasInode re-check could be fooled by inode number reuse: a
lookup landing between the forget and the check makes the number look alive
and the stale entry stays, keyed to someone else's inode. Do not check after
the fact — RemovePath now runs the retention callback inside its critical
section, where the forget that releases under the same lock cannot have run
and cannot be missed. Publishes need no such fence: their open descriptor
keeps the kernel from issuing the final forget in the first place.
Claude-Session: https://claude.ai/code/session_01GYqLENjZzbV5hgt4L8cSAK
* s3: share the per-key object authorization across copy and delete
AuthorizeCopySource and AuthorizeObjectDelete both authorize a key the request
URL does not name by evaluating the bucket policy and IAM against a synthetic
per-key request; only the method and action differed. Extract that into
authorizeObjectKeyAction and make the two callers thin wrappers. No behavior
change.
Claude-Session: https://claude.ai/code/session_01Qo7p6VsoWxMo8816ogJFk5
* s3: route POST Object uploads through the shared object authorization
POST Object uploads (presigned-POST / HTML form) authorized the write with only
the coarse per-identity Write action, unlike the other write paths which also
check the resolved object against the bucket policy and IAM. Route POST through
authorizeObjectKeyAction via a new AuthorizeObjectWrite so it is authorized like
the equivalent PUT.
Claude-Session: https://claude.ai/code/session_01Qo7p6VsoWxMo8816ogJFk5
* s3: test POST Object per-key authorization
Drives a signed POST upload and checks the per-key authorization decision for a
denied, permitted, and admin caller.
Claude-Session: https://claude.ai/code/session_01Qo7p6VsoWxMo8816ogJFk5
The version walk listed every directory with no prefix, transferring
all 1024-entry batches over gRPC and filtering gateway-side - and kept
paging past the point where names can no longer match. On wide
directories (many sibling orgs/jobs next to the requested prefix) that
is most of the transfer, decode, and CPU cost of every page.
Derive the next path component of the requested prefix per directory
level and hand it to the filer listing. A name holds no slash, so a
directory whose name does not start with the component cannot contain
a matching key and a file that does not cannot be one; stores with
native prefixed listing (sql, leveldb) turn this into a range scan and
stop the stream at the end of the prefix zone.
Claude-Session: https://claude.ai/code/session_01FquvGtTD2zA3uMZGQHAuV4
* volume: an already-deleted EC needle is not a delete failure
Deleting a needle that is already gone is what the caller asked for, and the
non-EC paths have always said so: BatchDelete reports StatusNotModified when
DeleteVolumeNeedle finds nothing to do, and DeleteHandler answers 404 from its
ReadVolumeNeedle pre-check. The EC branches had no such case, so ErrorDeleted
fell through to a generic failure -- 500 from both, and DeleteHandler also
counted it in VolumeServerFileWriteFailures, inflating a failure metric on a
replayed or duplicated delete.
The filer already tolerates this by string-matching "already deleted" on the
result, which leaves an error message load-bearing; the status is now right at
the source instead.
Claude-Session: https://claude.ai/code/session_01P3pE6J2UPFp6G3ksfMV4s1
* volume: close the EC fixture's disk location
Close stops the location's disk-space goroutine and releases the mounted EC
volume's file handles, which otherwise live until the test binary exits.
Claude-Session: https://claude.ai/code/session_01P3pE6J2UPFp6G3ksfMV4s1
* s3api: drop two redundant filer round-trips per listed version entry
ListObjectVersions paid two avoidable getEntry calls while walking a
bucket, both re-fetching data the walk already held:
- getObjectVersionList re-read the .versions directory entry that every
caller had just received from listing the parent directory (or from
its own sibling probe). Pass the entry down instead: one RPC saved
per object listed.
- getObjectOwnerFromVersion, on a version with no stamped owner,
re-fetched the same version entry its OwnerID had been extracted
from. The refetch cannot answer differently, so data written before
owners were stamped cost one futile RPC per listed version, forever.
All round-trips on this path are sequential, so on large versioned
buckets (Veeam-style workloads) they add up to a visible share of
per-page latency and gateway CPU.
Claude-Session: https://claude.ai/code/session_01FquvGtTD2zA3uMZGQHAuV4
* s3api: treat a nil .versions entry as an empty version list
filer_pb.GetEntry's contract permits (nil, nil) for an absent entry, and
the old internal lookup answered that case with an empty list. Keep that
answer now that the entry arrives from the caller.
Claude-Session: https://claude.ai/code/session_01FquvGtTD2zA3uMZGQHAuV4
The .versions lookup retried every error through the full backoff
ladder, so a missing key spent 12.7s (8 attempts, 100ms..6.4s) before
the pre-versioning fallback could answer. NotFound is an answer, not a
transient failure: gate the retries on isRetryableFilerErr, the same
classifier retryFilerOp already uses, which also stops retrying for
callers whose context is canceled or past its deadline.
GetObject already treats NotFound on .versions/ as definitive; this
brings the retention/tagging/ACL/attributes/delete/copy paths that go
through getLatestObjectVersion in line with it.
Claude-Session: https://claude.ai/code/session_01FquvGtTD2zA3uMZGQHAuV4
* filer.sync: count received, processed, and failed events and export in-flight jobs
The metadata processor admits at most -concurrency jobs and blocks the
subscription stream past that, so the backlog lives in the source filer's
metadata log and cannot be counted here. What can be measured honestly:
events read off the stream, replication outcomes, and worker saturation.
in_flight_jobs pinned at the concurrency limit means the sync itself is
the bottleneck; near zero means it is caught up or starved by the source.
Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw
* filer.sync: export replication lag in seconds
Lag is now minus the freshest of the processed watermark and the last
idle heartbeat: the watermark stops at the last real event, so a quiet
caught-up stream would otherwise show phantom lag. A ticker drives the
gauge because the offset callback only fires while events flow and
freezes exactly when the workers are saturated. Until the first event
or heartbeat the gauge stays unset rather than reporting lag against a
zero or stale resume offset.
Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw
* filer.sync: track replicated data sizes alongside event counts
An event count hides that 32 in-flight jobs can be 32 KB or 300 GB. Byte
counters mirror the event counters, and in_flight_bytes pairs with
in_flight_jobs. An event's size is the chunk delta - new chunks the old
entry does not already have - so deletes, renames, and attribute-only
updates count zero and byte rates reflect data movement, not metadata
churn.
Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw
* grafana: chart the new filer.sync metrics
The lag panel reads lag_seconds directly instead of deriving it from
sync_offset, and the sync row gains event rate, throughput, and the
in-flight jobs and bytes gauges.
Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw
* filer.sync: a pinned failure keeps showing as lag
An idle heartbeat means the stream is consumed, not that every event
replicated. While a permanent failure pins the watermark, letting the
heartbeat advance lag_seconds or the sync_offset gauge would report a
caught-up stream with an unreplicated event in it, so both now ignore
heartbeats until a restart replays the failure.
Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw
* filer.sync: in-flight gauges survive subscription retries
A subscription retry builds a new processor sharing the gauge children
while the old processor's jobs may still be draining, so setting the
gauge from either side's local count clobbers the other. Each job now
increments and decrements for itself, keeping the total truthful across
generations.
Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw
* master: keep the periodic growth scan to data centers hosting the layout
The rack-aware scan planned growth for every data center in the topology,
so a collection pinned to one DC (fs.configure -dataCenter) sprouted
volumes in all the others within one scan cycle. Plan only for data
centers already hosting the layout's volumes; an empty DC gets its
volumes from the DC-constrained assign that first asks for them. The
lastGrowCount divisor likewise counts only the racks the scan can plan
for.
Claude-Session: https://claude.ai/code/session_01J22TVTyoCMzdHyJirsLMG5
* master: pin periodic must-grow growth to a single-DC layout's data center
The must-grow and crowded paths of the periodic loop grow with no
DataCenter, so even with the scan fixed a pinned collection's volumes
could still land in any DC once lastGrowCount demands more writables.
Stamp the grow request with the layout's data center when its volumes
all live in one; layouts spanning DCs keep unconstrained growth.
Claude-Session: https://claude.ai/code/session_01J22TVTyoCMzdHyJirsLMG5
* master: never pin growth of a cross-DC-replicated layout
A layout whose replication spans data centers cannot legitimately live
in one DC; observing a single hosting DC there means the other DCs are
down. Do not encode that outage as a placement constraint.
Claude-Session: https://claude.ai/code/session_01J22TVTyoCMzdHyJirsLMG5
* master: bound the hosting-DC walk by the answer it needs
listVolumeDataCenters walked every location of the layout under
accessLock — ~190ms for a million volumes, twice per layout per cycle,
stalling assigns behind the read lock. Stop once enough distinct DCs
answer the caller's question: two for the single-DC check, the
topology's DC count for the scan. A spanning million-volume layout now
finishes in microseconds; only a layout truly confined to fewer DCs
still pays a full walk, the same cost class as the under-replication
count this loop already takes each cycle.
Claude-Session: https://claude.ai/code/session_01J22TVTyoCMzdHyJirsLMG5
Three dispatchers reuse one CommandEnv: the interactive shell, the master's
maintenance script runner, and the plugin worker's admin script handler. The
first two were fixed; this is the third.
It changes nothing today -- ForceNoLock already exempts this path -- so it is
here to keep the rule the same everywhere rather than resting on that exemption
staying in place.
* volume server: open volume files with O_NOATIME
Nothing reads the atime of .dat, .idx, .sdx, or EC files, but every
needle read still dirtied the inode: even relatime writes atime on the
first read after each write, so an actively written volume paid a
metadata write per read/write cycle, and strictatime mounts paid one
per read. Open the serving handles with O_NOATIME, falling back to a
plain open when the file belongs to another owner (EPERM).
Claude-Session: https://claude.ai/code/session_015uVY4diBgEn3VYQoc2eMuD
* seaweed-volume: mirror the O_NOATIME volume file opens
Same change as the Go volume server: serving handles for .dat, .idx,
.sdx, .ecx, .ecj, and shard files open with O_NOATIME on Linux, with a
plain-open fallback on EPERM.
Claude-Session: https://claude.ai/code/session_015uVY4diBgEn3VYQoc2eMuD
* route the tier-down and recreate .dat opens through the no-atime helper
Review caught the Rust tier-down swap opening the local .dat directly.
The Go swapToLocalDatBackend and the zero-length read-only .dat
recreate in maybeWriteSuperBlock had the same gap: all three install
long-lived serving handles.
Claude-Session: https://claude.ai/code/session_015uVY4diBgEn3VYQoc2eMuD
* master: scope the startup capacity shed to a truly empty topology
The retryable "no volume server capacity registered yet" shed checked
capacity for the requested disk type, so a cluster serving only other
media -- where that capacity will never register -- shed every assign
until the client's deadline instead of failing fast. An unsteered write
to such a cluster hung for its full HTTP deadline and surfaced "context
deadline exceeded" in place of "No writable volumes". Shed only while
no disk type has any registered capacity, and name the unserved medium
in the fast failure.
Claude-Session: https://claude.ai/code/session_01TF7FQghfDkpdoZgakTMX4R
* master: name the unserved medium for every fail-fast caller
The diagnostic sat in the growth-initiator block, so a follower joining
an in-flight growth and a growth-disabled master failed the same way
with only the generic pick error. Wrap at the fail-fast break instead,
which every caller reaches, and cover all three paths in the test.
Claude-Session: https://claude.ai/code/session_01TF7FQghfDkpdoZgakTMX4R
noLock says "this invocation changes nothing" -- volume.balance, volume.move,
volume.copy, volume.merge and volume.fix.replication all set it for a dry run,
and none clears it. The CommandEnv is created once and reused by both
dispatchers, the interactive shell and the master's maintenance script runner,
so a simulation left every later command unlocked:
volume.balance -noLock # changes nothing
volume.move ... # mutates, and skips its lock
Reset before dispatch in both, where the invocation begins. forceNoLock is
untouched: that is set once, deliberately, for a trusted path.
* s3: resolve admin bucket subresources to their specific S3 actions
Encryption, requestPayment, publicAccessBlock and ownershipControls
requests reached the policy engines as s3:*, so only a policy granting
all of s3 could authorize them. Map each subresource to its AWS action,
with DELETE sharing the PUT permission as AWS does.
Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ
* s3: authorize CreateBucket as s3:CreateBucket in the policy engine
A plain bucket-level PUT is registered with ACTION_ADMIN, which resolved
to s3:*, so no attached policy short of s3:* could match it. Federated
sessions whose policy explicitly allowed s3:CreateBucket were always
denied while the same policy worked for object operations. Resolve it to
s3:CreateBucket, like DeleteBucket already resolves.
Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ
* filer_pb: walk a re-delivered directory only once in TraverseBfs
A directory handed back twice by a listing (a page-boundary race with
concurrent renames, or a store whose ordering misbehaves) was enqueued
twice; the second walk re-lists the same subtree and can keep the
traversal from ever terminating.
Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ
* filer_pb: fail a directory listing whose pagination stops advancing
A full page ending on the very name the cursor started from re-fetches
the same page forever; a store whose listing order does not advance past
the cursor turns any full-directory read into a silent infinite loop.
Return an error naming the stuck cursor instead.
Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ
* shell: skip foreign-collection manifests in fs.mergeVolumes
Every manifest chunk in the namespace was resolved, downloading its
manifest needle, even when the merge plan only touches one collection.
Sub-chunks live in the manifest's own collection, so a manifest on a
volume outside the plan's collections cannot reference a source volume;
skip it and spare a cluster-wide download pass that looks like a hang
after the real moves finish.
Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ
* sts: session duration no longer clamped to the web identity token exp
The assumed-role session lifetime is governed by DurationSeconds and the
configured tokenDuration/maxSessionLength, matching AWS. Clamping to the
already-verified token's exp made short-lived id_tokens (GitLab issues
~2-minute ones) yield unusable sessions regardless of configuration.
Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ
* sts: cover session duration against short-lived web identity tokens
The mock OIDC provider now carries the token exp through to the identity
like the real provider, so the integration test would catch the clamp.
Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ
* get volume topology info without volume details
Signed-off-by: lou <alex1988@outlook.com>
* master: rename VolumeListRequest.without_volumes to topology_only
The field shapes the reply rather than selecting volumes, and it leaves
out the ec shards too, which the old name denied. Match the message's
*_only style and say what a master that predates the field does with it.
Claude-Session: https://claude.ai/code/session_01QHnaNRgxnjzZsiz7WTFML5
* master: refuse topology_only combined with a volume selector
A topology_only request that also names a collection or volume ids
contradicts itself, and answering either half in silence surprises the
caller. Answer InvalidArgument from both VolumeList and its stream,
before the stream sends its header.
Claude-Session: https://claude.ai/code/session_01QHnaNRgxnjzZsiz7WTFML5
---------
Signed-off-by: lou <alex1988@outlook.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
mq: remove emptied topics after the cleanup iteration, not inside it
cleanupIdlePartitions called manager.topics.Remove from inside
manager.topics.IterCb. IterCb holds the shard's read lock while running
the callback, and Remove takes the same shard's write lock, so removing
an emptied topic self-deadlocked the cleanup goroutine. The pending
writer then blocked every later reader of that shard, permanently
hanging ListTopicsInMemory and, for shard-mates, TopicExistsInMemory.
On the Kafka gateway this surfaced as flaky e2e consumer-group tests:
one minute after any earlier topic went idle, the broker's first
'Removing empty topic' wedged the map, every gateway
ListTopics/TopicExists RPC burned its full 5s timeout, Metadata could no
longer finish inside kafka-go's 5s coordinator deadline, and consumer
groups looped in PreparingRebalance until the test timed out.
Collect the emptied topic keys during the iteration and remove them
afterwards via RemoveCb, re-checking emptiness under the shard lock so a
topic that just gained a partition is kept.
Claude-Session: https://claude.ai/code/session_014yA6c8JQcY6MqPXCT13yYA
An assign arriving before any volume server has heartbeated saw zero
available space and failed outright with a plain error no client retries,
so the first write to a fresh bucket answered 500 while the cluster was
still starting. Distinguish a topology with no registered capacity from a
genuinely full one: fail fast only when registered capacity is exhausted,
and shed ResourceExhausted otherwise so the client's retry budget rides
out the startup window.
Claude-Session: https://claude.ai/code/session_018G9kWFgy8BaBAEkYV3YL9n
* ec: refuse to mount a 0-byte shard file when the index has entries
The startup scan already skips (and eventually deletes) zero-sized shard
files as residue of a failed copy, but the mount RPC path opens the file
directly with no size check, so an explicit VolumeEcShardsMount over a
truncated file registers a size-0 claim. A registered empty shard serves
nothing while advertising ownership: with placement pinned to the owning
disk, it would keep attracting re-copies to a file that was never valid.
The one legitimate 0-byte shard is the empty volume's: encoding a volume
with no live needles produces a 0-byte .ecx and 0-byte shards, and that
mount must keep working (TestMountEcShards_EmptyEcxMountsSuccessfully).
So the gate compares against the index: AddEcVolumeShard (Go) and
EcVolume::add_shard (Rust) refuse a 0-byte shard file only when the
volume's .ecx has entries. Go's AddEcVolumeShard grows an error return
for this; the loader cleans up the refused shard and, when it just
created the EcVolume, unregisters that too. The mount loop already
collects non-ENOENT failures per disk and keeps scanning, so a sibling
disk holding a real copy still wins.
Regression tests in both trees: an empty shard beside an index with
entries is refused and leaves nothing registered; an empty shard of an
empty volume still mounts.
Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9
* ec: release the duplicate shard when a mount retry re-loads it
Review follow-up: AddEcVolumeShard keeps the existing shard and reports
added=false for a shard this disk already registered, but the loader
discarded that result, so every retried LoadEcShard leaked the duplicate
it had just opened — an fd and a mount-gauge increment per retry. Release
both and return the existing volume. Regression test pins the gauge.
Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9
* ec: close the test DiskLocation instead of only its EC volumes
Review follow-up: DiskLocation.Close() also stops the background
goroutine NewDiskLocation starts; closeEcVolumes left it running for the
rest of the test process. Both uses are this PR's own tests.
Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9
* rust: unregister the just-created EcVolume when its first mount is refused
Review follow-up: when the first mount of a volume rejects its shard
(e.g. the new 0-byte-beside-nonempty-index refusal), the Rust mount path
had already inserted the EcVolume and propagated the error without
removing it — a zero-shard registration advertising a mount that serves
no data while pinning the .ecx/.ecj descriptors (and, since placement's
mounted tier keys off it, steering shard placement at this disk). Remove
it on the way out, exactly as the Go loader already does; a volume that
already holds shards keeps them (the RPC's first-error-aborts contract).
Regression test covers both.
Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9
* rust: skip already mounted shards on a mount retry
Review follow-up: EcVolume::add_shard replaces self.shards[id] for a
shard the volume already holds, and the mount loop then bumps the
ec_shards gauge although the mounted count did not grow — gauge drift on
every mount retry, and a serving fd swapped for no reason. Skip shard
ids the volume already reports, mirroring Go's AddEcVolumeShard
added=false handling. Regression test pins the gauge across a duplicate
mount (unique collection label: the gauge is process-global and tests
run in parallel).
Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9
* ec: pin auto-selected shard placement to the disk that already owns the shard
A multi-disk server legitimately mounts one EC volume on several disks, so
FindEcShardTargetLocation's per-volume tiers tie at "mounted" and the
free-shard-count tie-break decides — pointing at whichever disk is emptier,
not at the disk that already holds the shard being placed. A re-copy of a
shard the server already has (a retried ec.balance / ec.rebuild move) then
lands on a sibling disk, and both disks register the same (volume, shard id):
the shard is reported to the master from two disk ids, and which claimant
serves reads or survives a later unmount/delete becomes an accident of
Locations order.
Add a tier above "mounted": a disk that already claims one of the shard ids
being placed wins, ahead of the space filters too — re-copying in place
needs no new shard slot, and a genuinely full disk should fail the write
rather than silently split the claim. Applied to the Go selector and the
VolumeEcShardsCopy auto-select (ReceiveFile refuses mounted EC volumes, so
no claim can exist there) and mirrored in the Rust volume server.
Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9
* ec: refuse a copy batch whose shards are already owned by different disks
Review follow-up: ownership-aware selection ranks a mixed-owner batch
(shard 0 on disk A, shard 2 on disk B — the legitimate multi-disk spread)
into one destination, so the copy would still duplicate the losing disk's
claim. No production caller sends such a batch (balance moves one shard,
rebuild and encode copy shards the target lacks), so fail closed: report
every owning disk via Store.EcShardOwnerDisks and refuse the copy with an
error naming them, telling the caller to split per shard or pass disk_id.
Go and Rust, with unit tests for the owner-reporting contract.
Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9
On 32-bit systems the generation fields becomes unaligned. Not an issue
on 64bit.
Discovered on 32bit raspbian, validated by manual patch (in addition to
test).
* ec: uniform shard block layout
An EC volume is striped as 1GiB blocks until less than one row remains, then
1MiB blocks, and consecutive blocks land on different shards. With ec.encode's
-fullPercent 95 against the 30GiB default limit, ~30% of every volume sits in
that 1MiB tail, so a 4MB filer chunk there is five stripes on five servers.
New encodes now use one block per shard, sized ceil(datSize/dataShards) rounded
up to 1MiB and recorded in the .vif (EcShardConfig.block_size, also carried by
the .ecsum manifest). A needle now maps to one shard unless it is larger than
the block or straddles a boundary. The chosen size equals the legacy layout's
padded shard length for every input, so shard sizes, capacity math, and the
shard-size credibility checks are unchanged; only the byte placement moved.
Reads, decode, and scrub resolve the block sizes from the volume's .vif;
absence keeps the legacy interpretation, so existing EC volumes read exactly as
before. Rebuild is layout-agnostic. weed fix -ecx recovers the layout from the
.vif, else the .ecsum sidecar, and with neither de-stripes under both candidate
layouts and keeps the one that indexes more valid needles.
Same change in the Rust volume server, which now also streams the encode in
256KB sub-batches like Go instead of allocating whole blocks, and computes the
large-row count as shardSize/largeBlock to match Go on exact multiples. On a
26MB fixture both encoders produce byte-identical shards, and a Go-written .vif
parses in Rust with the block size intact.
* ec: resolve the rust ecx rebuild through the recorded layout
The Rust rebuild path regenerated a lost .ecx by scanning the logical .dat
through a hand-rolled pure-1MiB striping, which was already wrong for legacy
volumes with large-block rows and is wrong for any uniform volume with a block
past 1MiB. Route the scan through locate_data with the .vif-recorded block
size, the same mapping the read path uses. Also seed the new tests' random
data instead of the deprecated global math/rand.Read.
* ec: fail the Rust ecx rebuild on any shard read error
A read error mid-scan published the entries collected so far as a
successful .ecx, and read_at's byte count was ignored so a legal short
read passed as complete — a truncated or failing shard could produce a
silently incomplete recovery index. Exact-read semantics in
read_from_data_shards, error propagation in the needle walk, and a
truncated-shard regression test.
* ec: fail the mount on an unreadable or malformed vif
Both servers silently fell back to the legacy layout when an existing
.vif could not be read or parsed. Every new encode records a positive
uniform block size there, so the fallback mounted the same shards with
legacy offset math and could return wrong data. Absent stays legal
(legacy volumes predate the sidecar), and a zero-byte stub still reads
as absent (Go's MaybeLoadVolumeInfo convention, now mirrored in Rust);
a present-but-unreadable or malformed .vif fails the mount instead.
* ec: bound the reconstruct fan-out of one needle's intervals
A degraded interval fans out a read to every reachable shard location, each
with a buffer the size of the interval. Reading a needle's intervals in
parallel multiplied that by the interval concurrency: a needle spanning 8
blocks could hold 8 x MaxShardCount remote reads and buffers at once, where
the sequential version peaked at MaxShardCount. Give each needle a single
reconstruct budget its intervals share, held for the buffer's lifetime, so
separate reads stay independent but one read cannot multiply its own
fan-out.
* ec: drop the duplicated shard-size formula
calculateExpectedShardSize reimplemented the padding rule that
UniformBlockSize already owns — TestUniformBlockSizeMatchesLegacyShardSize
asserts the two agree for every input — so a change to the rule would have
had to be made in both. Defer to the helper, keeping the historic answer for
an empty .dat.
* ec: resolve the shard block layout from whatever records it
Four places still answered the layout question by inference when a record of
it was available, or accepted an answer that was not one:
- A mount with no .vif defaulted to the legacy layout; the bitrot sidecar
records the same config at encode time, so take it when present, as
weed fix -ecx already does. The vif itself is now parsed once per mount
rather than twice.
- The Rust ecx rebuild derived its row count from the padded shard extent,
which under the legacy layout reads a shard that is an exact large-block
multiple as one row too many. Pass the encode-time .dat size from the .vif
and keep the extent as the fallback.
- weed fix -ecx read the block size outside the EC-config guard (collapsing
the unknown sentinel into a definitive legacy), only wrote the recovered
layout back when the .vif was absent rather than unusable, and broke a
scan tie by candidate order instead of the documented reach.
- The uniform layout tripped writeDatFile's large-block ambiguity guard,
which cannot apply when the large and small blocks are the same size.
* ec: give the index-recovery tests a parseable vif
The fixtures wrote the literal bytes "volinfo" as the source .vif and the
recovery copies it verbatim, so the receiving server then mounted the volume
from a .vif it could not parse. That used to pass by silently defaulting to
the legacy layout; a mount now refuses a vif it cannot read, which is what
the tests were exercising all along without meaning to.
* ec: validate the layout a vif records, not just its syntax
Review follow-ups on the mount-strictness change:
- A .vif can parse and still record a block size no encoder could have
produced (negative, or not a whole number of small blocks). Both servers
took it and mapped every read through it. ValidateBlockSize / the Rust
mirror now refuse the mount, the same way an unparseable vif does; 0 stays
valid as the legacy two-tier layout.
- The bitrot-sidecar fallback accepted parity_shards == 0 and summed the
counts in their own width, so values near the ceiling wrapped past the
MaxShardCount bound. Require both counts and sum in a wider type.
- weed fix -ecx treated a config with only DataShards > 0 as usable, so a
half-written .vif suppressed the recovery paths AND survived the rewrite.
Require a complete, in-range config before trusting it.
- Returning the vif-load error left the .ecx and .ecj descriptors open;
repeated mount attempts on malformed metadata could exhaust them.
* ec: refuse to act on a layout the metadata does not establish
- The worker encode only logged a failed .vif write and skipped it in the
distribution set, and treated the .ecsum write as best-effort. A worker
whose disk filled after the much larger shards landed could still
distribute, mount, verify shard inventory, and delete the source replicas —
leaving holders with shards whose geometry nothing records. Both writes and
both inclusions are encode success conditions now.
- A generation-matching .ecsum that disagreed with the .vif geometry only
disabled checksums in Go, and in Rust was not compared at all, so
protection stayed On while reads used the other layout. Both files record
the layout their generation was encoded with, so a disagreement now fails
the mount.
* ec: reject an invalid recorded block size in weed fix -ecx
A .vif with valid shard counts but a negative or unaligned block size was
marked usable: a positive invalid value pinned the scan to a geometry that
de-stripes to garbage, and a negative one ran the dual scan but left the
invalid .vif in place afterwards. Validate it with the same rule the mount
applies, and when it fails leave the layout unknown so the scan recovers it
and the file is rewritten.
* ec: validate the sidecar layout weed fix -ecx recovers from
The .ecsum fallback was taken on DataShards > 0 alone, so a CRC-valid
sidecar carrying the wrong generation, an incomplete ratio, or an unaligned
block size would pin the reconstruction to one incorrect uniform-layout
candidate instead of letting the dual scan decide. Require generation 0, a
complete in-range ratio, and a valid block size; anything less leaves the
layout unknown, which is the answer that still recovers by scanning.
* ec: let only a genuinely absent sidecar choose the legacy layout
With no .vif the bitrot sidecar is the only record of a volume's layout, and
the mount fallback read a failed load, an unusable config, or a sidecar
stamped for another generation as "assume legacy". A uniform generation-0
volume could therefore mount with legacy or another generation's geometry and
answer reads with the wrong bytes. Present-but-unusable now fails the mount;
only actual absence keeps the legacy defaults. Shared as
EcShardConfigFromSidecar so every caller reads the sidecar the same way.
* ec: treat a recorded-but-impossible layout as corruption, not as legacy
- A .vif whose ecShardConfig is PRESENT but records an impossible ratio was
answered with the default 10+4 and the legacy block layout, in both
languages. That reads a uniform volume's shards at the wrong offsets and
returns the wrong bytes. Only an entirely absent config still means "this
predates the record"; a present one that cannot be true fails the mount.
- The shard-count bound summed two uint32 counts as int, which wraps on a
32-bit build: 0x7fffffff + 0x7fffffff lands at -2 and slips under
MaxShardCount. ValidEcShardCounts sums in uint64, and every EC call site
that checked a recorded ratio now goes through it.
* ec: rebuild on the geometry the sidecar records, and flag it when it disagrees
The rebuild RPC passes BackgroundECContext, so RebuildEcFiles resolves the
layout itself — and it resolved a missing or invalid .vif to the default 10+4
with the legacy block size. Two consequences: a 12+4 volume was reconstructed
through a 10+4 matrix, which produces wrong bytes and never regenerates
shards 14-15; and the chosen geometry then contradicted a valid uniform
sidecar, which loadRebuildSidecar reported as BitrotOff — silently skipping
the input and regenerated-shard checksum checks precisely when the volume had
already lost its metadata.
The layout now resolves from the bitrot sidecar (found across the server's
disks, not just beside the base name) before falling back to the defaults,
and a present-but-impossible ratio fails instead of being replaced. A sidecar
that contradicts the chosen geometry is BitrotInvalid, which the existing
unsafeIgnoreSidecar override still lets an operator push past.
* ec: let the Rust rebuild read metadata off a sibling disk
read_ec_shard_config searches only the location the rebuild writes into, so a
volume whose .vif or generation-0 .ecsum sits on another of the server's
disks resolved to the default 10+4 with the legacy block layout — the Rust
half of the geometry-guessing the Go rebuild just stopped doing. It then
reconstructs a custom-ratio or uniform volume through the wrong
Reed-Solomon matrix and de-striping geometry.
The rebuild now looks for the .vif in its own location and then each sibling,
falls back to the generation-0 sidecar wherever that lives, and only defaults
when neither exists anywhere. The encode-time .dat size the ecx rebuild needs
is resolved the same way.
* ec: resolve a rebuild's vif from every directory that may hold it
RebuildEcFiles probed only <data-base>.vif. The caller knows the selected
location's index directory and the sibling locations, but passed neither for
metadata: additionalDirs carried shard directories only, and were searched
for shards and the checksum sidecar. A split -dir/-dir.idx layout, or a disk
holding only shards, therefore resolved a pre-sidecar custom-ratio volume to
10+4 and reconstructed through the wrong matrix — never regenerating shards
14-15.
The caller now hands over the index and sibling directories, and the resolver
probes the vif across all of them, matching what the Rust resolver already
does for both the vif and the sidecar.
* ec: make every rebuild consumer agree on the layout it resolved
- The post-rebuild bitrot backfill re-derived the geometry from this
directory's .vif alone and dropped the block size entirely, so a rebuild
that resolved its layout from a sibling, the sidecar, or a uniform vif wrote
a manifest describing a DIFFERENT layout — one later mounts reject, or that
covers only the default shard count. The layout is resolved once now,
through an exported ResolveRebuildECContext, and the rebuild and the
backfill share that answer.
- The Rust rebuild collected only each location's data directory, so a
sibling's INDEX directory — where a split -dir/-dir.idx layout keeps
.ecx/.ecj/.vif — was never probed, and a custom-ratio volume still resolved
to 10+4 with the legacy layout. Both directories of every location are
carried now, deduped against the rebuild's own.
- A shard delivery can bring the checksum manifest with it, but the receive
path only writes the file: a server that already had the volume mounted kept
its resolved protection state (off) until a remount. The mount RPC
re-resolves it once the shards it describes have been added.
* ec: cover the rebuild's directory search with tests
Reviewers flagged the sibling index directory twice, and the fix that
closed it had no test of its own: the assembly sat inline in the rebuild
handler, reachable only through a gRPC call against a populated store.
Lifting it into rebuildSearchDirs / select_rebuild_location makes the
rule assertable — a sibling contributes BOTH its data and its index
directory, a shared index directory is listed once, and the rebuild's own
data directory never repeats.
Writing the Rust cases surfaced that the two implementations do not agree
on where the rebuild's own index directory belongs, and both are right:
Go's resolver takes a single directory list, so that directory has to be
inside it, while Rust's takes the rebuild's data and index directories as
their own arguments and would search them twice. The tests now state
which contract each side is holding to, so neither drifts into the
other's shape.
Pure refactor otherwise; no behaviour change.
* ec: search the index directory for the layout sidecar
The Rust resolver looked for the generation-0 .ecsum in the rebuild's
data directory and the sibling list, but not in the rebuild's own index
directory — while the .vif lookup directly above it did, and Go's
findBitrotSidecar has always checked both bases. On a split -dir/-dir.idx
location that directory is where the metadata lives, and callers leave it
out of the sibling list precisely because it is passed here separately,
so nothing searched it.
With no .vif anywhere the sidecar is the only surviving record of the
layout. Missing it resolved a 12+4 uniform volume to 10+4 with the legacy
striping — the test added here fails with (10, 4, 0) against the old
code — and the rebuild then reconstructs through the wrong matrix and
writes .ecx offsets that no reader can follow.
* ec: let the rebuild see its own index directory
The Rust rebuild takes a single flat directory list — the shape Go's
RebuildEcFiles uses — so it cannot be handed the rebuild location's index
directory separately the way the layout resolvers are, and the handler
was passing the sibling list, which deliberately omits exactly that
directory. On a split -dir/-dir.idx location that is where .ecx and .vif
live, so the shard and index lookups could not see them.
Go has always carried that directory in additionalDirs; this lines the
two call sites up.
* ec: let a config-free vif fall through to the layout sidecar
A .vif that carries no ecShardConfig answers nothing about the layout, so
it is no more informative than an absent one — but both trees treated its
mere existence as the end of the search. Go went straight to the 10+4
legacy defaults without consulting the sidecar at all; Rust returned
whatever ec_shard_config_from could make of a single directory. A 12+4
uniform volume with a legacy config-free vif therefore resolved as 10+4
legacy, and every read landed at the wrong shard offset.
The sidecar lookup was also single-directory on both sides, while a split
-dir/-dir.idx layout keeps .vif and .ecsum with the INDEX. Go's
findBitrotSidecar has always taken both bases; the callers here passed
only the data base, and the Rust bitrot resolver derived its path from
the data base alone. Rust's layout resolver now takes a candidate
directory list — data, index, then any siblings — and searches all of it,
which also removes the early return that made the vif's presence
decisive.
load_vif_info_across_dirs reported `dir` even when load_vif_info had
found the vif in `dir_idx`. Nothing reads that field today, so this
changes no behaviour; it stops the next caller that resolves the rest of
the volume's metadata against the answer from being sent to a disk
holding none of it.
Absence stays legal throughout: a volume with neither record is genuinely
legacy. Present-but-unusable still fails the mount, now in the
config-free-vif branch too.
* ec: activate a delivered sidecar on every per-disk runtime
A vid mounts as one EcVolume per disk, each with its own resolved
protection state, but the post-delivery reload used the first-match
lookup and so touched exactly one of them. The siblings kept reporting no
protection until a remount — and since shard distribution deduplicates
the metadata files onto the first target disk for a node, the runtime
that got the .ecsum is not necessarily the one the lookup returns.
Iterate every runtime instead, via a new FindAllEcVolumes and its Rust
mut equivalent. Combined with each runtime now resolving its sidecar
against its index directory as well as its data directory, a server
sharing one -dir.idx across its disks activates all of them from the
single delivered copy.
The Rust volume server had no post-mount reload at all; it gets one here,
matching Go.
* ec: resolve the delivered sidecar across every EC metadata directory
Reloading every per-disk runtime, added last round, did not by itself
make the delivered manifest reachable. Startup mirroring copies
.ecx/.ecj/.vif to every shard-bearing disk so each mounts
self-contained, but deliberately not .ecsum, and a repair delivers
exactly one copy. Each runtime was resolving against its own two
directories, so every sibling of the disk that received the file kept
reporting no protection however often it reloaded.
Resolve one authoritative copy across every EC metadata directory
instead of duplicating the file. Mirroring .ecsum would have to keep
pace with a file that is rewritten as shards are repaired, and would not
help the reported case at all: the delivery happens at runtime, and
mirroring only runs at startup.
The regression test pins both halves — a reload restricted to the
volume's own directories still finds nothing, and the same reload
given the server's metadata directories turns protection on.
* ec: ask every directory before writing a TOFU baseline
After a rebuild the opportunistic backfill asks whether this volume
already has a checksum manifest, and answered from the data base alone.
A split -dir/-dir.idx layout keeps the sidecar with the index, and a
multi-disk server may keep it on a sibling, so an existing manifest read
as absent.
The consequence is worse than a missed read. On a false "no" the backfill
writes a fresh sidecar at the data base from whatever the shards say right
now — and the data base is the first candidate every resolver checks, so
that TOFU baseline shadows the real manifest rather than sitting beside
it. A shard that was silently corrupt gets blessed, and the record that
would have caught it stops being consulted.
FindBitrotSidecar exports the search the package already used internally,
so the question is asked of the data base, the index base and the sibling
disks — the same candidates the rebuild resolves its layout from.
* ec: refuse a shard block size no encoder could have produced
weed fix -ecx derived one from the raw shard extent, so a truncated or
partially copied shard wrote a .vif that NewEcVolume then permanently
refuses — the volume the tool was run to rescue could never mount again.
An extent that is not a whole number of small blocks cannot have come
from a uniform encode, so it is no longer offered as a candidate, and
nothing unvalidated reaches the .vif.
Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7
* ec: derive the .vif's dat size and block size from one measurement
VolumeEcShardsGenerate stat'ed the .dat before the encode while
WriteEcFiles stat'ed it again to size the blocks. A write landing
between the two produced a .vif whose own two fields describe different
files. WriteEcFiles now leaves both on the context, and fills a
placeholder context in place so the caller can read them back.
Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7
* ec: keep the source volume until every holder serves its shard layout
The uniform layout rides in a .vif field older volume servers never
knew: they discard it, mount the shards as legacy and return wrong bytes
with nothing erroring, and the shard files are the same length either
way so no other check notices. The upgrade order lived only in the
release note. VolumeEcShardsInfo now reports the block size the holder
actually serves, in both the Go and Rust servers, and the pre-delete
verification refuses to drop the source unless every reachable holder
echoes the one the shards were encoded with — while a rollback still
exists. A server that predates the field answers 0, which is the
negative answer.
Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7
* ec: drop the rebuild's dead block-size parameters
generateMissingEcFiles never reads largeBlockSize/smallBlockSize —
Reed-Solomon reconstruction is layout-agnostic — so passing the legacy
constants only advertised a layout the rebuild does not use. Also move
UniformBlockSize's doc off ValidateBlockSize.
Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7
* ec: warn about EC defaults only when the mount used them
The "vif file not found, using defaults" warning fired even after the
bitrot sidecar supplied a non-default layout, sending anyone triaging
wrong bytes after the legacy layout the volume never mounted on.
Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7
* ec: stat the distributed bitrot sidecar once
The strict check re-stat'ed the file immediately before the stat that
already gates inclusion, and a failed sidecar write now fails the encode
outright, so the first could only fire on a deletion between the two
lines.
Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7
* ec: say what the reconstruct budget actually bounds
A shard's buffer stays in bufs until its interval reconstructs, which is
after the read that filled it released its permit, so the semaphore
bounds round trips in flight and not retained bytes. Peak memory is the
intervals reconstructing at once times the shards each reaches times the
interval size.
Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7
* test: let the fake volume server report its delivered EC layout
The pre-delete verification now asks each holder which shard block
layout it serves, and a fake that always answered "unset" looked exactly
like a volume server too old to know the field. Distribution ships the
.vif to every holder alongside its shards, so read the layout back out
of it as a real holder does.
Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7
* filter by volume ids
* master: carry the volume ids VolumeList asks about in one repeated field
One id and a list of them ask the same question, so field 2 holds the list
rather than standing beside a second field that supersedes it.
Claude-Session: https://claude.ai/code/session_011qAmAdhrYvnzGkw7A9N4mP
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* Introduce a new `READS` scrub mode.
`READS` performs a full volume scrub but, unlike `FULL`, it will attempt to
reconstruct data for missing/damaged shard intervals from other shards in the cluster
when necessary.
The goal of this check is to ensure that EC volume contents _are readable by Seaweed_
even on a degraded storage state, by exercising parity data which is not read in `FULL`
mode. This is useful not only to validate data is user-readable, but also to detect potential
parity shard issues which may be difficult to pinpoint otherwise - particularly for older
volumes lacking sidecar data, and hence unaffected by `CHECKSUM` scrubs.
For regular volumes, this operation is equivalent to `FULL`.
Example:
```
> ec.shard.unmount --volumeId=1 --shardId=0,3,11 --delete --apply
Live shard topology for volume ID 1 (14 shards):
0@10.200.18.89:9001
1@10.200.18.89:9002
2@10.200.18.89:9003
3@10.200.18.89:9004
4@10.200.18.89:9005
5@10.200.18.89:9006
6@10.200.18.89:9007
7@10.200.18.89:9008
8@10.200.18.89:9009
9@10.200.18.89:9013
10@10.200.18.89:9010
11@10.200.18.89:9011
12@10.200.18.89:9012
13@10.200.18.89:9020
Will unmount + delete 3 shard(s):
0@10.200.18.89:9001
3@10.200.18.89:9004
11@10.200.18.89:9011
Unmounting shard 0@10.200.18.89:9001 for volume ID 1...
Deleting shard 0@10.200.18.89:9001 for volume ID 1...
Unmounting shard 3@10.200.18.89:9004 for volume ID 1...
Deleting shard 3@10.200.18.89:9004 for volume ID 1...
Unmounting shard 11@10.200.18.89:9011 for volume ID 1...
Deleting shard 11@10.200.18.89:9011 for volume ID 1...
All done!
> ec.scrub --volumeId=1 --node=10.200.18.89:9002 --mode=full
using FULL mode
Scrubbing 10.200.18.89:9002 (1/1)...
Scrubbed 6 EC files and 1 volumes on 1 nodes
Got scrub failures on 1 EC volumes and 1 EC shards :(
Affected volumes: 10.200.18.89:9002:1
Affected shards: 10.200.18.89:9002:1:0
> ec.scrub --volumeId=1 --node=10.200.18.89:9002 --mode=reads
using READS mode
Scrubbing 10.200.18.89:9002 (1/1)...
Scrubbed 6 EC files and 1 volumes on 1 nodes
```
* ec: report the shards a READS scrub had to rebuild
A READS scrub that recovers an interval was recording nothing, so a volume
missing three shards came back clean and nobody repaired it. The unreadable
shard is now recorded before the rebuild is attempted: READS reports the same
broken shards as FULL and differs only in whether the needles themselves
failed, which is the signal worth having - shards are gone, data is still
there.
forceDeletedNeedlesCheck now applies to READS as well, in the shell and in the
RPC guard: it runs the same needle walk as FULL.
Regenerated the proto instead of hand-editing it, so the pancis typo (which
protoc-gen-go-grpc emits into eight other files here) and the header whitespace
stay as generated.
Mirrors into the Rust volume server, which also now honors
force_deleted_needles_check rather than hardcoding it off.
Claude-Session: https://claude.ai/code/session_014yMNebkUjSbx9sfUCWJJtq
* ec: answer a deleted needle from a READS rebuild as deleted
#11020 gave the Rust recovery a deleted flag alongside its bytes, and it
answers a deleted needle with no bytes at all. The READS scrub appended that
empty answer, which does not compile against the new signature and, once it
did, would leave the needle short and report the size mismatch as damage.
Zero-fill the interval instead, the way the direct read beside it already
does: the assembled needle then reaches read_bytes as the delete-state
mismatch the walk already tolerates. Go takes the same branch off the flag
its recovery returns, rather than discarding it.
Claude-Session: https://claude.ai/code/session_014yMNebkUjSbx9sfUCWJJtq
---------
Co-authored-by: Lisandro Pin <lisandro.pin@proton.ch>
* master: bound each volume server DeleteCollection, and finish the fan-out
A collection delete fanned out to every volume server holding it with
context.Background(), so a server that accepted the connection and then
went quiet held the whole delete open with nothing to end it. Each RPC is
bounded now, on the same budget allocateVolumeTimeout gives the other
master-to-volume-server admin RPC. The volume server runs the delete to
completion regardless of the request context, so giving up costs the
confirmation and not the deletion.
The walk itself is the caller's, not a per-server one:
- It outlives the caller. A cancelled request must not abandon a
destructive fan-out part-done, with volumes left behind and no request
still running to come back for them.
- It no longer stops at the first server that refuses, which left the
collection on every server after it in the list. The first failure is
still what is reported, and the collection stays in the topology so a
later delete comes back for the rest.
- It sends one RPC per server rather than one per replica.
ListVolumeServers reports a node once for every replica it holds, while
DeleteCollection removes the whole collection from the server it
reaches, so a collection with thousands of volumes repeated the same
whole-collection delete thousands of times over.
Both passes run too. Returning after a failed normal pass left the
collection's EC shards in place with nothing left to retry them.
Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP
* master: delete the EC shards behind /col/delete too
The HTTP handler carried its own copy of the volume-server walk and only
ever ran the normal pass, so a collection deleted through it kept its EC
shards. It shares the gRPC path now, which also gets it the bounded RPCs
and the one-per-server fan-out.
Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP
* filer: bound the collection delete a bucket delete leaves behind
Deleting a bucket entry deletes its collection afterwards, deliberately
detached from the request so a client that hangs up cannot strand the
bucket's volumes. Detached meant unbounded, though: with the master down
or mid-election the wait for a leader has nothing to end it, so the
handler parks, and the client retrying behind it parks another.
It keeps outliving the request and now carries a deadline of its own. The
budget bounds the wait, not the work: the master keeps deleting on its own
fan-out once asked, so giving up costs the confirmation.
Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP
* s3api: bound the collection RPCs a bucket creation and deletion issue
Neither carried a deadline, so a transient failure anywhere down the chain
held the S3 request open until the client gave up on it. Both budgets are
taken outside the filer failover walk, so one budget covers the whole walk
rather than granting each filer a fresh one.
The walk itself stops when that budget is spent, and stops without blaming
anyone: the caller's own expiry is not evidence against the filer that was
answering, and the next filer has no time left to answer in either.
Recorded as a filer failure, a slow master upstream would flag every filer
in the walk, and the three failures that open the circuit take unrelated
object reads down with them.
Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP
* s3api: a failed collection listing no longer fails a bucket creation
PutBucket lists collections to notice a leftover one it is about to reuse.
The result feeds a warning and nothing else -- s3a.exists is what decides
whether the bucket already exists -- yet a transient failure of that
listing returned 500 and refused the creation. It is advisory now, so a
failure is logged and the creation continues, exactly as it does when the
listing returns false.
Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP
* s3: reject a part whose upload was aborted while its body was in flight
The upload-exists check runs before the part body is read. An abort answered
during the read deletes the upload directory, and the part write that follows
re-creates it, so the aborted upload is listed nowhere yet completes.
Re-check after the write: only createMultipartUpload stamps the destination
key on .uploads/<id>, so a directory without it is one the part write
resurrected. Drop it along with the part and answer NoSuchUpload.
Claude-Session: https://claude.ai/code/session_01ByZ49KQdUtHhmjG6SpNczT
* s3: reject a copied part whose upload was aborted mid-copy
UploadPartCopy has the same window as UploadPart: the upload-exists check
runs before the bytes are copied, and the part write that follows re-creates
the directory an abort removed. Both the re-encryption and the raw-copy path
re-check before answering.
Claude-Session: https://claude.ai/code/session_01ByZ49KQdUtHhmjG6SpNczT
* s3: do not complete an upload whose directory holds no upload record
A .uploads/<id> directory that a part write created rather than
createMultipartUpload carries no destination key, no owner and no
encryption settings. Completing one turned stray parts into an object;
answer NoSuchUpload instead.
Claude-Session: https://claude.ai/code/session_01ByZ49KQdUtHhmjG6SpNczT
* s3: log the part left behind when the resurrected directory survives
abortMultipartUpload can fail to remove what the part write re-created. The
client still hears NoSuchUpload, since the upload is gone either way and a
retry would only write another part, but the leftover is worth a line.
Claude-Session: https://claude.ai/code/session_01ByZ49KQdUtHhmjG6SpNczT
* ec: split the shard-interval recovery into a gather and a rebuild
Recovering an interval is now one function doing the local seeding, the waved
peer fetch, the shard accounting and the Reed-Solomon rebuild, under a memory
budget. Splitting the gather from the rebuild makes the rebuild a plain
function over a set of intervals, which is testable on its own and reusable by
the parity checks a full scrub wants.
The rebuild refuses a parity target, and the caller checks that before the
gather so a doomed target costs no fan-out. ReconstructData rebuilds data
shards only, so asking it for a parity shard returned no error and left the
slot nil, and the caller copied that out as a successful read of zeroes. Only
data shard ids reach here today, so this is a guard, not a live fix.
Claude-Session: https://claude.ai/code/session_014yMNebkUjSbx9sfUCWJJtq
* ec: rebuild only the EC shard the read asked for
ReconstructData rebuilds every missing data shard. The gather stops as soon as
DataShards intervals are in hand, so on a distributed volume it routinely
finishes holding parity where data is missing -- and each of those data shards
is then rebuilt into an interval-sized buffer, decoded, and never read. Ask for
the one shard the read needs.
The budget covers it now too: DataShards gathered plus the one the rebuild
allocates. It never covered the rebuild's output, and with ReconstructData that
output was up to ParityShards buffers.
The required mask is Total() long rather than DataShards. reedsolomon documents
both lengths, but its presence scan walks every shard and indexes the short
mask past its end, so the documented short form panics whenever a parity shard
is absent - which here it usually is.
Claude-Session: https://claude.ai/code/session_014yMNebkUjSbx9sfUCWJJtq
* Re-check an EC shard map a failed read has disproved
A read that fails against a cached location drops that shard from the map,
which leaves it one short of complete -- and a map one short is trusted for
seven more minutes. So a moment's trouble between volume servers cost
minutes in which every read of that shard skipped the direct fetch and paid
for a Reed-Solomon recovery instead, at DataShards times the memory and the
peer load.
Mark the map when a read disproves it, and re-check a marked map on the
same eleven-second footing as one that never had enough shards to begin
with. The mark clears on refresh, so it buys one prompt re-check rather
than a master lookup per read. The tiers move into a helper; they were
three overlapping conditions in one expression, and the reading of them
was not obvious.
Rust keeps the entry rather than dropping it -- a dead peer fails fast on
the next attempt, and it was the freshness window, not the entry, hiding a
shard that had moved.
Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN
* Invalidate the location of an EC shard whose own read failed
Recovery fans out to the other shards, so the one whose direct read just
failed is the only location nothing ever invalidates: a shard that moved to
another server was reconstructed on every read until the map's own window
expired, up to thirty-seven minutes for a map still complete. Mark the map
there too. The entry stays -- a moved shard's old holder fails fast, and
the next refresh is seconds away.
Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN
* Consume the stale mark before the lookup, not after
A read that fails while the master is answering has disproved the very map
that answer is about to install, and clearing the mark on the refresh's
return swallowed it. Clear it where it is acted on instead. A lookup that
then fails loses the mark, which costs nothing: the refresh time is only
advanced on success, so the next read looks up regardless.
Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN
* Judge the shard map and consume its mark in one critical section
Reading the mark and clearing it were two separate acquisitions, so a mark
raised between them was cleared by a refresh that had not seen it. In Go
that gap was a few instructions; in Rust the mark was read when the read
first snapshotted the volume and cleared at the decision point, with the
local interval reads in between. Take both under one hold. Rust needs a
mutex rather than an atomic to do it, and no longer carries the mark
through the snapshot.
Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN
* Put the stale mark back when the lookup does not answer for it
Consuming the mark up front assumed the lookup would supersede it. A
lookup that fails, or comes back with fewer than DataShards holders,
supersedes nothing: the map is unchanged, its refresh time unadvanced, and
with the mark gone the map a read had disproved is trusted for its full
window again on the strength of a lookup that never landed. Put the mark
back on both branches.
Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN
* mount: implement fallocate instead of reporting it unsupported
Fallocate answered ENOSYS, so the kernel marked the mount as having no
fallocate and returned EOPNOTSUPP. glibc then fell back to its emulation,
which preads a byte from every block already inside the file to see if it
is allocated; on a write-only descriptor that pread is EBADF, and
posix_fallocate returned it.
Volume space is assigned when a write is flushed, so nothing can be
reserved up front: a range inside the file is answered OK untouched, and
one past the end grows the file the way a truncate would. A mode we
cannot honor is refused with ENOTSUP, not ENOSYS, so the kernel keeps
sending the ones we do.
Claude-Session: https://claude.ai/code/session_01XG6sAAdkqTwWffqK5h4rH9
* mount: let a fallocate that allocates nothing past the quota and worm guards
A range already inside the file, and any FALLOC_FL_KEEP_SIZE request,
reserve no space and rewrite no entry, but the preflight refused them
with ENOSPC on a full mount and EPERM on a worm-enforced file. Decide
the no-op first and guard only the growth.
Claude-Session: https://claude.ai/code/session_01XG6sAAdkqTwWffqK5h4rH9
* mount: charge a fallocate growth to the uncommitted byte counter
Write charges the counter by how much the file grew, so the writes that
fill a range fallocate already extended charge nothing and the real-time
quota check never sees that data — only the periodic filer refresh does.
Count the growth where it happens.
Claude-Session: https://claude.ai/code/session_01XG6sAAdkqTwWffqK5h4rH9
* mount: charge a truncate-up growth to the uncommitted byte counter
Same gap Fallocate had: Write charges the counter by how much the file
grew, so the writes that fill a range ftruncate already extended charge
nothing and the real-time quota check never sees that data. Count the
growth where it happens; a shrink still leaves the counter alone, since
it is only ever raised and then reset by the periodic filer refresh.
Claude-Session: https://claude.ai/code/session_01XG6sAAdkqTwWffqK5h4rH9
* s3: stop retrying a delete the filer refused for a non-empty folder
The filer looked and the children are there, so the answer will not change.
retryFilerOp spent six attempts and up to 3.1s of backoff on it before the
caller could act on the condition it was already holding.
Claude-Session: https://claude.ai/code/session_01XqaJrwgXQ5GSUpyzRbe5nD
* s3: thread the request context through the unversioned delete path
doDeleteEntry issued every DeleteEntry on context.Background(), so an S3
client that hung up left the gateway working on its behalf, out of reach of
both cancellation and the per-request retry allowance that
DeleteMultipleObjectsHandler installs.
Claude-Session: https://claude.ai/code/session_01XqaJrwgXQ5GSUpyzRbe5nD
* s3: treat a cancelled filer RPC as terminal, not transient
isRetryableFilerErr matched context.Canceled and DeadlineExceeded by
sentinel, which only holds while the error is still local. Once it has
crossed gRPC it is a status, so an abandoned request was retried six times
on behalf of a caller that had already gone.
Claude-Session: https://claude.ai/code/session_01XqaJrwgXQ5GSUpyzRbe5nD
* s3: replay a delete whose reply the transport dropped
A delete is idempotent at the filer, which answers an entry that is already
gone with an empty resp.Error, so a reply lost in transit can be reissued
rather than surfaced. Surfaced, it becomes a 500 on the bucket delete, which
boto3 resends and is then answered NoSuchBucket, or a per-key InternalError
inside the 200 of a multi-object delete, which no SDK retries at all.
The replay runs through retryFilerOp, so it draws on the allowance the
request already installs rather than paying a backoff per key, and stops for
a caller that has gone. rm and rmObject re-enter WithFilerClient per attempt,
so each one walks the failover list again on a connection the failed attempt
had invalidated; the multi-object loop holds one client for the batch, so
there the replay reuses it.
Classification stays structural. The filer reports its own refusals in
resp.Error, which carries no status and has the deleted path - and, for a
recursive delete, the children it stopped on - formatted into it, so no key
name can steer the decision either way.
rm and rmObject now take the caller's context. Cleanup and rollback paths
pass context.Background() deliberately: they have to run whether or not the
caller is still waiting.
Claude-Session: https://claude.ai/code/session_01XqaJrwgXQ5GSUpyzRbe5nD
* s3: share one retry allowance across multipart completion cleanup
The unused-entry loop deletes once per entry, and each delete now retries,
so a filer that stays unavailable held the response for 3.1s per entry after
the object was already committed.
Claude-Session: https://claude.ai/code/session_01XqaJrwgXQ5GSUpyzRbe5nD
* Replay metadata log chunks the way the mount reads every other chunk
The subscription's log-chunk replay built its own lookup, which always
resolves volume server addresses. A mount started with
-volumeServerAccess=filerProxy cannot reach those, so every fresh
subscription failed on the previous minute's persisted segment and
resubscribed a second later, forever. Take the lookup from the caller
instead; the mount hands over the one it uses for file reads, which also
keeps publicUrl and the bounded location cache in play.
Claude-Session: https://claude.ai/code/session_01NGqrxYj7cHUpSrL249n3Z6
* Keep a log chunk read failure off the filer connection
A metadata subscriber reads persisted log chunks over HTTP from volume
servers and hands whatever went wrong back as the subscription's error.
"connection refused" from a volume server then matched the transport
patterns that decide a gRPC channel is dead, so every failed replay
closed the shared filer ClientConn and cancelled the assign and upload
RPCs riding on it with "the client connection is closing". Mark those
read failures so they are judged for what they are.
Claude-Session: https://claude.ai/code/session_01NGqrxYj7cHUpSrL249n3Z6
* Reconstruct an EC shard from the shards already on this server
recoverOneRemoteEcShardInterval only ever fanned out to the cached shard
locations, so a server holding shards of the volume still fetched them
over gRPC from itself -- and when the peers were unreachable it could not
reconstruct at all, even holding the whole volume on local disk. Seed the
Reed-Solomon buffers from the locally mounted shards first; each one is a
peer round trip, and an interval-sized buffer, the fan-out no longer needs.
Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN
* Fetch only the EC shards reconstruction still needs
The recovery fan-out read every surviving shard, so a 10+4 volume pulled
13 interval-sized buffers to feed Reed-Solomon 10 -- a third more memory
held, and a third more load asked of peers that were, by definition,
already having trouble. Fetch what is missing, and widen only when some of
those reads fail. A shard reporting the needle deleted ends the walk: the
rest would only answer the same.
Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN
* Bound the bytes EC recovery holds in flight
Recovery is the one read path that multiplies the served bytes: it holds
an interval-sized buffer per shard until Reed-Solomon runs, and a peer that
is slow to fail keeps them all alive for the whole gRPC timeout. Nothing
bounded how many of those fan-outs ran at once, so a transient problem
between volume servers turned every read into a DataShards-fold allocation
and the server died of it -- 64 concurrent 4MB intervals pin 3.6GB, and
that is a small burst.
Charge each recovery against a process-wide budget, so a burst queues on
the semaphore instead of on the heap.
Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN
* Answer a deleted EC needle as deleted, not as a failed recovery
A holder reporting the needle deleted is authoritative: deletes are never
invented and never undone. Recovery already collected that flag, then
dropped it on the branch where too few shards came back -- so a read of a
deleted needle that had to recover surfaced as "cannot recover shard", and
the volume server answered 500 where it owed a 404. Carry the flag out of
the shortfall, and let it decide ahead of the error it came with.
Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN
* Check the encode run of a locally seeded EC shard in Rust
The Rust recovery seeded Reed-Solomon straight from the mounted shards,
without the encode-run check the remote reads and Go's
readLocalEcShardInterval both apply. A volume remounted from a newer
encode between the read's snapshot and its recovery would have fed
mixed-generation bytes into the reconstruction.
Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN
* Say what the recovery budget actually guarantees
Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN
* Seed Rust EC recovery from shards on every local disk
find_ec_volume returns the first disk's EcVolume, so a reconciled volume
whose shards are split across data dirs had the siblings ignored and could
report "cannot recover" while holding enough shards locally. Resolve each
shard together with the disk that owns it, the way Go's recovery already
does, and check that owner's encode run.
Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN
* Name the failure when the source cluster cannot locate a chunk's volume
LookupFileId formatted a nil err into the message it returned, so the only
thing a caller could do with "no locations for this volume" was match on the
text. Return a typed error instead.
Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK
* Fail a source chunk read on a failure status instead of copying the error page
ReadPart never looked at the response status, so a volume server answering 404
for a needle vacuum had removed came back as a successful read whose body was
the error page. The caller counted those bytes as file content and reported a
size mismatch — a corruption claim about data the source had simply lost — and
a 404 from one replica ended the search instead of trying the next.
Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK
* Stop retrying a chunk the source cluster can no longer produce
A chunk whose volume vacuum has removed fails the same way on every attempt, but
the retry loop had no way to say so and kept going forever. The sync job holding
it never finished, so it pinned the offset watermark at the event ahead of it and
filer.sync never checkpointed again — alive, quiet, and permanently behind.
Wait the source out for a grace period long enough to cover a volume server
restart or a master failover, then give up and mark the failure permanent.
Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK
* Let replication continue past an entry whose source data is gone
An entry the source can no longer read holds the sync offset forever: the event
fails on every replay, so the checkpoint never moves past it and every later
event stays uncheckpointed, however long the sync keeps running. Nothing brings
those bytes back, so skip the entry with an error naming it and carry on.
Skip only while the source is demonstrably still serving other chunks. A volume
with no locations reads the same whether it was vacuumed away or every replica is
down, and during a cluster-wide outage that answer comes back for every chunk —
skipping then would drop live files wholesale.
Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK
* Propagate a missing source chunk instead of waiting when supersession is unverifiable
An incremental sink's dated target keys cannot be mapped back to a source path,
so nothing here can tell a chunk the source lost from one a later version already
replaced. Waiting out the grace period would stall every vacuumed needle for half
an hour; hand the failure to the caller, which has the event's real source key.
Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK
* Wait out a gone volume once, not once per file it held
A volume vacuum removed took every file it held with it, and each chunk was
timing its own grace period. With a bounded chunk executor those waits serialize,
so one gone volume holding many files stalls the sync for far longer than the
grace period — the wedge again, only slower.
Track the wait per source volume on the sink instead: the first chunk to find it
unlocatable starts the clock, every later chunk inherits it and gives up as soon
as it has run out, and a chunk the source does serve clears it.
Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK
* Probe the source with a read, not a lookup, before writing an entry off
A lookup only proves the source master still has the topology. If every volume
server is unreachable while the master still lists them, the probe passed and the
sink wrote off an entry whose data was merely out of reach. Read the probe chunk
instead, and say in the log that the entry stays unreplicated.
Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK
* util, pb: classify a filer error by the status the server sent
DoSeaweedListWithSnapshot wrapped a failed ListEntries with %v, dropping the
gRPC status, so IsTransientError fell back to matching substrings against a
message that now held the caller's path. Keep the status with %w and let it
decide, reading the server's own text rather than the wrapper's.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* s3: keep the bucket and prefix out of the list retry decision
A bucket named transport, or a prefix under logs/unavailable/, made a
PermissionDenied listing look transient and got it retried; a key holding the
not-found sentence suppressed a retry that should have run. Both checks now
read the filer's status, and only fall back to the text when there is none.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* filer, s3: classify a delete failure before the path is wrapped into it
The filer put the non-empty-folder marker behind its own "delete directory %s"
wrapper and the gateway matched it as a substring, so a key named after the
marker turned a real delete failure into the demote-the-marker no-op and the
request answered 204. Keep the marker leading the message that crosses the
wire, turn it back into a sentinel where the response is read, and match that.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* s3tables: report a delete the filer rejected
deleteDirectory discarded DeleteEntryResponse and checked only the
transport error, so DeleteTable, DeleteNamespace, DeleteView and
DeleteTableBucket answered 200 for a delete the filer refused. Call
filer_pb.DoRemove, which reads resp.Error and still treats a missing
entry as success.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* admin: report a delete the filer rejected
The bucket delete, the file browser handlers and the topic retention
purger all discarded DeleteEntryResponse, so a delete the filer refused
came back as success. Call filer_pb.DoRemove, which reads resp.Error.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* credential: report a delete the filer rejected
DeleteUser, DeletePolicy and the full-sync cleanup loops discarded
DeleteEntryResponse, so a rejected delete answered success and left the
credential file in place. The service account path in the same store
already read resp.Error; the rest now do too, via filer_pb.DoRemove
where not-found is already tolerated.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* shell: report a delete the filer rejected
remote.configure -delete, remote.cache and the remote metadata sync
discarded DeleteEntryResponse, so a rejected delete printed as removed.
Call filer_pb.DoRemove, which reads resp.Error.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* mq: report a delete the filer rejected
The consumer offset group purge and the coordinator assignment delete
discarded DeleteEntryResponse. Call filer_pb.DoRemove, which reads
resp.Error.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* iam: count only the revocation entries the filer actually deleted
The expiry sweep discarded DeleteEntryResponse, so a rejected delete was
counted as purged and the entry stayed. Call filer_pb.DoRemove, which
reads resp.Error, matching the role and provider stores beside it.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* mount: fail rmdir when the unary fallback delete was rejected
The streaming branch turns DeleteEntryResponse.Error into an error, the
unary fallback dropped it, so rmdir of a non-empty directory answered OK
off the stream and ENOTEMPTY on it. Surface it in both.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* s3tables: fail DeleteTableBucket when the directory delete is refused
The handler only failed when both the leaf entry and the directory
delete failed, so a refused bucket directory delete still answered 200
with the bucket in place. The directory is the bucket, so it decides;
the leaf entry stays best-effort.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
Every key in a multi-object delete drives its own retryFilerOp, so a filer
that is briefly unhealthy multiplied one op's ~3.1s of backoff by a key
count the client picks. The batch now carries a single allowance in its
context, sized to one op's worst case; once it is spent the remaining keys
fail fast with a per-key error instead of holding the request goroutine.
A single-object delete carries no allowance and keeps its full retries.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* wdclient: bound the wait for a master leader by the caller's context
WithClient waited on GetMaster with context.Background(), so a caller that
arrived while no master leader was known parked in a 200ms poll loop until one
appeared, whatever deadline it had already set on the RPC. Each retry above it
then left another goroutine in the same wait.
Take the context in WithClient and WithClientCustomGetMaster and hand it to
GetMaster, and stop the retry loop once it is done. The dial keeps
context.Background(): fn brings its own RPC context, so a cancellation seen
here cannot be attributed to the shared connection.
Call sites pass whatever they hold: the request context in the filer's
CollectionList, DeleteCollection and Statistics handlers and in the credential
store's propagation, the operation context in the shell's s3.bucket.delete and
the kafka gateway's broker and filer discovery, and context.Background() where
there is none - the shell commands, the admin dashboard wrapper, and the
exclusive locker's initial lease. The locker's release keeps its own
uncancelled context so a slow unlock cannot turn into a ghost lock.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* wdclient: test that WithClient gives up with the caller's context
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* wdclient: cut the master retry backoff short when the caller gives up
util.Retry sleeps unconditionally between attempts, so a transient error
arriving just before the caller's deadline still cost it a full backoff step.
Use the context-aware util.RetryWithBackoff, the same helper the volume lookup
in this file already uses.
Two call sites went with it: the shell's lock-holder lookup builds its three
second bound before WithClient so it also covers finding the leader, as its
comment already promised, and the filer's post-delete collection cleanup goes
back to an uncancelled context - the entry is already gone, so a caller that
hung up must not leave the collection behind.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* wdclient: test that a cancel during backoff ends the retry
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
HeadBucket treated any lookup error as ErrNoSuchBucket, so a transient
filer failure answered 404 instead of 500 and clients stopped retrying.
Split the two cases the way the bucket policy handlers already do.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* volume: stop the .vif guard depending on which entry the scan handed over
A volume has both an .idx and a .vif, and loadExistingVolume skipped a .vif
next to an .ecx as EC shard metadata. That was only ever correct because
os.ReadDir sorted .idx ahead of .vif: an interrupted encode, where the .idx is
still there, has to reach validateEcVolume to be reclaimed. Ask for the .idx
instead of trusting the order.
Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy
* volume: walk volume directories in batches instead of listing them whole
os.ReadDir builds, and sorts, a slice of every entry before the caller sees
the first one. A disk holding millions of volumes has a .dat, .idx and .vif
per volume, so each startup scan costs hundreds of MB of peak heap that the
runtime is slow to hand back -- and there are several of them before the
first volume loads.
Walk in batches instead, and keep only the entries each scan acts on:
loadAllEcShards now sorts and stats the shard and index files alone rather
than every file on the disk.
Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy
* volume: skip the sibling-.dat scan when no EC volume is loaded
pruneIncompleteEcWithSiblingDat only ever prunes EC volumes that are loaded,
but it first walks every disk and keys a map by every .dat on the server. On
a store with no EC volumes at all that is millions of map entries built to
answer no question.
Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy
* volume: stop keeping a departure message for every volume
The report state held a VolumeShortInformationMessage per volume copy so a
departure could be named, but almost no volume ever departs. Hold a handle to
the identity instead -- volumes share very few distinct ones -- and build the
message on the way out.
Measured over a populated report state: 195 -> 83 bytes per volume.
Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy
* rust volume: stop keeping a whole volume message per volume held
The send loop kept a VolumeInformationMessage for every volume just to notice
mounts and unmounts, and rebuilt the map from scratch on every beat. Keep the
identity a delta names, which is what the Go report state keeps for the same
reason.
Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy
* rust volume: keep only the EC files the shard scan acts on
load_all_ec_shards named every file on the disk twice -- once in the dedup set
and once in the sorted vector -- before deciding it only wanted .ec?? and .ecx.
Filter while reading instead. Mirrors the same change in loadAllEcShards.
Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy
* volume: share the strings every .vif repeats
A tiered volume's .vif names its replication and its backend, and every decode
allocates a fresh copy, so a server holding millions of them holds millions of
copies of the same handful of names. Route them through the interning table
the volume info decode already uses. The remote key names one volume and is
left alone.
Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy
The router matches bucket subresource routes in registration order while
the IAM action resolver matches its own list in a different order, so a
request carrying two operation subresources is authorized as one
operation and served as another. `PUT /bucket?policy&tagging` resolves to
s3:PutBucketTagging and runs PutBucketPolicy, letting an identity
delegated bucket tagging install an arbitrary bucket policy. The same
mismatch reaches PutBucketCors, PutBucketLifecycle, PutBucketVersioning,
PutObjectLockConfiguration, PutBucketRequestPayment and the policy and
cors deletes.
Reject the ambiguity where the other pre-routing checks live, so neither
list has to stay in step with the other. Keys that modify an operation
rather than select one -- versionId, partNumber, prefix -- still combine
freely.
* shell: remove the directories emptied by volume.fsck's filer entry purge
volume.fsck -findMissingChunksInFiler -reallyDeleteFilerEntries deleted the
orphan entries but left their parent directories behind, so a namespace
accumulated empty directories that had to be cleaned up by hand.
Remember the parent of every purged entry and, once the purge is done, walk
up from each one deleting the directories that are now empty. The delete is
non-recursive, so the filer itself rejects a directory that still has
children; a bucket and a directory that is an S3 object of its own are left
alone.
Claude-Session: https://claude.ai/code/session_01BncsNo2RVANCDtdbw96Kfc
* shell: keep a directory volume.fsck saw change under it
The empty-directory sweep read the entry to spot an S3 directory key object
and then deleted unconditionally, so a directory promoted to an object in
between was removed anyway.
Delete with the mtime the lookup returned, leaving the filer to skip a
directory that has changed since.
Claude-Session: https://claude.ai/code/session_01BncsNo2RVANCDtdbw96Kfc
* shell: leave a directory volume.fsck just saw written for the next run
The mtime the delete is conditioned on has second resolution, so a write
landing in the same second as the one already on the directory is
indistinguishable from it and the directory would still be deleted.
Skip a directory modified within the last few seconds. A write after the
lookup then always carries a later second than the one the delete carries,
and the sweep picks the directory up on the next run.
Claude-Session: https://claude.ai/code/session_01BncsNo2RVANCDtdbw96Kfc
* shell: skip a directory volume.fsck cannot condition a delete on
A zero mtime disables the delete's condition at the filer, so a directory
whose entry carries none was removed unconditionally and a concurrent
promotion to an S3 object went with it.
Leave such a directory alone.
Claude-Session: https://claude.ai/code/session_01BncsNo2RVANCDtdbw96Kfc
* shell: hold volume.fsck's quiet period to the cutoff second itself
Mtime keeps whole seconds, so a directory whose mtime lands on the cutoff
second was written up to a second after it. Skip that directory too, so the
quiet period fails closed.
Claude-Session: https://claude.ai/code/session_01BncsNo2RVANCDtdbw96Kfc
* s3: enforce object lock when deleting a directory marker
The key "dir/" is deleted the unversioned way, ahead of the branches
that enforce Object Lock, so a principal with plain delete permission
could remove a key the gateway was reporting as COMPLIANCE-retained --
retention set through PutObjectRetention is stored on the directory
entry and served back by GetObjectRetention, only the delete ignored it.
The same path also takes any key ending in "/" regardless of size, while
a PUT only makes a marker of one up to 1KiB. A larger one is a genuine
versioned object, and deleting it here dropped its whole history after
the versioned delete of the same key had been refused.
Enforce in the marker delete itself, so the single, versioned and
multi-object delete paths are all covered.
* s3: apply object lock headers on a directory marker PUT
The trailing-slash branch runs before the versioning and Object Lock
handling, so it accepted x-amz-object-lock-* headers and stored none of
them: a bucket owner could believe a key was retained while nothing
recorded it, and an invalid mode or a past retention date that a regular
key rejects came back 200 here.
Validate the headers the way the regular path does, store what they ask
for beside the owner the same callback already sets, and refuse to
replace a key that is already retained.
* s3: check every version a marker delete would remove
The marker delete clears any history under the key in one recursive
removal, while the lock check ahead of it resolves the latest version
only. A version retained under an unretained one was taken with the
rest, so enforce against each version the removal covers.
* test: pin the marker lock refusals to AccessDenied
A bare require.Error passes on any failure, including one that has
nothing to do with the lock. Assert the code, the key the batch delete
reports, and that the marker survives each refusal.
* s3: check the history entries a version list leaves out
The version list skips an entry without a version id, while the removal
takes it with the rest, so an entry an older build left unnamed escaped
the check. Walk the history directly instead, and refuse when an unnamed
entry is still under a retention or a legal hold of its own.
* s3: let a governance bypass reach an unnamed history entry
The unnamed branch refused every active retention, so a caller allowed
to bypass governance could not clear one, which the named path lets
through. Refuse a legal hold and compliance mode as before, and take the
bypass into account for governance.
* s3: keep the object lock decision in one place
The unnamed history entry had to repeat the retention and legal hold
rules inline because the enforcement helper only takes a key to look up.
Split the part that judges an entry out of it and call that from both.
* s3: guard a marker PUT on the entry it replaces
The overwrite check resolved the key's latest version, but mkdir builds
a fresh entry for the marker itself, dropping the lock metadata the old
one carried. Once the key had a history, an unlocked version answered
for a retained marker and a plain PUT replaced it. Judge the entry the
write is about to replace instead; a versioned write of the same key
still adds a version, which is its own to allow.
* s3: guard a marker delete on the entry it removes
The check ran against the key rather than the entry, so once the key had
a history it answered with a version and the retention recorded on the
marker itself went unseen. Judge the entry that is about to be removed,
the same way the PUT side now does; the versions under it are still
covered by the walk that follows.
* s3: take the object write lock for a marker PUT
The overwrite check read the entry that the mkdir after it replaces, so
two marker PUTs could both pass while one was still unlocked. The marker
delete already runs under this lock; hold it across the check and the
mkdir so the entry cannot change in between, and so the two paths are
serialized against each other.
DeleteTable authorizes the named table, then recursively purges the data
path derived from its stored MetadataLocation. That location is supplied
by the caller at create/register time and never bound to the table, so a
tenant allowed to drop one table could point it at a table in a sibling
namespace and have the delete destroy that table's catalog entry and
data files.
A legitimately decoupled location -- a rename source, or a leftover the
name was reused over -- has had its catalog attributes stripped, so a
surviving metadata marker identifies a path that belongs to another
entry. Refuse those, alongside the existing ancestor refusal.
* volume: build the guarded remote storage client through a shared helper
Fold the endpoint validation, credential check and rebinding-safe dialer
that FetchAndWriteNeedle applies before dialing a caller-supplied remote
storage endpoint into a single BuildGuardedRemoteStorageClient helper, so
other callers that dial the same endpoints can reuse it. No behavior
change on this path.
Claude-Session: https://claude.ai/code/session_01AiH1FU3rmshSbFFTbJpaZN
* filer: build the remote-mount stream client through the guarded helper
streamFromRemote serves a cold remote-only entry straight from its mounted
origin. Build its client through BuildGuardedRemoteStorageClient so the
same endpoint checks the volume server applies cover this read path too.
Claude-Session: https://claude.ai/code/session_01AiH1FU3rmshSbFFTbJpaZN
* s3: build the remote-mount stream client through the guarded helper
openRemoteStream serves a remote-mounted object straight from its origin
when the local read cannot. Build its client through the same guarded
helper so the endpoint checks apply here as well.
Claude-Session: https://claude.ai/code/session_01AiH1FU3rmshSbFFTbJpaZN
* s3tables: share one CreateTable authorization gate
CreateTable and RegisterTable each carried their own copy of the name
validation, policy load and permission check. Fold them into
authorizeCreateTable, and expose it on the Manager for callers that write
into a table bucket before the table itself is registered.
Claude-Session: https://claude.ai/code/session_01QiJkka1T2NAWDWq4JQ8Vuy
* iceberg: authorize a table create before it writes
Stage-create returns before the S3Tables registration that authorizes a
create, and the plain create writes its metadata file before reaching it,
so a caller who may not create the table could still leave a staged
template, a marker and a v1.metadata.json in the target bucket - and get
vended credentials for a location of their choosing. Run the CreateTable
gate as soon as the table is known to be absent.
Claude-Session: https://claude.ai/code/session_01QiJkka1T2NAWDWq4JQ8Vuy
* iceberg: authorize a create-on-commit the same way
A commit against a table that does not exist creates it, writing the
metadata file first and only then reaching the registration that checks
the caller may create it. Denied callers saw a 500 for what is a 403.
Claude-Session: https://claude.ai/code/session_01QiJkka1T2NAWDWq4JQ8Vuy
* iceberg: pin that identity actions reach the create gate
The manager request is built from the caller's own context, so an identity
whose actions carry the permission still passes. Worth a test: a fresh
context here would silently deny every such caller.
Claude-Session: https://claude.ai/code/session_01QiJkka1T2NAWDWq4JQ8Vuy
* mount: serve metadata ops from the open handle of an unlinked file
ftruncate on a descriptor whose file was unlinked failed with ENOENT:
maybeReadEntry resolved the inode to a path first, and unlink had already
dropped it. GetAttr worked around that with its own handle fallback;
SetAttr and the xattr handlers had none.
Look the handle up first and let it answer whether or not a name still
points at the inode. GetAttr keeps reporting nlink 0 there, now off the
empty path.
Claude-Session: https://claude.ai/code/session_01U1R8BM4bVT46KwPDEj2Ega
* mount: read an open handle's attributes under the handle lock too
GetAttr held only the LockedEntry lock, which covers the async uploader's
chunk appends but not Write or the metadata flush: those rewrite size,
times and the whole chunk slice under the handle lock, so FileSize could
walk a slice mid-reassignment. The branch this replaced took both locks;
take both here, outer handle lock first, as Read and Lseek do.
Claude-Session: https://claude.ai/code/session_01U1R8BM4bVT46KwPDEj2Ega
* mount: report nlink 0 from SetAttr for an unlinked open file
The kernel caches the attributes a SETATTR reply carries, so an ftruncate
on an unlinked file left fstat reporting nlink 1 until the cache expired,
even though GetAttr had it right. Both replies go through the same rule.
Claude-Session: https://claude.ai/code/session_01U1R8BM4bVT46KwPDEj2Ega
* webdav: name the entry, not its path, in a listing
DAV:displayname carried the full path of every entry. A client that
takes displayname for the child's name - Windows Explorer does - then
looks for /dir/name under /dir and finds nothing, so a folder shows up
empty while the root, where the two spellings differ only by a leading
slash, still lists.
Readdir now builds its entries with toFileInfo like stat does, so a
listing and a lookup describe a child the same way, and the wrapper that
was trimming the sub-folder back off a name goes away with it.
Claude-Session: https://claude.ai/code/session_01XCeuCWpF9xo9CfyHvCQE9c
* webdav: derive an ETag when nothing hashed the entry
Uploads through this gateway carry no content MD5, so filer.ETag comes
back empty and every file in a PROPFIND answered with an empty
DAV:getetag, which is not a valid entity-tag. Report it as unimplemented
instead, the way the sub-folder wrapper already did, and webdav falls
back to modification time and size. The wrapper's copy went with it - it
swallowed the stat error a caller was meant to see.
Claude-Session: https://claude.ai/code/session_01XCeuCWpF9xo9CfyHvCQE9c
* s3: keep a missing object a 404 under If-Match and If-Unmodified-Since
GET and HEAD resolved the target before evaluating the conditional headers, and
a missing target failed If-Match and If-Unmodified-Since outright, so absence
surfaced as 412 PreconditionFailed. AWS reports the missing object instead:
404 for HeadObject, NoSuchKey for GetObject, and 412 only when a live object
fails the condition. Clients cannot tell absence from a stale precondition
without an extra racy HEAD, so OpenDAL disabled its four conditional
stat/read capabilities against SeaweedFS.
A precondition now only fails against an object that exists; a missing one --
including a latest version that is a delete marker -- returns NoSuchKey.
Claude-Session: https://claude.ai/code/session_01X4kEbuwxd9DFsTnSXjfjgv
* s3: evaluate a conditional read against the version the request names
GET and HEAD resolved the latest version before evaluating the conditional
headers, so a request carrying versionId had its If-Match compared against a
different version than the one it was asking for: a live version whose ETag the
client held failed once a newer version -- or a delete marker -- became the
latest. resolveObjectEntry now resolves the named version on a versioned bucket,
the way DELETE already does.
A named version that resolves to nothing is left to the handler, which alone
knows whether the bucket is versioned and so whether it owes NoSuchVersion.
Claude-Session: https://claude.ai/code/session_01X4kEbuwxd9DFsTnSXjfjgv
* volume: resolve the disk IO slow-latency threshold per disk
volume.toml keys [volume.disk.io.slow.latency] by disk type, but the
threshold was chosen once per server by switching on the raw -disk flag.
-disk is comma-separated, one entry per -dir, so a multi-disk server
matched no case and silently took the hdd threshold.
Carry the table on DiskIOProbeConfig and resolve it in CheckDiskSpace
from the location's own DiskType. A type with no entry keeps falling
back to the hdd threshold.
* volume: run the disk IO probe on multi-directory volume servers
The probe was disabled whenever more than one -dir was configured,
because a single server-wide slow-latency threshold could not describe
disks of different types. The threshold is per disk now, and the rest of
the probe already is: diskRegistry is keyed by directory, each
DiskLocation runs its own CheckDiskSpace, and Store consults
isDiskUnavailable per location.
* volume: reject duplicate -dir entries
Nothing deduplicated -dir, so the same directory listed twice produced two
DiskLocations that each loaded every volume in it, appending to the same .dat
under two independent locks. Compare directory identity with os.SameFile
rather than the path, so a symlink or bind mount aliasing an earlier entry is
rejected as well.
* volume: cover the per-disk slow-latency handoff
SlowLatencyFor has a test, but nothing asserted that CheckDiskSpace feeds it
the location's own disk type. Probe through a seam so the resolved threshold
is observable, and check hdd, ssd, nvme, the empty type, and an unlisted tag.
* mount: let a rename remove its source at the source's own version
A rename stamps the source and takes the name away at the same log position,
so the removal reaches the meta cache carrying exactly the version the source
already records. The version gate read that as a write already reflected and
dropped it, while the destination half of the same event still applied -- the
source stayed cached beside the destination, and readdir and stat went on
serving a name the filer no longer had:
gate dropped removal of /winfsp-test-TestRenameOverExisting/src
eventTs=1787761708173717200 record=1787761708173717200
floor=1787761708173717200 tombstone=false
A removal asks a different question from a write. An entry still present at
exactly that version has the write reflected but not its removal, so only a
strictly newer record fences one out; a tombstone is the removal already
reflected and goes on fencing as before.
* mount: sweep a section's vanished name recorded at the snapshot
The refresh deletes the names its listing did not return, but asked the gate
whether a write at the snapshot was reflected. A name recorded at exactly that
version has the write reflected and not its removal, so it survived the sweep
and stayed cached until some later event happened to touch it.
Same reading as the rename source a commit earlier: the call site removes, so
it asks about a removal.
* s3: keep verifying the request host when externalUrl is set
externalUrl was the only host candidate once set, so a client that dialed
the gateway directly instead of through the proxy always got
SignatureDoesNotMatch. Make it lead the candidate walk instead: every
candidate still needs a valid signature, and the request-derived hosts are
already trusted when the flag is unset, so a mixed proxy plus in-cluster
topology can now advertise a public endpoint and verify both planes.
* s3: cover virtual-hosted addressing behind externalUrl
The old pin also rejected an external client that signed
bucket.api.example.com, since only the bare externalUrl host was ever
tried. The candidate walk covers it; pin the case down.
* operation: give an encrypted chunk the plaintext ETag
With -encryptVolumeData the volume server stores ciphertext, so it cannot
echo a Content-MD5 back and the chunk lands with an empty ETag. Every ETag
derived from those chunks then comes out empty for a single chunk, or
d41d8cd98f00b204e9800998ecf8427e-N for several.
The caller already hashes the plaintext to send as Content-MD5, so keep that
digest as the chunk ETag instead of dropping it, and compute it for a
WantMd5 caller under cipher too.
* s3: re-encrypt a part copy from a volume-encrypted source
UploadPartCopy raw-copies source chunks when neither side uses SSE, which
also caught -encryptVolumeData sources. Those chunks are ciphertext a
whole-chunk cipher key decrypts, so copying a byte range out of one and
keeping the key leaves a destination that fails authentication on GET, and
the copied chunks carry no ETag for the part result to report.
Route them through the re-encrypting path already used for SSE: it reads the
source as plaintext, hashes the part, and writes the destination under the
gateway's own encryption.
* s3: fetch only the range a part copy asked for
The re-encrypting UploadPartCopy path opened the source at offset 0 and threw
the prefix away, so assembling an object part by part read the source once per
part. Now that volume-encrypted sources take this path too, that is the common
case rather than an SSE corner.
The chunk stream already seeks, so hand it the range.
* s3: reject an unsatisfiable copy-source-range
A part copy has no way to report a short part, so a range reaching past the
source cannot be clamped the way a GET clamps one. The fast path silently
produced a part shorter than asked for, or an empty one; the re-encrypting
path pads with zeros, so a 2 MiB source copied as bytes=1048576-9999999 came
back as 1 MiB of data followed by 7.5 MiB of nothing.
Answer InvalidRange instead, which is what s3-tests'
test_multipart_copy_invalid_range expects.
The test truncates a good .sdx and asserts the result still looks fresher
than its .idx, on the reasoning that truncation bumps the mtime. That holds
only at the filesystem's timestamp granularity: where both writes land in the
same tick the precondition fails and the run reports a failure that says
nothing about the code under test — as it did on CI. Backdate the .idx the
way the sibling stale_sdx_is_regenerated already does.
* mount: stop a replaced rename destination from flushing over the rename
Rename replaces whatever the destination held, which deletes that entry, but
only the source handle was told. A handle still open on the replaced entry
went on flushing its metadata under that name, and on Windows -- where the
close carrying the flush runs after the application's CloseHandle has already
returned -- the flush landed after the rename and put the destination's old
content back:
dir Rename old_entry:{name:"src"} new_entry:{name:"dst" ... inode:...3416}
doFlush /dst fh 1521468582993181449
/dst saveToStorage 1,6872462993 [0,3)
flushMetadataToFiler /dst inode 11939747521756968515
InsertEntry /dst
The next read of the destination returned the content the rename was supposed
to replace. Unlink already handles this with markHandleDeleted, which raises
the flag under the handle's flush lock so a flush already writing finishes
first and any later one sees it; a rename that replaces an entry deletes it
just the same, so it now does likewise.
Verified on the Windows runner: TestRenameOverExisting 300/300, where the same
loop reproduced the corruption twice without this.
* test/winfsp: say which layer kept a renamed-away name
The failure only reported the stat. Which layer answered narrows the search a
lot: a listing reads no per-path cache, the mount's own forgets within a
second, and a name that survives both is still in the meta cache.
* mount: keep the destination barrier honest when the rename does not happen
Two gaps in the barrier the previous commit put in front of a replaced rename
destination:
The flag was raised before the filer rename, which can still fail. The
destination then stays exactly where it was, with its handle marked deleted
and its dirty metadata silently dropped from then on, so a rename that
returned an error has to put the flag back.
The handle was only found through the path mapping, which Forget drops while
the handle is still open. The source side already falls back to the inode the
entry carries; the destination now does the same, off the entry the sticky-bit
check had already loaded.
* mount: let only the caller that raised a delete mark lift it
Restoring the destination handle after a failed rename cleared isDeleted
outright, so an unlink that marked the same handle in between lost its mark and
a later flush could write the unlinked entry back.
Every raise of the flag already happens under the handle's flush lock, so
counting them there is enough to tell one caller's mark from another's: the
rename lifts only the mark it made itself.
* mount: drain the destination flush before marking it deleted
A flush already queued for the destination belongs to the entry as it stands.
Marking first meant the drain waited on a flush that then skipped its metadata
as deleted and released its handle, so a rename that failed afterwards had
nothing left to restore and the queued update was gone, its chunks orphaned.
Draining first lets that flush finish as itself, before the rename has taken
anything away.
* filer: cover the storage rule TTL on the object transaction write path
An object written through ObjectTransaction used to land with ttlSec 0
even under an fs.configure TTL rule, while the same object written
through CreateEntry got the rule's TTL. Guard the shared stamping so the
two paths cannot drift apart again.
* filer: apply the path's storage rule to an appended entry
AppendToEntry resolved the storage option from the path - so its chunks
land on a TTL volume under an fs.configure TTL rule - but never stamped
the rule's TTL on the entry it creates, leaving an entry that outlives
its data. Route it through applyStorageDefaultsToEntry, which now feeds
the entry's own TTL into the option so the placement an existing entry's
appended chunks get is unchanged.
* filer: apply the path's storage rule to a completed TUS upload
The PATCH path resolves the storage option from the target, so a TUS
upload into an fs.configure TTL prefix writes its chunks to a TTL volume,
but completion built the final entry with ttlSec 0 - the entry outlived
the data it pointed at. Stamp it through applyStorageDefaultsToEntry,
which also subsumes the hand-rolled read-only check and supplies the
rule's name-length limit.
* filer: apply the destination's storage option TTL to a copied entry
The copy handler re-uploads the source's chunks under the destination's
storage option, so a copy into an fs.configure TTL prefix already lands
its data on a TTL volume. The entry, though, carried the source's ttlSec
- 0 for a source outside the prefix, or the source's own TTL where the
two rules differ - so it never expired with the data it pointed at. Take
the TTL from the same option the chunks were placed with, after the
data-only copy has restored the destination's metadata.
* master: cap the reported capacity at what the disks hold
Statistics reported max volume count times the volume size limit, which is
how many volumes the cluster is allowed to place, not how much space it has.
A cluster given far more slots than its disks can fill reported a capacity it
could never reach -- 65536 slots at 30GB read as 1.9PB on a 460GB disk -- and
the number never moved, since writing data changes neither the slot count nor
the size limit.
The volume servers already report each filesystem's total and free bytes in
their heartbeats, so bound the answer by what they say is left.
* mount: keep the last known sizes when filer statistics fails
A failed Statistics call returned before df's answer was filled in, so a
mount whose filer or master was briefly unreachable reported an empty
filesystem rather than the sizes it already had.
* master: drop the disk ceiling when a volume server does not report
A cluster part way through an upgrade has volume servers that predate the disk
bytes in the heartbeat. Summing only the ones that answered left the quiet
server's free space out of the total, and the server holding the room is
exactly the one that could make the cluster read as full.
Answer with the disks only when every one of them reported.
* rust volume: fail closed on sorted-index failures and reconcile tier-up
Follow-ups to the .sdx sorted needle map (#10951):
- get() folded open/read failures into None, so an EIO, a torn .sdx, or a
failed pooled reopen answered reads with NotFound and let do_delete_request
acknowledge the delete as Ok(0) without writing a tombstone. It now returns
io::Result and every caller propagates; redb's get() had the same shape and
is fixed with it. is_file_unchanged cannot propagate, so it reports unknown
and logs rather than treating an unreadable index as proof of a change.
- A delete whose .idx append landed but whose .sdx mark failed left the map
still resolving the old live entry, so deleted content stayed readable until
a reload. The map now records the tombstone before touching .sdx and only
clears it once the mark lands; lookups consult that first and report the
needle deleted, which is what the next reload concludes anyway.
- Mode reconciliation ran one way. Entering remote mode made use_sorted_index()
true, which returned early, so a volume tiered while the server runs kept its
in-memory map and pinned .idx descriptor until restart — the RAM and fd win
never applied. It now reconciles in both directions.
- Tier-down dropped the remote reference before the fallible refresh, so a
failure left volume_info local, the remote backend attached, the .vif still
remote, and a retry reporting "already on local disk". The transition is
snapshotted and rolled back.
- The read-only fallback set no_write_or_delete but left no_write_can_delete,
so metrics and mode checks called the volume delete-capable while every
delete was refused.
* rust volume: count a sorted-map delete against the durable .idx append
The deletion counters sat after the in-place .sdx mark, so a mark that
failed left them at their pre-delete values while the tombstone was already
durable in .idx — and with retries now idempotent, nothing applied them
later either. Heartbeats, status responses, and the garbage calculation
would report the volume as free of that garbage until a reload.
Move them to the append that makes the delete durable, which is also what a
reload of .idx would count. Covered by a test that injects a mark failure
through a cfg(test) seam: no portable filesystem trick reproduces it, since
a read-only .sdx fails the borrow long before the mark.
* rust volume: hide a pending tombstone from the sorted-map scans too
The overlay that keeps a needle deleted after a failed .sdx mark was only
consulted by get(). visit_live_entries still read the stale valid record
straight off .sdx, so ascending_visit, iter_entries and save_to_idx all
reported the needle live — and compaction takes iter_entries for the
complete live set, so it would copy the deleted content forward and
save_to_idx would write it back into the rebuilt .idx as live.
Snapshot the overlay once per scan and skip its keys, which is the same
conclusion the next reload reaches from the .idx tombstone.
* rust volume: quarantine a durable write whose index lookup fails
The prior-mapping lookup that decides whether to index a fresh append runs
after the record is already down and flushed, so a failing lookup leaves
exactly the state a failing put leaves: a durable .dat record nothing
indexes. The put path marks the volume read only for it; this one returned
the error and kept taking writes, and the next append would bury the
orphan mid-file where the .dat tail check on reload cannot see it.
Give it the same treatment.
* mount: name the disk after the mounted path
Finder and Explorer labelled every mount with the filer address, so two
mounts from one filer were indistinguishable. Use the mounted path's last
segment, the way df already shows it, and keep the filer address only for
a whole-tree mount.
* mount: let a given mount option override the default
The options from -o were placed before the ones this mount derives, so
a volname or iosize given on the command line lost to the derived value.
Append them last, matching the Windows adapter.
* mount: document what labels the disk
* worker: move the collection filter parser into weed/util/wildcard
The parser sits beside the volume-list filtering it was written for, in
weed/plugin/worker, which imports weed/shell — so the shell commands that
parse the same filter three other ways can never call it. Move it down to
weed/util/wildcard, next to the comma-separated wildcard helper it already
replaced, leaving the behavior unchanged.
* shell: parse every collection filter the same way
The shell parsed a collection filter three ways: compileCollectionPattern
compiled one regex for ec.encode, ec.decode, volume.balance and the tier
commands; volume.list and volume.deleteEmpty matched a single wildcard; and
volume.tier.move, volume.fix.replication and volume.configure.replication
called filepath.Match on their own. None of them took a list, so
"ec.encode -collection=a,b" selected nothing, the same way the admin UI did.
They all go through the shared matcher now: a comma-separated list of names,
"*" and "?" wildcards, "_default" for the collection with no name, and regex
entries. The one thing that stays per-command is what an empty value means -
every collection for -collectionPattern, the unnamed collection for the ec
and tier -collection flag - so compileCollectionPattern keeps that mapping.
The matchers are compiled once per command instead of once per volume, and a
regex entry now has to match the whole name unless it anchors itself, so
-collection=bucket no longer picks up mybucket2.
* shell: keep dots in collection names, and commas inside a regex
A dot no longer marks an entry as a regex, so a collection named "my.bucket"
matches itself and not "my-bucket" - the difference decides which volumes
volume.deleteEmpty and volume.tier.move touch. A dot still counts when it is
quantified, so "bucket.*" stays a prefix regex.
The comma split also leaves alone the commas inside a character class or a
repetition count, so "bucket[0-9]{1,3}" stays one entry instead of becoming
two broken fragments.
* shell: let a regex entry match its own spelling
A collection named after regex syntax, say "logs(2024)", was unreachable:
the entry compiled to a pattern that matches "logs2024" instead. Match the
entry verbatim as well, so naming a collection always selects it, whatever
characters it holds.
* shell: reject a collection filter that names no collection
A value of "," parsed to no entries and then matched every collection, so a
typo widened ec.encode or volume.deleteEmpty to the whole cluster. Only a
genuinely empty filter means "all collections"; anything else has to name one.
* shell: keep commas inside a regex group out of the entry split
The split already left alone the commas inside a character class or a
repetition count, but not the ones inside a group, so "bucket(foo,bar)"
was cut into two fragments that no longer compile.
* shell: cover escaping a collection name that is not a regex
A name like "logs(2024" does not parse as a regex on its own; escaping it,
"logs\(2024", reaches it. Pin that so the escape hatch does not regress.
* shell: split entries only on commas inside a closed regex construct
An unmatched "{" or "[" made the splitter swallow every comma after it, so
"foo{bar,videos" became one entry that matches neither collection - the
silent no-op this filter work exists to remove. A construct now has to close
before its commas stop separating entries.
* shell: skip character classes while scanning a regex group
A ")" inside a class is a literal, so "(a[)],b)" ended its group early and
split into two fragments that no longer compile.
* shell: cover escaping a comma inside a collection name
A comma separates entries, so a name holding one is reached by escaping it.
* shell: follow the regexp parser when scanning a character class
A "]" leading a class is a member of it, and a POSIX class such as
"[:alpha:]" carries its own "]", so stopping at the first one cut a valid
filter like "(a[]),],b)" into fragments and rejected it.
* s3: deny anonymous requests when the identity config loads no identities
Naming a config file is the operator asking for authentication. A file that
yields no identity - an unpopulated secret mount, or a mistyped top-level key
the proto parser silently drops - left the gateway open to every anonymous
caller: ListBuckets returned 200, and anonymous PUT could create buckets and
write objects.
* s3: name the unknown top-level keys in an identity config
The proto parser discards what it does not recognise, so a mistyped
"identites" loads as an empty config. Naming the dropped keys at startup turns
the resulting lockout into a one-line diagnosis.
* s3: isolate the auth-enforcement tests from AWS environment credentials
* s3: use a singular "identity" as the unrecognised-key example
Codespell rejects the misspelling the example used.
* s3: cover the empty identity config alongside the unrecognised key
* s3: cover a config file whose body is an empty object
* rust volume: search .sdx for read-only volumes instead of holding the index
The Go volume server loads every read-only volume through SortedFileNeedleMap:
the index lives on disk as a sorted .sdx, a lookup is a binary search, and
since #10950 no descriptor is held between lookups. The Rust server had no
counterpart. Read-only volumes built a full in-memory CompactNeedleMap, and
cloud-tiered ones — noWriteCanDelete, so not the read-only branch — went
through the writable path and pinned an .idx append handle on top of it. At the
hundreds of thousands of tiered volumes a real server carries, that is an index
in RAM and a descriptor each, for volumes nobody reads.
Port the sorted map and the bounded handle pool. A tiered volume now costs zero
descriptors and zero index bytes when idle; the pool keeps the hot handles open
so a busy volume does not pay an open() per needle. Handles are Arc<File>, so
an eviction cannot close one a reader still holds.
The generated .sdx is byte-identical to Go's — same sort, same last-write-wins,
same dropped tombstones — so a volume moved between a Go and a Rust server reads
whichever copy is already on disk. A test pins the bytes against a Go-generated
fixture.
* rust volume: fail compaction on an unreadable .sdx, and rebuild the map on tier-down
Two ways the sorted map could lose data.
iter_entries swallowed read errors and returned however many entries it managed
to collect. Compaction takes that vector for the complete live set, so a
truncated .sdx or a mid-scan I/O fault would commit a volume missing every
needle past the failure. Return a Result instead and abort. redb's
collect_entries dropped errors the same way on the same path, so it goes with
it.
Tier-down clears the remote mode and publishes the volume as writable, but the
map it booted with is the read-only sorted one. Its put always fails, so the
first write would append to the local .dat and then fail to index it, leaving
bytes nothing references — and a non-fsync write repeats it. Fold the
reopen_idx_for_write swap into refresh_remote_write_mode so the map always
matches the mode it just published; a rebuild that fails pins the volume
read-only rather than letting it take writes it cannot record.
Go reaches neither: its tier-down leaves noWriteCanDelete set, so the volume
stays read-only until a reload or an explicit mark-writable, which already goes
through reopenIdxForWrite.
* rust volume: keep read-only volumes mountable on a read-only index dir, and batch the .sdx scan
Building .sdx writes to the index directory, and load_index_sorted_file also
created a missing .idx there. A volume whose index sits on a read-only mount
took both paths and failed to load, where before it mounted read-only off an
in-memory index and served reads. Create the .idx only where deletes are
allowed, and fall back to the in-memory map when the sorted one cannot be
built, so a directory nobody can write costs memory rather than availability.
The end-to-end scan behind iter_entries, ascending_visit and save_to_idx read
one entry per syscall. Read 1024 at a time instead, the batch size
idx::walk_index_file uses. Positional reads, not a cursor: the handle is shared
with any other borrower.
Also gate the Go byte-parity fixture on the 5bytes feature it describes, which
is otherwise dead code in a 4-byte-offset build.
* rust volume: roll back a failed writable mark, and rebuild a torn .sdx
set_writable clears the read-only flags before it can know the rest will
succeed, but only the map rebuild rolled them back. An .idx writer that fails to
attach left the volume advertising writable over a needle map with no writer, so
puts landed in memory and were gone after a restart — the exact failure the
function exists to prevent. The read-only-mount fallback made it reachable: that
path loads an in-memory map with no writer attached. All three steps now run
behind one rollback point.
A .sdx whose length is not a whole number of entries was accepted as long as it
looked fresh, and truncation is what makes it look fresh. The entry count then
floored, hiding the last needle from lookups and from compaction, which would
commit the shorter set. Treat a torn file like a stale one and rebuild it from
.idx. Go writes .sdx in place rather than through a temporary, so a crash
mid-generation is a real way to produce one.
Appends now start at the last whole .idx entry too, so a torn tail there is
overwritten by the next tombstone instead of misaligning every row after it.
* rust volume: trim a torn .idx before writing to it, keep delete-only volumes online, count sorted-map deletes
Three from review.
Flooring the sorted map's append offset only protected its own positional
writes. Every writable path appends at EOF instead, so a partial row left by a
short write pushed the next row off alignment and the following load parsed the
rest of the file as garbage. Drop the partial row before attaching any writable
index writer — it is unrecoverable anyway, and every loader already skips it.
Go refuses to load such a volume at all; trimming keeps it mountable with the
rows before the tear intact.
The unwritable-index-dir fallback stopped one step short for volumes that allow
deletes, which is every tiered one: the in-memory loader opens .idx read-write
there and fails on the same directory that just refused the .sdx, so the volume
stayed offline. Give up the deletes instead — without a writer no tombstone
could be recorded anyway — and a remount on a writable directory restores them.
Sorted-map deletes left the counters untouched, so a tiered volume reported
itself garbage-free until it restarted. They now land where a reload would put
them: the tombstone is another .idx row, and both it and the row it supersedes
count as deletions under the rule the load-time metric applies. Go skips this
too, and should not.
The collection filter was parsed twice with two syntaxes: the master-side
volume listing compiled the whole string as one regex, while EC encode and
EC balance detection split it on commas and matched each entry as a
wildcard. A volume had to pass both, so "collection-a,collection-b" matched
nothing (no collection is named that), and the ALL_COLLECTIONS sentinel,
which the master side skips, dropped every volume at the task side.
Parse it once, in one place: a comma-separated list where an entry is a
name with optional * and ? wildcards, or a regex when it carries regex
syntax. A regex entry now has to match the whole name unless it anchors
itself, so listing a collection no longer picks up its longer namesakes.
* filter volume by local or remote storage name
Signed-off-by: lou <alex1988@outlook.com>
* fix SelectsEverything
Signed-off-by: lou <alex1988@outlook.com>
* keep the proto sync out of this change
The branch copied weed/pb/*.proto over their seaweed-volume and Java
counterparts and regenerated every .pb.go with a different protoc and
protoc-gen-go-grpc. DiskStatus.error arriving that way broke the Rust
build, and the rest is toolchain churn in files this change has nothing
to say about.
---------
Signed-off-by: lou <alex1988@outlook.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
A read-only or cloud-tiered volume loads a SortedFileNeedleMap, which held
both its .idx and its .sdx open for the life of the process. On a server with
~600K tiered volumes that is 1.2M descriptors before a single read, enough to
exhaust the fd limit and take the listeners down. The .dat is not the problem:
a tiered volume serves it from the remote backend.
Neither index file is needed except while a lookup is in flight, so borrow them
from a bounded process-wide pool instead. An idle volume now holds zero
descriptors; a busy one keeps its handles hot rather than paying an open() per
needle. Reads borrow O_RDONLY, so a volume on a read-only mount answers lookups
that previously failed at load. Sync tracks whether a tombstone was appended,
which also drops the fsync-per-volume storm at shutdown.
* master: estimate a hintless assign's size from the volume's average file size
An assign that carries no dataSize hint charged a flat 1MB per file id
against the volume's effective size. A small-file workload overpays by
orders of magnitude: bulk-writing 4KB files marks volumes holding a few
hundred MB of real data as crowded and then full, so the master grows
unnecessary volumes and, once every volume is spuriously full, fails all
assigns. Estimate from the volume's own average file size instead, and
keep the 1MB fallback only for volumes with no history.
* master: decay pending assign sizes for volumes gone quiet
The decay that corrects pending assign estimates runs only when a
heartbeat reports the volume, and a heartbeat only reports a volume
whose content changed. A volume held out of the writable list takes no
writes, so once inflated estimates mark every volume full, nothing is
ever reported again, nothing decays, and the cluster refuses all writes
until a restart. Run the decay from the master's periodic loop for
volumes no heartbeat has reported within two pulses, feeding the last
reported size back through the same path an unchanged heartbeat would
take.
* master: trim the comments on the assign size estimate
* master: keep the periodic decay out of the replica-dedup window
UpdateVolumeSize ignores a report arriving within two seconds of the last
one, so replicas of the same volume do not each halve the pending
estimate. The periodic decay went through the same path and stamped that
window, so a real heartbeat landing right behind it was dropped along
with its reported size and compact revision. Only a volume whose content
changed is reported at all, so nothing would send that size again and
the master kept a stale one. Let the dedup window belong to volume
server reports alone.
* master: let the decay read the size record under the lock it mutates
The periodic decay picked its volumes under a read lock and replayed
them under a write one, carrying the size it had read across the gap. A
heartbeat landing in between was rolled back: the replay wrote the older
size and compact revision over the fresh ones, and a compaction report
lost that way is never resent, since only a volume whose content changed
is reported. The decay has no size of its own to contribute, so it now
reads the record under the same lock it mutates.
* master: let a heartbeat that beat the decay stand for the cycle
The decay chooses its volumes under a read lock and applies them under a
write one. A heartbeat landing in that gap already did the halving the
cycle owed, so applying the decay on top of it halved twice and forgot
pending bytes the volume has not written yet - the double-halving the
replica-dedup window exists to prevent. Both callers now give way to a
report already handled for this cycle; only a real report still advances
lastUpdateTime, so a quiet volume keeps decaying every pulse.
* master: keep genuinely full volumes out of the decay pass
A volume the disk really did fill keeps its fullSince set for good, so it
was selected every pulse for a decay that cannot help it: UpdateVolumeSize
refuses to recover a volume whose reported size is at the limit, and
replaying a size that cannot move leaves the record as it found it. Full
and quiet is the ordinary resting state of a cluster, so this was most of
the pass, taking the layout write lock away from the heartbeats to do
nothing. On a million tracked volumes with a hundredth of them phantom-full
it costs ten thousand write locks a pulse instead of a million.
* master: put the stale-replay test back on the path it guards
Giving the decay the dedup window left this test short-circuiting there,
so it no longer reached the locked read it was written for and passed
with that read removed. Age the record past the window, which is the only
case where reading it under the lock is what saves the report.
* grpc: a non-cancellable context is no evidence of a stale channel
shouldInvalidateConnection only invalidates on Canceled/DeadlineExceeded
while the context handed to WithGrpcClient is still live, so that an RPC
timing out on its own does not close the shared cached ClientConn and
cancel every other in-flight RPC on it. context.Background()/TODO never
expire, so Err() stays nil forever and that guard always answered
"invalidate" - and Background is what almost every caller passes, the S3
gateway included.
One S3 request whose RPC rode an abandoned HTTP request context therefore
closed the shared filer connection, and every multipart part in flight
died with "the client connection is closing", surfacing to the client as
400 InvalidRequest.
Only a cancellable context bounds an RPC attempt, so require one before
reading it. A genuinely stale channel (a peer restart behind a stable L4
endpoint) surfaces as Unavailable, which invalidates on its own branch.
* grpc: a bystander of a connection teardown is not a stale-channel witness
gRPC raises ErrClientConnClosing locally, before an RPC reaches the wire,
when this process has already closed the ClientConn. Every caller that
touches a channel during another goroutine's teardown gets it, so reading
it as a stale-channel signal lets one teardown re-arm itself across the
whole herd of callers it just cancelled.
The cached-connection version check keeps those callers from closing a
replacement channel, but the streaming path invalidates by address alone
and has no such guard.
* grpc: end a stream without dropping the peer connection under it
A streaming caller gets its own ClientConn, but on any error it also drops
the cached non-streaming ClientConn every request handler shares with that
peer, to recover a peer restart hidden behind a stable L4 endpoint. Any
error includes the ordinary ones: a metadata subscription that reached its
stop point, a follow callback that refused an event, a caller that gave up.
The S3 gateway follows filer metadata on such a stream and reconnects
forever, so each ordinary end of it cancelled every S3 request in flight
against the filer. Drop the shared channel only for errors that say the
peer went away, which is what invalidation is for.
* test: close the connections the cascade tests leave cached
Each test swaps in a fresh connection cache and restores the previous one,
dropping its own entries without closing them, so the ClientConn's
transport and reconnect goroutines outlive the fake filer they dialed.
* grpc: say why ErrClientConnClosing's deprecation notice does not apply
It points at codes.Canceled, which is the code this function exists to
disambiguate. Only the message distinguishes a teardown a caller merely
walked into, so the sentinel stays.
* filer: store TUS sub-chunks through the regular chunk writer
A TUS sub-chunk was written with one assigned file id, retried up to
three times against that same id, and abandoned on failure: an attempt
that had landed on some replicas left a needle no session record and no
entry ever references, unreclaimable by vacuum.
dataToChunkWithSSE, which the regular write path uses per chunk, assigns
a fresh file id per attempt and hands back the file ids of failed
attempts, which are now freed the way the regular write path frees them.
* filer: retry a chunk write on a fresh volume when the server 5xxs
The filer's chunk writer assigns a fresh file id per attempt but only
retried transient network errors, so a volume filling up and turning
read-only mid-write failed the whole request even though the very next
assignment would have landed elsewhere. Every other write client already
routes this through ShouldReassignUpload; the filer's own write path now
does the same, for regular uploads and TUS sub-chunks alike.
* filer: export the chunk deletion queue
The filer test harness in weed/server builds filer.Filer as a struct
literal, so any code path reaching DeleteChunks dereferenced a nil
queue. Exported like the neighboring DeletionRetryQueue so the harness
can arm it.
* filer: complete a TUS upload whose chunk records overlap
A PATCH retried while its predecessor was still storing a sub-chunk -
a proxy timeout with an immediate retry is enough - records the same
range twice. HEAD computes Upload-Offset as the covered watermark and
reported the upload fully received, but completion demanded exactly
adjacent records and failed every attempt: the client concluded success
from offset == length, no entry was created, and the session eventually
expired, turning the entire upload into deleted needles for the vacuum
to chew through.
Completion now validates gapless coverage with the same watermark HEAD
uses. A record extending coverage joins the entry - the read path
resolves partial overlaps by ModifiedTsNs, and the raced copies carry
identical bytes - while a fully covered duplicate is freed once the
entry lands.
* filer: allow one mutating TUS request per session at a time
Nothing stopped two PATCHes from writing the same range concurrently:
both loaded the same offset, both passed the conflict check, and both
recorded their sub-chunks. A client whose request timed out in a proxy
retries immediately while the server side is still storing the buffered
sub-chunk, which is exactly that race.
A session now accepts one PATCH or DELETE at a time, the way tusd locks
uploads; a concurrent one is refused with 423 Locked, which TUS clients
retry, and HEAD keeps answering so progress polling is unaffected. The
chunk state is loaded under the claim, so a retried PATCH sees every
record its predecessor left and conflicts cleanly instead of duplicating
data.
* test: cover a TUS PATCH raced by its own retry
Stalls a PATCH mid-body over a raw connection, retries the same range
while it is in flight, and expects the retry refused with 423 Locked;
the upload then resumes from the reported offset and the final content
must be intact.
* filer: never free a TUS duplicate the entry still references
Coverage is computed from ranges, so a record fully covered by another
is treated as a duplicate no matter which needle it names. A malformed
record naming a file id the entry keeps would have had that needle freed
right after the entry landed - the corruption this change set exists to
stop. The duplicates are now freed in one batch, skipping any file id
the entry references; their records go with the session directory.
* test: bound the raw TUS connection reads
http.ReadResponse on the stalled PATCH's connection blocked until the
whole go test timeout if the filer never answered.
* filer: free the needles of chunk write attempts a retry replaced
A volume server stores the needle locally and only then fans out to the
replicas, so a replication failure 5xxs with the data already written.
Each attempt assigns its own file id, so once a later attempt lands
elsewhere nothing references the earlier ones: the caller only sees the
chunk that succeeded, and the failed ids were dropped.
They are now freed the way the caller frees them when the whole write
fails. Retrying on a 5xx makes this reachable on every read-only or full
volume, which is exactly the condition that filled the reporter's
volumes.
A \\server\share -dir was passed to WinFsp as a plain mount point, which
treats it as a directory path on an actual remote server and fails. Turn it
into the VolumePrefix option instead, so the mount registers with the WinFsp
network provider: the UNC path is then reachable from every logon session,
which a drive letter mounted from a service is not, and each user can map
their own drive letter to it.
Comments only, no rendering change: the values paragraphs compress to
the density of the file around them, the env-var note becomes a
template comment instead of leaking into the rendered manifest, and the
two spots that invite a wrong simplification - the unconditionally
rendered -port.lance and the empty-placeholder platform guard - each
get their one-line why.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* helm: serve the S3 gateway's Lance Namespace, on by default
Standalone `weed s3` serves the Lance Namespace API on 9101 unless told
not to, so the chart defaulting s3.lancePort to 9101 matches weed's own
posture instead of hiding the port behind a null. The flag is always
rendered, so lancePort: 0 reaches weed as -port.lance=0 and genuinely
disables the namespace rather than silently falling back to the binary
default; 0 also drops the service port and the optional lanceIngress,
which otherwise mirror the iceberg wiring. The NetworkPolicy admits the
port the same way it admits icebergPort.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* helm: run the Lance maintenance worker beside the Go worker
The Go and Rust workers have no overlapping jobs - Go serves vacuum,
balance, EC and iceberg_maintenance, only /usr/bin/weed-worker serves
the lance_* family - so a cluster serving Lance tables needs both, not
an either/or switch. The worker deployment now adds a worker-lance
container whenever the namespace is reachable: worker.namespaceUrl, or
derived from the release's S3 service and s3.lancePort. Untouched Go
container; admin address derived the same way; mTLS flags point at the
already-mounted worker cert when security is on; metrics on their own
worker.lanceMetricsPort (9328, next in the 932x convention) with the
same health probes, service port and scrape endpoint the Go container
gets, and the worker NetworkPolicy admits that port exactly when the
container renders. The image carries an empty placeholder on armv7/386
where exec falls back to the shell and exits 0, so the command refuses
those platforms by name; s3.lancePort: 0 is the escape hatch there.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* filer.backup: key the checkpoint by source path and sink destination
The checkpoint id hashed only sink name + directory, so two backups to
different buckets or endpoints sharing a directory layout advanced one
checkpoint: whichever job was running pushed the shared offset forward,
and a stopped or failing job later resumed from the other's position,
silently skipping changes. Backups of different source paths to the same
destination shared a checkpoint the same way.
Each sink now reports a destination identity (endpoint or account,
bucket or container, directory) and the checkpoint is keyed by the
source path plus that identity. Reads fall back to the historical
name+directory key when the new key has no value, so existing backups
resume where they left off; writes go only to the new key.
* filer.sync: include the target path in the offset key
The offset stored on the target filer was keyed by source path and
source filer signature only, so two syncs from the same source cluster
and path to different directories on the same target cluster advanced
one shared checkpoint, and the slower one could resume past events it
never applied. The target path now participates in the key; "/" keeps
the historical form, and a sync with a non-root target path falls back
to the historical key once when its own key has no value yet.
* join checkpoint key fields with NUL so they cannot alias
A path or configuration value spelling out the separator could
concatenate two different field tuples to the same checkpoint key.
NUL cannot appear in a CLI path argument or any sane configuration
value, making the encoding injective.
* admin: offer copyable S3 object URLs in the bucket file browser
* admin: hide object urls when the bucket type lookup fails
* admin: ignore an s3.public_endpoint that is not an absolute http url
* mini: build the seeded s3 endpoint with JoinHostPort for ipv6
* admin: reject a query or fragment in s3.public_endpoint
* mini: drop the seeded s3 endpoint when a later run disables s3
* admin: reject userinfo and bare delimiters in s3.public_endpoint, redact the warning
* mini: pass its s3 endpoint as an admin option instead of mutating viper
* admin: keep the rejected s3.public_endpoint value out of the log
The SSE streaming path kept writing 200/206 from filer metadata before
fetching or decrypting anything, so a missing needle or failed decrypt
setup surfaced as a broken 200 body. Same deferral as the plain path:
the status commits on the first body write, and every failure before
that returns to the handler for a clean S3 error response.
streamFromVolumeServers wrote the 200/206 status from filer metadata
before any byte had been fetched from a volume server, so a missing or
corrupted needle surfaced as a broken 200 body and the request metrics
recorded a success. Defer the status commit to the first body write: a
failed first read now returns a clean 500 before headers, while the
wire timing of successful responses is unchanged since net/http buffers
the status line until body bytes arrive anyway.
* filer: keep the sentinel when CreateEntry reports an update failure
CreateEntry flattened the error UpdateEntry wraps, so errors.Is stopped
matching and ErrExistingIsDirectory and ErrExistingIsFile never reached
the S3 mapper, which answered a retryable 500 instead.
* s3: let a key that is a prefix of other keys be an object
S3 keys are flat, so "a/b" and "a/b/c" are independent objects that
coexist in either write order. The filer stores a key as a path, so one
of them has to live on the directory the other is nested under.
Writing the nested key first refused the prefix key outright. Writing it
second promoted the file to a directory, which kept its data but lost the
key: an empty object left nothing to recognise it by and disappeared, and
one with data listed under a trailing slash it never had.
Mark the directory that carries such a key, and write the object onto it
when the path is already a directory. The mark makes an empty prefix
object visible to listings and readable by GET and HEAD, keeps the empty
folder cleaner off it, and lists it under the key it was written with.
Deleting the key strips the mark back off along with the data.
* filer: keep a TTL off a directory that stands for an object
An expired entry is deleted a row at a time, so expiring a directory
removes it and leaves everything under it unreachable. Promoting a file
to a directory carried its TTL across, and a promoted file is exactly the
one that has keys nested under it.
Drop the TTL on promotion, and leave one an older build wrote alone. The
lifecycle worker still expires the object, through the delete that leaves
the directory behind.
* s3: delete the null version of a key other keys are nested under
The routed delete cannot remove an entry that other keys live under, and
answered a retryable 500 rather than falling back to the lock path the
unversioned delete already falls back to. That path then looked the entry
up under the bucket with the whole key as its name, so the demote wrote it
back one directory too high and failed as not found.
Fall back on any non-precondition error, and split the key before deleting
it. Trailing-slash directory markers with children reach the same delete.
* filer: keep the sentinel when MkFile and Mkdir report a create failure
Same flattening one layer out: every mkFile caller lost the sentinel, so
a CopyObject onto a key that other keys are nested under answered a
retryable 500 where a PutObject of the same key answers 409.
* s3: copy and rename a key that other keys are nested under
Such a key is stored on the directory those keys live in, and copy and
rename both refused it: the source lookup maps every directory entry to
NoSuchKey, so a key a plain GET serves could not be copied or moved, and
the destination side refused it as a directory conflict.
The source is read through a view of the entry as the object it names.
The destination is written the way a PutObject of that key writes it. A
rename at either end copies the object's own data across and strips it off
the source key rather than going through AtomicRenameEntry, which moves a
directory by moving everything under it - the nested keys are not part of
what is being renamed.
* mini: reserve the admin gRPC port instead of binding it late
Port selection probes every port with a throwaway listener and closes it.
Master, filer, volume and S3 bind a moment later, but the admin waits for
all of them first and only then binds its worker gRPC port, roughly two
seconds in. That port defaults to the admin http port + 10000, which lands
inside the Linux ephemeral range, so one of the cluster's own outgoing gRPC
dials can take it during the gap and the admin dies on bind, taking the
worker with it.
Keep the listener from the availability check and hand it to the admin.
* mini: clear the admin gRPC reservation before retaking it
A rerun inside one process would otherwise inherit the closed listener of
the previous run whenever the reservation fails, and the admin would accept
it and only find out inside Serve.
* mini: snapshot the admin options for the startup goroutine
The cleanup path read the package-level options long after the goroutine
started, so a later in-process run could have its reserved listener closed
by the previous run.
* ec: read a needle's intervals in parallel
A needle spanning more than one EC block gets one interval per block, and
consecutive blocks live on different shards. We read those intervals in
sequence, so a 4MB chunk landing in a volume's 1MB small-block region cost
five round trips to five different servers.
Read them concurrently into disjoint slices of a single buffer, at most 8 in
flight. Same change in the Rust volume server's phase C.
* ec test: seed the random payload instead of the deprecated rand.Read
* admin: treat a missing S3 Tables policy as an empty load, not an error
The bucket/table policy GET relayed the backend's 404 NoSuchPolicy to the
dialog, whose loader treats any non-OK response as a load failure and
keeps Save and Delete blocked. A bucket or table without a policy could
never be given one. Return policy null instead, the same contract
ShowBucketPolicy uses for classic buckets.
* admin: reject policy documents the structured editor would misread
A top-level JSON array passed the object guard (typeof [] is 'object')
and loaded as a zero-statement policy, which the next commit would
rewrite to an empty document. Object elements in Action/Resource were
coerced to '[object Object]' and saved that way on the s3tables surface,
which stores policies verbatim. Both now throw, which routes the
document to the JSON tab like other unrepresentable shapes.
* admin: let the JSON tab save documents the structured editor can't model
Save with the JSON tab active required a round-trip through
policyDocToEditorState, so exactly the documents the dialogs shunt to
'JSON tab only' mode (unrepresentable Effect, Resource+NotResource, and
the like) could never be saved - Delete was the only mutation left.
Invalid JSON still blocks; an unrepresentable document now saves and the
editor state stays marked unparsed.
* admin: pin the policy editor to what each consumer's backend supports
The s3tables evaluator has no NotResource/NotPrincipal fields - it
silently drops them, turning Allow+NotResource into allow-everything and
making Deny+NotPrincipal inert - and it only matches s3tables: actions
against s3tables ARNs, while the editor suggested s3: actions and
arn:aws:s3::: resources. New registerPolicyEditor knobs: allowNegation
hides the Not* modes and routes documents using them to the JSON tab;
resourceSuggestions pins the Resource autocomplete to the open
resource's ARN; the S3 Tables dialogs get an s3tables-only action
datalist. requirePrincipal now also hides NotPrincipal, which
policy_engine.ValidateBucketPolicy always rejects, and the client-side
check requires Principal specifically to match that server rule.
* admin: save S3 Tables policies from a button, not form submission
The multi-input structured editor sits inside a form whose Save button
was type=submit, so Enter in any single-line editor input - accepting an
autocomplete suggestion, say - implicitly submitted whatever half-built
statement the editor held, and the backend stores the document verbatim.
A lone statement with no Principal matches nobody, locking out every
non-owner. Save is now an ordinary button and the form ignores
submission.
* admin: block zero-statement policy saves
Committing the active tab before the emptiness check made 'Policy JSON
is required' dead code: an empty editor serializes to {"Statement":[]},
which the s3tables backend stores verbatim - evaluated default-deny for
every non-owner, while the statement-count column keeps showing 'Not
configured'. All three policy dialogs now refuse a save with no
statements and point at Delete instead. The classic bucket modal only
gained a clearer message; the server already rejected the document.
* admin: guard S3 Tables policy mutations against stale and overlapping requests
The save/delete completions ran against whatever resource the shared
modal happened to show by then: a slow PUT for one bucket would hide the
modal mid-edit of another and misattribute its alerts, a late DELETE
cleared the shared textarea over the newly opened resource with its
loaded flag set, and nothing stopped a double-click from firing two
overlapping mutations. Ported the classic modal's pattern: capture the
target on start, flag the mutation in flight with the buttons disabled,
and only touch the UI when the completion still matches the open
resource. Success now reloads the page, which also keeps the Policy
column's statement count honest.
* admin: confirm before deleting an S3 Tables policy
Delete Policy sat next to Save and fired on a single click; with
default-allow enabled one stray click silently dropped the resource
policy and left the bucket open to every principal. Same confirmation
the classic bucket modal already has.
* admin: let a corrupt stored bucket policy be shown, fixed, and deleted
A stored document the decoder rejects made the policy GET 500, and with
the loaded flag never set the modal blocked both Save and Delete - the
one policy an operator most needs to remove was the one they couldn't,
even though the delete path never reads the document. The GET now
returns the raw bytes alongside a null policy; the dialog hands them to
the JSON tab and unblocks the buttons.
* admin: url-encode the bucket name in the policy API calls
The filer lists any directory under the buckets path, names S3 would
never allow included; one carrying '#' or '%' broke the fetch URL or
addressed a different name than the modal shows.
* admin: drop stale edit-policy responses on the IAM policies page
The same race the bucket and S3 Tables dialogs already guard against:
open one policy's editor while its GET stalls, open another, and the
late response populates the editor under the second policy's name -
Update then saves the first policy's statements over the second.
* admin: warn before a bucket policy save drops unsupported fields
The editor tracks unmodeled top-level keys precisely so
confirmPolicyFieldDiscard can warn before the server's Version+Statement
decode discards them, but only the IAM page called it; the bucket modal
saved a pasted document with e.g. a console-generated Id without a word
while the editor kept displaying the field.
* s3: enforce the bucket policy size cap on both surfaces
The 20KB cap lived only in the admin UI, so a larger policy stored via
the S3 API displayed there but could never be re-saved, desyncing the
two writers the cap comment claimed could not desync. The constant now
lives in policy_engine next to the shared validator and PutBucketPolicy
rejects oversized documents with PolicyTooLarge, matching AWS.
* admin: ship the policy editor's fieldset styles with the editor
The .policy-stmt-* rules that undo Bootstrap's full-width legend reset
stayed behind in policies.templ when the editor markup moved to the
shared script, so the bucket and S3 Tables dialogs rendered Actions/
Resource/Principal as full-width jumbo headings. PolicyDatalists is the
component every consumer already renders once; the styles live there
now.
* s3: mirror bucket policy changes into the IAM store from the metadata subscription
The advanced-IAM path appends the bucket-policy:<bucket> document to
every STS/session evaluation, but only this gateway's own PutBucketPolicy
maintained that mirror - a policy tightened or created through the admin
UI (or another gateway) never reached it, so revoked access stayed live
indefinitely, and the delete side was an unimplemented TODO in any case.
The metadata subscription now diffs the stored policy on every bucket
entry change and updates or removes the mirror, covering all writers and
deletion with one mechanism; IAMManager gains the missing
RemoveBucketPolicy.
* admin: deduplicate the bucket policy write path
Set and Delete carried line-for-line identical filer closures;
bucketPolicyMutation already treats nil as clear-the-key. The shared
helper sits below Set's validation, since ValidatePolicy cannot take the
nil document Delete passes.
* s3: drop ValidateBucketPolicy's re-checks of ValidatePolicy rules
Both callers run ValidatePolicy first, which already enforces the
version and at-least-one-statement rules; the duplicates were dead code
with drifted error text.
* admin: seed a new statement's Resource from the pinned suggestions
A fresh statement on the S3 Tables dialogs started with no resource row
at all; seed it with the broadest pinned ARN the same way cfg.bucket
already seeds the classic modal.
* admin: refuse to save Not* fields the backend would silently drop
Hiding the NotResource/NotPrincipal modes was not enough where negation
is disallowed: the JSON tab accepts any valid document (that is its
job), and a statement's Advanced-fields box can reintroduce the keys, so
an s3tables save could still store fields the evaluator drops - turning
Allow+NotResource into allow-everything. commitPolicyActiveTab now runs
a final document-level check over what would actually be saved; Delete
stays available for cleanup.
* s3: move the IAM bucket policy mirror on a bucket rename
A same-directory rename delivers one event carrying both entries, and
the byte-equality short-circuit skipped the new name's mirror when the
policy was unchanged - while the replayed delete for the old name
removed its mirror, leaving the renamed bucket unmirrored. The mirror
decision is now a pure function that removes the old name and writes the
new one regardless of byte equality, with the rename cases unit tested.
* s3: backfill the IAM bucket policy mirror on lazy bucket loads
The metadata subscription only mirrors changes, so a policy that
predates the IAM integration never reached the bucket-policy:<bucket>
mirror and its grants did not bind on the IAM path until the policy was
next modified. The gateway is deliberately lazy at startup (nothing
lists all buckets), so the backfill hooks the same place a bucket's
policy first becomes known: the cold bucket-config load. EnsureBucketPolicy
writes only when no mirror is stored, so repeat loads cost one cached
read.
* s3: reconcile the bucket policy backfill against concurrent changes
The backfill's check-then-write could race an event-driven mirror update
or removal and re-store bytes that were already stale, with no later
event to heal it. EnsureBucketPolicy now reports whether it wrote, and a
write is reconciled against a fresh authoritative entry read: a changed
policy is re-mirrored, a removed one is removed. Anything changing after
that read fires its own event, which finds the backfill's write already
present and supersedes it. The backfill also carries the entry's raw
bytes rather than a re-marshaled document, so the reconcile can
byte-compare.
* s3: prime the bucket policy mirror before advanced-IAM authorization
The backfill ran from the lazy bucket-config load, but IAM authorization
evaluates the bucket-policy:<bucket> mirror before any handler runs - a
grant carried only by a not-yet-mirrored policy denied forever, and the
denied request never reached the code that would have loaded the bucket.
authorizeWithIAM now primes the bucket config first (an in-memory cache
hit once warm), and the backfill runs synchronously on the cold load so
the very first authorization already sees the mirror.
* admin: honor a persisted or admin.toml maintenance enabled=false
The startup path discarded an operator's enabled=false twice over:
ApplyDefaultsToProtobuf treated the bool zero value as unset and applied
the schema default of true, and a force-enable migration block flipped
any survivor. With the legacy /maintenance UI routes gone, nothing could
write the config either, so the maintenance system ran unconditionally.
Keep the persisted enabled flag across schema-default application in
LoadMaintenanceConfig, drop the force-enable block, and add a top-level
[maintenance] enabled key to admin.toml as the config surface, persisted
through SaveMaintenanceConfig like the per-task settings. Absent config
still defaults to enabled.
* admin: track presence on the maintenance enabled flag
A plain proto3 bool cannot distinguish an operator's persisted false
from a legacy file that simply omits the field, so honoring false would
have silently switched maintenance off for configs written before the
toggle could be persisted. Make the field optional: files that predate
presence tracking keep the enabled default, while a file that explicitly
persists the toggle is honored either way.
* admin: manage S3 bucket policies from the admin UI
Bucket policies were only manageable through the S3 PutBucketPolicy API;
the admin UI had no equivalent to the quota/owner/lifecycle editors it
already offers. Add GET/PUT/DELETE for a bucket's policy, sharing the
exact validation the S3 gateway uses.
- Extract validateBucketPolicy/validateResourceForBucket out of
s3api_bucket_policy_handlers.go into policy_engine.ValidateBucketPolicy /
ResourceMatchesBucket so both the S3 API and the admin UI enforce
identical rules.
- weed/admin/dash/bucket_policy.go: Get/Set/DeleteBucketPolicy, writing
through ObjectTransaction + PATCH_EXTENDED (the lifecycle pattern) so a
concurrent owner/quota/lifecycle change on the same bucket entry isn't
clobbered. Propagation to every S3 gateway is automatic via the existing
filer metadata log subscription. The S3 gateway's IAM policy mirror is
deliberately not replicated here (its delete path is already an
unimplemented TODO on the S3 side).
- New GET/PUT/DELETE /api/s3/buckets/{bucket}/policy routes, CSRF-guarded
on writes.
- Bucket list and details modal now show a statement-count badge, read
from the entry already fetched (no extra RPC).
- UI: a JSON-textarea policy editor modal, matching the lifecycle modal's
structure.
* admin: reuse the visual policy editor for bucket policies
Extract the structured policy editor (add/remove statement, action/
resource/principal rows with autocomplete, JSON tab kept in sync) out of
policies.templ's inline script into a shared
weed/admin/static/js/policy_editor.js, and wire the bucket policy modal
in s3_buckets.templ up to it instead of a bare JSON textarea.
- registerPolicyEditor(which, config) replaces the hardcoded create/edit
id derivation with a per-instance config (textarea/tab/body ids,
datalist ids, requirePrincipal, bucket). The IAM policies page keeps its
exact pre-extraction ids via two registerPolicyEditor calls, so its
markup is unchanged.
- New policy_datalists.templ exposes the three shared <datalist>s
(actions/resources/principals) as @PolicyDatalists(), now rendered by
both policies.templ and s3_buckets.templ.
- requirePrincipal seeds new bucket-policy statements with Principal: "*"
and adds a client-side check before save (the server, via
policy_engine.ValidateBucketPolicy, remains the actual authority); the
bucket config pins the Resource autocomplete to the open bucket instead
of fetching every bucket in the cluster.
- layout.templ loads policy_editor.js globally, after admin.js/
modal-alerts.js (basePath/escapeHtml/showAlert) which it depends on.
3a (the extraction) is a byte-preserving move verified against the
unchanged policies.templ behavior before layering 3b's parameterization
and the bucket-policy wiring on top.
* admin: migrate S3 Tables bucket/table policy editors to the shared editor
Third consumer of the shared visual policy editor: the S3 Tables bucket
and table policy modals (a bare JSON textarea each) now get the same
structured Editor/JSON tabs as the bucket policy and IAM policy pages,
via registerPolicyEditor('s3tablesBucketPolicy'/'s3tablesTablePolicy',
{ textareaId: ... }). Storage and validation are untouched - S3 Tables
policies still go through their own s3tables.PolicyDocument type and the
s3tables.policy extended attribute, unrelated to policy_engine and
s3-bucket-policy; only the editor UI is shared.
Fix a real bug surfaced by adding this second load path: the bucket
policy modal (and the naive first draft of this s3tables port) called
commitPolicyTextareaToEditor() right after a GET and then force-switched
to the Editor tab. commitPolicyTextareaToEditor() is designed to leave
the current tab in place and the editor state untouched when a document
fails to parse (so an in-progress edit survives a bad tab switch), so
forcing the Editor tab afterwards could show empty/stale editor state
that a careless Save would then serialize over a perfectly valid but
structurally-unusual stored policy. Add
loadPolicyTextareaIntoEditor(which) to policy_editor.js, which has no
"current tab" to defer to and instead falls back to the JSON tab with an
alert on a document the structured editor can't represent - the same
safety editPolicy already had in policies.templ - and use it at all three
"populate the editor right after a GET" call sites (bucket policy,
S3 Tables bucket policy, S3 Tables table policy).
* admin: show policy statement count on the S3 Tables buckets page
Mirrors the "Policy" column already added to the classic S3 buckets
list: a clickable badge with the statement count when the table bucket
has a resource policy, "Not configured" otherwise. S3 Tables policies
are a separate mechanism (s3tables.PolicyDocument under the
s3tables.policy extended attribute) from the S3 bucket policy work
elsewhere in this branch (policy_engine.PolicyDocument /
s3-bucket-policy), so this is a parallel implementation of the same
pattern rather than shared code.
- S3TablesBucketSummary gains PolicyStatementCount, populated in
GetS3TablesBucketsData from entry.Entry.Extended[s3tables.ExtendedKeyPolicy]
via the new extractS3TablesPolicyStatementCountFromEntry - no extra RPC,
the entry is already fetched for ExtendedKeyMetadata.
- The badge reuses the existing .s3tables-bucket-policy-btn class, so it
opens the same policy modal as the row's action button with no JS
changes.
* admin: don't let a failed policy GET open the door to an empty overwrite
loadS3TablesBucketPolicy/loadS3TablesTablePolicy cleared the textarea,
then unconditionally called loadPolicyTextareaIntoEditor() regardless of
whether the GET actually succeeded - including when fetch() rejected or
the response was not ok, silently logged to console only. That leaves
the structured editor holding a legitimate-looking empty policy
({version, statements: []}), with the Editor tab active by default.
If Save is then clicked, commitPolicyActiveTab() serializes that empty
state into the textarea as `{"Version":"2012-10-17","Statement":[]}` -
a non-empty string - before the "Policy JSON is required" guard ever
sees it, so the guard passes and the transient load failure gets
written over whatever policy was actually stored.
Add s3tablesBucketPolicyLoaded/s3tablesTablePolicyLoaded, set true only
once a GET has actually completed (ok, including a genuinely empty
policy) and false on any failure path (fetch rejection or a non-ok
response, which previously fell through silently). Both submit handlers
now check the flag before touching the editor at all, and a failed load
surfaces via alert() instead of only a console.error - the user
previously had no visible indication the load had failed.
Verified with a jsdom simulation driving the real rendered page against
a stubbed fetch: a failed GET followed by Save now sends no PUT at all
(previously it sent Statement: []); a successful GET followed by Save
still PUTs the loaded policy unchanged.
* admin: address code review findings on the policy editor
1. policy_editor.js: policyEditors is only pre-populated for 'create'/
'edit'; every other `which` (bucket, s3tablesBucket, s3tablesTable)
stays undefined until its first successful async load. Nothing in
this file enforces that a page hide its Editor/JSON tabs and
Add-statement button until that load completes - the S3 Tables policy
modals don't - so a click in that window (e.g. Add statement, or
switching to the JSON tab) threw "Cannot read properties of undefined
(reading 'unparsed')". Add policyEditorState(which), which lazily
initializes a default state, and route addPolicyStatement, the
jsonTabBtn 'show.bs.tab' handler, commitPolicyActiveTab, and
renderPolicyEditor through it. Verified with a jsdom simulation
against a never-resolving fetch: the exact click threw on the
pre-fix code and no longer does.
2. s3_buckets.templ: the bucket-policy Save handler checked the
textarea for emptiness before calling commitPolicyActiveTab(), which
is what actually serializes the structured Editor tab's fields into
that textarea. A policy entered entirely through the Editor tab (the
primary path - never touching the JSON tab) left the textarea at
whatever it was at load time, so creating a new policy this way hit
"Enter a policy document" and Save silently did nothing. Move the
commit before the emptiness check, preserving the existing alert and
early-return. Verified with a jsdom simulation: Add-statement then
Save (no tab switch) now PUTs the entered statement; before the fix
the same sequence never reached fetch().
3. s3tables_buckets.templ / s3tables_tables.templ: the policy Editor/
JSON nav-tabs were missing the ARIA roles Bootstrap's own tab pattern
expects (role="tab"/"tabpanel", aria-selected, aria-controls,
aria-labelledby) - screen readers had no way to tell these were tabs
or which pane went with which button. Added the standard Bootstrap 5
tab markup to both.
* admin: guard policy load/save flows against overlapping requests
1. s3tables.js: loadS3TablesBucketPolicy/loadS3TablesTablePolicy had no
protection against overlapping loads. Opening one bucket's (or
table's) policy dialog and then another's before the first GET
resolved let the late response write its document into the shared
textarea and mark the dialog "loaded" while it was now targeting the
second resource - a subsequent Save would then push the first
resource's policy onto the second. Add a per-load monotonic sequence
number (s3tablesBucketPolicyRequestSeq / s3tablesTablePolicyRequestSeq,
the same pattern already used for the classic bucket-policy load in
s3_buckets.templ); a response is only applied - textarea, loaded flag,
editor state - if its captured sequence still matches the latest one
issued.
Verified with a jsdom simulation: bucket A's policy load (artificially
slow) followed immediately by bucket B's (fast) previously left A's
policy in the textarea once A's late response landed; it now correctly
keeps B's.
2. s3_buckets.templ: the bucket-policy Save button lives outside the
(initially hidden) editor wrapper, so it stays clickable while a load
is still in flight - the existing policyRequestSeq guard only protects
the *load* from a stale response, not Save from firing before any
load for the current bucket has completed. Add bucketPolicyLoaded,
reset before each GET and set only once the matching response lands,
and check it at the top of the Save handler.
Verified with a jsdom simulation: clicking Save immediately after
opening the dialog, before a (deliberately never-resolving) GET
settles, now sends no PUT; a normal load-then-save sequence still
PUTs the loaded policy unchanged.
* admin: address further code review findings on the policy editor
1. s3tables.js: loadS3TablesBucketPolicy/loadS3TablesTablePolicy only
reset the JSON textarea when a new load starts; the structured editor
kept showing the previously loaded resource's statements (Editor tab
is the default active one) until the new fetch resolved. Call
loadPolicyTextareaIntoEditor() against the now-cleared textarea
immediately, so switching resources visibly resets the editor right
away instead of only once its own load completes. Verified with jsdom:
opening bucket A (loads fully) then bucket B (GET never resolves) no
longer leaves A's statements visible in B's editor.
2. s3tables.js: deleteS3TablesBucketPolicy/deleteS3TablesTablePolicy had
no loaded-state check, so a failed GET (which already blocks Save)
left Delete fully able to remove the resource's stored policy sight
unseen. Add the same s3tablesBucketPolicyLoaded/s3tablesTablePolicyLoaded
guard Save already uses. Verified with jsdom: delete after a failed
load now sends no DELETE; delete after a successful load is unaffected.
3. s3_buckets.templ: the bucket-policy Editor/JSON nav-tabs were missing
the same ARIA roles already added to the S3 Tables policy tabs in an
earlier round (role="tab"/"tabpanel", aria-selected, aria-controls,
aria-labelledby) - this instance was out of scope for that review
comment but is the same gap. Bootstrap's own tab.js already manages
aria-selected on tab switch once the attribute exists, so no extra JS
was needed.
4. s3_buckets.templ: neither the bucket-policy Save nor Delete handler
guarded against a double-click, or against firing while the other was
still in flight - two overlapping PUT/DELETE requests for the same
bucket could land in either order. Add a shared
bucketPolicyMutationInFlight flag: set (and both buttons disabled)
before each fetch, cleared (and buttons re-enabled) on failure so the
user can retry, left set through the existing success hide-and-reload
path, and also reset when a new bucket's dialog opens so an abandoned
in-flight request from a closed dialog can't leave the buttons stuck
disabled. Verified with jsdom: double-clicking Save now sends exactly
one PUT, and a Delete click while that PUT is still pending sends no
DELETE.
* admin: scope bucket-policy mutation completions to the bucket that started them
1. The previous round's fix reset bucketPolicyMutationInFlight whenever a
new bucket's policy dialog opened, to avoid leaving Save/Delete stuck
disabled if the modal was closed mid-request. That traded one bug for
a worse one: if bucket A's PUT/DELETE was still in flight when the
user opened bucket B's dialog, the reset let B's Save/Delete fire
immediately, and A's completion handler - unaware anything had
changed - would still hide the (now B's) modal and reload the page
out from under whatever the user was doing with B, on success, or
alert a message with no bucket context, on failure.
Stop resetting on reopen, so a pending mutation for a previous bucket
keeps this bucket's Save/Delete blocked until it settles (matches the
"preventing overlapping mutations" the review comment describes).
Instead, capture policyEditorBucket as targetBucket right before each
fetch and compare it against policyEditorBucket again in the
completion handler: the in-flight flag is always released so the
buttons never get stuck, but the modal-hide/reload/alert only fire if
this bucket is still the one showing; a stale completion for an
abandoned bucket just logs to the console instead.
Verified with a jsdom simulation: opening bucket B while bucket A's
Save is still pending leaves B's Save button disabled and a click on
it a no-op; once A's PUT resolves, B's button re-enables but no
modal.hide()/reload() fires (previously both fired unconditionally).
2. bucketPolicyDeleteBtn had no bucketPolicyLoaded check, unlike Save -
a failed GET blocked Save but left Delete free to remove a policy the
client never actually saw (the same gap already fixed for the S3
Tables policy modals in an earlier round). Added the same guard,
ahead of the confirm() dialog. Verified with jsdom: Delete after a
failed load now sends no DELETE request.
* admin: fix spelling mistake
The storage rule's fsync decision was computed by the filer
(detectStorageOption -> rule.Fsync) and applied on the filer's own HTTP
write path, but was never carried onto the chunk uploads S3 issues: the
AssignVolumeResponse had no fsync field, so the s3api client could not
learn the decision, and the chunked upload URL was hardcoded without it.
Every S3 write to a path with fsync configured went to the volume server
as a non-fsync write.
Carry the decision through the assign response:
- filer.proto: AssignVolumeResponse gains bool fsync, filled from the
storage option the assign resolved.
- operation.AssignResult gains Fsync, so uploadChunk can append
?fsync=true to the volume server upload URL (single and replica
fan-out paths).
- The S3 PUT/UploadPart assignFunc, the S3 copy path, the admin file
browser upload, and the Iceberg worker assign functions all forward
the response field.
Adds TestUploadReaderInChunksAppendsFsyncWhenAssigned.
* erasure_coding: one home for the shard-count to volume-slots conversion
* ec: refund the cleared leftover shards' slots in the encode source health check
The Lance namespace gateway took the request-body location field, trimmed a
trailing slash, and passed it straight to the marker sink. That location feeds
TableDataDirFromMetadataLocation, which joins it under /buckets and collapses
any ../ segments, and writeMarker's CreateEntry then auto-creates every missing
parent. A caller could point the location at another tenant's bucket, or escape
/buckets entirely, and plant a fixed-name marker (recursively creating the
parents) or hide a victim's live table with .lance-deregistered.
Confine the declared location the way the Iceberg gateway already does: require
an s3:// URI whose bucket is the caller's own and whose path carries no
traversal segment, on both the declare and register handlers.
A client that reconnects before the old handler exits re-registers the
same client name, and addClient overwrites the map entry. The old
handler's deferred deleteClient then closed whatever channel the map
held under that name: the new, live stream's. Receiving from a closed
channel returns nil immediately and forever, so the new handler's send
loop degenerated into sending empty responses at wire speed, pinning a
core on each side until the client killed the connection.
deleteClient now closes the channel its own handler registered and
leaves the map entry alone unless it still points to that channel. This
also closes the previously orphaned old channel, whose drain goroutine
used to leak. The send loop treats a closed channel as an exit instead
of a message stream.
s3api: no filer failover after fn has consumed part of a response
withFilerClientFailover replays fn verbatim on the next filer, so a filer
that died mid-stream followed by a healthy peer returned success with the
callback's closure-captured accumulator holding the dead filer's prefix
twice; the per-attempt accumulator in listWithRetry could not close this,
because the replay happens inside a single attempt. Track delivery on the
connection handed to fn: once a unary reply or streamed message has reached
the callback, surface the transport error unwrapped instead of failing
over, and let callers replay from a clean slate. A filer that fails before
delivering anything fails over exactly as before.
* telemetry: sync the server module to go 1.26
The root module moved to go 1.26 but the telemetry server module, which
replaces seaweedfs with the repo root, stayed on 1.25.8, so go refuses
to build or test it until the directive catches up.
* telemetry: confirm a cluster after a week of reports, not two days
Two days of history still lets recurring CI and demo clusters into the
confirmed fleet: anything torn down and rebuilt across a UTC midnight
counts. Requiring seven distinct UTC days keeps the fleet charts and the
version/OS distributions to clusters that actually stay up; real
clusters qualify after their first week, and the fallback to all active
clusters while none is confirmed is unchanged.
* filer, volume server: return 416 when no requested range overlaps the content
* seaweed-volume: return 416 when no requested range overlaps the content
* server: check the range test error, use the request context, fix the no-overlap comment boundary
* master: never re-seed a raft cluster over committed state
-raftBootstrap deleted logs.dat, stable.dat and snapshots on every start and
then bootstrapped a fresh cluster. Since hashicorp raft only snapshots after
8192 log entries, the TopologyId lives in the log, not in a snapshot, so the
pre-wipe snapshot recovery found nothing and each restart minted a new cluster
identity. A master that came up while it could not reach its peers seeded a
rival cluster; when the two logs met, SetTopologyId's split-brain guard fatally
stopped every master holding the other id, and the master layer crash-looped
with no quorum.
Bootstrapping is genesis. Drop the wipe and the inline bootstrap. The first
master in -peers already mints a cluster once it has confirmed no peer has a
leader, so the flag has nothing left to do and is now ignored; keeping that one
master the sole bootstrap authority is what stops a partition from minting two
clusters, so the flag must not widen it either. A master with state rejoins its
peers, and one whose data dir was reset is admitted by the sitting leader
instead of forking again.
* test: cover -raftBootstrap restarts in the multi-master suite
Three masters start with -raftBootstrap, the way the helm chart renders it on
every master on every roll, and the cluster has to hold one TopologyId after
they all restart. /dir/status is proxied to the leader, so each master's own
view of the identity is read out of its log, which is where a fork shows up.
Before the fix the hashicorp case minted a new id on each restart.
* fix(mount): reply to LINK with the kernel node id, not the stored inode
Link() answered the kernel with out.NodeId = oldEntry.Attributes.Inode.
That attribute is a mount-runtime number and only entries created through a
mount carry one. An entry written by the S3 API, WebDAV or a direct filer
call persists inode 0, so the LINK reply named node id 0, which the kernel
rejects as invalid_nodeid and reports as EIO. The hard link itself had
already been written to the filer, which is why it looked correct again
after a mount restart.
The same stale number was also used as an inodeToPath key. AddPath(0, path)
filed the new link under inode 0, so a later Lookup on that name handed the
kernel node id 0 as well, and a LOOKUP reply carrying node id 0 means no
such entry.
in.Oldnodeid is the node id the kernel already holds for the source, and it
is the key inodeToPath is indexed by, so use it for the reply, for AddPath
and for the sibling sync.
Fixes#8404
* test(mount): cover the sibling sync in Link with a third hard link
The two existing cases never reach the body of syncHardLinkSiblings: with
two links the source alias and the name just created are both in skipPaths,
so the loop iterates over nothing and a change to that site goes unnoticed.
A third link leaves one name that no other part of Link() writes.
The new case drives three links off one source. It guards against covering
nothing (it fails if every path turns out to be a skipPath), checks that
every name of the file reports nlink 3, and then drives the sync with both
candidate keys to pin down which one it has to be: keyed by the source's
persisted Attributes.Inode, which is 0 for an entry written outside a mount,
GetAllPaths has no path to walk, while the kernel node id reaches the
sibling.
That second half is driven directly because Link() alone cannot tell the two
keys apart. The meta cache keeps one blob per hard link id (FilerStoreWrapper
setHardLink/maybeReadHardLink), so a read of any sibling returns the
attributes of the last write to any of them whether or not the sync ran.
* s3api: retry a transient failure when listing multipart uploads/parts
A blip on the way to the filer failed the whole ListMultipartUploads or
ListParts request. Both reported failure points sit inside one streaming
listing: the ListEntries call that opens the stream, and the stream.Recv
calls that drain it. Neither retried, so a single Unavailable answer from
a filer that was restarting turned into a 500 for the S3 client.
Replay the listing instead, bounded to three attempts with a 100ms
backoff that doubles. Only a transient failure is replayed. A not-found
answer stays authoritative so the empty-list branch still works, and
every other error still reaches the client on the first attempt.
This is scoped to (*S3ApiServer).list rather than added inside
DoSeaweedListWithSnapshot, which mount, the shell and the other object
listings share, and where a retry after a partial stream would
re-deliver entries the callback had already seen. Within one call to
list, a replay is safe: it collects into a fresh slice each time, so it
can neither duplicate nor drop entries.
That guarantee does not extend past this function. withFilerClientFailover
already re-runs its callback against the next filer on any non-NotFound
error without resetting the caller's accumulator, so on a multi-filer
gateway a mid-listing failover can itself produce a duplicated result
with err == nil, independent of this change and not fixed by it. Noted
in the PR rather than silently left for someone to rediscover.
Fixes#7221
References #7235
* s3api: move the listing retry inside list itself
---------
Co-authored-by: Junker der Provinz <jdp@braethoria.com>
* admin: honour persisted task configs when building the maintenance policy
buildPolicyFromTaskConfigs passed a literal nil to vacuum, erasure_coding
and balance LoadConfigFromPersistence. Those functions look for their
LoadXTaskPolicy() accessor via a type assertion, which a nil interface can
never satisfy, so every call fell through to NewDefaultConfig() and the
policy came back with the compiled-in defaults - Enabled: true among them.
A task disabled on disk was therefore still scheduled, and the only trace
was a glog.V(1) "Using default ... configuration" line.
Thread the real ConfigPersistence through instead. There are two copies of
this function: the one in weed/admin/dash builds config.Policy on the
normal admin startup path and can simply take cp as its receiver, and the
one in weed/admin/maintenance is the fallback used when the config carries
no policy yet, which now receives the store from NewMaintenanceManager.
weed/admin/dash already imports weed/admin/maintenance, so the maintenance
side has to keep the duck-typed interface{} parameter that the task
loaders already use rather than importing the concrete type back.
The store is only handed over when a data directory is configured: an
unconfigured one has nothing to read, and a typed nil pointer would pass
the loaders' type assertion and then panic on first use.
Fixes#10874
* admin: restore the maintenance scan cadence after an error backoff
scanLoop shortens its ticker to the error backoff delay after a failed
scan, but it decided whether to replace the ticker by comparing the
target interval against the configured scan interval instead of against
the interval the ticker was actually running at. Once the errors stopped,
getScanInterval returned the configured interval again, the comparison
came out false, and the ticker was left at the backoff delay - so a
single transient scan failure pinned the scanner to one scan per second
for the rest of the process lifetime. That is the ~1/second cadence in
issue #10874: 658 KB/s of "Cancelled N stale pending balance tasks
before re-detection" and 193k orphaned task files over two days.
Track the interval the ticker is running at and compare against that, so
both entering the backoff and returning to the normal cadence replace the
ticker.
While in here:
- defer ticker.Stop() bound the ticker that was current when the defer
was registered, so every replacement ticker leaked on return. Wrap it
in a closure.
- running was written by Start/Stop and read by all three background
loops without synchronisation. Guard it with the existing mutex, fold
the running check in triggerScanInternal into the lock it already
takes, and make Stop a no-op when not running so a second call cannot
close the stop channel twice.
Refs #10874
* admin: make the maintenance policy actually reach the task detectors
Loading the persisted task configs into the maintenance policy only
matters if something reads that policy, and nothing did.
MaintenanceIntegration pushes the policy into every registered detector
and scheduler through interface{ SetEnabled(bool) } and
interface{ SetMaxConcurrent(int) } type assertions. Every task registered
through base.RegisterTask is backed by base.GenericDetector and
base.GenericScheduler, and neither implemented either method, so all four
assertions failed silently for every task on every startup. The policy's
enabled flag reached nothing: ScanWithTaskDetectors gates on
detector.IsEnabled(), and the queue's policy lookups for max concurrent
and repeat interval are fallbacks that only fire when the scheduler
reports zero, which the generic scheduler never does.
Add the setters, delegating to the TaskConfig.SetEnabled the interface
already declares and to TaskDefinition.MaxConcurrent, which is what
GetMaxConcurrent returns.
Applying the policy required three more fixes, because with the
assertions working the policy could now do damage as well as good:
- IsTaskEnabled reports false for a task type the policy has no entry
for, so applying it unconditionally would have disabled every task the
policy does not list. Skip task types with no policy entry: no entry
means no opinion, not disabled.
- ec_balance was exactly such a task. It is registered like the other
three but had no entry in the policy builder and no accessor on
ConfigPersistence at all, so its configuration could never be
persisted. Add SaveEcBalanceTaskPolicy/LoadEcBalanceTaskPolicy, the
task_ec_balance.pb file, the SaveTaskPolicy dispatcher case, and the
policy entry.
- InitMaintenanceManager ran before loadTaskConfigurationsFromPersistence,
which replaces each task's whole config object, so the policy was
applied and then immediately thrown away. Swap the order. Both read the
same files, so the policy is now the last writer and stays
authoritative.
MaintenanceManager.UpdateConfig also updated the queue's and the
scanner's policy but not the integration's, so a policy changed at
runtime never reached the detectors. Add MaintenanceIntegration.SetPolicy
and call it.
While building the policy, stop hand-copying each task's fields and use
the task's own ToTaskPolicy(). The hand-written version was a second
definition of every task's policy and had already lost the erasure coding
preferred tags and replica placement and the balance IO rate limit. For
the same reason, the "nothing persisted yet" branches of
LoadVacuumTaskPolicy, LoadErasureCodingTaskPolicy and
LoadBalanceTaskPolicy now derive from each task's NewDefaultConfig()
instead of a third hand-written copy. Those copies had drifted, so with a
data directory but no config file on disk the effective defaults differed
from what the task and the admin UI schema both advertise:
vacuum scan interval 24h -> 2h
balance scan interval 6h -> 30m
balance imbalance 0.1 -> 0.2
erasure coding scan interval 168h -> 1h
erasure coding fullness 0.90 -> 0.95
erasure coding min volume 1024MB -> 30MB
Finally, weed/admin/dash and weed/admin/maintenance each carried a copy
of the policy builder and they had already diverged. Export the
maintenance one as BuildPolicyFromTaskConfigs and have dash call it.
Refs #10874
* worker: warn when a config store cannot supply a task's persisted config
LoadConfigFromPersistence logged a single glog.V(1) "Using default X
configuration" for every way of not loading anything, so the bug in
issue #10874 - a store handed in that the type assertion rejects, leaving
a task running on compiled-in defaults - looked exactly like the normal
"no data directory configured" case. The reporter had to read the source
to work out why their disabled task kept running, and asked for this
specifically.
Separate the cases. A non-nil store that does not provide the accessor is
always a wiring bug and is now logged at warning level, naming the type
and the missing method. A read error or a policy that will not apply is
also a warning. No persistence configured, and a store with nothing saved
yet, stay at V(1): those are normal.
Refs #10874
* admin: stop GetTaskPolicy panicking on a maintenance policy that is nil
GetTaskPolicy dereferenced its MaintenancePolicy argument to look at
TaskPolicies, so IsTaskEnabled, GetMaxConcurrent and GetRepeatInterval
all took the admin process down when handed a nil policy. A nil policy is
not a programming error here: MaintenanceConfig.Policy is unset until
something builds one, DefaultMaintenanceConfig returns a config with no
policy at all, and UpdateConfig installs whatever config it is given.
Found by calling IsTaskEnabled with the policy from a freshly defaulted
MaintenanceConfig.
Treat a nil policy as "no entry": no task enabled, the safe concurrency
default of 1, and a repeat interval of 0 so callers fall back to their
own default instead of reading DefaultRepeatIntervalSeconds off nil.
Also add the startup test this was found with. It walks the admin
server's startup sequence over a data directory that has balance saved as
disabled and checks the state that decides whether issue #10874 happens:
the balance detector reports disabled, vacuum stays enabled, and tasks
whose config was never saved keep their compiled-in default.
Refs #10874
* admin: document the synchronisation SetPolicy would need beyond startup
ConfigureTasksFromPolicy now really writes TaskDefinition.Config and
TaskDefinition.MaxConcurrent, which the scan loop reads through
detector.IsEnabled() with nothing synchronising the two. Every caller
runs during admin server startup today, before the scan loop exists, so
there is no live race - but the next caller has to add the locking, and
the same already applies to UpdateAllConfigs replacing the whole config
object. Write it down at the seam instead of leaving it to be
rediscovered.
Refs #10874
s3api: retry a transient failure when listing multipart uploads/parts
A blip on the way to the filer failed the whole ListMultipartUploads or
ListParts request. Both reported failure points sit inside one streaming
listing: the ListEntries call that opens the stream, and the stream.Recv
calls that drain it. Neither retried, so a single Unavailable answer from
a filer that was restarting turned into a 500 for the S3 client.
Replay the listing instead, bounded to three attempts with a 100ms
backoff that doubles. Only a transient failure is replayed. A not-found
answer stays authoritative so the empty-list branch still works, and
every other error still reaches the client on the first attempt.
This is scoped to (*S3ApiServer).list rather than added inside
DoSeaweedListWithSnapshot, which mount, the shell and the other object
listings share, and where a retry after a partial stream would
re-deliver entries the callback had already seen. Within one call to
list, a replay is safe: it collects into a fresh slice each time, so it
can neither duplicate nor drop entries.
That guarantee does not extend past this function. withFilerClientFailover
already re-runs its callback against the next filer on any non-NotFound
error without resetting the caller's accumulator, so on a multi-filer
gateway a mid-listing failover can itself produce a duplicated result
with err == nil, independent of this change and not fixed by it. Noted
in the PR rather than silently left for someone to rediscover.
Fixes#7221
References #7235
* admin: count plugin-runtime workers in worker metrics
The admin server keeps two worker registries: the legacy maintenance-worker
map, filled by workers registering over the worker gRPC stream, and the plugin
worker registry, filled by workers started as `weed worker`. Both the
SeaweedFS_admin_workers_connected / SeaweedFS_admin_worker_slots gauges and the
dashboard's Workers card read only the legacy map, so a cluster that runs the
admin and its workers as separate components reported 0 workers even while its
workers showed up on the plugin pages and ran scheduled jobs.
Aggregate both registries instead. The two are merged by worker ID: `weed mini`
starts both runtimes out of one working directory, so they share the persisted
worker ID and must not be counted twice. For such a worker the slot numbers
still come from the legacy registry, which keeps mini's existing readings.
Plugin workers report their slots in the heartbeat, so detection and execution
slots are summed from there; a worker that has connected but not yet sent a
heartbeat counts as connected with zero slots.
Fixes#10525
* admin: clamp negative worker-reported slot values in metrics merge
A plugin worker's self-reported heartbeat slot counts are untrusted
input; clamp them to 0 before summing so a stale or misbehaving
worker can't drive the aggregate gauge negative, matching the same
defensiveness already used in registry.go's own slot arithmetic.
* admin: add visual iam policy editor
Add a structured, tabbed editor (Editor / JSON) for creating and editing
IAM policies in the admin dashboard, alongside the existing raw-JSON
textarea:
- policies.templ: per-statement cards for Sid, Effect, Action, and
Resource, with unmanaged fields (Principal, NotPrincipal, NotResource,
Condition, or anything else) preserved verbatim in a per-statement
"advanced fields" JSON box so nothing is lost on round-trip. Switching
tabs commits and reparses in both directions. Restored the "Use Sample
Policy" button, now filling both the structured editor and the JSON
tab. The "Validate" button now calls the existing but previously
unused POST /api/object-store/policies/validate endpoint instead of
doing JS-only checks.
- Progressive Resource ARN autocomplete: suggests bucket names first,
then once "bucket/" is typed, suggests bucket/* plus the bucket's
direct subfolders, drilling down one path segment at a time as the
user types further "/" characters.
- New GET /api/files/list-folders endpoint (file_browser_handlers.go)
backing the folder autocomplete: wraps the existing file browser data
function and returns just the subdirectory names as JSON, scoped to
paths under /buckets.
- Action-name suggestions (datalist) for the Action field, sourced from
the existing s3_constants.S3_ACTION_* constants plus new
s3_constants.S3TABLES_ACTION_* constants (extracted from the s3tables
operation dispatch switch) so the suggestion list can't drift from the
strings the engines actually understand.
- policy_handlers.go: ValidatePolicy now accepts a statement with only
NotResource set (previously required Resource), matching
policy_engine.validateStatement and the fact the new editor makes such
statements reachable from the UI.
- Tests: ValidatePolicy behavior, route registration for the policy API
and the new list-folders endpoint, list-folders path scoping, and the
action-suggestion list's shape.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* admin: fix XSS, cache poisoning, and cap overshoot in policy editor
Address code review findings on the IAM policy editor added in the
previous commit:
- policies.templ (displayPolicyDetails): escape every interpolated
policy value (Sid, Effect, Action, Resource, policy name, and the raw
JSON document) before assigning to innerHTML. Policy documents can
come from other admins or an import, so an unescaped field could
execute script when the "View" modal renders it.
- policies.templ (policyEditorStateToDoc): reject JSON arrays in a
statement's "advanced fields" box, not just invalid JSON. `typeof []
=== 'object'` was true, so a JSON array was assigned to the statement;
subsequent property assignments (Sid, Effect, ...) landed on the array
object but JSON.stringify of an array only serializes numeric indices,
silently dropping them.
- policies.templ (loadPolicyFolderNames): on a failed folder lookup,
remove the cache entry instead of permanently caching the empty
fallback, so a transient network/server error doesn't block retries
for the rest of the page's lifetime.
- file_browser_handlers.go (ListFolders): stop appending directory
names as soon as the running count reaches maxListFoldersEntries,
instead of only checking the cap after a full page is processed,
so the returned list never exceeds the configured cap.
Regenerated policies_templ.go with the already-stamped templ v0.3.1001
to keep the diff scoped to this file.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* admin: stop policy editor from clobbering the active tab and dropping malformed advanced fields
Address two review findings on the IAM policy editor (Issue 3, stored-XSS
in displayPolicyDetails, was already fixed by the previous commit and is
unchanged here):
- createPolicy, updatePolicy, and validatePolicyDocument always committed
the structured editor's (possibly stale) state into the JSON textarea
before submitting, even when the user had just edited the JSON tab
directly. That silently discarded the user's JSON edits and
validated/saved the old structured-editor state instead, which could
leave broader permissions in force than intended.
Added commitPolicyActiveTab(which), which commits whichever tab is
currently visible into the other side instead of unconditionally
overwriting the JSON tab from the editor: if the JSON tab is active it
parses that JSON back into the structured editor (without touching the
textarea itself), otherwise it serializes the structured editor into
the textarea as before. All three call sites, plus the JSON-tab
"show.bs.tab" handler, now use this and abort with an alert if the
currently active tab's content can't be committed.
- policyEditorStateToDoc silently continued with an empty object when a
statement's "advanced fields" box held invalid JSON, so switching
tabs, validating, or saving would drop Principal/NotResource/Condition
from that statement without telling the user. It now throws (with the
statement number and parse error) on invalid or non-object JSON there,
and callers surface that via showAlert and abort instead of proceeding.
Regenerated policies_templ.go with the already-stamped templ v0.3.1001.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* admin: keep unmanaged top-level policy fields across editor tab switches
policyDocToEditorState only carried Version and Statement into editor
state, so any other top-level key (e.g. Id) present in the JSON tab was
silently rewritten away as soon as the user switched to the Editor tab
and back. Capture those keys in state.otherFields and merge them back in
policyEditorStateToDoc before Version and Statement are written, so the
two tabs stay faithful to each other and the editor never rewrites text
the user typed.
Note this is editor fidelity only: the admin API's
policy_engine.PolicyDocument carries just Version and Statement, and
DocumentJSON is never populated, so such fields are still discarded by
the server once a policy is saved. Making them survive a save would
require a backend change, which is out of scope here.
Regenerated policies_templ.go with the already-stamped templ v0.3.1001.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* admin: warn before a policy save discards unsupported top-level fields
The editor round-trips unmanaged top-level keys (e.g. Id) between the
Editor and JSON tabs, but the admin API's policy_engine.PolicyDocument
carries only Version and Statement, so the server drops them on save and
the user saw no indication.
Added confirmPolicyFieldDiscard(), called from createPolicy and
updatePolicy after the active tab is committed (so the field list is
accurate whichever tab is showing). It names the fields that will be
lost and lets the user confirm or cancel. Not wired into
validatePolicyDocument, which doesn't persist anything.
Chose the warning over the alternative of persisting these fields
through the backend: policy_engine.PolicyDocument is shared by the S3
bucket-policy engine and IAM evaluation, so extending it would change
the stored document shape for every policy in the codebase - far beyond
the scope of this editor.
Regenerated policies_templ.go with the already-stamped templ v0.3.1001.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* admin: reject malformed Effect and Resource/NotResource conflicts in policy editor
Two review findings on the IAM policy editor:
- policyDocToEditorState defaulted any non-"Deny" Effect (missing,
misspelled, wrong case) to "Allow". A statement meant to be "Deny" with
a typo like "deny" would silently become a permissive "Allow" instead
of being rejected. It now throws on anything but an exact "Allow" or
"Deny", naming the offending statement and value.
commitPolicyTextareaToEditor catches this the same way it already
catches invalid JSON: alert the user and keep the JSON tab active
instead of switching to the Editor tab with wrong data.
- policyEditorStateToDoc could save a statement with both Resource (from
the structured field) and NotResource (surviving in the "advanced
fields" extras from before the user switched to using Resource) set at
once - a contradictory combination neither the admin's ValidatePolicy
handler nor policy_engine's evaluator rejected. When the structured
Resource field is non-empty it now deletes any leftover NotResource
from extras, consistent with the file's existing rule that structured
fields take precedence over extras. Mirrored the existing
Principal/NotPrincipal exclusivity check in
weed/admin/handlers/policy_handlers.go's ValidatePolicy to reject the
same combination server-side, since create/update perform no
validation at all. Deliberately left policy_engine.validateStatement
(used by the S3 bucket-policy PUT handler for every bucket policy in
the product) unchanged - extending that shared validator is a larger,
separate change outside this admin-editor fix's scope.
Added a handler test for the new Resource+NotResource rejection.
Regenerated policies_templ.go with the already-stamped templ v0.3.1001.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* admin: add NotResource support to the visual policy editor
Since Resource and NotResource are mutually exclusive (enforced by a
previous fix), NotResource could previously only be set through the raw
JSON in a statement's "advanced fields" box. Promote it to a first-class
mode of the structured editor:
- The static "Resources" label is now a Resource/NotResource dropdown;
the same list of values underneath is reused for either key depending
on the selected mode, with a short form-text explaining the semantics.
- NotResource is added to POLICY_STATEMENT_KNOWN_KEYS, since it's now a
managed field like Resource rather than something that falls through
to extras.
- policyDocToEditorState derives resourceMode from which key is present
on load, and throws (same handling as the existing malformed-Effect
case: alert, keep the JSON tab active) if a hand-edited document has
both Resource and NotResource on one statement, since that can't be
represented by the dropdown.
- policyEditorStateToDoc writes only the key matching the selected mode,
replacing the previous one-directional "delete NotResource whenever
Resource is set" fix with mode-driven logic that also deletes Resource
when NotResource is selected.
- displayPolicyDetails (the read-only View modal) now shows the actual
NotResource values with a distinct label instead of a static
"(NotResource used instead)" placeholder.
No backend changes: the server-side "cannot specify both" check added
previously in policy_handlers.go's ValidatePolicy already covers this.
Regenerated policies_templ.go with the already-stamped templ v0.3.1001.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe
* admin: reject non-object policy documents; catch Principal/NotPrincipal conflicts server-side
Two review findings:
- policyDocToEditorState treated a top-level JSON value that wasn't an
object (null, or a bare string/number/boolean) as an empty statement
list instead of failing explicitly. If the user typed e.g. "hello" or
42 in the JSON tab and switched to the Editor tab, their input was
silently discarded and replaced with an empty policy - the same class
of "guess instead of reject" bug fixed for malformed Effect and
Resource/NotResource conflicts previously. Added an explicit check
that throws for null/scalar input, while leaving array and object
document shapes accepted exactly as before.
- weed/admin/handlers/policy_handlers.go's ValidatePolicy checked the
Resource/NotResource conflict by non-empty length
(len(...Strings()) > 0), which misses a statement where Resource is
explicitly present but an empty list (e.g. "Resource": []) alongside a
non-empty NotResource. Switched that check to field presence (!= nil),
matching how policy_engine's own validateStatement already treats
Principal/NotPrincipal exclusivity. Also added the equivalent
Principal/NotPrincipal presence check to this handler, which had none
before - the advanced-fields box in the visual editor lets a user set
both today, and nothing server-side caught it. The existing
non-empty "Resource or NotResource is required" check is left as a
length check, since an empty array shouldn't count as "provided".
Added test cases for both conflict checks in policy_handlers_test.go.
Regenerated policies_templ.go with the already-stamped templ v0.3.1001.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe
* admin: add Principal/NotPrincipal support to the visual policy editor (v1, AWS-only)
Adds a first, deliberately narrow structured editor for a statement's
Principal/NotPrincipal, left out when NotResource support was added:
- A Principal/NotPrincipal mode dropdown mirrors the existing
Resource/NotResource one (same mutual-exclusivity handling: the two
fields can't be set at once, and switching modes reuses the same
value list).
- A simple repeatable text-value list feeds a single {"AWS": [...]}
object on save - always the AWS type, never the "bare" (untyped)
SeaweedFS-extension shape. Per policy_engine's allowedPrincipalKeys,
Service/Federated/CanonicalUser also parse successfully, but nothing
in the S3 bucket-policy evaluation path ever sets a real caller's
principal to a service name, an OIDC provider ARN, or a canonical
user ID, so only AWS is functionally meaningful today - out of scope
for this v1.
- On load, only the exact {"AWS": ...} single-key shape is unwrapped
into the structured field and removed from "extras". Anything else
(bare string/array, a different single type key, or several type
keys at once) is left untouched in "extras" exactly as before, with a
visible warning under the dropdown so the user knows a
Principal/NotPrincipal exists but isn't shown there. Saving with the
structured field left empty never touches whatever's already in
extras, so a preserved complex form isn't silently dropped just
because the user didn't touch this field.
- The read-only View modal now displays Principal/NotPrincipal for any
shape (via a small generic summarizer), not just the AWS-simple one.
- Generalized the action/resource field-to-state-key mapping (used by
commitPolicyEditorForm and the add/remove-item click handler) into a
shared lookup table instead of stacking another ternary, now that a
third field (principal) exists.
Regenerated policies_templ.go with the already-stamped templ v0.3.1001.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe
* admin: support the bare "*" wildcard Principal in the visual editor
"Principal": "*" (and NotPrincipal: "*") is the standard AWS shorthand
for "everyone" and is common in real bucket policies, but the v1
Principal/NotPrincipal editor only recognized the {"AWS": ...} object
form, leaving a bare "*" statement's principal hidden in Advanced
fields.
parseSimpleAwsPrincipal now also accepts the bare string "*" as a
simple, structurally-editable value. On save, a principal value list
containing exactly ["*"] is written back as the bare "*" string
(matching the common convention) rather than wrapped as {"AWS": "*"};
anything else still wraps under AWS as before. Updated the field's
form-text hint accordingly.
Regenerated policies_templ.go with the already-stamped templ v0.3.1001.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe
* admin: add Principal field autocomplete backed by users + IAM roles
Adds a datalist-backed autocomplete for the policy editor's Principal/
NotPrincipal text fields, sourced from a new API listing existing
identities:
- weed/admin/dash/principal_suggestions.go: AdminServer.GetPrincipalSuggestions
combines S3 user ARNs (via the existing GetObjectStoreUsers +
iam.UserArn) with IAM role ARNs (via integration.NewFilerRoleStore /
ListRoles, reusing the exact same construction already used in
iam_manager.go - no new dependency risk introduced). Role ARNs are
reconstructed from the role name using SeaweedFS's default
arn:aws:iam::role/<name> convention rather than fetching each role's
stored definition, since this only backs a suggestion list. Role
listing failures are logged and swallowed rather than failing the
whole request - an incomplete suggestion list is fine, blocking
policy editing over it is not. Service accounts are deliberately not
listed separately: a service account's ARN is identical to its parent
user's, already covered by the user list.
- weed/admin/handlers/policy_handlers.go: GetPrincipalSuggestions handler
exposing this as {"principals": [...]}.
- Route registered at the API root (GET /api/principals) rather than
under policyApi's "/object-store/policies" prefix, since that
subrouter's existing "/{name}" GET route would shadow any
single-segment GET route registered after it (the same class of
gotcha previously seen with "/validate").
- weed/admin/view/app/policies.templ: a shared, lazily-fetched-once
policyPrincipalSuggestions datalist (flat list - unlike the
progressive per-folder Resource ARN autocomplete, users/roles aren't
hierarchical), wired into policyListRowHtml for field:"principal" and
populated on input/focus, with "*" always offered first.
Added tests for the new ARN-construction helper and route registration.
Regenerated policies_templ.go with the already-stamped templ v0.3.1001.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe
* admin: fix fieldset/legend styling in the structured policy editor
Bootstrap's form reset stretches <legend> to the fieldset's full width
(float: left; width: 100%), which loses the native "notch in the
border" look and makes each section's label bar as wide as the card.
Add two scoped classes: .policy-stmt-fieldset (border, rounded
corners, spacing between sections) and .policy-stmt-legend (undoes the
float/width so the legend hugs its content, with a little padding).
Applied to the three per-statement sections (Actions,
Resource/NotResource, Principal/NotPrincipal), replacing the ad hoc
"border rounded" utility classes that were doubling up with the
fieldset's own border. Also gave the "Advanced fields" <details> a
small top margin to match the new spacing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe
* admin: suggest bucket/* alongside the bucket itself in Resource autocomplete
At the bucket-name stage of the Resource field's progressive
autocomplete, only "arn:aws:s3:::bucket" was offered. Add
"arn:aws:s3:::bucket/*" right alongside it, since granting access to
everything in a bucket is the more common case and previously required
typing a "/" first to reach the folder-level "*" suggestion.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe
* admin: keep an unparseable policy in the JSON tab instead of wiping it
editPolicy() built the structured state inside the fetch .then, so a
policy the editor cannot model threw into the sibling .catch, which
alerted and called hide(). Showing the alert at that moment left the
modal on screen with an empty editor and the document only in the JSON
tab, and Save Changes then serialized the empty state over the policy.
Reachable two ways, since neither create path rejects these: the admin
API never validates on create, so "Effect":"allow" is stored as typed,
and policy_engine.validateStatement lets Resource and NotResource sit
in the same statement.
Hand the document to the JSON tab instead, which is what that tab is
for, and mark the state so nothing serializes the placeholder over it.
* admin: validate a policy document before saving it
Validation was wired only to the Validate button, so nothing stopped a
document the server's own validator rejects from being stored. With the
structured editor supplying the boilerplate and required dropped from
the textarea, opening the modal, typing a name and clicking Create
Policy was enough to save a statement-less policy.
Share validatePolicyJSON with the two save paths and abort on failure.
* admin: bound the folder autocomplete listing
maxListFoldersEntries caps the folders collected, but nothing capped the
entries paged through to find them, so a bucket holding only flat object
keys - no subfolders to count - was walked to the end, 200 entries per
round trip, behind one keystroke. Measured against an in-process filer:
6 entries 0.5ms, 3k entries 7.7ms, 30k entries 53ms, all of it linear in
the directory rather than in the answer.
Cap the scan as well, and let GetFileBrowser take a prefix so the segment
the user is still typing is filtered by the filer instead of by paging.
The same 30k directory now answers in 0.6ms once a prefix is typed.
* admin: clean the path before scoping list-folders to /buckets
util.CleanWindowsPath only rewrites backslashes, so "/buckets/../etc"
walked straight past the prefix check the endpoint relies on for its
scope. Nothing leaked - filer paths are literal keys, so the traversal
resolved to nothing - but the check reads as a boundary and wasn't one,
and the test asserting it didn't cover the one input that would try.
validateAndCleanFilePath in the same file already does this.
* admin: stringify policy values before escaping them
escapeHtml calls text.replace directly, and the Sid, the per-item action
and resource inputs, and the View modal's Resource/NotResource all pass
values straight out of JSON.parse. A policy carrying "Sid": 5 or
"Action": [1] threw "text.replace is not a function" and took the render
with it. escapedJoin already coerced; use it everywhere and coerce the
editor state at the point it's built.
* admin: only show the NotResource hint in NotResource mode
The hint rendered unconditionally, so it sat under a selector reading
"Resource" telling the user the statement applies to everything except
what they'd listed. Redraw the card when the selector changes so it
follows the mode.
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* install.sh: install the Rust maintenance worker
The release publishes weed-worker but the installer only knew weed and the
Rust volume server, so the one binary that cannot be built without a Rust
toolchain was the one you had to download by hand.
--component all skips it on a platform it has no build for rather than failing
an install that already put two binaries in place; asking for it by name there
still says so.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* install.sh: clean each component's temp directory as it finishes
The EXIT trap is per-process, so installing more than one component left every
extraction but the last behind. Cleaning at the end of the function keeps the
trap for the paths that exit early.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* worker: compile plugin.proto with the protoc that ships with the build
seaweed-volume already does this: protoc-bin-vendored carries the binary, so
the build needs no package manager and every build sees the same version. An
explicit PROTOC still wins, which is what lets the lance crates - whose own
build scripts read the same variable - share it.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* ci: point the worker builds at the vendored protoc
The jobs installed protobuf-compiler for lance's build scripts. They read
PROTOC, so pointing it at the binary protoc-bin-vendored already puts in the
registry serves them without a system package - one less apt call on the way
to a release, and the same protoc a developer's build uses.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* docs: say what the worker build needs from protoc
The lance crates' build scripts are the ones that need it, not ours, and they
take the same vendored binary.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* worker: name the binary weed-worker
It is the Rust side of `weed worker`, the way weed-volume is the Rust side of
`weed volume`, and lance is the first family of jobs it carries rather than
the only one it ever will. The crate keeps its own name: when a second family
arrives the bin target moves to a crate of its own, under this name.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* docker: ship the Rust maintenance worker in the image
Lance table buckets need a worker that can read the format, and until now the
only way to get one was a Rust toolchain and a cargo build. It now sits at
/usr/bin/weed-worker beside the Rust volume server, reached as
`docker run chrislusf/seaweedfs worker-rust --admin host:23646` — the verb
mirrors volume-rust, so plain `worker` still runs the Go one.
Taken pre-built or not at all: the lance jobs pull in arrow and datafusion, far
too large a tree to compile inside the image build, so an architecture CI did
not build for gets the empty placeholder the entrypoint refuses to exec, the
way the Rust volume server already does.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* ci: build the Rust worker for the container images
The same native cross-compile the volume server uses, so the release, latest
and dev images all carry it on amd64 and arm64. The artifact holds both
binaries now, so it is named for that rather than for the volume server.
Only the release directory each job builds is cached: with a debug profile
beside it the worker's target/ reaches 24GB, against a 10GB cache budget.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* ci: publish Rust worker binaries with the release
Linux amd64 and arm64 only: the worker runs beside the cluster it maintains,
and its dependency tree makes every extra target an expensive build.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* ci: build and test the Rust workers on change
Nothing built seaweed-worker in CI, so the release and the container images
would have been the first place a break showed up. Tests run in release too,
rather than compiling lance, arrow and datafusion again in another profile.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* docs: say how to get a released worker
Neither the image nor the release tarballs were mentioned; a toolchain and a
cargo build read as the only way in.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* ci: install protoc for the Rust worker builds
lance's crates compile their own protos, and unlike seaweed-volume they do not
vendor a protoc to do it with, so every job that builds the worker failed at
lance-encoding's build script.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* ci: do not persist credentials in the worker release checkout
The upload step is handed a token explicitly; a cargo build script should not
find another one sitting in the checkout's git config.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* docker: keep the worker's argument boundaries
Unquoted $@ splits on whitespace and expands globs, so an argument carrying
either arrived as something else.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* helm: install chart CI against an image tag that exists
The release bumps appVersion on master well before the container build
publishes that tag, and the chart CI runs on the bump commit, so every
release turns it red with ImagePullBackOff. Resolve the tag first and
fall back to latest while the new one is still building.
* helm: run the chart CI when the workflow itself changes
* helm: bound the registry lookup in the chart CI
An unbounded curl can hold the job, and the log did not say why the tag
was rejected. Cap it and print the status.
* filer: keep the TUS sub-chunks that already landed when a write fails
A PATCH is split into 4MB sub-chunks, and each one is recorded in the
session as soon as it is stored. The session listing is what HEAD reports
as Upload-Offset and what the final entry is assembled from, so a record
is a promise that the data behind it exists.
When a later sub-chunk failed - a read-only volume, or a client that hung
up mid-body - the error path deleted the needles of every sub-chunk the
same PATCH had written but left their records in place. The resuming
client was then told to continue past bytes the filer had just queued for
deletion, and the upload completed into a gapless manifest pointing at
needles that were gone: HEAD returned the right size, GET died mid-body
once a vacuum reclaimed them.
Recorded sub-chunks now stay, which is what resumption expects: the
client picks up at the offset the session reports, and an upload that is
abandoned frees its chunks with the session.
* filer: drop a TUS chunk's record before freeing its data
filer.CreateEntry can return an error with the entry already inserted -
the parent-directory pass runs after the insert and keeps the entry when
it fails. A failed saveTusChunk therefore does not mean the record is
absent, and deleting the needle outright left the same corruption the
resume path used to cause: a session record pointing at data that is gone.
Remove the record first and only free the needle once it is gone. A
record lost with its data still stored merely leaks, which the vacuum and
fsck paths already account for.
* test: cover a TUS PATCH that is cut off mid-body
Resets the connection after one 4MB sub-chunk has landed, resumes from the
offset the session reports, and vacuums before reading the file back, so
anything the filer deleted behind a kept record shows up as a short read.
* test: drive the Lance namespace with Spark
The counterpart of catalog_spark, which does this for the Iceberg REST
catalog. Spark is the engine most likely to be pointed at a lakehouse,
and it reaches the Lance catalog through the connector's DSV2 catalog -
org.lance.spark.LanceNamespaceSparkCatalog with impl=rest - over the same
routes every other client uses.
SHOW NAMESPACES -> ['`sparklance-lcephd80`.ml']
SHOW TABLES -> ['sparklance-lcephd80$ml$embeddings']
count -> 3
filtered -> [(2, 'two'), (3, 'three')]
count after a second commit -> 4
The second insert is there on purpose: a store that cannot order commits
fails on the second one, not the first.
Two things the run settled that were guesses beforehand. CREATE TABLE
works, because the connector declares through the namespace and writes the
data itself rather than pushing Arrow at the server. And SHOW TABLES
returns the namespace's own identifiers - bucket, namespace and name
joined by the delimiter - not bare Spark table names.
Credentials go under the catalog's storage.* prefix, which is handed to
lance as object_store options; a gateway without STS vends none, the same
trap the LanceDB suite documents.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: verify the Lance table bucket was actually created
weed shell prints a command's own failure and still exits 0, so the harness
would go on to blame Spark for a bucket that was never made.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: bound the Docker probe
An unhealthy daemon makes docker version hang, and the probe runs before the
test has a timeout of its own.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: keep the ivy cache under the user's cache directory
It is mounted into a container running as root, so a shared temp path lets
another local user pre-create it and choose what Spark loads.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: assert the vector column's type, not only its name
A column that came back as array<double> or array<string> would still be
called vector and still pass.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: read the dataset off its location for real
The catalog being optional is the property that lets duckdb and pandas read
these tables; it was asserted in a comment and printed, never exercised.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: do not persist credentials in the Spark Lance checkout
The job only uploads a log on failure; nothing in it pushes.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: say the hosts in the README are placeholders
The suite passes dynamically allocated host.docker.internal ports.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
The db credential secret and the filer's WEED_MYSQL_* env were gated on
filer.enabled alone, so a filer on mongodb, redis, postgres or leveldb2 got a
generated mysql secret it never reads - kept forever by resource-policy: keep -
plus a mysql-db-host pointing nowhere.
Gate all of it on WEED_MYSQL_ENABLED, which is how the store is selected, plain
keys and secret-backed ones alike. An enable flag the chart cannot read - a
valueFrom, or one in secretExtraEnvironmentVars - counts as selected, so nothing
is dropped from a filer that is actually on mysql.
* admin: add bucket lifecycle rule editing
* address greptile's comments
* more small fixes
* coderabbit's comments
* more comment fixes
* more fixes
* more
* maybe last
* last ?
* 14850
* 14851
* filer: stamp the content MD5 on every SaveInsideFiler write
An entry's ETag falls back to Attributes.Md5, so conditional writers key
IF_ETAG_MATCH off it. SaveInsideFiler carried the looked-up attributes
forward without refreshing the hash, leaving it describing whatever the
previous writer stored: a later conditional write matched the stale hash
and overwrote content that had already changed.
* s3api: give the bucket lifecycle constants and the write route key one definition each
The extended-attribute keys, the XML size cap and the object-write ring key
prefix were each spelled out in two places, so the admin dashboard's copies
could drift from the gateway's. Move them to the packages both sides already
import and alias them where the short local name reads better.
* admin: patch the bucket entry's lifecycle keys instead of rewriting the entry
The save read the bucket entry, edited its extended map and wrote the whole
entry back, guarded by IF_UNMODIFIED_SINCE. Nothing that writes a bucket
entry advances its mtime - not the S3 gateway's patchBucketEntry, not
SetBucketOwner, not SetBucketQuota - so the guard never fired and the stale
snapshot reverted whatever else had changed since the lookup.
Send the PATCH_EXTENDED mutation the S3 gateway already uses for these keys:
the filer re-reads and merges under the bucket path lock, so only the two
lifecycle keys move. That removes the reason for the mtime snapshot, the
verification retry loop and the compensating restore of the cleared day-TTL
rules, which the migration now logs instead.
* s3api: run the delete-lifecycle day-TTL migration through the shared helper
DeleteBucketLifecycleHandler kept its own copy of the read-strip-write
sequence the put handler now shares, including a missing return that let a
ToText failure persist a truncated filer.conf and write a second response.
It also wrote the whole file back unconditionally, reverting any concurrent
edit; the shared helper writes conditionally.
* admin: answer 404 when a lifecycle request names a bucket that does not exist
Every SetBucketLifecycle failure came back as 500, including the lookup miss
for an unknown bucket, so a client or monitor read a caller error as a server
fault and retried it.
* s3api: emit lifecycle XML a client would recognize
Two changes to what MarshalCanonical writes, both visible through
GetBucketLifecycleConfiguration, which replays the stored bytes verbatim:
stamp the S3 namespace on the root, and put a size range under <And>. A
<Filter> carries one predicate, so two size bounds side by side is a shape
AWS does not document. Parsing still accepts either.
* admin: fix the lifecycle editor's handling of stored status, deletes and empty saves
Four things the editor got wrong:
A stored <Status> the S3 API never validated, say 'enabled', left both radio
buttons unchecked, so reading the form threw on a null querySelector result
and Save did nothing. Collapse anything but an exact 'Enabled' to 'Disabled',
which is what the engine already does with it.
Deleting a rule re-rendered an open edit form from the snapshot taken when
editing began, discarding what had been typed; every other transition folds
the form in first.
The Transition warning only matched a bare <Transition>, missing the form
with attributes, self-closed or namespace-prefixed.
Saving an emptied rule list clears the configuration through a path with no
prompt, next to a Delete-all-rules button that asks.
Also collapses the three divergent copies of formatBytes on this page to one.
* filer: stop the day-TTL migration from deleting an operator's path rule
The migration removed every rule under the bucket's path that carried a day
TTL in the bucket's collection. The add path it is retiring used
AddLocationConf, which merged its TTL onto whatever already sat at the
prefix, so a rule can hold operator settings the lifecycle path never wrote -
a disk type, WORM retention, a read-only flag, a placement pin. Deleting the
whole rule to retire its TTL took those with it, leaving objects under that
prefix on defaults nobody asked for.
Delete only rules shaped like ones the add path created from scratch;
anything else keeps its settings and loses just the TTL.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
The failover harness treated an open volume server port as readiness, but the
master only learns of a volume server from its heartbeat. A lone master refuses
heartbeats until its bootstrap check elects it, and the servers back off and
retry, so registration lands seconds after the ports answer. Tests that started
writing in that window assigned against an empty topology, which fails with
"no free volumes left" and reaches the mount as ENOSPC.
* volume: clear per-collection metrics when a collection leaves a server
The read-only and disk size gauges are only ever set for collections the
heartbeat still finds here, and nothing zeroes the rest. volume.balance marks a
volume read-only to move it, so the last heartbeat that saw it counts it
read-only - and if it was the collection's last volume on that server, that
count stands until the process restarts. The dashboard then shows read-only
volumes that volume.list -readonly cannot find anywhere.
Remember what each heartbeat set, and drop what is gone on the next one.
* volume: stop the read-only volume count from wrapping at 256
The per-collection counters were uint8, so a server holding 256 read-only
volumes of one collection reported zero of them.
* volume: read the read-only flags once when counting them
The heartbeat asked IsReadOnly for the verdict and then read noWriteOrDelete
and noWriteCanDelete straight off the volume, unlocked, so the reasons could
disagree with the verdict they were explaining. Take them together, under one
lock. The location is now nil-checked rather than skipped by short-circuit
evaluation, so a volume that has not joined a disk location yet stays safe.
* volume: let only a surviving volume keep its collection reported
A volume being deleted for expiry still made an entry in the read-only counts,
which is what the cleanup reads as "this collection is still here". The
collection's last volume could go and its series would stand for one more
heartbeat. Count the survivors only.
* volume: size a collection from the volumes it still has
The size totals are rebuilt from scratch every heartbeat, so subtracting a
volume that is about to be deleted took the surviving volumes' sizes down with
it: a collection keeping a small volume and losing a larger one reported the
difference, or lost its entry and kept the previous heartbeat's number.
* volume: cover the deleted bytes total in the surviving volume test
Deleted bytes are totalled the same way as sizes and were going unchecked, so
the test now leaves deleted needles on both volumes and pins that gauge too.
* s3: resolve ${VAR} in static config credentials from the environment
A deployment that keeps its S3 keys in a secret store had no way to hand
them to the gateway: -config takes a file, so the keys had to be written
into that file. Let a key in the static config name an environment
variable instead, and drop any credential whose reference stays unset so
the placeholder never becomes a usable key.
* helm: source the generated s3 identities from an existing secret
The only way to reuse credentials that already live in a Secret was to
hand-author the whole seaweedfs_s3_config JSON, since the literal keys in
values.yaml end up in git and a lookup-based keyRef renders empty under
helm template and Argo CD. Let s3.credentials.admin/read name a Secret and
its keys instead: the generated config references them as ${VAR} and the
gateway resolves them from the environment, so nothing is read from the
cluster at render time.
* s3: treat an empty environment value as an unresolved credential reference
A secret store can hand over a key that exists but is blank. Resolving it
would leave an access key whose signing secret is empty, so count it as
unresolved and drop the credential.
* helm: render the s3 secret when only the all-in-one auth flag is set
The all-in-one deployment mounts the s3 secret whenever any of the three
enableAuth flags is set, but the secret itself only rendered for the s3 and
filer flags, so allInOne.s3.enableAuth on its own left the pod waiting on a
secret nothing creates.
* helm ci: check the credential wiring on every workload that mounts it
The render check only looked at the standalone s3 deployment and only at
one of the four variables, so a helper that bound a variable to the wrong
secret key would still pass.
* helm: create the all-in-one s3 secret for every flag that mounts it
The all-in-one pod mounts the secret on any of the three enableAuth flags,
so keying its creation off allInOne.s3.enableAuth alone still left
filer.s3.enableAuth without filer.s3.enabled pointing at a secret nothing
creates. Mirror the deployment's own condition instead, and check each
flag renders both the mount and the secret.
* s3: reject a malformed credential reference instead of keying on it
A typo such as ${MY-VAR} matches no substitution, so it survived expansion
and the placeholder itself became the access key the gateway accepted.
Require every ${ in a static credential to open a well-formed reference.
* release: cut the whole release from the version bump workflow
The bump workflow stopped after pushing the version commit, and the rest was
manual: create the release, then run "Prepare release" in the csi-driver and
the operator. It now pushes the tag itself, which is what starts the binary,
container and helm workflows, creates the release with generated notes, and
dispatches the other two repositories, waiting for both.
Pushing the tag and reaching the other repositories both need RELEASE_PAT;
GITHUB_TOKEN raises no events that start workflows.
* release: tighten the release workflow after review
Check out master explicitly: a dispatch can select any branch, and the tag,
the commit and the release would then come off that branch while the
downstream job dispatches master.
Scope contents:write to the job that pushes; the downstream job talks to the
other repositories with RELEASE_PAT and needs nothing here.
Wait for the module proxy to serve the release commit as the tip before
dispatching, instead of priming it and hoping. The dispatched workflows pin
seaweedfs with `go get -u ...@latest`, so a stale tip means they release
against a pre-release commit, silently.
Identify the dispatched run by diffing the run list against the snapshot
taken before dispatching, rather than assuming the newest run is ours.
* release: wait on the downstream release, not on the run that makes it
A dispatched run cannot be told apart from a concurrent one: the API does not
report the inputs a run was dispatched with, so watching "the run that appeared
after mine" can watch someone else's and report their result as ours.
Wait for a release to appear in the downstream repository instead. That is the
thing being waited for, and it holds however many runs are in flight.
* test: read Lance tables from DuckDB
The LanceDB and Spark suites go through the catalog. DuckDB does not: its
lance extension reaches the data over S3 with no namespace involved, which
exercises the other half of the design - a table bucket's layout is a
valid Lance dataset directory, so a table stays readable when the catalog
is not in the path.
scan_rows=128
scan_columns=id,title,vector
filtered_rows=5
nearest=1,0,2
It also pins the one place the layout costs us. DuckDB's replacement scan
recognises a dataset by a .lance path suffix, and tables created through
this catalog deliberately have none: the catalog entry is the dataset
directory, a table name may not contain a dot, and a suffix would leak
into ARNs and policies. So __lance_scan is the way in, and the bare
SELECT ... FROM 's3://...' form does not see these tables.
The test asserts both halves - a suffixed path is read, a suffix-less one
is not - so if the extension ever recognises a bare directory, it fails
and says to update the documentation rather than leaving it wrong.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: require the catalog error from the suffix-less read
Any failure satisfied the old check - a missing extension, bad credentials,
an unreachable endpoint - so the assertion could pass without the
replacement scan ever classifying the path.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: verify the Lance table bucket was actually created
weed shell prints a command's own failure and still exits 0, so the harness
would go on to blame DuckDB for a bucket that was never made.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: bound the Docker probe
An unhealthy daemon makes docker version hang, and the probe runs before the
test has a timeout of its own.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: order the aggregates the assertions read
string_agg over an unordered relation may return the names, and the vector
search's ids, in any order, so the expectations could fail on a run where
nothing changed.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: do not persist credentials in the DuckDB Lance checkout
The job only uploads a log on failure; nothing in it pushes.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* master: answer with the leader raft already knows
Topo.Leader() backs off for up to 20 seconds waiting for an election.
Callers that a health probe or a client is blocked on cannot afford that:
/cluster/status, /cluster/healthz and /readyz all sit past the probe
timeout of both the helm chart and the operator, so a master that is
still joining looks dead rather than joining, and the kubelet restarts
it. informNewLeader and SendHeartbeat hold the client on a master that
cannot serve it, exactly when it should move on to find the one that can.
Answer these from MaybeLeader instead, which reports what raft knows
right now. MaybeLeader takes over the "am I the leader myself" fallback
that Leader() used to apply on top of it, so one non-blocking call is
still correct; Leader() keeps the backoff for callers that must wait.
* master: let the leader admit a master that starts with no raft state
Neither raft implementation lets a server outside the configuration
campaign: goraft's promotable() requires a non-empty log, and hashicorp
rejects vote requests from a candidate that is not in its configuration.
A master that comes up with fresh state therefore cannot elect itself in
— the leader has to pull it in. Nothing did.
The peer list is static, rendered from the replica count, so scaling it
up leaves the sitting leader running the old list with no idea the new
masters exist. Under goraft they wait forever. Under hashicorp they are
worse off: each bootstraps a cluster of its own from the new list, and
two of them form a quorum next to the live leader, with their own
TopologyId. That is the split brain SetTopologyId kills a master over.
Admit the peer where it registers instead. Only the leader gets past the
IsLeader check in KeepConnected, and a joining master's client lands
there, so that is the moment it joins. The broadcast OnPeerUpdate rides
on is not enough on its own: it only reaches masters already connected,
which is why a leader that came up first missed both newcomers.
RaftAddServer grew a goraft branch on the way, so cluster.raft.add stops
silently doing nothing on the default raft, and RaftRemoveServer with it.
Bootstrapping is now one call for both implementations, made only after
the peers confirm nobody has a leader, and retried until this master is
in rather than checked once and dropped.
* master: do not evict a peer that is still in -peers
The hashicorp leader drops a master from the raft configuration as soon
as it stops answering pings. A master that is merely restarting answers
nothing, so an ordinary bounce shrinks the quorum behind the operator's
back — and then races its own return: the master comes back, registers,
gets re-admitted, and the eviction lands after it.
A randomized start/stop walk lands on it. Two of three masters running,
the leader evicts the one that just went down, the restart re-adds it,
the removal commits late and takes the leader's own leadership with it.
What is left is a two-server configuration whose other half is down, and
a running master that nobody will ask for a vote — no quorum, no way
back until the third master returns.
-peers is what declares membership. updatePeers already reconciles the
configuration against it on every leadership change, and an operator who
really means to drop a master can say so with cluster.raft.remove, so
keep the eviction for masters that are no longer listed at all.
* test: bounce masters at random and hold the election to it
Twelve rounds of stopping or starting a random master, on both raft
implementations, checking the two things an election must never get
wrong: two masters claiming leadership at once, and a quorum that comes
back without agreeing on one. The cluster's identity has to survive the
whole walk, since a master that re-mints a TopologyId is the split brain
SetTopologyId kills its peers over. The seed is random and logged, so a
failure names the walk that reproduces it.
Below a quorum the walk moves straight on. A master that has lost its
quorum cannot commit anything, and goraft only checks whether it still
has one on an election-timeout ticker, after its peers have been quiet
for a full timeout — measured taking over 30 seconds to step down. That
direction belongs to TestTwoMastersDownAndRestart, which was giving it
ten seconds and would have started failing on a slower machine; it now
waits on that behaviour explicitly rather than sleeping twice and hoping.
WaitForTopologyId returns the id it waited for. Reading it separately
raced the leader applying the raft entry that carries it, which shows up
as an empty id right after an election rather than as a wrong one.
* fix(filer): stop logging a held aggregated read as an error
An aggregated subscriber may not read past the peers' low-watermark, and
it stops at the first entry beyond it by returning a sentinel from the
read callback. LoopProcessLogData logs every callback error, so on a
cluster that keeps writing - where there is almost always an entry newer
than the watermark - every read wrote an ERROR line naming the entry it
stopped at, thousands per minute per filer.
Mark the stop as control flow: an error wrapping StopReadingError is
handed back to the caller unlogged, and the held-read sentinel wraps it.
* fix(filer): release an aggregated watermark hold on peer progress
A held read waited on the aggregated buffer's data channel, which the
next write signalled - but a write cannot release a hold, only a peer
reporting further progress can. On a cluster that keeps writing the loop
therefore re-ran a whole pass per arriving event, log file listing and
all, and held again on the same entry every time.
Signal held readers from the meta aggregator instead, whenever a
low-watermark rises: a peer reporting, or one dropped past its removal
grace. The retry interval stays as the backstop for what no watermark
covers. Count the holds so a parked subscriber stays visible.
* fix(filer): floor how often an aggregated watermark hold releases
Peers advance their delivery watermark on every event they stream, so
releasing a hold on every advance is the same pass-per-event storm as
releasing on every write, just without the log lines - and each pass
lists a day of log files.
Floor the release at 20ms. Advances inside the floor collapse into one
release, which then delivers everything they covered.
* fix(filer): pace a peer's delivery claim by what its subscribers hold at
A filer's local metadata stream carries an idle heartbeat to its peer
aggregators, and each peer turns it into that filer's delivery
low-watermark. Aggregated subscribers hold at the minimum across peers,
so a filer quiet enough to fall back on the heartbeat parked every
subscriber in the cluster up to a keepalive interval - 5 seconds -
behind live writes. With nine filers, most of them quiet at any moment,
the minimum sat there permanently.
Pace that heartbeat at 200ms once the filer has peers. It stays a
keepalive, at the keepalive interval, for a filer with none.
* fix(filer): wake each aggregated hold on its own watermark
A persisted-log read is held by what the peers have flushed, an
in-memory read by what they have delivered, but both parked on one
channel closed whenever either minimum rose. Peers advance their
delivery watermark on every event they stream, so a flush-held reader
woke at the coalescing floor to re-list a day of log files and park
again on the same entry - the storm this set out to fix, in the one
place asymmetric peer progress still reached.
Signal the two separately and park each read on the one that bounds it.
* lance worker: share the integration tests' scaffolding
The recorder that keeps what a handler sent, the config builder and the
storage-option fallback all lived inside compaction.rs, so a second test
binary would have had to copy them. They move to tests/common.
The fallback now reads AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and
AWS_ENDPOINT_URL from the environment, defaulting to what it used before.
A harness can then point these tests at a gateway that checks what it is
given rather than one that accepts anything.
* lance worker: maintain one named table, for a harness to drive
Compacts and cleans up whatever WEED_LANCE_TABLE names, through the
handlers' own detect-then-execute path: a proposal the worker would not
have made is not one worth running.
The existing tests seed the tables they check. This one deliberately does
not, so a harness that has already written a table and knows what is in it
can have the real handlers maintain it and then read it back.
* test: take a table through its whole life, for Iceberg and Lance
Created in the catalog, filled by a real client, maintained by the worker,
read again, dropped. The step nothing was checking is the read after
maintenance: compaction once rewrote every dictionary-encoded column onto
a single value and shipped, because the maintenance tests were thorough
about sequence numbers, manifest entries and metadata versions and none of
them opened the parquet file the worker had just written.
So the assertion is a tally - row count, the cardinality of each
dictionary-encoded column, and an md5 over whole rows - taken before
maintenance and again after, required to be equal. The cardinalities name
the failure that happened; the digest catches a rewrite that keeps every
column's cardinality and hands the values to the wrong rows. A compaction
that merged nothing fails rather than passes, or the read afterwards is
checking a file the worker never wrote.
The Iceberg half runs two clients. DuckDB is the one the corruption was
reported against and the only one here that writes the deprecated
PLAIN_DICTIONARY encoding, which parquet-go normalizes away on write, so a
Go writer cannot produce it. PyIceberg writes the modern spelling. Pinning
parquet-go back to v0.30.1 fails the DuckDB half and passes the PyIceberg
one, which is why both are here.
Lance maintenance lives in the Rust worker, so it runs there where cargo
is installed and through the two lance calls those handlers wrap where it
is not. WEED_LANCE_MAINTENANCE picks one instead of letting the test guess.
* ci: run the table lifecycle tests
CI maintains the Lance table through the lance library rather than the
worker: a cold build of the lance crate costs more than the glue it would
be checking, and the worker's own tests cover its handlers.
The suite drives the Iceberg maintenance worker, so a change to it now
triggers this workflow too.
* test: let the lifecycle harness fail instead of skipping
Setup failures all exited zero, so a cluster that would not come up, or a
port allocation that lost, reported a green run for code nothing had
executed. That is the failure mode this whole directory exists to close,
and it was in the harness itself.
Only a checkout without a weed binary skips now, and it runs the tests so
each one says so rather than the package quietly passing. Everything else
fails.
The filer existence probe gets a deadline while I am here: it ran without
one, so an unresponsive filer would hang the suite past every timeout the
clients have.
* test: make the lifecycle checks check what they claim to
Three of them could pass without having looked.
The DuckDB skip matched "syntax error", "not implemented" and "Failed to
load" anywhere in the output, in any phase. A parse error in the SQL this
test generates, or a refusal from our own catalog, would have taken the
only coverage of the PLAIN_DICTIONARY encoding out of CI and left it
green. It now matches the extension failing to install, and only in the
phase that installs it. Everything past LOAD is ours and fails.
The digests covered id, category and value. Compaction rewrites the whole
row, so a defect confined to ts, or to a Lance vector, changed nothing
either side of maintenance. Every persisted column goes in now, ts as
microseconds so no timezone sits between the two runs.
The Lance drop check caught every exception as proof the dataset was
gone. pylance turns credential and transport failures into the same
ValueError, so it only accepts the message that means not found.
* docs: say up front which maintenance path the Lance half takes
The opening summary said the worker maintains both tables. It maintains
the Iceberg one always and the Lance one only where cargo is installed,
which is not what CI does.
* volume: start a volume's batch write worker on first use
Mounting a volume started a goroutine parked on a 128-slot channel, plus
the 128-entry batch slice it had already allocated. That is around 6.7KB
per volume the server pays whether or not the volume ever takes a write:
7231 bytes per mounted volume, of which 4101 is goroutine stack.
Only a write that asks for fsync ever reaches the worker, and a
remote-tiered or read-only volume never can. Create the channel and its
goroutine on the first such request instead, and let a write arriving
after Destroy fall back to the inline path rather than queue onto a
worker that has gone.
Measured over 20000 mounted volumes: 7231 -> 1269 bytes each.
* volume: update the heartbeat report state in place
Every heartbeat built a second map of what it was about to tell the
master, holding a freshly allocated short information message per volume,
then swapped it in over the old one -- and computed departures through a
third map of the live volume ids. A server holding 2M volumes rebuilt all
three every VolumePulsePeriod for a report that usually says nothing.
Number the heartbeats instead and mark the entry already held with the
pass that found the copy, so a quiet volume costs a map lookup and no
allocation. Departures are the entries a pass did not mark; the live-id
map is now built only when there are some, sized to them.
Measured over 10000 mounted volumes: 436 -> 196 bytes allocated per
volume per heartbeat.
* volume: fill one volume information message per heartbeat, not per volume
The heartbeat built a message for every volume held so it could hash it,
then dropped all but the few it had something to say about. At 2M volumes
that is 2M messages allocated every VolumePulsePeriod to send almost none
of them.
Fill a message the caller supplies instead, and replace it only when the
heartbeat keeps it, so a server with nothing to report fills the same one
all the way through.
Measured over 10000 mounted volumes: 196 -> 4 bytes allocated per volume
per heartbeat, and a heartbeat runs a third faster.
* volume: drop the per-volume trace from the heartbeat's status read
glog.V(4).Infof evaluates its arguments whether or not the verbosity is
on, so every volume boxed its id into a fresh interface slice on every
heartbeat: 759 of the 773 allocations a 1000-volume heartbeat made, for a
line that at this scale would print millions of unreadable rows.
Measured over 1000 mounted volumes: 4776 -> 1792 bytes and 759 -> 14
allocations per heartbeat, which no longer grows with the volume count.
* seaweed-volume: mirror the in-place heartbeat report state
Same change as the Go volume server: number the heartbeats and mark the
entry already held with the pass that found the copy, instead of building
a second map of hashes and swapping it in.
The volume snapshot must leave the reporting state as it found it, so it
keeps asking through changed() while a real heartbeat marks through
record().
* volume: refuse writes to a closed volume instead of dereferencing nil
Close and Destroy leave the needle map and data backend nil, but a caller
that already holds the volume can still reach the write path, where both
are used unguarded: a write racing a volume deletion took the server down.
syncDelete has always checked; syncWrite and the batch worker had not.
Reachable before this series and now also from the inline fallback a
durable write takes when the worker has gone.
* seaweed-volume: guard the report state with one mutex, as Go does
The full-list flag and the generation that answers it have to move
together. Split across separate atomics they cannot: a request landing
between begin's two reads returns full == false with the generation it
just raised, and one landing between commit's read and its clear is
marked answered by a heartbeat that carried no list. Either way the
resend is dropped.
Neither is reachable today -- every caller reaches this through the
store's RwLock, the flag setters under a read lock and the heartbeat
build under a write lock, so they cannot interleave. The type should not
depend on that being true two files away, and Go holds a single mutex
over exactly these fields.
* test: build the servers under test to match the harness's offset size
The mixed Go/Rust suites run both servers against one dataset, so both
have to agree on the offset width. They did not: the harness built Go
with no tags, 4-byte offsets, while the Rust crate defaults to its 5bytes
feature, and the Rust server then refused the .vif the Go server had just
written -- "bytes_offset mismatch: found 4, expected 5".
Build each side to match the offset size the test binary itself was
compiled with, so a plain `go test` and one with -tags 5BytesOffset both
get a matched pair.
* feat(s3): serve from remote on local read failure
When a locally-cached chunk of a remote-mounted object becomes unreadable
(volume server down/restarting, or an evicted needle 404ing under
retry-backoff), fall back to serving the object from its mounted remote
instead of erroring. A bounded pre-flight probe makes a stuck volume trip
the timeout rather than stalling the request.
Gated by -localReadFallbackToRemote (default off) with
-localReadFallbackTimeout (2s default), so existing deployments are
unaffected until they opt in.
* fix(s3): register local-read-fallback flags for mini/server/filer
The mini, server and filer launchers build S3Options directly and only
populate the flag pointers they register. Without registering the two new
flags there, startS3Server dereferenced nil pointers and crashed at boot,
failing every integration suite that runs `weed mini`.
* fix(s3): treat a zero-byte probe read as unreadable
A read that returns no byte -- whether it reports io.EOF or no error at all
-- means the offset is not locally readable, so the probe must fall back to
the remote rather than proceeding to stream a truncated response. Only a
returned byte (including the object's final byte with a trailing io.EOF)
counts as readable.
* s3: finish a mid-stream local read failure from the remote mount
The pre-flight probe only proves the byte at the requested offset readable.
A multi-chunk object can still lose a later chunk after the 200/206 and its
Content-Length are committed, which truncated the body with no fallback.
Resume from the mounted remote at the byte the local copy stopped at, so the
response still carries the declared length. A short local read that surfaces
as a clean EOF is treated the same way instead of silently truncating.
* s3: fall back to the remote mount without a CLI switch
Serving a remote-mounted object from its authoritative remote is what the
read should have done all along -- the alternative is a 500 on an object the
cluster can still reach -- so make it the behavior instead of two new flags,
with the probe bounded by a constant.
* s3: trim the comments on the fallback path
* filer: report only the contiguous prefix when a parallel chunk read fails
The parallel branch of doReadAt fans the chunk reads straight into their own
windows of the output buffer, then sums every task's bytesRead. A middle chunk
failing while a later one succeeds therefore returned a length covering a hole
the reader never filled, handing the caller zeros in the middle of otherwise
valid data.
* s3: only splice the remote onto a local prefix while it is the cached generation
Eligibility establishes a size match, not byte identity: a remote key
overwritten with same-size content between the cache fill and the fallback
would have finished the response with bytes from a second generation, under
the first one's ETag. Stat the remote before resuming and keep the local
error when it no longer matches -- a truncated body is a visible failure,
a spliced one is not.
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* deps: upgrade parquet-go to v0.32.0
Iceberg compaction writes the merged file with the schema of its first
input, encodings included. parquet-go before v0.31.0 took the deprecated
PLAIN_DICTIONARY encoding that DuckDB writes at face value and encoded
those pages as plain int32 indices, but the spec gives PLAIN_DICTIONARY
the same bit-width-prefixed RLE layout as RLE_DICTIONARY. Every
dictionary-encoded column in a compacted file then decoded onto a single
dictionary entry, and anything past one page failed to decode at all.
* iceberg: cover compaction of dictionary-encoded input
The fixture is a DuckDB-written file, so it carries the PLAIN_DICTIONARY
encoding a Go writer will not produce.
* iceberg: tally whole rows in the dictionary merge test
Counting each column on its own passes a merge that remaps names while
leaving their cardinality intact.
The toast and modal changes edited static/js/ without rerunning
gen_static_gz.go, so the embedded assets served to browsers still carry
the old scripts and TestStaticGzMirror fails on master.
* admin: raise nested modals above the ones already open
Bootstrap gives every modal and every backdrop the same z-index, so a
modal opened while another is showing paints behind it and its buttons
cannot be clicked. Viewing an access key secret and then copying a field
left the confirmation stuck behind the details modal with no way to
dismiss it.
Give each nested modal, and the backdrop Bootstrap creates for it, a
z-index above what is already on screen, and put back the scroll lock
that Bootstrap drops as soon as any one of them closes.
* admin: confirm clipboard copies with a toast
The access key details modal offers three copy buttons, and each one
raised a modal that had to be dismissed before the next copy. Confirm
with a toast instead, so the credentials stay in view and nothing has to
be clicked away.
* mini: state the format of a -tableBucket, do not infer it
A table bucket holds one format and that format decides which catalog can
serve it, but the flag only took names. The format came from
miniTableBucketFormat(): Iceberg whenever its port was up, Lance only when
it was not. So -tableBucket=vectors on a default mini quietly made an
ICEBERG bucket that the Lance namespace then refused every table in, and
the only way to get a Lance one was -s3.port.iceberg=0, which buys it by
deleting the other catalog. One flag, two meanings, decided by an unrelated
port.
Each entry is now name[:FORMAT], unsuffixed meaning ICEBERG as before:
weed mini -tableBucket=warehouse,vectors:LANCE
Both catalogs stay up and both buckets are reachable. A name whose format
has no endpoint here is skipped with a warning rather than created out of
reach, and the Iceberg-only S3_TABLE_BUCKET default-routing hint gets the
Iceberg names alone, without their suffixes.
* mini: do not reuse a table bucket that holds another format
CreateTableBucket answers BucketAlreadyExists on the name alone, so
-tableBucket=vectors against a bucket created as LANCE logged "already
exists" and moved on, and the Iceberg default-warehouse hint then pointed
at it. Every table create against that catalog fails with "table bucket
vectors holds LANCE tables", far from the flag that chose it.
ensureMiniTableBuckets now reads the format of a bucket it did not create,
warns when it is not the one asked for, and returns only the buckets that
hold what was requested. S3_TABLE_BUCKET is seeded from that list, so an
unprefixed Iceberg request falls back to its own default rather than
committing into a Lance bucket. A bucket predating declared formats reports
an empty one and still accepts either.
* mini: normalize S3_TABLE_BUCKET whichever way the spec arrived
The rewrite that keeps Lance names out of the Iceberg default warehouse only
ran when the flag supplied the spec. Set the variable directly, as the docker
quickstart does, and it reached the catalog untouched: S3_TABLE_BUCKET=
vectors:LANCE,warehouse made the unprefixed default the literal string
"vectors:LANCE", a bucket no lookup finds, while warehouse sat behind it.
The variable is both mini's input and the catalog's routing hint, so it is
now always rewritten from the buckets that came back holding Iceberg tables,
and unset when there are none rather than left pointing somewhere stale.
* mini: reuse a table bucket only when its format reads back
An ordinary S3 bucket wearing the name answers CreateTableBucket with the
same BucketAlreadyExists as a table bucket does, and the format lookup that
follows returned "" for a failed read exactly as it does for a bucket
predating declared formats. So -bucket=data -tableBucket=data reported
nothing and published data as the Iceberg default warehouse, where every
unprefixed request 404s on a bucket that is not a catalog.
The lookup now returns its error, and only a bucket that reads back as the
format asked for is reused. Anything else is left alone with a warning
naming why, rather than routed to and discovered later.
* test: drive the Lance namespace with LanceDB
The Iceberg catalog is checked against Spark, Trino, ClickHouse, Doris,
Dremio and RisingWave. The Lance one had only its own reference client,
which is the same thing as checking it against ourselves.
LanceDB connects with connect_namespace("rest", ...), which speaks the
routes this catalog implements, so the suite exercises the protocol rather
than our idea of it: list the catalog, open a table through it, read the
schema, run a vector search and a filtered scan, create a table, and read
the same dataset straight off its URI with no catalog at all.
table_names -> ['lancedb-p0guidmm$ml$embeddings']
open_table -> 64 rows
search -> [1, 0, 2]
create_table -> 4 rows, listed by the catalog
direct read without the catalog -> 64 rows
Seeding is pylance, because the namespace records where a table lives and
does not carry its data. That split is the design rather than a limit of
the test.
One interop note the test encodes: a gateway without STS vends
storage_options carrying an endpoint and a region but no credentials, and
LanceDB uses what the namespace vends on some paths. The container gets
credentials in its environment as well, which is what a deployment without
STS would do.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: pin the LanceDB client, and index before searching
Three from review.
The client's dependencies were unpinned, so an unrelated upstream release
could change what an old commit reproduces. Pinned to the versions this
suite was verified against; the client is as much the thing under test as
the server.
The search was called ANN and was not: without an index LanceDB scans.
The test now builds an IVF_PQ index over 1024 rows first, which is worth
more than the wording fix - an index writes into a directory of the table
that the S3 door has to admit, and that guard has refused a Lance
directory before. It builds, covers all 1024 rows, and searches.
The assertion moved with it. Demanding the exact nearest neighbour was
right for a brute-force scan and wrong for a quantized index, which
answered 0 as readily as 1; both are correct, so the check is now the
neighbourhood.
And the pushdown check accepted any failure. It now requires the refusal
to be the catalog's Unsupported and requires that nothing was left behind,
or, when the client falls back, that the table is complete.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
Nothing displayed the worker metrics, and the panels that look like they
did are about something else: Workers Connected, Worker Slots and Worker
Events in the Admin / Maintenance row read SeaweedFS_admin_*, which the
older maintenance queue feeds. A cluster running plugin workers - Go or
Rust - reads zero there while they are connected and busy.
A Plugin Workers row graphs what the workers themselves publish: how many
are connected, jobs and their failures, detection and proposal rates, job
duration, slot usage, stream events, and what the Lance jobs reclaimed.
The panel worth having is Objects Seen vs Skipped, since a sweep with
nothing to do and a sweep that could read nothing report the same number
of proposals.
Also a commented scrape target in the sample Prometheus config. It is 9328
rather than 9327: the sample compose already gives 9327 to the S3 gateway,
so the port the worker's own usage text suggests collides with it on a
single host.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* seaweed-worker: serve health, readiness and metrics
A Rust worker had no surface of its own. If it wedged, the only signals
were its stdout and whatever admin could infer from a stream that had gone
quiet; nothing could be scraped and nothing could be alerted on.
--metrics-port serves /health, /ready and /metrics, the same three the Go
worker serves under -metricsPort, so one scrape config covers workers in
either language. Off by default, loopback unless --metrics-ip says
otherwise, since the endpoint is unauthenticated. Names follow the Go
convention, SeaweedFS_worker_*.
The counters live in core and are raised where the stream already knows
what happened - connect, close, detection, execution, preview - so a
worker for another format gets them without writing any of this. Slots are
published from the heartbeat that already computes them, so a scrape and
the admin UI cannot disagree.
The pair worth having is objects_seen_total and objects_skipped_total. A
sweep that proposed nothing because there was nothing to do and a sweep
that proposed nothing because it could not read anything are the same
number of proposals; they are not the same event, and until now only a log
line told them apart.
The Lance jobs add what they reclaimed - fragments, rows brought under an
index, versions, bytes - on the same registry, so one endpoint serves both.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* seaweed-worker: fix the metrics address, the count, and a dead field
Three from review.
--metrics-ip ::1 failed at startup: the address was built by joining host
and port with a colon, and "::1:9327" is not an address. It is parsed as a
host and combined with SocketAddr::new now, so an IPv6 literal works, with
or without the brackets an operator will reasonably type after seeing one
in a URL.
proposals_total counted before the send rather than after, so a stream
that closed mid-sweep left the counter claiming proposals admin never
received.
And MeteredSender carried a Metrics clone and a job type it never read,
kept alive by two statements that existed only to silence the warning
about them. Everything is recorded by the caller, so both are gone.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
Every directory-state lookup went through path2inode, the map that holds one
full path per inode in the table, and then through dirStates. Directories now
carry their own path and are indexed by it directly.
There are orders of magnitude fewer directories than files, so this map stays
small whatever the mount holds, and it is what a file needs before it can stop
carrying a full path of its own: a child's path is its parent's plus its name.
No behavior change - the two indexes are asserted to agree.
* topology: refresh oversized mark on every heartbeat
The oversized flag on a volume location was only set when the volume was
registered (RegisterVolume). A volume that later grew past the size limit
kept its stale "not oversized" mark, so the heartbeat path
(ensureCorrectWritables) kept re-adding it to the writable list while
RecordAssign removed it on every assign - a writable/unwritable flip loop
that let writes continue past the limit and made vacuum race in-flight
writes.
Refresh the mark from each heartbeat's reported size in both heartbeat
paths (ApplyVolumeChanges and SyncDataNodeRegistration), mirroring what
RegisterVolume already did at registration time. A volume that grew past
the limit now stays unwritable, and one that shrank back clears the mark
and can recover.
* topology: order heartbeat writable correction after decay and honor cooldown
Review feedback (Greptile, CodeRabbit) on the oversized-mark refresh:
1. Greptile: clearing the oversized mark before EnsureCorrectWritables let
the delay-unaware helper re-add a just-compacted volume to writables,
bypassing capacityRecoveryDelay. ensureCorrectWritables now checks
fullSince and skips the re-add while the cooldown is pending, so a
volume removed for capacity only recovers through UpdateVolumeSize's
heartbeat recovery path.
2. CodeRabbit: in the full-heartbeat path the mark was refreshed after
the writable correction, so a newly oversized volume stayed writable
for an extra heartbeat cycle. The standalone changedVolumes loop is
merged into the volumeInfos loop and EnsureCorrectWritables now runs
after UpdateOversizedState + UpdateVolumeSize in both heartbeat paths,
using the freshly refreshed mark.
3. TestHandlingVolumeServerHeartbeat used a size (254320) that is past
the test's volumeSizeLimit (32768); it only passed because the stale
mark hid the oversized state. Sized down to 30000 to keep testing the
add/remove flow, and added TestEnsureCorrectWritablesHonorsRecoveryCooldown
covering the cooldown window and the recovery after it.
* topology: do not restore a still-crowded volume after the cooldown
Greptile review: after capacityRecoveryDelay elapses, ensureCorrectWritables
could restore a volume whose effective size is still past the crowded
threshold. UpdateVolumeSize refuses the recovery (effectiveSize > crowded
threshold -> setVolumeCrowded + return false), but the cooldown check in
ensureCorrectWritables only looked at fullSince, so once the delay passed
it re-added the volume even though capacity tracking still considers it
crowded.
Check the crowded mark before re-adding: a volume UpdateVolumeSize just
marked crowded must not be restored here, otherwise assignments resume
while the volume is still flagged for growth.
Adds TestEnsureCorrectWritablesDoesNotRestoreCrowdedVolume: effectiveSize
decays to 10500 (past the 9000 crowded threshold) after a report of 8000,
and ensureCorrectWritables keeps the volume unwritable past the cooldown.
* ci: trigger re-run of flaky FUSE jobs
* topology: gate the writable restore on the limit, not on crowded
A crowded volume is above the growth threshold, not full, and is normally
writable. Refusing to restore one locks it out for good: nothing writes to
a volume that is not writable, so its size can never fall back under the
threshold. Gate on the same size the assign path uses to remove it.
* topology: let only the heartbeat refresh set the oversized mark
Registration also set it, from whatever VolumeInfo it was handed. The
incremental path builds that from a short heartbeat message, which carries
no size, so every arrival announcement cleared the mark and handed the
volume back to the writable list until the next full report.
* topology: use the re-resolved layout after a dropped one is replaced
A layout dropped with its collection makes RegisterVolume refuse, and the
full heartbeat then re-registered against a fresh layout but kept applying
the size, oversized and writable updates to the dropped one.
---------
Co-authored-by: hzsunchao <hzsunchao@corp.netease.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* iceberg: skip tables the maintenance worker does not own
A Lance dataset registered through the Lance namespace's Iceberg REST adapter
arrives as an Iceberg table with a placeholder schema and table_type=lance, and
keeps its fragments under data/ - the same subdirectory the orphan cleaner
walks. Every fragment is unreferenced by the Iceberg metadata, so a maintenance
pass deletes the dataset. Views share the entry shape and were only skipped
because parsing their metadata happened to fail first.
Gate the scan and the execution path on the entry actually being an Iceberg
table. Maintenance is off by default, so this was latent rather than live.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* s3tables: let a table declare a format the catalog does not interpret
CreateTable accepted ICEBERG and nothing else. A Lance table has no metadata
file for the catalog to maintain - the entry records a name and the dataset
root, and the client owns everything under it - so accept LANCE, and carry the
declared format on the entry instead of hardcoding it back on the way out.
ListTables now reports format and metadataLocation, so listing a catalog that
holds both kinds takes one pass rather than a GetTable per row. AWS omits both
fields; adding them is additive.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* s3tables: move the in-memory filer into its own package
The Lance namespace tests need the same harness, and copying it would leave two
of them to keep in step. Extracted as it was, plus the two fidelity gaps that
only surface once a paginating caller uses it: ListEntries ignored
startFromFileName and limit, so a caller that paginates re-read the first page
until it hit its own cap and reported the same entry over and over, and
GetFilerConfiguration was missing, which CreateTableBucket needs to resolve the
buckets directory.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* lance: serve the Lance Namespace REST spec
A second catalog surface beside the Iceberg one, over the same table buckets:
the namespace and table metadata operations, the $-delimited identifier codec,
the spec's numeric error model, the directory-catalog marker files, and
storage_options vending through the STS path the Iceberg catalog already uses.
Listens on -port.lance, 9101 by default, and inherits ARNs, policies and tags
from the storage layer, so a Lance table needs no second permission model.
Identifiers map bucket / namespace / table onto the three levels Lance clients
already use, which is why there is no warehouse selector to invent. The data
plane needs Lance format support that does not exist in Go and answers with the
spec's Unsupported code rather than a bare 404.
Two things it deliberately will not do: create a table bucket as a side effect
of creating a namespace inside one, since a bucket carries its own policy and
lifecycle, and resolve an Iceberg table's location for a Lance client, which
would hand it a table another engine owns.
The design note this follows is in design-lance-catalog.md, including the
.lance directory suffix it proposed and this does not implement.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* mini: give the Lance port the same treatment as the Iceberg one
The flag was registered but nothing else knew about it, so mini would start the
server without reserving its port, waiting for it, or saying where it is. Adds
it to the startup service list, the conflict resolver, the gRPC allocator's
reserved set, the readiness wait, the stop reporting and the banner.
The admin server still takes only the Iceberg port, because there is no Lance
page for it to link to.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* lance: stop deregister and repoint from deleting the dataset
Deregistering preserves data by definition, and this did the opposite: the
catalog entry is the dataset directory, so DeleteTable took the files with it.
Registering over an existing name had the same shape, destroying the dataset
the name used to hold. Found by driving the running server rather than the
in-memory filer, where both looked like success because the table did stop
being listed.
Deregistering is now a state on the entry - the marker file hides it, and
declaring or registering the name again brings it back. Repointing a name at
another dataset is an UpdateTable against the version token, so neither dataset
loses files. Drop is left alone; it is the operation that does remove data.
The storage endpoint now falls back to the advertised -ip where the Iceberg
derivation gives up. An Iceberg client brings its own s3.endpoint and
advertising the wrong one hijacks it, but storage_options is the only place a
Lance client learns where the store is, and without it object_store quietly
talks to real AWS.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* s3tables: refuse to create a table over one of another format
Creating a table that already exists is idempotent, and that path returned the
existing table without looking at its format. A Lance declare over an Iceberg
table answered 200 and handed back a directory Iceberg owns, so the client
would write its dataset on top. The view check immediately above it already
guards the same class of collision.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* s3tables: let a table bucket hold a format other than Iceberg
The S3 door validated every object written into a table bucket against
Iceberg's file layout, so a Lance client could not write its dataset at all: it
got 403 on data/*.lance, on _versions/, and on the _transactions/ directory it
turned out to write as well. Table buckets were only neutral containers by
intention; in practice they were Iceberg-shaped and enforced as such.
The allowed set is now the union of what the supported formats write, because
the validator runs where the table's format is not in hand. Underscore-prefixed
directories are treated as belonging to the format, since enumerating them
means guessing at the next one - _transactions is exactly the one this missed -
and their contents are checked only for traversal. Iceberg writes none of them,
so it loses nothing. Marker files at the table root are admitted too, which the
namespace/table/dir/file shape had rejected as too shallow.
Describe also honours the request-body spellings of with_table_uri,
load_detailed_metadata and check_declared. The spec puts them in the query
string, but real clients send both.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* design: record what the implementation found
The table bucket being an Iceberg-shaped container, enforced at the S3 door,
was the premise this design never questioned and the one that had to change
before anything worked end to end.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* iceberg: prove the data loss the foreign-format guard prevents
The guard landed with a unit test for the predicate and nothing showing what it
saves. These seed what the Lance namespace's Iceberg REST adapter actually
leaves behind - an Iceberg table with a placeholder schema and table_type=lance
whose directory holds a Lance dataset - and assert both halves: orphan
collection does flag the dataset's fragments, because the Iceberg metadata
beside them references nothing, and the scan never reaches the table. An
ordinary Iceberg table in the same shape is still scanned, so the guard is not
just skipping everything.
Confirmed against a running gateway first: our Iceberg catalog accepts the
adapter's registration, and a real Lance client then writes a dataset into that
table's location.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* s3tablestest: make the in-memory filer safe to race against
Two gaps that only matter once a test drives concurrent writers, which is what
an exclusive create has to be tested with: the entry map had no lock, and
CreateEntry ignored O_EXCL entirely, so both writers of the same name would
have won and the test would have passed while proving nothing.
The BeforeUpdate hook runs before the lock is taken. Its whole purpose is to
land a competing write in a handler's read-to-write window, and that write
needs the lock the hook would otherwise be holding.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* lance: make the namespace an external manifest store
Lance commits a version by writing _versions/{v}.manifest with
put-if-not-exists. The S3 layer in front of this same filer evaluates
If-None-Match by looking the entry up and then writing without a precondition,
so two writers can both pass the check and one commit is lost. The filer itself
has the primitive: CreateEntry with o_excl.
Adds the four version operations a Lance client actually calls - create, list,
describe and batch-delete - recording one entry per version under
_lance_versions/, and advertises managed_versioning so the client routes its
commits here. Reserving a version is the exclusive create, so exactly one of
several racing writers wins and the rest rebase.
Off by default, behind -lance.managedVersioning. Turning it on moves where a
table's version history lives, and a reader that does not come through this
namespace no longer sees all of it; that is the operator's call, not a default.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* design: record what managed versioning does and does not reach
The first commit through a namespace-backed store works and is recorded the way
the protocol specifies. Later commits do not, because lance 4.0.0 refuses
put_if_exists on that path in its own code, so the feature is capped upstream
rather than here.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: integration tests for the Lance namespace
Everything this surface got wrong so far - a deregister that deleted the
dataset, an S3 door that refused every Lance file, a version reservation that
could not actually be exclusive - passed against an in-memory filer first. So
these run against a live gateway, and where the claim is about data they check
storage rather than visibility.
Five Go tests on the shared harness: namespace and table lifecycle including
that deregister keeps the bytes and drop removes them, that a Lance client
cannot resolve or declare over an Iceberg table, that a Lance dataset's files
get past the table-bucket layout guard while junk still does not, and that
eight writers racing for one version produce exactly one winner.
One Docker-gated test drives the real Lance client, which is the only way to
check that the location and storage_options the namespace vends are between
them enough to write and read a dataset. It overrides the endpoint with the
container's view of the same gateway, because the shared harness binds a
wildcard address and so vends none.
The harness gains a Lance port and turns managed versioning on; the flag
touches nothing outside that surface.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* s3tables: a directory with no namespace metadata is a missing namespace
Three callers resolved a namespace by reading its metadata attribute and each
tested only for a missing entry, so a directory that carried no metadata came
back as an internal error saying "attribute not found". Creating a table under
a namespace that does not exist answered 500.
Collapses the three copies into one helper that reports both conditions as
absent, which is what they are: a directory without namespace metadata is not a
namespace.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* iceberg: stop reporting storage-layer refusals as server faults
writeManagerError recognised a missing table bucket and sent everything else to
500, so a missing namespace, a duplicate name and a commit conflict all reached
the client as InternalServerError with nothing to act on. Creating a table in a
namespace that does not exist is the case that turned up: 500 where the spec
wants 404 NoSuchNamespaceException.
Maps the storage error types onto the exception names this package already
uses, and keeps the existing bucket message, which explains how to select a
table bucket.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* iceberg: skip a foreign-format table by name, not by failing to parse it
A table the namespace created as LANCE carries no Iceberg metadata, so the
worker skipped it only because the parse failed, and logged that as damaged
metadata. The catalog records the format on the entry and this never read it.
Reading it turns an accident into a decision, and separates a mixed catalog
from a corrupt one in the logs. The property check beside it still covers the
other shape: a real Iceberg table wearing table_type=lance, which is what the
Lance namespace's Iceberg REST adapter writes.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* design: answer whether a Lance table needs maintenance
It does, and index optimization has no Iceberg equivalent: rows written after
an index was built are not covered by it, so a vector search quietly misses
them. None of the three jobs can run in the Go worker, and there is no useful
subset, because deciding what an old version still references means parsing
Lance manifests. Version cleanup at least has an answer that needs nothing from
us - Lance can enable it on the dataset itself.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* design: the Lance maintenance worker is a plugin worker, in Rust
Framing it as a sidecar was wrong. plugin.proto already defines a
language-agnostic gRPC contract for external maintenance workers, and
"weed worker -admin=..." is the Go reference implementation of it from outside
the admin process. seaweed-volume already compiles protos out of weed/pb with
tonic_build, so a Lance worker is that build plus plugin.proto and the lance
crate.
Scheduling, retries, dedupe, progress and the admin settings page all come from
the protocol: a worker that answers RequestConfigSchema with a descriptor gets
its configuration form rendered without a line of Go.
The data plane is the part that genuinely does need a process answering HTTP,
and this had the two conflated.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* seaweed-worker: Rust plugin worker workspace, with Lance as the first one
plugin.proto is language-agnostic and the Rust toolchain was already in the
tree, so a Lance maintenance worker needs no new integration surface: core is
the contract and nothing else, and a worker crate beside it supplies handlers
and a binary. A second worker is a new member here rather than a fork of the
protocol, which is why this is seaweed-worker and not seaweed-lance-worker.
Verified against a running admin: it connects, is accepted, and admin prefetches
descriptors for lance_compact, lance_optimize_indices and lance_cleanup_versions,
so their settings pages render from the Rust side without a line of Go. The
stream stays up across heartbeats.
The job bodies are stubs that report failure. Doing the work means adding the
lance crate and opening the dataset, and claiming success before that would be
worse than saying so.
Two things running it caught that reading the proto did not: the admin address
has to be converted to the gRPC port the way pb.ServerToGrpcAddress does, or the
dial fails as an h2 frame error; and the generated field names differ from the
Go ones in several places, so JobCompleted carries success rather than a state
enum.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* lance worker: implement compaction
Detection lists tables from the namespace, opens each one, and proposes a job
for any with more fragments than the policy allows; opening a dataset reads its
manifest and not its data, so a sweep stays cheap. Execution re-resolves the
table rather than trusting what detection saw - it may have been repointed, and
the vended credentials expire - then compacts and reports the fragment counts
either side.
Verified against a live gateway: a twelve-fragment dataset became one fragment
with all twelve rows intact. The test drives the handler directly and skips
unless WEED_LANCE_NAMESPACE names a namespace, the way the Go integration tests
skip without Docker.
Running it turned up a gap the design had not: a gateway without STS vends no
credentials at all, so the worker could not open anything and detection quietly
proposed nothing. --access-key/--secret-key are the fallback, and whatever the
namespace vends still wins over them.
Two API assumptions did not survive contact either. Datasets open through
DatasetBuilder::with_storage_options, not ReadParams, and lance 10's
ObjectStoreParams has no storage_options field at all.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* lance worker: implement index optimization and version cleanup
Index optimization is the job with no Iceberg equivalent: rows appended after
an index was built are invisible to a search of it until this runs. Detection
reads num_unindexed_rows from each index's statistics and proposes a table once
more rows sit outside its indices than the budget allows; a table with no
indices is skipped, which is different from one whose indices have fallen
behind.
Cleanup applies a retention window, refusing rather than silently dropping a
tagged version, and leaving unverified files alone because they may belong to a
commit still in flight.
Both verified against a live gateway: 512 uncovered rows became 0, and a
fourteen-version table lost its old ones. Each test now seeds what it needs,
including building an IVF_PQ index and appending rows outside it. The first
version of these depended on state a script had left, so the second run found
the work already done and asserted nothing - a test that passes by doing
nothing is worse than no test.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* lance: answer an empty catalog with an empty list, not null
ListAllTables built its result from a nil slice, so a namespace holding no
tables answered {"tables":null} on a field the spec marks required. A generated
client may decode that differently from an empty list. Found running the
namespace on a dev box, where the catalog was empty.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* admin: give Lance maintenance its own scheduler lane
Lane assignment is a hardcoded map, so the three lance_* job types fell through
to the default lane. That lane serialises its work under the cluster admin lock
because volume management shares global state, which would queue a table's
compaction behind volume balancing for no reason - Iceberg has its own
lock-free lane for exactly this.
Adds the lane, maps the three job types to it, and puts it in the sidebar
beside Iceberg and Lifecycle. The lane routes were already generic, so only the
nav was hand-written.
The lane-coverage test spelled out the three known lanes, so a fourth failed
it. It now checks against AllLanes(), which is the property it was reaching for
and does not need editing next time.
Found by connecting the Rust worker to a real admin: it registered fine and its
job types were known, but they were filed under "default" and had no page.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* lance worker: log what detection saw
"Detection proposed nothing" and "the worker could not read the table" look
identical from the admin side, and the second is what a missing credential
produces. One line per table separates them.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* lance worker: fix a leaked heartbeat and a silent reconnect loop
spawn_heartbeat returned a handle to an empty task rather than the ticker it
had just spawned, so aborting it aborted nothing and every reconnect left
another heartbeat running against a dead channel.
A stream that admin closes cleanly is not an error, but reconnecting in silence
hides why. Two workers sharing an id evict each other forever and the log shows
nothing but a login every five seconds - which is exactly how this presented on
a dev box, and it took a look at the admin's own log to see it. The message now
names the id to check.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* lance: a namespace cannot be created without its parent
Storage keeps a namespace's parts flattened, so creating "a.b" with no "a"
was accepted and left an intermediate that only existed inside a name. Listing
derives child names by slicing those parts, so it reported "a", while describe
and exists on "a" both answered 404 - a client walking the tree got a 404 on
something the listing had just handed it.
The spec asks for NamespaceNotFound when the parent is missing, which is also
what keeps listing and describe telling the same story.
Namespaces created through the S3 Tables API still bypass this, so listing
keeps deriving intermediates rather than hiding whatever is already there.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* admin: say why a non-Iceberg table shows no schema
The table pages read Iceberg metadata for schema and snapshots, and a Lance
table has none, so both panels rendered "No schema available" - which reads as
an empty table rather than a table this page cannot describe. The dataset
behind the one that prompted this holds 1024 rows.
The format is already on the entry and shown two rows above, so the empty
states now use it: the catalog records where a LANCE table lives, not what is
in it.
Reading the schema for real needs Lance format code, which is the same wall as
the data plane.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* seaweed-worker: run rustfmt over the workspace
Committed the crates unformatted, so `cargo fmt --all --check` failed on
files nothing had touched since.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* plugin: let a worker report what it saw about an object
Admin cannot read a Lance table: it knows where the dataset lives and
nothing else, so the details page had a location and two empty panels.
The worker already opens every dataset during detection to decide whether
it needs compacting, so it knows the schema, the row count and the
fragment count at that moment. It just had no way to say so.
Add a WorkerObservations body to the worker stream. Admin caches the last
observation per object and serves it back, timestamped, for display;
nothing schedules from it. The Lance compaction sweep reports what it
opened, and the S3 Tables details page fills its schema panel from the
cache when it has no metadata of its own, badged with when the worker
looked and which worker it was.
Nothing about this is Lance-specific past the reporting side, which is
the point: any format admin cannot parse can describe itself the same way.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* design: record the observation channel
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* plugin: ask a worker for sample rows of a table admin cannot read
Browse Data reads an Iceberg table's Parquet files directly, so it shows
real rows. For a Lance table it showed "Table has no Iceberg metadata"
and an empty grid, because there is no Go Lance reader and never will be
one worth maintaining.
The worker has the reader. Add RequestObjectPreview / ObjectPreviewResponse
to the stream, mirroring the config-schema round trip that already exists,
and give the Rust worker a PreviewProvider that scans the dataset and
formats the rows with Arrow's own formatter, so a vector column reads as a
vector. Admin picks the worker from the observation store: whichever one
last described this table is the one that can read it.
Unlike an observation the rows are not cached. They are the table's data
rather than a description of it, and a copy sitting in admin would be both
stale and nobody's business. The page fetches on load, bounded at 200 rows
and a 15 second round trip, and drops the snapshot and data-file panels
that only mean something for Iceberg.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* design: record the preview channel
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: disable the lance listener when two gateways share a host
* test: keep AllocatePorts away from the lance default port
* s3tables: let a table bucket declare the format it holds
A bucket is a catalog, and a catalog serves one protocol. Format was
recorded per table, so nothing could answer "where do I point a client at
this bucket" without opening a table first, and an empty bucket had no
answer at all.
CreateTableBucket takes an optional format, stored with the rest of the
bucket metadata and returned by Get and List. Empty means ICEBERG, which
is what AWS S3 Tables serves and therefore what an SDK that has never
heard of the field means. CreateTable refuses a table of another format,
and CreateView refuses outright in a bucket that is not Iceberg, since a
view is Iceberg metadata.
Buckets that already exist carry no declaration and keep accepting
anything, so nothing is migrated and nothing that worked stops working.
The Lance namespace declares LANCE for the buckets it creates, which is
what stops one of them being described to a client as an Iceberg catalog.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* admin: take the Lance port the way it takes the Iceberg one
The UI cannot name the endpoint that serves a Lance bucket without it,
and every format-aware page below needs to.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* admin: show which format a table bucket holds
The bucket list printed an Iceberg endpoint for every bucket, including
ones holding Lance datasets, where that endpoint serves nothing. It was
the most visible place the UI assumed one format.
The list gains a Format column and its endpoint column follows the
bucket's declaration. The banner names both endpoints rather than
asserting everything is Iceberg, and says so only for the servers that
are actually running. Create Bucket picks a format with two cards rather
than a dropdown, since what matters is not the name but which clients can
read the result, and the endpoint under them updates as you choose so the
operator leaves the modal knowing where to point one.
A bucket from before the declaration existed shows "unset" in an outline
badge, explained on hover. It is a fact about the bucket's age, not a
fault, so nothing nags about it.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* admin: carry the bucket's format into the pages inside it
Namespaces and tables are reached through a bucket, so both now say which
catalog they belong to rather than making you go back up to find out. The
tables list gains a Format column and a Rows column filled from what a
worker last observed, since for a format admin cannot read that is the
only row count there is; a table nothing has looked at shows a dash, not
a zero.
Create Table stops offering a choice the bucket has already made: in a
declared bucket the format is fixed and says why, and only an undeclared
one still offers both. Before this the select had exactly one option,
hardcoded, which made a Lance table impossible to create from the UI at
all.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* admin: let the table page speak the table's own format
Partitions and Snapshot History are Iceberg's shape. Rendering them empty
for a Lance table reads as a fault; a Lance table has neither, and says
so by not showing them. In their place is a Versions panel, which is what
that format calls its history, carrying the worker's timestamp so it is
clear the numbers are a cached look rather than something read live.
The breadcrumb carries the format badge, so the page names what it is
looking at before you read a panel and wonder why it is empty.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* admin: show how to connect to either catalog, and group the two format workers
The client examples on the buckets page were Iceberg's alone, so the one
thing an operator wants after creating a Lance bucket - what to type to
reach it - was not written down anywhere in the UI. Both formats now get
a pair of snippets, and only for a server that is running.
In the Workers menu, Iceberg moves below Lifecycle so it sits next to
Lance: the two table-format workers together, the two cluster-wide ones
above them.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* shell: create a table bucket of either format
s3tables.bucket -create takes -format, so a Lance bucket can be made
without going through the UI. The integration harness passes it too: its
Lance tests were creating Iceberg buckets and getting away with it only
because nothing checked.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* design: record that a bucket declares its format
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* lance: drop managed versioning; the store already orders commits
The namespace offered itself as an external manifest store, so that a
commit could reserve a version through a real put-if-not-exists. That was
designed around a gateway that no longer exists: If-None-Match: * is
reduced to a filer WriteCondition and evaluated at the object's owner
under its per-path lock, or under the object write lock on the fallback
path. Sixteen writers racing one fresh key get a single 200 and fifteen
412s, every time.
Lance needs nothing else. commit_handler_from_url hands every s3:// dataset
a ConditionalPutCommitHandler, which puts with PutMode::Create, which
object_store sends as If-None-Match: *. So the feature solved a problem
this store does not have, while moving a table's version history out of
the dataset and into the catalog - and lance could not use it past the
first commit anyway, since its own namespace-backed store answers
"put_if_not_exists is not supported" to the second.
The version operations answer Unsupported with the rest, managed_versioning
is false, and the flag is gone. In place of the reserve-once test there is
one that races eight writers at the manifest key through S3, which is the
path a commit actually takes.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* lance worker: honour the version floor, the slot limits, and a shutdown
Five findings from review, all of them things the worker claimed to do and
did not.
The version floor was checked when a cleanup job was proposed and ignored
when it ran, so a table whose versions had aged past the retention window
in between could be taken below the count the operator asked to keep.
Execution now computes the floor itself and passes it as before_version;
CleanupPolicy ANDs its clauses, so a version has to be both too old and
below the floor to go. Both settings are clamped to the range the form
offers, since Duration::hours panics on a large enough value and a
negative min-versions wraps to a huge usize.
Admin's shutdown was answered by returning from the stream, which the
reconnect loop read as a healthy close and logged straight back in: the
worker could not be stopped. serve_once now says which of the two
happened.
The advertised concurrency limits bounded nothing - every request spawned
a task - and the heartbeat reported zero slots in use whatever was
running. Both now go through semaphores sized from the limits, with the
permits held for the life of the request and reported in the heartbeat.
A namespace call had no timeout, so a gateway that accepted the connection
and went quiet held a detection slot forever. And one table whose stats
could not be read failed the whole sweep, losing the proposals for every
table already scanned; it is now skipped and warned about, like a table
that cannot be opened.
The tests drove one shared catalog concurrently, which is why one of them
asserted "no proposals at all" and passed by luck. They now take a lock
and judge only their own tables.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* admin: fix the review findings on the format-aware pages
The endpoint hint in Create Bucket built its HTML by concatenating the
bucket name the operator is typing, so a name like <img onerror=...> ran
in the admin origin as they typed it. It is built from DOM nodes now.
A preview reply looked its channel up under the lock and then sent outside
it, which Shutdown can close in between: a Gosched in that gap panics with
"send on closed channel" every time. The send now happens under the lock.
Observations were looked up by path alone, so a table dropped and remade
in another format at the same path was described by the observation left
behind. Lookups now have to agree on the format.
Also: the Lance namespace caps a request body rather than reading whatever
arrives; the details action no longer says "Iceberg" over a Lance table;
mini stops advertising a catalog port when it is not running S3; a format
whose server this cluster does not run cannot be picked in the modal or
accepted by the API, since a bucket nothing can reach is not worth
creating; and the unused catalogPortFor helper is gone.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* lance worker: let the control stream use mTLS
The channel was hardcoded to http://, so off loopback the stream carried
preview rows and execution commands in the clear - and a cluster with grpc
TLS turned on would refuse the worker outright.
--tls-ca, --tls-cert and --tls-key take the same certificates the Go
worker reads from the [grpc.worker] section of security.toml, and must be
given together: a CA on its own would quietly mean one-way TLS, which a
mutual setup rejects anyway. Without them the stream stays plaintext,
which is what the Go worker also does when nothing is configured.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* lance: answer null properties rather than an empty map
The catalog does not keep a table's properties. Declare echoed the
request's back and describe answered {}, both of which claim they were
stored and are empty. Null says the catalog does not keep them, which is
what the spec distinguishes and what is true here.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* lance worker: test the slot accounting
The heartbeat reporting and the waiting are the two things the semaphores
are for, and neither is observable from outside without catching a sweep
mid-flight.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: fix the mixed-format catalog test, and name the binary it drives
The integration suite passed locally and failed in CI on
TestLanceRefusesIcebergTables. Both were right: CI builds the binary
first, my tree had one from the day before, so locally the test drove a
gateway with no format enforcement at all.
The test itself no longer holds as written. It made a bucket, put an
Iceberg table in it, and checked the Lance surface hid it - but a bucket
that declares LANCE now refuses the Iceberg table outright. The invariant
still matters from the other side, so it starts from an Iceberg bucket
instead: Lance must not describe or list a table whose format it does not
serve, and must refuse to declare one beside it.
The harness now prints which weed binary it is about to run and when that
was built. `make test` rebuilds first; a plain `go test` will happily
drive a weeks-old binary and report a pass for code it never ran, which is
exactly what happened here.
Also make the row-limit conversion in the preview request explicitly
bounded: CodeQL flagged the int-to-int32 conversion, and clamping by
reassignment beforehand is not a form it recognises.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* lance: prove concurrent commits are kept, and preselect the only format on offer
Two more from review.
The commit test asserted that exactly one writer wins the conditional PUT,
which is the mechanism, not the claim. The claim is that nothing is lost:
the losers see the conflict, rebase and commit again. So there is now a
test that has eight writers append to one dataset at once and counts the
rows afterwards - all eight batches survive. That is also the sequence
managed versioning could not finish, since its store refuses the second
commit outright.
And when Iceberg's endpoint is not running, the format picker offered two
options with neither selected, so Create Bucket submitted no format at
all, fell back to ICEBERG, and was refused by the guard added last round.
Lance is preselected when it is the only format this cluster serves.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* Clamp the remaining worker settings, and bootstrap buckets in a served format
Compaction and index optimization read their thresholds and cast straight
to usize and u64, so a negative arrives as an enormous number and turns
the threshold into "never": compaction and reindexing both go quiet with
nothing to say. The cleanup job was fixed last round; these are the same
bug. Clamped to the values that stay meaningful rather than to what the
form offers - zero uncovered rows is a real setting, meaning reindex as
soon as anything is not covered, so the floor there is zero and not the
form's thousand.
mini pre-creates the buckets named by -tableBucket, and did so without a
format, which now means Iceberg. Started with the Iceberg endpoint off
and the Lance one on, that left buckets nothing could reach and which
refused every Lance table. It takes the format from the endpoint that is
actually running, and creates nothing when neither is.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* s3: allow-unordered is a listing parameter, not an unimplemented subresource
The guard that stops a bucket GET with an unknown subresource from being
answered with a listing does not know about allow-unordered, so it answers
501 NotImplemented - to a parameter the listing handlers already read and
already validate against delimiter.
This is why test_bucket_list_unordered and test_bucket_listv2_unordered
fail in the Ceph s3-tests suite. They fail on master too; this is not a
Lance change and can be taken on its own.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
The guard that stops a bucket GET with an unknown subresource from being
answered with a listing does not know about allow-unordered, so it answers
501 NotImplemented - to a parameter the listing handlers already read and
already validate against delimiter.
This is why test_bucket_list_unordered and test_bucket_listv2_unordered
fail in the Ceph s3-tests suite. They fail on master too; this is not a
Lance change and can be taken on its own.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
Multipart parts stage under /buckets/<bucket>/.uploads/<id>/, so the filer
resolved filer.conf storage rules against that path when the gateway assigned
volumes for them. A rule scoped to a key prefix - fs.configure
-locationPrefix=/buckets/b/data/ -ttl=30d - then matched a small object but not
the parts of a large one, so an object whose entry carried the rule's TTL had
its bytes spread over TTL-less volumes.
Assign part chunks against the destination object's filer path instead, the way
the x-seaweedfs-destination header made the filer resolve it before the S3 write
path moved off the filer proxy. Covers PutObjectPart and both UploadPartCopy
paths. The part entry itself is still written under .uploads, so a read-only
rule there still rejects it.
The lifecycle XML Expiration.Days TTL keeps passing 0 for parts: that rule
targets the user-visible object key and would start its clock before
CompleteMultipartUpload.
* fix(s3): stamp the gateway's own ids on single-shot PutObject entries
putToFiler builds the entry in the gateway now instead of proxying a PUT to
the filer, and it hardcoded Uid/Gid 0 while every sibling write path stamps
filer_pb.OS_UID/OS_GID. On a non-root deployment that leaves single-shot PUTs
and multipart parts owned by root while directories and completed multipart
objects keep the real ids, so a mount reader can list the tree but gets EACCES
on every open once objects are not world-readable.
* fix(s3): stamp mode and ownership on copy destinations
CopyObject and UploadPartCopy build the destination attributes themselves and
then assign them over the entry filer_pb.MkFile just stamped, so the copy
landed with mode 0000 and uid/gid 0 - unreadable on a mount even by the filer's
own user. Build the destination with the same mode PutObject resolves for the
request and the gateway's own ids.
* fix(filer): watermark-bound aggregated metadata subscription against multi-source merge races
The aggregated metadata subscription (SubscribeMetadata) merges per-filer
sources that become readable at independent paces, but tracks its progress
with a single scalar cursor. Once the cursor passes a timestamp T, anything
a source materializes below T afterwards is silently skipped: a peer
recovering from a stall re-inserts its backlog late (late ring merge), and
a source's flush can land a log file, or a later chunk of the same file,
after a subscriber's disk pass listed the files (late persisted-log
landing). This is the residual documented in #10501.
Bound the subscriber's two read paths by what every source has provably
made visible, each with its own watermark:
- Delivery low-watermark -> in-memory reads. The meta aggregator tracks,
per subscribed peer (self included), the newest timestamp received on
that peer's stream - real events, or idle heartbeats (peer streams now
opt into ClientSupportsIdleHeartbeat). The aggregated ring is complete
up to the minimum across peers; in-memory reads hold at it.
- Flush low-watermark -> persisted-log reads. Each filer reports its local
log-buffer flush watermark on its stream: a new flushed_ts_ns response
field, carried on idle heartbeats and on periodic flush reports (gated
on ClientSupportsIdleHeartbeat). Disk passes freeze the minimum across
peers before listing the log files and hold at it; the day-boundary
cursor jump and the metadata-chunks ref listing are bounded the same
way, the latter at minute-file granularity.
- Held reads keep the cursor at the last entry actually delivered and
retry; the retry re-lists the log files, which is what picks up a
late-landing file. Both watermarks are relaxed by the settled horizon
(2 x LogFlushInterval) as a liveness escape, so a peer stalled beyond it
delays subscribers by at most the horizon instead of forever - any loss
that escape allows was unconditional before.
With reads held at the flush watermark, a disk advance below it is proven
complete on every peer's disk, so the unproven-crossing counter now only
counts crossings the horizon escape allowed past a stalled peer.
Live delivery on the aggregated stream may lag by up to the idle-heartbeat
interval when some peers are quiet; SubscribeLocalMetadata consumers are
unaffected.
* fix(filer): resume evicted aggregated readers from an original-space disk anchor
The aggregated ring rewrites out-of-order peer arrivals to its head, so a
subscriber tailing it advances its cursor in bumped (arrival) timestamps,
while persisted logs keep original timestamps. When a slow reader's unread
window is evicted (e.g. a peer backlog flooding in after a stall) and the
reader falls back to disk, resuming from the bumped cursor skips every
original-space entry below it that memory never delivered - reproduced as
a ~66% silent loss on a 3-filer cluster with one peer's stream frozen for
~70s while the subscriber lagged.
Track a disk anchor: the newest original-space position the stream is
proven complete through. Disk passes advance it directly; contiguous
memory reads advance it to the peers' delivery low-watermark observed
before the read (per-peer streams are ordered, so everything with an
original timestamp at or below that watermark had already arrived and was
delivered). A reader kicked off the ring resumes the disk pass from the
anchor instead of the bumped cursor - redelivering what memory already
sent is within the subscription's at-least-once contract, skipping what
it never sent is not.
* fix(filer): close review findings on the peer-watermark subscription bounds
Four correctness holes found in review, one generated-file cleanup:
- The flush-through claim could assert durability for events still on
their way into the buffer: an event is timestamped before notification
work that can block, and only then appended. Track stamped-but-unappended
events on the Filer (the stamp shares a lock with the reader, and appends
are bumped monotonically past the buffer head), and cap the reported
flush watermark just below the oldest in-flight stamp.
- Removing a peer deleted its watermark entries while its stream kept
running: its next signal recreated the deleted entry, which then pinned
the low-watermark forever once the stream died. Watermarks now advance
only for tracked peers, and peer removal cancels the subscription
context so the stream stops feeding the aggregated buffer promptly.
- The pipelined sender folded flush reports (TsNs 0 reads as far behind)
into batch Events tails, where the aggregator's nil-notification guard
dropped them - a busy backlog replay could starve the flush watermark
until the settled-horizon escape opened a loss window. Control messages
are now unbatchable on the sender, and the receiver also reads watermark
state off nested batch entries as belt and braces.
- A give-up skip's cursor was not anchored, so the next eviction rewind
undid the counted decision and re-entered the same park forever when the
evicted window carried bumped timestamps. The anchor now follows give-up
skips; an anchored cursor makes the rewind a no-op and keeps the gap
machinery's re-arm onto the retained window reachable.
- Regenerated-file churn from a different protoc-gen-go-vtproto version is
dropped: the vtproto file is upstream's, plus only the flushed_ts_ns
marshal/size/unmarshal cases in the same generator style.
New tests pin the in-flight floor, the no-resurrection rule for removed
peers, and that control messages are never nested in batches.
* fix(filer): keep a removed peer's watermarks through a grace period
Deleting a peer's watermark entries the moment the master removes it
reopened the loss the watermarks exist to prevent: a filer frozen or
partitioned long enough to miss master heartbeats is removed from the
cluster, its unflushed events still exist, and with its entries gone the
low-watermarks snap forward to the healthy peers - subscribers advance
past the absent peer's window and its late-landing log files are silently
skipped. Reproduced on a 3-filer cluster: freezing two filers for ~70s got
them removed ~28s in, and a catching-up subscriber lost their entire
overlapping window.
Removal now only marks the peer; its watermarks keep participating in the
low-watermarks for a grace period (2 x LogFlushInterval, matching the
subscribe loops' settled horizon, which already bounds a stale watermark's
influence meanwhile). A re-added peer clears the mark and continues its
values monotonically - the flap case costs nothing. A peer that stays gone
is dropped when the grace expires, so a decommission cannot pin the
low-watermarks, and a dropped peer's straggling signals cannot resurrect
its entry.
* fix(filer): cap delivery heartbeats by the in-flight floor; harden stamps
Second review pass on the watermark bounds:
- Idle heartbeats on the local stream claimed delivery-completeness
through "now" while an event could still sit stamped-but-unappended
behind blocking notification work. A peer aggregator turns that claim
into its delivery low-watermark, so it could advance (and anchor
credits with it) past an event that had not been streamed yet. The
heartbeat timestamp is now capped just below the oldest in-flight
stamp, like the flush claim already was.
- In-flight stamps are forced monotonic against the registry's own
history, so a wall-clock step backwards cannot slip a new stamp under
an already-sampled floor. The cross-goroutine ordering still shares
the meta log's global forward-clock assumption; the comments now say
so instead of overclaiming.
- Duplicate removal notifications no longer refresh a removed peer's
grace deadline: the first removal time wins, so a decommissioned peer
cannot sit in the watermark sets forever on repeated updates.
- A failed buffer append clears the event's in-flight stamp on purpose:
the event is dropped from the change stream entirely (a pre-existing
defect of the append path, loudly logged), and a watermark waiting for
it would pin this filer's claims forever. The comments now state the
decision instead of implying the failure cannot happen.
* docs(filer): tighten the watermark comments
Comment-only: compress the narrative comments added on this branch down
to their load-bearing invariants, and fix one stale sentence (peer
removal no longer deletes the watermark entries immediately). No code
changes.
* fix(filer): subscribe to the local filer before remote peers
Self's events reach the aggregated buffer only through the aggregator's
own subscription to it, but bootstrap only seeded the peers the master
already listed - and self's master registration races that listing, so
the watermark set could hold remote peers without self. Once the remotes
signalled, the low-watermarks would claim completeness for a stream that
was still missing a merge source, letting aggregated subscribers advance
past the local filer's events before its subscription started.
Seed self first, unconditionally: before that the watermark set is empty
(a documented safe state - reads hold at the settled horizon), and after
it the set can never be remotes-only. The later master update for self,
or a duplicate in the listed peers, is a no-op via the already-followed
check in OnPeerUpdate.
* fix(filer): fence watermark claims against wall-clock regression
Record issued heartbeat/flush claims in the in-flight registry and stamp
later events above them, so a backward clock step cannot land an event
under a watermark a peer has already advanced to.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(filer): re-check the buffer head after fencing heartbeat claims
An event appended between the caught-up check and the delivery claim
was covered by the claim but not yet sent on the stream. The claims
fence later stamps, so re-checking the head after them proves every
covered event was already sent before the heartbeat.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(filer): cross the aggregated ring's pre-subscription range only on proof
The eviction gate and the gap proofs read "nothing evicted yet" as "memory
holds everything after the cursor". That is false for the merge-fed
aggregated ring, which is born empty while every peer's history sits on
disk: before the ring's first real eviction, a subscriber whose cursor was
still below the bounded chunk pass's listing stop was served the ring's
earliest entry inclusively, silently skipping the withheld pre-restart
files - and the idle-wait callback credited the delivery low-watermark to
the disk anchor in the same disconnected state.
Mark everything at or below the subscriptions' start as evicted when the
aggregator is built, credit the anchor only once the run is connected to
the ring, and give the aggregated gap pass a real proof to cross the
marked boundary with: each disk pass's proven coverage (the peer flush
low-watermark capped by the pass's listing bound). An empty pass whose
proof reaches the eviction watermark crosses to it silently - no park, no
loss counter - so the mark costs a bounded catch-up delay instead of the
15-minute give-up.
* fix(filer): keep shipped chunk tails at or below the hold point
A log file spans past its named minute (window start plus up to a flush
interval), and chunk-mode clients apply a shipped file whole - so a file
tail past the hold point can become a persisted client checkpoint beyond
what every peer has proven, and a crash inside that window resumes past
another peer's late-but-in-contract flush. Stop the ref listing a minute
plus a flush interval below the hold; the withheld band is served by the
memory pass (ring retention far exceeds it) or by later passes as the
hold advances, so freshness is unchanged. A frozen peer flushing one
window that spans its whole freeze can still overshoot; that residual is
bounded by the freeze and needs a crash inside it.
* docs(filer): trim the review-fix comments
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* mount: discard a path-cache insert that raced a purge
The Windows adapter's walk resolves a component with a Lookup RPC and inserts
the result holding no lock, so a purge can land in between - and what the walk
just resolved is then the very name the purge removed. Anything opening the
old path concurrently with a rename repopulates the cache with the vacated
name, which the next stat is served from for up to a second. The release path
already guards its equivalent insert; the walk had nothing.
The cache counts purges now. A resolve snapshots the generation before its
lookups and insert discards the entry when any purge ran in between, parking
the reference in the graveyard so the in-flight caller keeps a valid inode
either way.
Seen once in CI as TestRenameOverExisting failing with 'source survived the
rename': every SeaweedFS layer is synchronous with the rename, but a
background open of the source - an antivirus scan of the just-written file
fits - can requalify the stale name through this window. The assertion also
reports what stat returned now, and whether it persisted, so a recurrence
indicts a specific layer instead of reading as a mystery.
* mount: cover the path-cache discard by key, and let a discard rest
Review follow-ups. The generation was global, so any purge between a walk's
snapshot and its insert discarded the entry whatever its name - and an open
retries resolve-then-steal only four times before failing with EIO, so
sustained unrelated churn could fail opens of untouched paths. Purges are
remembered by key now and only one that covers the inserted name discards it;
past the remembered window the insert is discarded without a check, which only
costs a retry.
A discard that itself tripped the sweep also handed its own reference straight
to forget while the walker was still using the inode. The graveyard holds two
generations now, so an appended reference always survives the sweep of the
call that appended it - which the displaced-entry and purge paths needed too.
Also restores the original path-cache test suite this branch had overwritten
instead of extended, and rewords the semantics-test failure so it no longer
claims the source survived when stat returned a transient error.
* mount: take an open's reference directly instead of stealing it back
resolveAndSteal cached the final component only to steal it back, so an open
depended on that insert surviving whatever purges raced it - four attempts and
then EIO. The keyed purge window narrowed how often an insert is discarded,
but past the window the discard is blind again, so the cliff had only moved.
A cached entry is still stolen; anything else is now looked up directly, with
the caller owning the reference from the start. No retry loop, and no way for
churn - covered, unrelated or overflowing the window - to fail an open.
Also covers the whole-cache purge: purge of the root with prefix set clears
every entry, but the covers check tested for a '/'-prefixed key that a
normalised key never has, so it covered no in-flight insert at all.
* mount: leave the target alone when a move has no source
MovePath cleared whatever sat at the target before it checked that the source
was still there, so a move it then declined to make had already taken the
target's mapping apart. The same rename reaching the table twice - once for an
open handle, once for the invalidation behind it - was enough to leave the
moved inode with no path at all.
* mount: move the inode table when a rename arrives from the cluster
A rename made by another client reaches this mount only as a metadata event,
and the only table update on that path sat inside the open-file-handle branch.
Every other inode the kernel still addresses by nodeid kept resolving to its
pre-rename path, so the next operation on it went to a path the filer no
longer has.
Move the entry for every rename invalidation. The filer sends one event per
moved entry, so a renamed directory's children follow their parent without a
descendant walk.
* mount: move the inode table exactly once per rename event
Making MovePath return early on a missing source was the wrong half to fix.
The source is also missing when the rename came from a client that never
visited it, and there the destination really was replaced and has to be
unlinked - the early return kept the name resolving to a file the rename
destroyed, so a dirty handle on it could still flush over what took its place.
The two cases are indistinguishable from inside MovePath, so leave it alone and
stop calling it twice: invalidateOpenFileHandle reports whether it moved, and
the handler moves only when it did not. Both paths mark a replaced file's
handle deleted, which the no-handle path previously did not do at all.
* mount: leave a rename alone unless the source is still ours to move
Two ways the fallback move could act on state it did not own. A handle whose
version guard skipped the event never reached RememberPath, so moving the table
under it left the handle flushing to the pre-rename path; the handle path owns
its inode's rename, so the fallback now runs only for an inode without one.
And the subscription can redeliver a rename once it falls out of the 4096-entry
dedup ring. A replay found no source and a live target, unlinked the mapping
the first delivery had just made, and marked the moved file's handle deleted -
worse than the stale path this set out to fix. Only a source still in the table
is moved now.
That gives up unlinking a destination the rename replaced when the source was
never visited here, which is where this started. It is what the mount already
did before this branch, and it is the safer of the two: retaining a stale name
costs a wrong lookup, while unlinking the wrong one costs a file's dirty data.
* mount: decide a rename move inside the table's lock
The source-presence check sat outside MovePath, so two invalidations for one
rename could both see the source and the loser would unlink what the winner had
just placed - the same damage the check was added to prevent. MovePath makes
the decision under its own lock now and reports that nothing moved.
The handle a rename destroyed is also marked from the caller rather than from
inside invalidateOpenFileHandle, which was setting isDeleted bare on a second
handle while holding the first one's lock. markHandleDeleted already takes the
lock the flush reads that flag under, and marking from the caller keeps it to
one handle lock at a time.
The invalidation test harness wires onEntryInvalidation now, the way the mount
does, rather than reaching past it.
* mount: apply a rename ahead of the handle's version fence
The fence exists so an old event cannot roll a handle's entry back to stale
content. A rename carries no content: it says the name the inode answered to is
gone. Skipping one on the strength of the fence left the inode and the handle
both pointing at a name the filer had vacated, and no fallback ran either,
since a handle owns its inode's rename.
Applied before the fence now, and only when MovePath reports the source was
still ours to move - which is what keeps a replayed rename from remembering a
path the handle has already moved past.
* Add monitoring label to filer servicemonitor
* helm: label the headless filer service, not the client one
The ServiceMonitor takes its job label from the service name, and the
bundled dashboard queries job="seaweedfs-filer". Selecting the client
service would have renamed the job and blanked those panels. The
headless service also publishes not-ready addresses, so it keeps
reporting while a filer is starting up or shutting down.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Four workflows ran apt to install libfuse3-dev before every FUSE job. Nothing
needs it: go-fuse implements the protocol in pure Go, no cgo in the tree
references fuse, and the package does not even provide the fusermount3 the
mount actually execs - fuse3 does, and it is already on the runner image, which
is why the setuid-repair step finds it.
So the step downloaded a dev package to build against headers no compiler ever
opened, and it is the step that has been hanging whenever the Ubuntu mirror
goes slow. Configuring /etc/fuse.conf is all that is left.
* rust: stop writing when a durable write's index flush fails
A durable write flushes the .dat, publishes the needle map row, then
flushes the .idx. If that last flush failed we returned the error and
carried on: the row stayed live, the volume stayed writable, and the
handler answered 500 without replicating. The primary then served a needle
its replicas never saw, for a write the client was told had failed - and
if the unflushed row was lost on restart, the durable .dat tail took the
volume read only anyway.
Taking the row back out is not an option: it means undoing published state
on a disk that is already failing, and a truncate afterwards would leave
an .idx row pointing past the end. So the volume stops taking writes
instead, the same as when the truncate after a failed .dat flush cannot be
done. Nothing more gets appended past a record whose index is in doubt,
and the master routes writes elsewhere once the volume heartbeats read
only. The divergence against the replicas is still there, but it is
bounded and it is visible.
A failed nm.put after the .dat is down leaves the same durable but
unindexed record, so it takes the same route.
* rust: drop the import the rollback removal left behind
NeedleValue came in with rollback_unflushed_write, which went away when
the durable path moved to flushing before it publishes. Nothing has used
the type since.
* rust: mark the test-only heartbeat helper as such
collect_heartbeat has only ever been called from the tests - the send loop
uses collect_heartbeat_with_snapshot, which it wraps - so a lib build
rightly called it dead code.
* rust: flush the index on a durable write that dedups
A durable write matching content already in the volume flushed the .dat
and returned before reaching the index flush. So a fsync=true write that
deduped against an earlier non-durable one was acked with the row that
indexes it still in the page cache - the same false promise the index
flush exists to rule out, and the same read-only volume on restart if the
row is lost.
The dedup path now flushes both files, and the quarantine on a failed
index flush moved into flush_idx so it applies wherever the flush is
reached rather than only at the one call site that had it inline.
The e2e workflow sets defaults.run.working-directory: docker, so the call I
added resolved to docker/docker/apt-install and every FUSE Mount run has
failed with 'sudo: docker/apt-install: command not found' since it merged.
Every workflow that builds the Rust volume server first installed protoc
from a package manager - twelve steps across apt, brew and choco. That is
37s per job on a good day, and this week archive.ubuntu.com stalled long
enough for four jobs to burn their whole timeout without reaching a build.
protoc-bin-vendored ships the compiler as a build-dependency, so it now
arrives through the cargo registry the workflows already cache and there
is nothing left to install. cargo build works on a machine with no protoc
at all, which is worth as much locally as it is in CI.
It also pins the version. The apt protoc on ubuntu-22.04 is 3.12, old
enough to reject proto3 optional, which is why build.rs passes
--experimental_allow_proto3_optional; the vendored one is 31.1. The flag
stays, since it costs nothing and keeps a build against an older PROTOC
working, and an explicit PROTOC still overrides the vendored binary for
packagers who supply their own.
The e2e job overwrote the runner's sources.list with two azure-only lines and
installed fuse from it, so the same mirror outage that took out the image
builds failed the step outright - this time on the runner rather than inside
the container, where the image-side fallback cannot reach.
Install through the same helper, and widen its rewrite to match any archive
host so it works whether the pristine list came from the base image
(archive.ubuntu.com) or from a CI runner (azure.archive.ubuntu.com). Keeping
the runner's original list also restores the security and backports pockets,
which the hand-written two-line replacement dropped.
Verified against the outage itself: with the pristine list pointed at Azure,
the build logged the skip after Azure timed out for real and installed from
archive.ubuntu.com.
The e2e image pointed both archive and security at azure.archive.ubuntu.com and
nothing else, and the samba and pjdfstest images inherit that list. When Azure
is unreachable the build has nowhere to go: Acquire::Retries just retries a dead
host, every package fails, and apt exits 100 before a single test runs. Two
different workflows lost runs to it tonight.
Install through a helper that starts from the pristine sources.list each time
and walks a list of mirrors, so Azure stays the preferred one - the reason it
was pinned in the first place - without being the only one.
Verified both paths against a real build: the normal one installs from Azure,
and with the first entry pointed at an unroutable host the fallback logs the
skip and installs from archive.ubuntu.com.
* Give the local Unix socket gRPC transport room to breathe
Unix socket buffers default small and never autotune: 208KB on Linux, 8KB on
macOS. Once the buffer cannot absorb what gRPC's loopyWriter emits for the
in-flight streams the writer blocks on Write, and since v1.82.1 grpc-go counts
per-RPC bookkeeping toward its control-buffer throttle, so both peers stop
reading and the connection deadlocks for good. weed mini wedged at roughly 320
concurrent S3 PUTs with every filer RPC parked in waitOnHeader and no handler
running.
Force 8MB on both ends of the sockets we open. Best effort, since a kernel may
clamp it lower; that only lowers the concurrency this survives. TCP loopback
never hit this because its buffers start large and grow.
* Set the buffer on accepted connections too
Linux does not carry the listener's SO_SNDBUF onto sockets returned by accept,
so only the dialing half was getting the headroom: measured 8388608 on the
dialed side against the 212992 default on the accepted side. Wrap the listener
and re-apply per connection. macOS inherits either way, which is why this did
not show up locally.
InodeEntry held its one path in a slice, so every inode the kernel references
cost a 16-byte backing array and a second heap object on top of the 32-byte
entry. The extra links of a hard-linked file now hang off a pointer instead,
which keeps the struct in the same 32-byte size class and leaves the ordinary
single-link file with nothing to allocate.
Populating the table with 1M children: 237.5 -> 221.5 B/inode at 85-character
paths, 301.3 -> 285.6 at 148.
readdir built dirPath.Child(name) for every child while entry.FullPath was
already that exact string, from NewFullPath in the meta cache store or from
FromPbEntry on the read-through path. One allocation per entry, and on a wide
tree with long paths that is most of what a listing allocates.
BenchmarkReadDirectory/kernel_readdirplus over 200k entries: 2,039,656 ->
1,839,318 allocs/op, 174.5 -> 167.8 MB/op, 152.6 -> 135.1 ms/op.
The Rust volume server ignored the fsync parameter completely: nothing
parsed it, and write_volume_needle -> write_needle -> append_needle never
flushed. So a ?fsync=true upload was acked out of the page cache, and
since ReplicatedWrite forwards the parameter, a Go primary handing a
durable write to a Rust replica got the same empty promise.
The upload handler now reads fsync the way Go's r.FormValue does, off the
decoded query fields, and threads it down to the volume. A durable write
appends, flushes the .dat, publishes the needle map entry, then flushes
the .idx, and only then is it acked. Nothing points at bytes that are not
down yet, so a failed flush only has to take its own append back off the
end - the index never moved and the volume's counters never saw the
rejected write. If that truncate cannot be done the volume stops taking
writes, rather than letting a later append bury the rejected record
mid-file where the tail integrity check cannot see it.
The .idx flush is what keeps the ack honest: load() rebuilds the map from
.idx, so an acked write whose row was lost comes back as a .dat tail the
integrity check cannot account for, and the volume loads read only.
A dedup hit flushes too: there is nothing to append, but the write it
matched may have been non-durable, and the caller is asking for the
content to be on disk.
Batched writes carry the flag per request rather than one flush per
batch, so the write queue's module doc no longer claims otherwise.
A single Read on the response body can return the last bytes together
with io.EOF, so asserting NoError on it fails even though the body is
complete. Use io.ReadAll, like every other test in this package.
* volume: forward fsync=true to replicas in ReplicatedWrite
When a write request carries fsync=true, only the primary volume server
flushed to disk: the replica fan-out URL in ReplicatedWrite only carried
type/ttl/ts/cm, so replicas always wrote without fsync even when the
client explicitly requested a durable write.
Forward the fsync request parameter to the replica volume servers so a
durable write means every replica has flushed to disk, not just the
primary. Replicas without fsync are untouched (zero behavior change).
* storage: flush a durable write inline while stopping
The fsync flag on the write path really selects the async batch worker,
and it was switched off once the store is stopping. So a fsync=true write
landing during the pre-stop drain got acked without ever being flushed -
and now that ReplicatedWrite forwards fsync, that covers replicas too.
Flush it inline instead of queueing it. The drain keeps accepting writes,
which is the whole point of preStopSeconds, and the ack still means the
.dat is on disk. If the fsync fails, the append comes back off the .dat
and the needle map goes back to what it pointed at before, so nothing
resolves to an offset past the truncated end.
* storage: make the store's stopping flag atomic
SetStopping runs on the signal handler goroutine while the write and
vacuum paths read the flag, so every read of it was racy. Nothing about
the shutdown ordering changes; only the flag itself is now safe to read.
* topology: check the errors the replication test was dropping
The mock replica ignored its response write and the mock master ignored
whatever Serve returned, so a broken mock would have shown up as a
confusing timeout rather than a failure. Also drops the explicit listener
close: grpc.Server.Stop already closes the listener it was given.
---------
Co-authored-by: hzsunchao <hzsunchao@corp.netease.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3: answer GetBucketReplication, GetBucketWebsite and GetBucketNotificationConfiguration
None of the three had a route, so they reached the unconstrained ListObjectsV1
catch-all and a client asking for a bucket's replication config got 200 and a
<ListBucketResult> back. Replication and website report their configuration as
absent the way AWS does; notification returns the empty configuration AWS
returns for a bucket with no events wired up.
* s3: stop an unrouted bucket subresource from being answered with a listing
ListObjectsV1 is the catch-all GET on a bucket, so every subresource without a
route of its own - ?torrent today, whatever AWS adds next - came back 200 with a
<ListBucketResult>. A client that asked for a configuration and got a listing
either fails its XML decode in a way that reads like corruption, or worse,
tolerantly parses it. Refuse the request instead.
The allow-list is the ListObjects parameters rather than the subresources,
so a new one fails closed. Presigned URLs sign their credentials into the
query string, so X-Amz-* and the SigV2 trio have to stay listable.
* filer: add filer.options.disable_remote_storage_deletion for cache-only deletes
Deleting a filer entry under a remote.mount path also deletes the backing
object from the remote store (maybeDeleteFromRemote). Deployments that use a
remote mount as a read-through cache in front of an authoritative,
externally-managed object store cannot allow this: the filer typically holds
read-only credentials, so the remote delete fails and the entire delete
errors out; and even where it would succeed, it destroys data the filer does
not own.
Add filer.options.disable_remote_storage_deletion (default false, so existing
behaviour is unchanged). When enabled, maybeDeleteFromRemote is skipped for
both single-entry and recursive folder deletes: local metadata and cached
chunks are still removed, but the remote object is left intact.
* filer: assert local removal in cache-only recursive delete test
The recursive cache-only delete test only checked that no remote delete
happened; it did not verify the local child and directory entries were
removed. Add FindEntry assertions so a regression that skips local
recursive deletion is caught.
* filer: reload the remote mount mapping when /etc/remote changes
The mapping was only read at startup, so remote.unmount left the mount live
in the filer: the purge that follows the mapping delete then went to the
remote store and wiped every object under the mount.
Rebuild the rules trie and the conf map from scratch on each load, since
ptrie cannot drop a key, and swap them under a lock.
* filer: drop the filer-wide remote deletion switch
With the mapping reloaded on unmount, the purge no longer reaches the remote
store, so there is nothing left for the switch to protect against.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
The cached readdir trims the head of the handle's entry stream as the
client walks past it; the read-through path never did, so a directory
too large to cache -- the only kind that takes that path -- was held
whole in the handle for the length of the walk. Hoist the trim to cover
both paths.
* mount: re-resolve volume locations after a failed chunk read
NewChunkGroup passed nil as the ReaderCache's CacheInvalidator, so
retryFetchAfterCacheInvalidation was dead code on the FUSE read path. A
mount that cached a volume's locations while one server was down kept
retrying that server after it died, then returned EIO, even though the
master and filer both resolved the live replica. The S3 gateway already
passes its filerClient; do the same for the mount.
* test: FUSE integration tests for volume server failover
One mount appends while a second tails, and a volume server is killed,
started or restarted mid-stream against a 001-replicated cluster of three
volume servers. Automates the scenario matrix reported for Docker Swarm
mounts, including the large-file variant and a no-chaos control.
* test: report the filer's own view when append content mismatches
A mismatch between what the writer wrote and what the reader sees can come
from either side's cache. Read the file back through the filer's HTTP
handler as well, and let the mount verbosity be raised from the
environment, so a failing run says which layer lost the data.
* test: wait for the reader mount to converge before comparing
A mount caches metadata for about a second, so reading the file the instant
the writer's last close returned can legitimately come back short. Poll the
reader until it matches or the timeout expires; content that is wrong rather
than merely late never converges and still fails, now with the writer's
mount and the filer's own view alongside it.
* test: detect a failover cluster child that exited at startup
Signal(0) succeeds for a zombie and nothing reaped these children until
shutdown, so a process that died on startup looked alive until the readiness
timeout expired. Reap each child as it is started and consult the result.
* test: read a file the killed volume server actually holds
Placement decides which two of three servers back each volume, so killing
volume N and reading readfile-N could pass without the victim ever holding a
replica of it. Resolve each file's volumes through the filer and the master,
and pick one the victim backs, preferring a file the reader has not cached.
* ci: stop persisting checkout credentials in the failover workflow
The job does not use the token after cloning. Also tag the README's command
block as bash and match the timeout the workflow actually uses.
* test: discard the ignored errors errcheck flags in the failover harness
* test: resolve manifests when mapping a file to its volumes
A manifest chunk's own fid names the volume holding the manifest, not the
volumes holding the data, so a large enough file would point the failover
victim at the wrong server.
* test: pin the stale-location recovery path with a primed reader
Reading a file for the first time after a server dies proves nothing: the
lookup is fresh and returns the survivor. Kill one holder and wait for the
master to drop it, read a file on that volume so the reader caches the lone
survivor, restart the first server, then kill the survivor. The reader's only
cached location is now dead while the data is live elsewhere, which is the
case the invalidator exists for: EIO without it, recovery with it.
* filer: re-look-up a chunk's locations as soon as they all fail
A read that fails against every location it was given is far more likely to be
holding a stale list than to be hitting a cluster that is briefly slow, but the
retry loops spent the whole backoff ladder, about 13 s, before the caller got a
chance to invalidate and look the chunk up again. Give the loops a refresh hook
and let the reader cache invalidate on the first fully failed pass, so recovery
starts in milliseconds. Clients without an invalidator keep the old behavior.
The filer's streaming read path has its own fetch loop and is not covered.
* webdav: give the chunk reader a bounded, invalidatable location cache
WebDav resolved chunk locations through filer.LookupFn, whose own doc asks
long-running processes to prefer wdclient.FilerClient: its cache is unbounded,
and it has no way to invalidate an entry, so the reader cache was constructed
with a nil invalidator and a WebDav server that had cached a location kept
reading from it after the volume moved or died. Use FilerClient, as the mount
and the S3 gateway already do.
* filer: refresh locations on the random-read path too
readChunkSliceAt bypasses the chunk cacher in random-access mode and fetches
the range directly, which left it without the invalidation the cacher does:
a random reader parked on a stale location had no way back at all. Hoist the
refresh hook onto the reader cache so both paths share it.
* filer: compare chunk locations as a set, not in order
Lookups shuffle the locations they return, so comparing positionally reads a
reshuffle of the very same replicas as a fresh set and spends an immediate
retry on locations that just failed. weed/filer already had an
order-independent comparison for this; move it next to the retry loops so
both callers share one helper.
* mount: re-resolve volume locations after a failed chunk read
NewChunkGroup passed nil as the ReaderCache's CacheInvalidator, so
retryFetchAfterCacheInvalidation was dead code on the FUSE read path. A
mount that cached a volume's locations while one server was down kept
retrying that server after it died, then returned EIO, even though the
master and filer both resolved the live replica. The S3 gateway already
passes its filerClient; do the same for the mount.
* test: FUSE integration tests for volume server failover
One mount appends while a second tails, and a volume server is killed,
started or restarted mid-stream against a 001-replicated cluster of three
volume servers. Automates the scenario matrix reported for Docker Swarm
mounts, including the large-file variant and a no-chaos control.
* test: report the filer's own view when append content mismatches
A mismatch between what the writer wrote and what the reader sees can come
from either side's cache. Read the file back through the filer's HTTP
handler as well, and let the mount verbosity be raised from the
environment, so a failing run says which layer lost the data.
* test: wait for the reader mount to converge before comparing
A mount caches metadata for about a second, so reading the file the instant
the writer's last close returned can legitimately come back short. Poll the
reader until it matches or the timeout expires; content that is wrong rather
than merely late never converges and still fails, now with the writer's
mount and the filer's own view alongside it.
* test: detect a failover cluster child that exited at startup
Signal(0) succeeds for a zombie and nothing reaped these children until
shutdown, so a process that died on startup looked alive until the readiness
timeout expired. Reap each child as it is started and consult the result.
* test: read a file the killed volume server actually holds
Placement decides which two of three servers back each volume, so killing
volume N and reading readfile-N could pass without the victim ever holding a
replica of it. Resolve each file's volumes through the filer and the master,
and pick one the victim backs, preferring a file the reader has not cached.
* ci: stop persisting checkout credentials in the failover workflow
The job does not use the token after cloning. Also tag the README's command
block as bash and match the timeout the workflow actually uses.
* test: discard the ignored errors errcheck flags in the failover harness
* test: resolve manifests when mapping a file to its volumes
A manifest chunk's own fid names the volume holding the manifest, not the
volumes holding the data, so a large enough file would point the failover
victim at the wrong server.
* test: pin the stale-location recovery path with a primed reader
Reading a file for the first time after a server dies proves nothing: the
lookup is fresh and returns the survivor. Kill one holder and wait for the
master to drop it, read a file on that volume so the reader caches the lone
survivor, restart the first server, then kill the survivor. The reader's only
cached location is now dead while the data is live elsewhere, which is the
case the invalidator exists for: EIO without it, recovery with it.
* filer: re-look-up a chunk's locations as soon as they all fail
A read that fails against every location it was given is far more likely to be
holding a stale list than to be hitting a cluster that is briefly slow, but the
retry loops spent the whole backoff ladder, about 13 s, before the caller got a
chance to invalidate and look the chunk up again. Give the loops a refresh hook
and let the reader cache invalidate on the first fully failed pass, so recovery
starts in milliseconds. Clients without an invalidator keep the old behavior.
The filer's streaming read path has its own fetch loop and is not covered.
* filer: refresh locations on the random-read path too
readChunkSliceAt bypasses the chunk cacher in random-access mode and fetches
the range directly, which left it without the invalidation the cacher does:
a random reader parked on a stale location had no way back at all. Hoist the
refresh hook onto the reader cache so both paths share it.
* filer: compare chunk locations as a set, not in order
Lookups shuffle the locations they return, so comparing positionally reads a
reshuffle of the very same replicas as a fresh set and spends an immediate
retry on locations that just failed. weed/filer already had an
order-independent comparison for this; move it next to the retry loops so
both callers share one helper.
* filer: re-list a folder after deleting it, and put it back if it is not empty
The emptiness check inside the delete and the removal of the folder entry are
not atomic, so an entry can land between them and be left reachable by its own
path but out of every listing. Looking again after the delete catches the ones
whose create event has not arrived yet, and does not depend on the event stream
or on the observation window holding.
* filer: create the directories holding an entry after the entry
A parent checked before the insert can be taken by the empty-folder cleaner
before the entry lands, which leaves the entry reachable by its own path but out
of every listing. Creating the parents afterwards cannot be undone by a delete
that was authorised before the insert, and pairs with the cleaner re-listing
after its own delete: whichever of the two acts second sees what the other did.
Going second means the entry is already stored when the parent fails, so it is
taken back out and the caller still sees the error it used to get.
* filer: narrow a directory that came back wider than the one it replaced
A writer recreating its own missing parent has only the entry it is inserting to
go on, so the directory it mints can grant access the deleted one denied - a
0700 folder comes back 0751. The cleaner read the real attributes before
deleting, so its restore now puts the original mode back instead of leaving the
inferred one in place. It only ever narrows, so a directory deliberately
tightened since is left as it is.
* mount: re-resolve volume locations after a failed chunk read
NewChunkGroup passed nil as the ReaderCache's CacheInvalidator, so
retryFetchAfterCacheInvalidation was dead code on the FUSE read path. A
mount that cached a volume's locations while one server was down kept
retrying that server after it died, then returned EIO, even though the
master and filer both resolved the live replica. The S3 gateway already
passes its filerClient; do the same for the mount.
* test: FUSE integration tests for volume server failover
One mount appends while a second tails, and a volume server is killed,
started or restarted mid-stream against a 001-replicated cluster of three
volume servers. Automates the scenario matrix reported for Docker Swarm
mounts, including the large-file variant and a no-chaos control.
* test: report the filer's own view when append content mismatches
A mismatch between what the writer wrote and what the reader sees can come
from either side's cache. Read the file back through the filer's HTTP
handler as well, and let the mount verbosity be raised from the
environment, so a failing run says which layer lost the data.
* test: wait for the reader mount to converge before comparing
A mount caches metadata for about a second, so reading the file the instant
the writer's last close returned can legitimately come back short. Poll the
reader until it matches or the timeout expires; content that is wrong rather
than merely late never converges and still fails, now with the writer's
mount and the filer's own view alongside it.
* test: detect a failover cluster child that exited at startup
Signal(0) succeeds for a zombie and nothing reaped these children until
shutdown, so a process that died on startup looked alive until the readiness
timeout expired. Reap each child as it is started and consult the result.
* test: read a file the killed volume server actually holds
Placement decides which two of three servers back each volume, so killing
volume N and reading readfile-N could pass without the victim ever holding a
replica of it. Resolve each file's volumes through the filer and the master,
and pick one the victim backs, preferring a file the reader has not cached.
* ci: stop persisting checkout credentials in the failover workflow
The job does not use the token after cloning. Also tag the README's command
block as bash and match the timeout the workflow actually uses.
* test: discard the ignored errors errcheck flags in the failover harness
* test: resolve manifests when mapping a file to its volumes
A manifest chunk's own fid names the volume holding the manifest, not the
volumes holding the data, so a large enough file would point the failover
victim at the wrong server.
* test: pin the stale-location recovery path with a primed reader
Reading a file for the first time after a server dies proves nothing: the
lookup is fresh and returns the survivor. Kill one holder and wait for the
master to drop it, read a file on that volume so the reader caches the lone
survivor, restart the first server, then kill the survivor. The reader's only
cached location is now dead while the data is live elsewhere, which is the
case the invalidator exists for: EIO without it, recovery with it.
* volume: accept only static-key gcs credentials on the fetch request
An inline credentials document of a federated type points the SDK at a url,
file or executable of the caller's choosing for the token exchange, so the
request-supplied value is no longer just a key.
* volume: guard the gcs token endpoint like the other remote endpoints
Inline credentials pick where the token request goes, so route the gcs client
through the same deny-list and rebinding-safe dialer used for S3 and azure.
* rust volume: pin that gcs has no credential-driven dial path
* volume: only check gcs credentials on a gcs remote conf
Only the gcs backend reads that field, so another backend carrying a stale
value should not fail the request.
* gcs: load credentials with the type the caller expects
The untyped loader is deprecated because it reads whatever the document
claims to be; callers handling credentials they do not control now name the
types they accept.
* arangodb: bind list prefix, start file name and collection into the AQL query
Concatenating them into the query text let a caller-supplied prefix or
start name close the string literal and append arbitrary AQL, which runs
with the filer's ArangoDB credentials against any collection.
* arangodb: bind the folder path and collection into the recursive delete query
A trailing-slash S3 key reaches DeleteFolderChildren through the
directory-marker cleanup, so quotes in the path could turn the filter
into a match-everything REMOVE over the whole bucket collection.
* arangodb: match the real directory prefix in the recursive delete
The prefix was built by re-joining the path segments with commas, so it
never matched a stored directory and the subtree sweep did nothing.
* worker: log the balance move stage through a constant format string
Go 1.26's printf analyzer now follows printf wrappers reached through an
interface, so passing the stage straight to Logger.Info is a vet failure.
* s3api: bracket the IPv6 host in the signature test URL
A bare IPv6 literal is legal in a Host header but never in a URL. Go 1.26
stopped parsing it leniently, so carry the two forms separately and set
r.Host to the value the client would actually have signed.
* mini: bracket IPv6 addresses in the readiness probe URLs
An IPv6-only host hands mini a bare literal, and %s:%d pasted it into a URL
unbracketed. Under Go 1.26 that URL no longer parses, so waiting for the
admin server never succeeds and mini refuses to start.
* readme: add RustFS to the file system comparison
* readme: note RustFS write amplification and rigid layout
* readme: correct RustFS version, parity and protocol details
* readme: merge the RustFS comparison into the MinIO section
* filer: restore a folder that received an entry while it was deleted
The empty-folder cleaner checks that a folder is empty and then deletes it,
and those two steps are not atomic. An entry created in between survives the
delete but loses the directory holding it: still readable by its own path, yet
absent from every listing until a later write happens to recreate the parent.
Record the folders deleted in each pass and re-check them on the next one,
putting back any that turned out to hold entries. The check waits a pass on
purpose - a writer looks up the parent before inserting the child, so checking
straight after the delete can still run ahead of the insert and see nothing.
Restoring a directory that holds entries is always correct, and restoring one
whose entry went away again just leaves an empty folder for a later pass to
collect, so the repair needs no locking or coordination.
Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r
* filer: keep failed restores queued and inherit the ancestor's ownership
Two gaps in the restore pass.
A folder whose count or restore hit a transient store error was dropped from
the tracking list and never looked at again, leaving its entries out of
listings until some later write recreated the folder - the very thing the pass
exists to avoid. Put those back for the next pass, still under the cap.
A restored folder was minted with a fixed mode and no owner, so a directory
that had been private came back world-readable and owned by root. Take the
mode and ownership from the nearest ancestor still present instead.
Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r
* filer: let the redis stores keep a directory listing that still has entries
On the redis stores the listing is not derived from the entries, it is the only
record that they sit under that directory. DeleteEntry opened by dropping it
outright, so an entry that arrived after the caller judged the directory empty
lost its membership and became unreachable: readable by exact path, absent from
every listing, and invisible to any later check, since counting the directory
reads the listing that was just destroyed. Nothing could detect or repair it.
Drop the listing in DeleteFolderChildren instead, alongside the children it
describes, and leave it alone in DeleteEntry. redis3 needs it explicitly, since
removeChildren clears the skip list nodes but not the list itself, and the plain
redis store was leaking the key entirely.
Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r
* filer: restore folders with their own attributes, and observe them for a window
Five gaps in the restore pass.
The restored directory was reconstructed from whatever ancestor happened to
still be present, and the mode was ORed with 0111 on the way. A private
directory under a world-traversable parent came back granting traversal it had
denied. Read the folder's own attributes before deleting it and put exactly
those back. That also removes the ancestor walk, which treated a transient
store error as "not found" and silently fell through to a broader ancestor.
A single check a pass later was not a delay at all. Ticker sends coalesce, so
when a pass runs long the next one starts immediately, and a writer already
past its parent lookup can insert after the check has read zero - after which
the folder was discarded for good. Keep each folder under observation for a
bounded wall-clock window and re-check it on every pass until it expires. This
narrows the exposure rather than closing it; only making the emptiness check
and the delete atomic would do that.
A delete that returned an error was never observed at all, though the redis
stores drop the folder before its parent-list member, so a failure return is
not proof the folder survived. Record the folder before the delete instead.
Restores now run shallowest first, so a folder taken by the parent cascade is
rebuilt with its own attributes before anything below it needs it as a parent.
Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r
* filer: recover a deleted folder from the create event for the entry that raced it
Checking each deleted folder on a timer was the wrong instrument. It cost a
listing per folder per pass, and it could only ever be a guess about when the
racing write would land.
The metadata stream already carries the answer. A folder is recorded before it
is deleted, so any entry that can be orphaned is created after that record and
its create event names that exact directory. Match the event against the
recently deleted folders and the folder is known to need putting back, rather
than inferred to.
The window stops being a guess at the race and becomes what it should be: how
far behind the event stream is allowed to run before a folder stops being
watched. Listing is now done once, for a folder an event has already named, to
skip the restore when the entry has since gone away again.
Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r
* filer: bound how long a folder is watched, and rebuild ancestors from themselves
Four gaps found reviewing the restore pass.
A folder whose restore kept failing was never let go: the written-to check ran
before the age check, so it was picked up, retried, put back, and counted again
on every pass for the life of the process. Apply the window first, whatever
state the folder is in.
At the cap, the folder being recorded was the one turned away, though it is the
one whose race is still live - the older entries are already close to ageing
out. Give up one of those instead, picked as the oldest of a small sample so
the cost stays flat under heavy deletion rates.
An ancestor taken by the same cascade was left to the descendant's restore to
recreate, which minted it from the descendant's attributes and handed back
access the ancestor never granted. Rebuild those from what they were, ahead of
anything below them.
Reading a directory's attributes assumed an entry came back. Some stores return
nothing with no error, so treat that as not found. The mode is also taken whole
rather than through Perm(), which was dropping setgid, setuid and sticky.
Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r
* redis3: take a directory listing left behind by a failed delete
Removing the last name deletes the list, and if that delete fails the header
survives pointing at a name that is gone. The retry finds nothing to remove,
reports no changes, and returns before reaching the delete, so the key stays
for good. Take it on that path too.
Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r
* filer: do not sweep children when deleting a folder non-recursively
doBatchDeleteFolderMetaAndData lists a folder and bails out if it has any
children, then calls Store.DeleteFolderChildren unconditionally. On the
non-recursive path that bulk sweep has nothing legitimate to remove: it only
runs once the listing came back empty, so the sole rows it can delete are
ones inserted after the check.
The S3 empty-folder cleaner deletes through this path, so a PUT landing
between the listing and the sweep loses its entry after the write was already
acknowledged. Neither side sees an error - the client has its 200 and the
cleaner logs an ordinary empty-folder deletion - and the chunks leak, since
the cleaner passes shouldDeleteChunks=false and nothing was enumerated to
collect. Workloads that scatter objects over many shallow prefixes empty and
refill those folders constantly, which is what makes the window reachable.
Sweep only when the delete is recursive, or when the whole-bucket shortcut
skipped the listing and depends on it.
Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r
* filer: pin the folder entry removal left by the racing-child test
The surviving entry is reachable by path but drops out of listings until the
folder comes back, and nothing in the test said so. Assert it, so the exposure
that remains after this change is visible rather than implied.
Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r
* iceberg: vend table-scoped credentials to clients that ask for delegation
The catalog recognised X-Iceberg-Access-Delegation: vended-credentials
and then deliberately said nothing, because it had nothing to vend: it
withheld even the S3 endpoint so the client would keep the credentials it
was configured with. That left every engine expecting the catalog to hand
out access - Snowflake, Databricks, Trino with vending, any multi-tenant
setup - needing static S3 keys distributed out of band.
Mint an STS session per request instead, scoped by a session policy to
the table's own prefix plus the bucket listing needed to resolve it, and
return it in the load response config and storage-credentials. The role
to assume is named by -s3.iceberg.credentialRole; its trust policy is
what decides whether a caller may assume it, and vending stays off until
it is set. A failed mint falls back to the old silence rather than
handing back an endpoint the client cannot sign for.
* iceberg: keep vended credentials inside the table prefix
Review follow-ups on credential vending:
Listing was granted on the bucket ARN with no condition, so a credential
vended for one table could enumerate every other table's object names.
Constrain s3:prefix to the table's own prefix, which the S3 gateway
already populates for list requests.
A table location carrying * or ? would have gone into the policy's
resource pattern unescaped and widened the session to sibling prefixes.
Refuse to vend for such a location rather than escaping it; nothing the
catalog generates contains those characters.
DurationSeconds skipped the 900..43200 bounds the other assume-role paths
enforce, so -s3.iceberg.credentialDurationSeconds could ask for a session
outside them. The check is now shared by all three entry points.
* iceberg: return the vended credentials from buildFileIOConfig itself
buildStorageConfig was a second name for what buildFileIOConfig already
did; it now returns the storage credentials alongside the properties, and
callers that only want the properties drop them.
* iceberg: split the vended bucket grants, and refuse a whole-bucket scope
The prefix condition sat on a statement that also granted
GetBucketLocation and ListBucketMultipartUploads, neither of which carries
an s3:prefix to satisfy it, so both were denied for every vended
credential. GetBucketLocation moves to its own unconditioned statement.
ListBucketMultipartUploads is dropped: Iceberg writers complete and abort
by upload id, and granting it either leaks in-flight keys bucket-wide or
breaks on the same missing prefix.
A table whose location has no prefix - one registered at the bucket root -
would have been vended read and write over every other table in the
bucket. Refuse, the way a location with wildcards is refused.
* iceberg: add view rename, scan-report and snapshots=refs to the catalog
Three gaps against the REST spec that clients hit in normal use:
Views had no rename, though tables did and views are stored the same way,
so the move is the same catalog-only pointer move. Tables and views share
a namespace directory, so both renames now refuse the other kind instead
of moving it.
Engines POST a scan or commit report after planning; a 404 there turns
into an error line per query. Accept the report and discard it - the
catalog keeps no metrics store.
LoadTable ignored ?snapshots=refs and always returned the whole snapshot
history, which is what clients use the parameter to avoid on long-lived
tables.
* iceberg: authorize view rename against the view ARN, tighten the metrics endpoint
Review follow-ups:
The shared rename checked the source against a table ARN whatever the
kind, so a policy scoped to a view's own ARN never matched and one
written for a table ARN was evaluated for a view. The entry kind now
carries the ARN builder.
The metrics endpoint truncated a report at 1 MiB and then failed to parse
it, answering 400 for a query that had actually succeeded. Read one byte
past the limit to tell "fits" from "cut short", and discard an oversized
report instead of rejecting it. Empty bodies and reports without a
report-type are now rejected, which the REST schema requires.
?snapshots= is defined for LoadTable, so it no longer filters what
CreateTable echoes back.
* iceberg: make a table commit a compare-and-swap
The catalog validated the caller's version token, ran its authorization
checks, and only then wrote the new metadata xattr. Two engines
committing against the same base both passed that check and both wrote,
so the second silently dropped the first one's snapshot. Both also derive
the same v{N}.metadata.json name and the file write overwrote, leaving
the surviving pointer aimed at the loser's metadata - and the loser's
conflict cleanup then deleted the winner's file.
Write the metadata file with an exclusive create and update the xattr
conditionally on the bytes the handler read, the way the maintenance
worker already commits. A writer that lost the race re-reads and retries,
and reports 409 CommitFailedException once out of attempts.
* iceberg: stage a commit under a unique name when the versioned one is taken
Two follow-ups from review of the commit compare-and-swap:
Refusing to overwrite v{N}.metadata.json also refused to get past a file
left behind by a commit that died between staging and updating the
pointer. Every later commit derived the same name, saw the collision, and
reported a conflict, so the table stayed uncommittable until an orphan
sweep removed the file. Stage under v{N}-{uuid} instead: neither writer's
file is overwritten and the catalog pointer still decides who won, which
is how the maintenance worker has always staged its own metadata.
metadataVersionFromLocation learned to read the version back out of that
name.
The conditional update guarded only the metadata attribute while the
write replaced the whole entry, so a policy or tag written in the same
window was silently reverted. Guard every catalog attribute, which turns
that into a conflict the caller retries on fresh state.
* iceberg: give saveMetadataFile the exclusive flag instead of a second name
saveNewMetadataFile, saveMetadataBlobExclusive and uniqueMetadataFileName
were three new names around one existing helper. The flag now rides on
saveMetadataFile and saveMetadataBlob, and the unique-name construction
sits where it is used.
* iceberg: reuse the filer CAS helpers #10773 added, and stage transactions exclusively
#10773 landed mutateEntryExtended, which already writes an entry back under a
whole-entry precondition and retries. Drop the helper this branch added and
route the table commit through it: the check that the metadata is still the
one this request read now lives in the mutation, where it sees current state.
The policy the request was authorized against is asserted too, so an
administrator restricting it mid-commit sends the caller back through
authorization instead of having a stale decision applied. Bucket and
namespace policies live on other entries and a single-entry precondition
cannot cover them.
Multi-table transactions stage their metadata exclusively for the same
reason single-table commits do, and carry the name they landed on into the
pointer flip.
#10774 gave buildTestMetadata its refs and age parameters while #10773
added a caller with the old arity. Each was green against a master that
did not yet have the other, and the merge of both does not compile, so
vet and the unit tests fail on master.
* iceberg maintenance: keep the snapshots that branches and tags pin
expireSnapshots only ever protected the current snapshot, so a snapshot
held by a tag or a non-main branch was expired once it aged out of the
retention window. iceberg-go's RemoveSnapshots drops any ref whose
snapshot is gone without complaint, so the tag disappeared and the files
behind it were deleted as unreferenced.
Protect every ref target, and honour a branch's own
min-snapshots-to-keep / max-snapshot-age-ms over the ancestors behind its
head. Detection skips pinned snapshots for the same reason: proposing a
job whose only outcome is a no-op keeps the worker busy forever.
* iceberg maintenance: re-plan when a ref appears mid-commit, and stop proposing no-op expiry
Three follow-ups from review of the ref-aware expiry:
The commit guard only compared the table head, so a tag created between
planning and commit could pin a snapshot the plan was about to expire.
Re-check the refs against the metadata the commit actually reads.
Detection now asks snapshotsToExpire what execution would remove instead
of approximating with its own count-and-age rules. Expiry always requires
a snapshot past the retention window, so a table over the quota whose
snapshots are all young was being proposed for a job that could only
no-op.
The branch retention test could not tell "retained the whole lineage"
from "honoured min-snapshots-to-keep", because the branch had exactly as
many ancestors as the count. Give it one more, and cover
max-snapshot-age-ms too. Both need snapshots genuinely older than a
retention window, which iceberg-go will not accept at build time, so the
fixture backdates the metadata after building it.
* iceberg maintenance: fold the metadata test builders back into one
buildTestMetadata, buildTestMetadataWithRefs, buildTestMetadataAged and
buildTestMetadataNow were four names for one thing. Keep the original and
give it the refs and age it needs.
* s3tables: add the maintenance configuration APIs
Stores the configuration verbatim as the wire shape under a new
s3tables.maintenance extended attribute, so Get hands back what Put took
and no translation layer can drift from the AWS model.
Nothing reads the configuration yet.
Put merges a single type into the stored map so configuring compaction
does not drop snapshot management, and asserts the attribute's prior value
so two concurrent Puts cannot silently clobber each other.
* iceberg: apply the maintenance configuration in the worker
The worker now reads the per-table and per-bucket maintenance
configuration written by the control plane, so the wildcard plugin config
is a default rather than the only setting a table can have.
Table properties still win by default, since a table declaring its own
layout is what every engine honours and the compactor has to agree with
whoever writes the files. Clearing table_properties_override makes the
maintenance configuration authoritative instead.
Status is not part of that contest: a disabled type drops its operations
and no property can re-enable them, so the operator's kill switch always
holds. Manifest and delete-file rewrites have no AWS equivalent and ride
with compaction.
Detection reads both attributes from entries it already lists.
* s3tables: report maintenance job status
The worker records the outcome of each run in its own extended attribute,
separate from the configuration so operator and worker writes do not
contend, and GetTableMaintenanceJobStatus reads it back.
Only the types a run touched are written, so a partial run cannot erase
what an earlier one recorded. The reader fills in the rest: Disabled when
the configuration switched a type off, Not_Yet_Run otherwise.
Status is advisory, so a lost race is logged rather than failing a job
whose work already committed.
* s3tables: route the maintenance APIs over REST
The five actions were only reachable by X-Amz-Target dispatch, which the
AWS CLI and SDK do not use for this service. They address the operations
by path, so the APIs were unreachable from any official client.
* s3tables: fix the table bucket ARN field name
GetTableBucketMaintenanceConfiguration emitted tableBucketArn where the
wire field is tableBucketARN, as every other response in this package
already spells it. Official SDK deserializers ignore the unknown key, so
the required field came back unset.
* s3tables: carry the compaction strategy through to the worker
IcebergCompactionSettings modelled only targetFileSizeMB, so a request
naming a strategy was accepted and then dropped on the way to storage.
The worker now maps binpack and sort onto its own rewrite strategy and
lets auto defer to the worker configuration.
z-order is rejected rather than accepted and quietly binpacked.
* s3tables: report bucket-level maintenance status
GetTableMaintenanceJobStatus read only the table's configuration, so
unreferenced file removal — which is configured on the bucket — reported
Not_Yet_Run or a stale success after an operator disabled it.
The merge helper now lives in this package and the worker shares it.
* iceberg: delete orphans only after the non-current window
AWS marks a file non-current once it has been unreferenced for
unreferencedDays, then deletes it a further nonCurrentDays later.
The cutoff was taken from unreferencedDays alone, so a 3/10 configuration
hard-deleted on day three and threw away the ten day recovery window.
remove_orphans deletes in one step rather than marking, so the cutoff is
now the sum of the two.
* s3tables: assert every attribute when rewriting an entry
UpdateEntry writes the whole entry back from the snapshot the caller
read, and its precondition only covers the keys the caller names. Both
maintenance writers named one key, so a job status write could revert a
maintenance configuration an operator had just disabled, turning an
advisory write into a silent re-enable.
Both now assert the entry's full attribute set, including the target key
when absent so a concurrent create also fails the precondition.
* s3tables: assert absent attributes when rewriting an entry
The precondition covered the attributes present when the writer read the
entry, so an attribute created between that read and the write was absent
from it. A first-time PutTableMaintenanceConfiguration disabling a type
therefore lands, passes the per-key checks, and is then deleted by the
stale whole-entry write.
Every attribute this package stores is now asserted, absent ones
included. The metadata commit and planning index writers rewrite the same
entries and had the same exposure, so both use the shared snapshot too.
* iceberg: implement the auto compaction strategy
auto was accepted, stored and read back, but left the worker on its own
default, so a sorted table configured as auto was compacted with binpack.
AWS defines auto as sorting tables that declare a sort order and
bin-packing the rest. That needs the table metadata, so the choice is made
where the rewrite plan is resolved: an unsorted table falls back to
binpack rather than failing the way an explicit sort request does.
* s3tables: validate the maintenance setting ranges
PUT accepted zero, negative and oversized values for every numeric
setting. The worker then ignores a non-positive value and saturates an
oversized one, so the configuration read back was not the one that ran.
AWS bounds all five to 1..2147483647, which is now enforced. The fields
are pointers so an explicit zero is distinguishable from an omitted one
and can be rejected rather than silently ignored.
* s3tables: give every entry writer the same compare-and-swap
updateExtendedAttribute asserted the entry's attributes, but the helpers
behind the metadata, policy and tag handlers still wrote the whole entry
unconditionally. Any of them could land on a stale snapshot and delete a
maintenance configuration an operator had just written.
They all share one read-modify-write loop now, so the precondition and
the bounded retry apply wherever an entry is rewritten.
* s3tables: move the maintenance configuration with a renamed table
RenameTable carried the metadata, version, policy and tags to the new
name but left the maintenance configuration and job status behind. A
table with snapshot management disabled came back enabled under its new
name, and the stale configuration stayed on the old name where a table
created there would inherit it.
The decoupled-delete cleanup left the same two attributes behind.
* s3tables: accept every AWS partition in ARNs
The route regexes and the ARN patterns both hardcoded arn:aws, so valid
aws-cn and aws-us-gov ARNs never reached a handler. The router now shares
the partition-tolerant prefix with the parser, and a generated ARN uses
the partition its region belongs to so it parses back.
* s3tables: generate ARNs in the region's partition
The handler's own ARN generators still formatted arn:aws directly rather
than going through the partition-aware builder, so a China or GovCloud
deployment routed the request but then returned a commercial ARN and
matched IAM policies against it.
The round-trip test missed this because parsing accepts any partition, so
it now asserts the prefix the region implies.
* s3tables: complete the ARN partition table
aws-iso-e, aws-iso-f and aws-eusc were missing, so eu-isoe-*, us-isof-*
and eusc-* regions fell through to the commercial partition.
* s3tables: do not let a rename swallow a concurrent maintenance write
Rename copied the source attributes early and cleared the source at the
end, so a Put landing in between missed the copy to the destination and
was then deleted by the cleanup. It succeeded and vanished.
The cleanup now clears the source only while it still holds exactly what
was copied, and returns a conflict otherwise. Put checks the catalog
identity inside the same conditional mutation, so it also cannot write to
a name that a rename or delete has already soft-deleted.
* iceberg: carry snapshot retention in milliseconds
Config stored retention as hours, so any sub-hour value would have to be
truncated to 0 and then clamped back up to the 168 hour default. Keep the
plugin config key in hours and convert once at parse time.
* iceberg: let table properties override the worker config
Every other Iceberg implementation lets a table's own properties win over
engine defaults; the worker ignored them entirely. A writer honouring
write.target-file-size-bytes and a compactor rewriting to the plugin
config's size would rewrite each other's output forever.
Resolved once per job rather than per operation, so compaction committing
new metadata mid-job cannot change the settings underneath it.
* iceberg: clamp the orphan cutoff so it cannot overflow
collectOrphanCandidates converts the cutoff to a time.Duration. Past
roughly 2.5 million hours that multiplication wraps negative, putting the
cutoff in the future so every file walked looks like an orphan and gets
deleted, including data a concurrent writer has not yet committed.
Reachable today through orphan_older_than_hours.
* ec.encode: require the shards to agree on size before deleting the volume
Before an encode deletes the volume it just encoded, it asks whether
enough shards exist and whether they are spread across nodes. Both are
questions about presence: nothing asks whether those shards are whole.
Every shard takes one piece of each block row, so they are all written to
the same length. One that disagrees was truncated, half copied, or landed
on a disk that filled up -- and counting cannot see it, so the source
volume is deleted on the strength of a set that cannot rebuild it.
Compare the sizes the cluster already reports (shard_sizes travels in the
heartbeat) and hold the deletion back when they disagree, naming the odd
shard and its holder. Sizes reported as zero are skipped rather than read
as a disagreement: a volume server that predates shard-size reporting, or
one that has not heartbeated them yet, must not strand every encode in
the volume-plus-shards state this check exists to avoid.
* ec.encode: judge shard sizes on the newest encode generation only
The size check collected every shard the master reports for the volume,
while the recoverability check beside it counts only the newest encode
generation. A re-encode can change the ratio, so an orphaned older
generation -- one the pre-encode sweep could not reach, but the master
still hears about -- has shards of a different length by nature. Merging
those into the comparison makes a healthy current set look inconsistent,
and because the orphan keeps being reported, every retry fails and the
encode is left holding the volume and its shards for good.
Collect sizes the way CollectEcShardBitsByNode collects bits: fenced to
the newest EncodeTsNs, with unstamped entries forming the one legacy
generation.
The harness kills shells mid-command, and the master releases the dead
session's lock only when it notices the connection is gone. That cleanup
lands after the harness has already re-acquired the lock, so it can clear
the lock this run holds and the next command refuses with
need to run "lock" first to continue
recoverInterruptedBalance answered that the way an operator would -- run
lock again and retry -- but the encode and decode recoveries called
shellCommand once and required success, so the same reap failed the run
outright. Move the retry into shellCommand: the reap can land during any
command that follows a kill, not only a balance.
A decode ends by deleting the shards it read, and the only thing standing
between that and a bad reconstruction is verifyDecodedVolumeBeforeDelete,
which asks whether .dat and .idx are non-empty. A .dat truncated to a
single byte passes, and the shards -- the only other copy of everything
past the cut -- are deleted on the strength of it.
The server already knows the answer it never checks: FindDatFileSize
returns the extent the EC index references, and WriteDatFile rebuilds to
it. Compare the two once the file is written and fail the decode instead
of reporting a short volume as a good one.
Longer than the extent still verifies -- padding is not missing data --
so only a genuinely short rebuild is rejected.
Needle counts cannot answer this: .idx is written from .ecx, so the count
matches by construction and a truncated .dat still reports every needle.
A decode deletes the shards only after the regenerated volume is mounted
and verified, so a run interrupted in that last phase leaves the volume
in place with its shards partway through deletion. The re-run then finds
both, tries to collect the shards again to rebuild a volume that already
exists, and fails on the first shard the interrupted run had removed:
generate normal volume 3 ...: ec volume 3 missing shard 6
Nothing recovers from there: the shard set is deliberately being
destroyed, so every retry fails the same way while the decoded volume
sits there, already complete.
Finish that cleanup instead. A volume beside the shards is not enough to
act on -- an encode interrupted before it deleted the original leaves the
same shape, as does a decode killed while generating, whose volume may be
half written -- so require a data shard to be gone. Only the deletion
phase removes one, and it is also exactly the state no decode can
recover from, so finishing is the only move left rather than a choice
between two. The deletion still runs behind
verifyDecodedVolumeBeforeDelete, the check that guards it in a normal
run.
* admin: show capacity per storage tier and stop counting remote-tiered bytes as local disk usage
A remote-tiered volume reports its cloud object's size, so summing volume
sizes inflated the dashboard's used-vs-capacity numbers (the local .dat is
gone after volume.tier.move). Split the accounting: DiskUsage now only
counts bytes on local disks, with the cloud bytes surfaced separately per
server and per remote storage name.
The dashboard gains a Storage Tiers table breaking volumes and EC shards
down by tier (each local disk type plus each remote storage), using the
per-disk-type statfs numbers already in the VolumeList response. The
volumes page badges remote-tiered volumes with their storage name, and
the EC shards page fills in real per-shard sizes instead of hardcoding 0.
* admin: review fixes for the tier capacity display
- A disk that predates disk_total_bytes now contributes its logical
bytes to the tier's DiskUsed, so a tier mixing old and new volume
servers doesn't underreport usage; the usage bar always reflects the
displayed Disk Used value (the DataSize fallback in UsagePercent is
gone, and the percent math is overflow-safe).
- getTopologyViaGRPC defaults a zero VolumeSizeLimitMb to 30000 MB like
GetClusterVolumeServers, keeping slot-based capacities consistent.
- The dashboard volume-servers column reads Usage / Capacity to match
its cell content, and the hdd disk-type default is shared between the
volumes-page badge and countUniqueDiskTypes.
* ec: bounded-exhaustive model check of the volume lifecycle
The randomized chaos harness samples the state space; this enumerates
it. The lifecycle is a state machine whose steps mirror the pipelines in
this package, and the checker explores every schedule within the bound:
a crash at every step boundary, an error return running the rollback
(itself crashable at every step), a volume-server restart applying the
startup reconciliation rules in every quiescent state, and the
prescribed restart-based recovery from every crashed state.
Checked in every reachable state: durability (a readable copy always
exists), at most one generation mounted, and — a property the sweep
discipline turns out to guarantee — at most one generation's files on
disk. From every quiescent state the recovery must converge to a clean
volume. Runs in well under a second.
* test: deterministic EC interruption matrix
Enumerate every phase of every interruptible EC operation and kill a
real weed shell exactly when the phase announces itself on the command
output, instead of at a random moment: four encode phases, four decode
phases, and the balance's move phase (set up with -rebalance=false so a
move is guaranteed). Each scenario prepares its precondition, kills at
the marker, runs the prescribed recovery, and verifies every stored byte
still reads back identical.
The interruption recoveries move out of the randomized ops into shared
chaosRun helpers both drivers use.
* test: make the randomized EC chaos walk opt-in
The systematic layers — the interruption matrix and the lifecycle model
check — carry the CI coverage deterministically; the randomized walk
stays for exploratory runs, behind EC_CHAOS_SEED.
* ci: bound the EC integration suite by the job budget, not go test's default
The suite with the interruption matrix runs close to the default 10m
binary timeout on slower runners.
* test: require every interruption-matrix marker to appear
A marker that never prints means a pipeline refactor renamed or dropped
the progress line; silently degenerating into a no-interruption run
would let CI pass without exercising the boundary the scenario names.
Also recheck the marker channel after the wait: a shell that prints and
exits at once makes both channels ready, and select picking the exit
case must not report a printed marker as missed.
* ec: let the encode's balance see a migrating volume's shards across disk-type buckets
Shard generation writes beside the source .dat, so a cross-tier encode
(source on hdd, -diskType=ssd) leaves the fresh shards in the source
disk-type bucket. The encode's internal balance ingested only the target
bucket, saw no shards, and planned no moves; the spread guard then
correctly aborted the encode (and before that guard existed, the shards
silently stayed clumped on the generation host in the wrong tier).
EcBalance now takes the encode batch as migratingVolumeIds and ingests
those volumes' shards from every bucket, while everything else keeps the
bucket filter so a plain ec.balance never drags deliberately tiered
shards onto another disk type. The in-memory model delete also becomes
bucket-agnostic: a node holds a given shard in exactly one bucket, and a
bucket-scoped delete missed cross-bucket moves in the dry-run model.
* volume: decode reads shard 0 from its resolved path, not the EC volume's base dir
On a multi-disk server a volume's shards can sit on several disks; the
store registers each shard with its own path and CollectEcShards resolves
them, but FindDatFileSize derived the .ec00 path from the EcVolume's base
directory. When shard 0 lived on a sibling disk, VolumeEcShardsToVolume
failed with 'open ...ec00: no such file or directory' and ec.decode
aborted.
* ec: decode re-copies shards the topology claims but the target does not hold
An interrupted earlier decode or balance can leave the master believing
the decode target holds a shard whose file never landed: the mount
registered but the partial copy was cleaned, or the file was swept. The
collect step took the topology's word for it, excluded the shard from
the copy set, and the decode failed with 'missing shard'. Probe the
target's live inventory (VolumeEcShardsInfo) and treat anything it
cannot serve as still-to-copy.
* ec: decode discovers shards across disk-type buckets
Shards sit wherever encode generation and balance left them: a
cross-tier encode leaves them in the source disk-type bucket, a partial
migration straddles buckets. ec.decode scoped its shard discovery to the
-diskType bucket and reported a decodable volume as having no shards at
all. Union across buckets, the way the encode's shard verification
already does.
* test: EC chaos lifecycle harness
Randomized, seeded sequences of the EC lifecycle against a live cluster
in the production-shaped layout: multiple data disks per server, a
separate -dir.idx directory so .ecx/.ecj sidecars are shared across
disks, and a tagged ssd tier. Operations cover encode (hdd and ssd
targets), balance, shard damage plus rebuild, decode, re-encode,
deletes, scrub, tier moves, crash-restarts, sidecar fault injections
(a data-dir .vif pushed into the shared idx dir; a stale-generation
shard planted beside a newer encode), and interruptions: a real weed
shell subprocess killed mid-encode, mid-decode, and mid-balance, with
the recovery re-run required to converge.
One invariant holds after every step: every stored byte reads back
identical and every deleted needle stays deleted. EC_CHAOS_SEED and
EC_CHAOS_STEPS make runs reproducible and scalable.
A known gap is tolerated and logged rather than fixed here: a shard
mounted on two disks of one node (orphan adoption after an interrupted
copy) is invisible to ec.balance's dedup and unaddressable by
ec.shard.unmount's shard@address form, so no cleanup path exists yet.
* test: fail payload-corruption checks on the test goroutine
t.Fatalf inside require.Eventually's condition runs on the poller's
goroutine, where Goexit kills only that goroutine and the corruption
message can be lost behind a generic timeout. Record the mismatch, end
the polling, and fail on the test goroutine. Also assert the full shard
count in the cross-bucket decode-discovery test.
* shell: move ErrorWaitGroup to weed/util
* shell: remove unused CandidateEcNode and EcRack types
* ec: extract EC orchestration logic from weed/shell into weed/ec
Move the EC node/topology model, balance engine, encode pipeline, decode
pipeline, and rebuild engine into a new weed/ec package so shell commands
and maintenance workers can share the logic. Shell commands keep flag
parsing and delegate through a small ec.Env (dial option, topology fetch,
volume locations, lock check). Tests move along with the code.
* shell: remove unused proportional-rebalance type stubs
* ec: move scrub, replication check, and shard unmount engines into weed/ec
* worker: share the EC generation-aware shard counter from weed/ec
* ec: gofmt
* shell: drop EC aliases with no remaining callers
* ec: guard a missing topology hook and nil disk entries in topology helpers
* ec: drop trailing newlines from decode error strings
* ec: re-check the shell lock before applying shard unmounts
* shell: trim -node entries in ec.scrub
* s3: add option to disable bucket auto-creation on upload
* command: expose -autoCreateBucket in s3, filer, server, and mini
* s3: apply the bucket auto-create policy to directory marker uploads
* s3: validate the bucket name before the auto-create disabled check
* s3: cover the disabled auto-create gate at all three upload entry points
sftp: url-encode the upload path so filenames can't inject filer query commands
The SFTP put handler concatenated the user-controlled filename straight into
the filer upload URL, so a name containing "?" was parsed as a query string.
Build the URL via url.URL{Path: ...} so "?" becomes %3F and stays a literal
path character.
* volume: validate replica upload targets in FetchAndWriteNeedle
The replica leg forwarded the fetched needle to a caller-supplied address
without checking it, so a malformed target could redirect the upload to an
unintended host or path. Require each replica target to be a bare host:port
whose host is not loopback / link-local / unspecified, reusing the address
deny-list; cluster peers legitimately sit on private networks, so RFC 1918 /
CGNAT stay allowed and -volume.allowUntrustedRemoteEndpoints still opts out.
Validate every target up front so a bad one fails the request before the local
write, and upload through a client that re-checks the resolved address at
connect time so a replica hostname cannot rebind to a blocked address after
validation. Mirrored in Rust (validation moved ahead of the local write; the
Rust S3 path's connect-time re-check is still a follow-up there).
* volume: only accept inline gcs credentials in FetchAndWriteNeedle
The gcs credentials value on this request could name a local filesystem path,
which the SDK reads from disk. Accept only inline JSON here; the server-side
GOOGLE_APPLICATION_CREDENTIALS env var still supplies a path. The Rust volume
server has no gcs backend, so there is nothing to mirror.
* remote_storage/azure: allow a per-request HTTP client
Thread an optional *http.Client through NewAzBlobClient and add
azure.MakeWithHTTPClient, mirroring the S3 backend. When set, the client
overrides the azblob transport so a caller can pin the dial path. The
existing makers pass nil, so behavior is unchanged.
* volume: extend the remote-endpoint guard to the azure backend
The endpoint validation and rebinding-safe dialer in FetchAndWriteNeedle
covered the S3-SDK backends. The azure backend also dials a caller-supplied
AzureEndpoint, so route both families through a single guardedRemoteClient
helper that returns the endpoint each backend dials and a constructor bound
to the guarded HTTP client. azure is guarded only when AzureEndpoint is set;
an empty endpoint derives the public host from the account.
-volume.allowUntrustedRemoteEndpoints still opts out.
* rust volume: assert the azure endpoint has no remote-client path
The Rust volume server has no azure backend, so make_remote_storage_client
rejects the type before any client is built. Add a regression test pinning
that invariant.
* volume_move: treat zero-sized EC shards as absent in move verification
A zero-sized shard file is residue of a failed operation (issue 10730),
not a shard - but VerifyEcShards only checked presence, so a copy that
landed as an empty file passed verification and the source was deleted
behind it. Size zero now reads as absent, with a distinct error naming
the zero-sized shard so the operator can tell a broken copy from a
missing one.
* storage: exclude zero-sized EC shards from rebuilds and clean up stale ones
The reproducer in issue 10730: a zero-sized shard file left by a failed
operation was selected as a Reed-Solomon input and failed the whole
rebuild with an input size mismatch, because input discovery checked
existence, not substance.
- RebuildEcFiles treats a zero-sized shard file as missing and
regenerates over it in place (the reclassified-corrupt path: temp
file beside the residue, atomic rename).
- The startup/rescan shard loader, which always skipped zero-sized
files, now deletes them once they are older than an hour - young
enough files can be an in-flight copy's just-created file, since the
same scan runs from LoadNewVolumes while serving.
Regression tests: a rebuild with one emptied shard regenerates it
byte-identical; the loader deletes a stale zero-sized shard and leaves
a fresh one alone.
* storage: age-check each zero-shard cleanup candidate individually
The shard scan merges the data and idx directory listings, so the
age-checked entry and a deletion candidate can be different files
sharing one name - a stale zero-sized file in one directory next to a
fresh same-named file in the other (possibly an in-flight copy's
just-created one) could get the fresh file deleted. Each candidate's
own modification time now decides, both directories are handled in one
pass, and the split-directory case is pinned by a test.
* redis2: route the orphan cleanup existence checks to the master
* scaffold: the redis_cluster2 read routing key is useReadOnly
* ci: run the gated redis store tests
* redis2: poll for the redis expiry instead of a fixed sleep
* redis2: assert the value key exists before testing its expiry
The Go volume server has VolumeConsolidateIndex, which moves a volume's
.idx out of the data directory into the configured -dir.idx directory
(where an EC decode/reconstruct can leave it co-located) and reloads the
volume in place. The Rust port's proto omitted the RPC entirely, so its
generated VolumeServer trait was one method short of Go's.
Add the proto message and rpc, the gated grpc handler, and
Store::consolidate_volume_index / Volume::relocate_index_to, mirroring
Go's Store.ConsolidateVolumeIndex and Volume.RelocateIndexTo -- including
the cross-device copy fallback and the reopen-against-the-old-dir path
when the move fails.
Integration tests cover the real move (index relocated, volume still
serves reads and the move is idempotent), the no-op paths (index already
in place, no separate idx dir) and the not-found error, plus the grpc
handler end to end.
A bounded LoopProcessLogData (stopTsNs set) on a buffer that never took a
write since process start fell into the ResumeFromDiskError branch, which
never checks stopTsNs when ReadFromDiskFn is nil and HasData() is false.
The read parked on the notification loop forever while the subscription's
idle heartbeats kept the stream looking alive, so a bounded
SubscribeMetadata pass on a freshly restarted idle filer never completed.
Terminate like the caught-up path does, returning a nil error: leaking
the pending ResumeFromDiskError would latch the filer's outer loop into
its gap machinery, which parks the bounded subscriber all over again.
* feat: throughput limits for replicate, EC shard, and worker-driven moves
VolumeCopy was the only rate-limitable transfer; EC shard copies,
replica creation, and worker-driven moves all ran at whatever the
receiving server's maintenance rate allowed, with no per-operation
control.
- proto: VolumeEcShardsCopyRequest and the balance / ec_balance task
params and configs gain io_byte_per_second; 0 keeps today's behavior
(the volume server's own maintenance rate governs).
- volume server: VolumeEcShardsCopy throttles with one WriteThrottler
per request, shared across the shard, .ecx, .ecj, .vif, and .ecsum
copies so the limit caps the transfer as a whole - the same shape as
VolumeCopy.
- volume_move: ReplicateVolume accepts the limit; EcMoveOptions carries
it through MoveEcShards/CopyAndMountEcShards into the copy request,
with fake-client tests asserting propagation.
- shell: ec.balance gains -ioBytePerSecond; volume.tier.move's
replication top-up honors the command's existing -ioBytePerSecond
instead of running unthrottled.
- worker: balance and ec_balance configs gain io_byte_per_second
(surfaced in the admin config schema), carried through detection and
plugin job parameters into task params and handed to the shared
mover; batch balance jobs inherit the limit from their detection
results.
The limit is per copy stream, so maxParallelization multiplies the
aggregate ceiling.
* worker plugins: expose io_byte_per_second in the plugin config and derive it
The plugin-driven detection path derives its task Config from the
plugin configuration values, and both balance and ec_balance left
IoBytePerSecond at zero there - a configured limit silently reverted
to the server maintenance rate. Both derive functions now read the
field (clamped at zero), and the plugin descriptors expose it with
defaults so the configuration form carries it.
rust volume: gate the remaining admin RPCs behind check_grpc_admin_auth
The Go volume server gates 29 destructive VolumeServer RPCs on the
-whiteList admin check; the Rust port only gated 14. Add the gate to the
other 15 -- batch_delete, read_all_needles, fetch_and_write_needle, the
EC-shard generate/rebuild/copy/unmount/to-volume RPCs, both tier-move RPCs,
volume_copy, volume_tail_receiver, set_state, scrub_ec_volume and
volume_needle_status -- so a configured whitelist restricts them the same
way it already does on the Go side.
check_grpc_admin_auth also required peer info before checking whether any
control was configured, unlike Go's `if vs.guard == nil { return nil }`.
Short-circuit when no whitelist and no signing key are set, so in-process
callers keep working with security inactive and only the gate ordering
changes for configured servers.
tests/admin_auth_coverage.rs mirrors the Go coverage test: every handler
must either gate or be listed as intentionally open with a reason, so the
two implementations can't silently drift apart again.
* redis2: derive the orphan cleanup keys inside the helper
* redis2: skip orphan cleanup in super large directories
* redis2: detach orphan cleanup from the request context and log a failed restore
* redis2: keep a directory member whose child index is still live
* redis2: run restore-path tests under both key prefixes and fix the test harness
* redis2: check cleanup errors in tests
* fix(redis2): remove orphaned directory index members on listing
ListDirectoryEntries skipped index members whose value key was gone and
left them in the ZSET, so the per-directory child index grew without
bound under any TTL workload. Mirror the ZRem the logical-expiry branch
already performs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(redis2): keep the index member when a concurrent insert recreates the value
The orphan cleanup removed the member unconditionally, so an InsertEntry
landing between FindEntry and the ZRem left a live value with no index
member, invisible to listings until another InsertEntry on that path.
UpdateEntry does not re-add it, so the loss persisted.
Restore the member when the value is present again after the removal.
The value key and the directory index key hash to different slots, so a
Lua script or MULTI over both is not available to the cluster store.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The unmount+full-teardown of EC shards was duplicated: the plugin-worker EC
task had unmountAndDeleteEcShards and the shell had unmountAndDeleteEcShardsQuiet,
byte-identical apart from a fence parameter and a sentinel error. That
duplication is how the teardown fence semantics drifted between the two paths.
Distribute, mount and verify already live in weed/storage/erasure_coding and are
shared by both callers; move the teardown there too, as UnmountAndDeleteEcShards
plus the shared ErrFullTeardownNotAcked sentinel. Both paths now call the one
function, so the fence semantics cannot diverge again. The shell keeps a thin
type-converting wrapper and aliases the sentinel; behavior is unchanged.
* EC worker: clear stale/interrupted shards at task start and on failure
The EC encode task cleared stale shards from a prior interrupted encode only
at 55% progress (after mark-readonly, copy, and generate), and used a
generation-fenced teardown. Two gaps left orphan shards behind:
- a retried encode's prior attempt carries the same admin-issued encodeTsNs,
and the server's teardown fence preserves same-or-newer generations, so the
prior attempt's shards were never cleared;
- shards left by an interrupted distribute often have an unreadable .vif
generation (the sidecar never landed), which the fence also preserves.
Both survive the next volume-server restart as orphans and make detection
refuse the volume (Manual intervention required).
Move the cleanup to a Step 0 preflight that runs before any destructive step,
and switch it to the server's blanket (generation-independent) teardown -- the
same wipe the shell ec.encode pre-cleanup uses. The admin dedupe key already
prevents a concurrent newer encode of the volume, and the blanket path aborts
rather than clobber a live newer mount.
Add rollbackDistribute: a failure after distribute begins but before verify
commits the EC copy now tears down the shards it wrote and restores the sources
to writable, so a terminally-failed encode (a single-attempt job, or the last
of a retry series, which has no successor preflight) leaves nothing behind.
The preflight also rejects a plan with no targets or no source before marking
the source readonly.
* EC worker: reject malformed targets and keep source readonly on incomplete teardown
Address review feedback:
- ensureCleanEcStart only rejected an empty target slice; a target with an
empty Node (or no shard ids) passed the length check, was then silently
skipped by cleanupStaleEcShards, and let Execute mark the source readonly
with nothing to distribute to. Validate each target before the first
destructive step. Add regression cases.
- rollbackDistribute marked the source writable even when the shard teardown
returned an error, exposing a writable source beside stale (possibly mounted)
shards -- reads/writes could diverge and orphan cleanup will not remove a
writable source. On an incomplete teardown, leave the source readonly for the
next preflight or an operator to reconcile.
* shell: roll back a failed ec.encode instead of leaving readonly volumes and orphan shards
ec.encode marks the source volumes readonly and generates EC shards before it
verifies the shards and deletes the originals. If any step in between failed,
the command just returned the error: the volumes were left readonly and the
partially-produced EC shards survived as orphans, cleaned up only by the next
ec.encode run (via clearPreexistingEcShards) if the operator retried.
Add a deferred rollback that runs when the batch fails before the originals are
deleted: it tears down the EC shards produced this run and restores the sources
to writable, reusing the existing clearPreexistingEcShards and
markVolumeReplicaWritable helpers. Once the shards are verified recoverable the
batch is committed to the EC copy and does not roll back. Both rollback steps
are idempotent, so a failure before the volumes were marked readonly is safe.
* shell: re-read volume locations when restoring writable in ec.encode rollback
Address review: rollbackFailedEcEncode restored writable using the location
snapshot taken before doEcEncode, but doEcEncode re-reads locations and marks
every replica of that later snapshot readonly. A replica added or moved in
between would be left readonly. Re-read locations in the rollback and fall back
to the pre-encode snapshot only if the re-read fails.
* refactor: extract remote mount resolution into shared helpers
* refactor: share the adaptive remote cache wait policy
* filer: stream cold remote reads from the origin while caching
* s3: stream cold remote reads from the origin instead of 503 retries
* test: cover the S3 origin stream-through path
* remote mounts: match on path components and prefer the longest mount
* fail short origin streams instead of silently truncating
* s3: try the origin before failing a cold read on a local cache error
* s3: gate origin streaming on the entry's resolved version
* return the cache RPC's NotFound as a canonical status and classify it everywhere
* filer: keep multipart-range cold reads on the retry path
test: assert EC shard identity and empty-view, not just counts, in lifecycle
Follow-up to the multi-disk EC lifecycle tests (#10721), addressing review
feedback.
The phase checks compared shard counts. A reconcile that put a shard on the
wrong disk, or loaded a different shard than the file on disk, keeps 6/5/3
right while corrupting the mapping. Compare the exact registered shard set per
disk at every phase instead, via a shared assertRegistered helper. The
cross-disk mount phase now also pins that shard 0 landed on disk2 with the
existing shards, not merely that it is findable.
The sidecar-disk-lost scenario only logged the registered view, so a change
that registered shards without reachable sidecars would pass despite the
documented expectation that the view stays empty. It now asserts
countRegistered == 0: a registered-but-unreadable shard is worse than an
unregistered one, because the master advertises it.
The first store's closer is now deferred as a closure the moment the store is
created, so a Fatalf in an early phase no longer leaks it and its
notification-drainer goroutine; the closure reads the reassigned variable so it
also covers the post-restart store.
* plugin scheduler: drain started jobs past the window close instead of killing them
* plugin scheduler: never drain-cap an attempt below its declared estimated runtime
* plugin scheduler: cap estimated_runtime_seconds before the Duration conversion
* operation: add shared volume_move package for volume and EC shard moves
The shell commands (volume.move, volume.balance, ec.balance, tier moves)
and the maintenance workers (balance, ec_balance) each carried their own
copy of the move RPC sequences, and the copies had drifted: the worker
verified the target before deleting the source but dropped the disk
type and IO throttle; the shell passed those but deleted the source
unverified.
volume_move.Mover carries the merged sequences, keeping the stricter
behavior from each side:
- LiveMoveVolume: check-then-hard-freeze the source (VolumeStatus's
IsReadOnly also covers low-disk and readonly-but-can-delete states,
which still accept needle deletes), copy with disk type and IO
throttle, tail, verify the target is not behind the source before the
destructive source delete (a target that is ahead holds writes it
accepted during the tail and the move commits to keep them), and
restore the source's writability when a failure precedes the delete
and this move did the freezing. Aborts clean up the incomplete target
copy; a failed cleanup or an ambiguous source delete keeps the source
readonly (ErrSourceKeptReadonly) so callers do not thaw a source next
to a possibly-authoritative copy. With a readonly source, an existing
or unknown-state target refuses the move outright: no client-side
observation can prove such a copy is a stale remnant rather than the
authoritative copy of an unfinished move.
- MoveEcShards: copy with the .ecx/.ecj/.vif/.ecsum sidecars, mount,
verify the target registered every shard before unmount+delete on the
source, and reject same-server moves (the EC delete is server-wide).
Server identity is the grpc endpoint (SameServer), so node:8080 and
node:8080.18080 compare equal while test servers sharing a degenerate
HTTP address stay distinct; addresses are validated non-fatally before
dialing and before being embedded in copy/tail requests, since both the
client dialer and the receiving server normalize them through a parser
that aborts the process on a malformed port. The Rust volume server's
codes.NotFound counts as a definitively absent probe answer alongside
the Go server's plain-error code Unknown.
All RPCs go through an injectable ClientFunc, so the sequences are unit
tested against a fake volume server client: RPC order, request fields,
and that verification failures keep the source intact.
* shell, worker: delegate volume and EC shard moves to operation/volume_move
LiveMoveVolume and the copy/tail/delete/mark-writable helpers become
thin wrappers over the shared mover, keeping their signatures; the EC
helpers keep their per-step output and delegate the RPCs. BalanceTask
and ECBalanceTask keep their parameter validation, progress reporting,
and guards (same-node cross-disk rejection, dedup keep-node
verification, shard ids range-checked before the uint8 narrowing) and
hand the RPC sequences to the mover. volume.tier.move skips its
thaw-on-failure when the mover deliberately kept the source readonly,
since reopening the replicas beside a possibly-authoritative target
copy would fork the volume.
The tail-failure tolerance moves inside the mover: a failed tail is
tolerated only when the volume was already readonly before the move
began, backstopped by a stability re-read across the idle window, so
volume.balance's -skipTailError-by-readonly heuristic and tier-move's
unconditional skip both become the same authoritative rule.
* volume_move: keep the source readonly when a failed copy leaves a target of unknown origin
A failed copy can leave a complete, mounted copy on the target (the
server finishes after the client loses the stream). The abort probed
the target only when its pre-copy state was known-absent; an unknown
prior state skipped both the probe and the cleanup and then reopened
the source - two writable replicas of one volume, diverging from the
next write on.
The abort now probes the target on every failed copy and restores the
source only when the target provably holds nothing. A copy whose
provenance cannot be proven (unknown prior state, a pre-existing
replica, or an unreachable target) is never deleted, and the source
stays readonly with ErrSourceKeptReadonly naming the recovery.
* test: teach the plugin worker harness the shared move sequence
The fake volume server lacked VolumeStatus, which the shared mover now
issues before freezing the source, and the batch execution test's
status-read accounting predates the pre-copy target probe and the
verification reads. Mirrors the harness the enterprise tree already
carries.
The existing multi-disk EC integration test asserts on shard counts. Counting
cannot tell a healthy volume from one a repair reassembled out of the wrong
inputs — both have fourteen shards. This drives the real shell commands
(ec.encode, ec.balance, ec.rebuild) against a live three-node, four-disk
cluster and reads the stored bytes back after every step, so a rebuild that
produced fourteen plausible-but-wrong shards fails here.
An 8 KB random payload is stored, then encoded, balanced, damaged (two shard
files removed and the servers restarted so the master relearns the reduced
set from disk), and rebuilt. The rebuild output matches the shape of the
support case that motivated this — "rebuildOneEcVolume", "missing shard N.0",
"copied N.1 from ..." — and the payload is verified identical after each of
upload, encode, balance, shard loss, and rebuild.
Two ordering facts the test pins, both of which cost real debugging time:
ec.rebuild is driven by the master's topology, not disk truth, so shards must
be relearned (via restart) before a repair can target the right set; and the
shell lock is dropped when the restart disconnects the master, so it has to be
retaken before the rebuild.
A multi-disk volume server keeps one .ecx / .ecj / .vif set per volume on a
single disk while ec.balance scatters the shards across the others. Every EC
operation on such a node crosses that split: startup registration, balancing
the sidecar disk's shards away, rebooting in that state, and mounting a shard
delivered to a disk that has no local sidecars.
Each of those transitions is handled by a different mechanism (per-disk scan,
cross-disk reconcile, mount-time .ecx lookup), individually tested but never
as the sequence a production node actually lives through — where the output
state of one transition is the input of the next. A regression in any hop
shows up as shards that exist on disk while the master's view says otherwise,
and every topology-driven repair then works against the wrong shard set.
The layout, volume id and collection mirror a support case. The second test
pins the failure floor when the sidecar disk itself dies: shards on the
surviving disks may drop out of the registered view, since nothing can read
them without the .ecx, but their files must survive so restoring the sidecars
restores the volume.
* mount: invalidate hot directory listings by section
A cached directory used to be dropped whole when it saw 64 changes in
2s: with a continuous writer the listing cycled through wipe, direct
listing and full rebuild for as long as the writer kept going, and
every sibling lookup fell through to the filer in between.
Split each cached listing into name-range sections of 1024 entries. A
burst of foreign changes invalidates just the section it lands in;
entries stay served and events keep applying, and the next readdir
re-lists only that range from the filer, reconciled through the version
gate so it cannot roll back newer applied events. Lookups in an
invalidated section read through until then. The mount's own writes no
longer invalidate anything: they are ground truth for its cache.
* meta_cache: drop the version floor with a deleted or moved directory
The other teardown paths already clear both maps; a floor left behind
here would fence the listing of a directory re-created at the same
path.
* mount: harden section refresh
An unversioned listing (pre-upgrade filer) now only fills gaps instead
of reconciling: without a snapshot to order against, an overwrite or
the deletion sweep could roll back an event applied after the listing.
The section table can be rebuilt or re-split between the listing and
its apply, so the refresh only marks fresh or splits when the section
still covers the range it read. Splicing bounds from a stale range
into a rebuilt table could leave them unsorted.
Bound the wait: a readdir gives a refresh five seconds before serving
the maintained-but-unverified cache. Bound the size: a range grown
past four sections aborts the refresh and drops the directory cache,
re-tiling it with a full rebuild, with that request served direct.
Cover the filer-facing path with a listing server: paging with the
snapshot pinned across pages, the section cutoff, no calls for a
fresh section, and the overgrown-range abort.
* meta_cache: make the section table a self-contained state machine
Churn counting, freshness, stale-range scanning and the refresh
completion with its guard and split now live on dirSections itself,
free of the lock, the store and the apply loop, so they test directly
with synthetic clocks and tables. MetaCache keeps thin wrappers that
hold its mutex and find the directory's table.
* meta_cache: keep section internals out of the apply request
The request now carries the completed build's table and one refresh as
opaque values built by section code, and the boundary-derivation rule
moves out of the build loop into a collector next to the rest of the
section logic.
* mount: fence refreshed sections with a snapshot floor
A refresh versioned the entries it fetched and tombstoned the ones it
swept, but a name absent from both cache and listing kept the old
directory floor, so a delayed event between the two snapshots could
resurrect it into a section already marked fresh. The section now
carries its own floor, consulted next to the directory floor, covering
every name in the range, present or absent — which also retires the
refresh's per-entry version stamps and sweep tombstones.
An unversioned listing sets no floor and vouches for nothing: it may
still fill gaps, but the section stays stale and reads through until a
filer that stamps snapshots re-validates it.
A listing's reach is unknowable up front — a resumed handle can skip
far ahead, and shrunken sections let one batch span many — so a
readdir now re-validates every stale section from its start name to
the end of the directory instead of the next two.
* mount: fence tombstoned names with floors and gate the reconcile
A tombstone answered for its name before the floors were consulted, so
one at an old position let through events the newer listing floor
should have fenced; a build never hit this because it prunes
superseded tombstones, which a section refresh does not. The version
gate now raises a tombstone to the floors like any other record.
With no per-entry versions, only the section floor fences a
reconcile's work, so a range the rebuilt or re-split table no longer
has must not touch the store either: the range check moves ahead of
the mutations, under the same lock the floor install holds.
An unversioned refresh no longer retries: the section is remembered as
unverifiable and skipped by the stale scan, or every batch of every
readdir would re-list the same ranges against a filer that cannot
vouch for them.
* mount: clear beaten unversioned markers and skip refresh mid-build
An unversioned marker outliving the snapshot write that replaced its
content bypassed the section floor the same way an old tombstone did,
letting a delayed pre-snapshot event roll the entry back. The refresh
now clears the marker when its write wins; pinned local-only entries
are not replaced at all, keeping their content and marker.
A rebuild wipes and repopulates the store off the apply loop, so a
refresh reconciling meanwhile could sweep children the build had
already inserted and let it publish the directory incomplete. The
refresh now skips a building directory, as events (buffered) and
purges (skipped) already do; its staleness dies with the build's
fresh table.
* mount: clear the unversioned marker only after its replacement lands
Clearing before the insert meant a failed write left the old local
content claiming the listing floors, fencing the very events that were
still entitled to correct it.
* meta_cache: rename the section state machine to sectionList
dirSections named both the type and the map of them.
* mount: raise the default cacheDirMaxEntries to 100000
The low ceiling guarded against whole-listing rebuild churn: a big
cached directory under writes kept re-streaming everything. Sectioned
invalidation ended that — a burst now costs one range listing — so the
remaining cost of caching a large directory is its one-time build,
comparable to the single direct listing that read-through mode pays on
every enumeration instead.
* meta_cache: cover section border and edge cases
A bound-named entry belongs to the section starting at the bound: the
neighboring refresh's sweep stops before it, its own section's covers
it. Churn past everything the build saw lands in the tail section, a
rename spanning two sections invalidates both, and a listed entry at
the section's end name is cut off with the ones beyond it.
* ec: confirm a surviving copy before deleting a duplicate EC shard
The dedup phase of EC balancing removes a shard it believes exists elsewhere.
It copies nothing first, so the shard surviving on another node is the only
thing that makes the delete safe -- and it took the plan's word for that.
The plan is built from the master's topology, which can name a location that
holds nothing: such a server answers "CopyFile not found ec volume id N" when
something later tries to read the shard there. A shard listed on a phantom
location and on a real one looks duplicated, so dedup deletes one of them. When
it picks the real one the last copy is gone, and the job reports success -- the
loss only surfaces later, as a rebuild that cannot assemble enough shards.
The move phase already refuses to work on trust: it verifies the shard
registered on the destination before removing the source. Dedup now holds to
the same standard. The planner records which node it chose to keep, and both
executors -- the worker task and the shell's ec.balance -- confirm that node
really holds the shard before deleting. A keep node that cannot be queried is
unknown rather than confirmed, and blocks the delete.
Tests drive the destructive path against an in-process volume server that
tracks what is actually on disk separately from what the plan claims, which is
the distinction the bug turns on. Without the guard, two of them fail by
deleting the only copy and returning success.
* ec: check the collection and bound the wait when confirming a survivor
Two gaps in the dedup survivor check.
The inventory RPC is keyed by volume id alone, so a server holding the same
number for a different collection answers "yes, I have that shard" to a
question about this one. Accepting that deletes the last real copy on the
strength of an unrelated volume. The response already carries the collection,
so verify against it rather than widening the RPC.
The shell path also queried on a background context, so a keep node that
accepts the connection but never answers would hang the whole balance run
instead of reporting that the survivor could not be confirmed. Bound it.
The check moves into VerifyShardsOnServer next to the existing helper, shared
by both executors, so the two paths cannot drift.
* remote_storage: build S3-compatible clients through one constructor
The eight non-s3 S3-SDK providers each duplicated the AWS session setup
and only the s3 maker could take a custom *http.Client. Route every
S3-compatible type (s3, wasabi, b2, aliyun, tencent, baidu, filebase,
storj, contabo) through MakeWithHTTPClient with a single options table,
and add S3CompatibleEndpoint so callers can resolve the endpoint a given
type dials. No behavior change.
* volume: apply the remote-endpoint check to all S3-compatible providers
FetchAndWriteNeedle validated the endpoint and used the pinned dialer only
for type "s3". Every S3-SDK backend (wasabi, b2, aliyun, tencent, baidu,
filebase, storj, contabo) dials a caller-supplied endpoint through the same
client, so gate on S3CompatibleEndpoint to apply the same check uniformly.
-volume.allowUntrustedRemoteEndpoints still opts out.
* volume: don't route the guarded remote-endpoint client through a proxy
The guarded client exists to dial the validated endpoint directly and
re-check the resolved IP at connect time. With http.ProxyFromEnvironment
set, the dialer only validates the proxy's address while the proxy
re-resolves the endpoint host, which reopens the rebinding window. Drop
the proxy on this path; operators that need one can opt out with
-volume.allowUntrustedRemoteEndpoints.
FileCount and DeleteCount were int, so each cost a word on every replica the
master holds. A volume caps at 30GB on a 4-byte-offset build and 8TB on a
5-byte one, and neither holds 4.29 billion needles.
That takes VolumeInfo from 120 bytes to 112, which is its own size class rather
than rounding up into the 128 one, so a replica costs 135.7 bytes in the map
instead of 151.7 -- about 25MB across the 1.6M replicas in a cluster the size
of the one this came from.
Counts are narrowed where they are read rather than assigned across, so a
report claiming more than a volume can hold pins at the ceiling instead of
wrapping to a small number.
* filer: stop silently dropping metadata replay failures from peers
When two filers do not share a store (e.g. one leveldb3 per pod), each
subscribes to its peers' metadata streams and replays their events
locally (meta_aggregator.go's maybeReplicateMetadataChange, wired into
doSubscribeToOneFiler). A failed Replay() was logged and then treated
as done anyway: processEventFn always returned nil regardless of the
replay outcome, and processOne advanced lastTsNs unconditionally. The
offset is the only record of subscription progress, so a dropped event
was gone for good - no retry, and nothing else ever observed it.
An entry that fails to replay this way diverges from its peer
permanently. This is how a bucket's quota (entry.Quota, carried on
peer events like everything else - see entry_codec.go's EqualEntry
comparing Quota, and FromPbEntry copying it in entry.go) can end up
different across filers indefinitely: one replay hiccup on one filer,
and its enforcement and any metric reading its own store diverges from
the others' with no signal anything went wrong.
Fix: replicateMetadataChange now retries a failure with util.Retry,
which already distinguishes transient errors (timeouts, connection
resets, throttling, ...) from everything else and bounds the backoff.
That covers the common case - a busy store, a blip talking to a
remote-backed backend - without changing behavior when replay
succeeds. An error that is not transient, or outlives the retry
budget, is not retried further: propagating it so the offset never
advances would stall this peer's entire stream behind one event that
may never replay, which is worse than the one entry staying stale.
Instead it is skipped, loudly - counted in a new
stats.FilerMetaAggregatorReplayFailures metric and logged at error
level - so the divergence is discoverable instead of silent.
Tested: go build ./... and go test ./weed/filer/... ./weed/server/...
Added meta_aggregator_replay_test.go: one test fails against the old
one-shot Replay call (a single transient failure is never retried, so
the store never converges) and passes with the fix; a second covers a
permanently-failing event completing quickly and being counted instead
of retried forever.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* filer: keep the test quota constant int64 for 32-bit builds
An untyped shift constant passed to t.Fatalf's ...any defaults to int and
overflows on 32-bit, failing go vet there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* filer: name the diverged entry in the give-up log line
event.Directory is only the parent (typically /buckets), so for any
directory with more than one child the previous log line could not say
which entry failed to replay - the exact thing the change exists to
make discoverable. Name comes from NewEntry, falling back to OldEntry
for deletes; both getters are nil-safe.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* chore(filer): trim metadata replay comments to the non-obvious why
Compress the added comments on replicateMetadataChange and its tests down to
the reasoning a maintainer cannot get from the code: why a retry-exhausted
failure is skipped rather than propagated, what the old one-shot Replay body
did that the test pins, and why the quota constant is typed int64. Drops
deployment-specific narration and restatement of the code. No behaviour change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* chore: restore load-bearing clauses trimmed in the comment pass
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* filer: document and test the multi-step DeleteEntry replay hazard
CodeRabbit flagged that FilerStoreWrapper.DeleteEntry skips the delete
once FindEntry reports the path already gone, and that the redis store
families remove the primary key before parent-directory membership.
Chained together, a delete that fails between those two steps is
retried as a no-op: the stale membership is never revisited, and
replicateMetadataChange now reports overall success for it without
incrementing FilerMetaAggregatorReplayFailures, whereas before this PR
every such failure was unconditionally logged. The underlying store
inconsistency is pre-existing (a single non-retried Replay already
leaves the same stale membership behind); what retry adds is that this
one case no longer surfaces it.
Making Replay atomic or teaching every store to repair secondary
mutations on retry is out of scope here. Instead: document the hazard
at Replay, filerstore_wrapper.go's DeleteEntry, and
replicateMetadataChange, and add a test against the real
FilerStoreWrapper (not a strawman) that pins down the current,
documented behavior.
* filer: trim replay retry comments and tests
Drop the comment-only hunks documenting the pre-existing DeleteEntry
partial-failure hazard, the test that asserted that hazard still exists,
and the second hand-rolled fake store. Reuse stubFilerStore for the two
retry tests.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* admin: make paths relative
* admin: make filer browser link and nav path checks prefix-relative
* admin: add isCurrentPath and currentPathStartsWith helpers
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* benchmark tool for mounted filesystems
* ci: on-demand mount benchmark, native WinFsp vs rclone plus a Linux reference
* windows mount: let the Windows cache manager cache file data
WinFsp only turns the cache manager on for a file when FileInfoTimeout
is infinite; at any finite value every application read and write is a
synchronous trip into the mount process at whatever size the application
issued. Metadata events already reach FspFileSystemNotify, which purges
a changed file's cached pages and attributes, so an infinite timeout
stays coherent. The dir listing, volume info and EA timeouts are pinned
to one second so they do not silently inherit the infinity.
* windows mount: cache resolved paths and attributes in the adapter
WinFsp addresses every operation by path and has no FORGET, so the
adapter walked the whole path through Lookup on each one, and in a
directory the filer has not listed yet every walk was a filer round
trip; nothing played the part of the kernel's dentry and attribute
caches. The path cache owns one lookup reference per entry the way the
kernel holds one until FORGET, serves attribute reads for files without
an open handle, and is purged by the mount's own mutations and by
metadata events, with the timeout as backstop.
* windows mount: keep a closed file's attributes cached
Open steals the path's cache entry for its handle and Release returned
the reference with a purge, so the stat that follows every copied file
walked to the filer again. Reading the handle's final attributes before
it goes away and moving the reference back into the cache serves that
stat locally, the way the kernel's attribute cache does after a close.
Only if the path still names that inode, though: WinFsp reports the
path the handle opened with, and after a delete-on-close or a rename
caching it would resurrect an entry that is gone.
* windows mount: persist entries at create, and let the flush stay at close
WinFsp posts the cleanup and close that carry the flush after
CloseHandle has returned, so deferring the filer entry to the flush let
everything that reads through the filer race an unflushed close: a
listing missed just-written files, and a directory rename moved a
directory on the filer before its newest child existed there, leaving
the straggler flush to recreate the child under the dead path.
Flush-at-cleanup is not the answer either: it makes every handle's
cleanup flush, and those flushes race the unlinks of delete-on-close,
re-inserting the entry the unlink just removed. Persisting the entry at
create takes the ordering question away.
* mount: flush written pages before a truncate shrinks past them
The shrink trims chunks, but written pages that have not become chunks
yet are invisible to it, so the next flush wrote them back and the file
grew again, resurrecting the truncated bytes. Windows hits this on
every write-then-shrink because its flush runs after CloseHandle, but
the gap is platform-neutral.
* mount: order a file's unlink against its in-flight flush
Unlink set the handle's deleted flag bare, so a flush already past its
own check of that flag wrote the entry back right after the delete
removed it, and a delete-on-close file outlived its last handle. The
flag is now set under the handle's flush lock and re-checked under it,
so a flush either completes before the delete or sees the flag and
skips. An eagerly created handle also starts clean: the dirty mark
existed to make the deferred filer create happen at flush, and eager
creates have nothing to flush.
* shell: volume.move restores source writability when aborted after the copy phase
* shell: volume.move removes the incomplete target copy when aborted before the source delete
* shell: give each abort cleanup RPC its own timeout
* filer: TUS creation accepts Upload-Concat partial uploads
* filer: TUS final uploads concatenate completed partials
* filer: TUS concatenation tests
* filer: consumed marker pins TUS chunk ownership on completion
* filer: TUS session delete decides chunk ownership after removing the session info
* filer: TUS completion persists the consumed marker before creating the entry
* filer: TUS completion re-verifies the session after persisting the consumed marker
* filer: serialize TUS session ownership transitions per filer
* filer: surface failed TUS consumed-marker rollbacks
* s3: a null object wins over a rescan when the latest-version pointer is absent
The read path already resolves an absent pointer this way; the listing-path
counterpart scanned .versions/ first and could surface an old version or
delete marker over the current suspended-versioning null object.
* s3: dedup a key against its .versions sibling in suspended buckets too
A suspended bucket keeps its .versions directories, so a suspended-versioning
null object and its .versions sibling emitted the same key twice.
* s3: retract a null object from the listing when a delete marker shadows it
Deleting a key whose null version predates versioning leaves the base-path
entry in place and records the delete marker under <key>.versions. The
listing appended the base-path entry and relied on the .versions sibling to
replace it, but a delete-marker current version emitted nothing, so the
deleted key stayed visible to ListObjects while GET and HEAD returned 404.
* s3: keep a key's .versions sibling on the same page as the key
When the page quota ran out between a base-path entry and its .versions
directory, the page ended with the stale entry and the next page skipped the
directory as a marker echo, so the replacement or retraction never happened.
* s3: the null version is not latest when the .versions pointer names a newer one
ListObjectVersions stamped IsLatest on every base-path null object, so a key
deleted after enabling versioning reported IsLatest on both the delete marker
and the null version.
* s3: test listing after a pre-versioning null object is delete-marked
* s3: find a key's earlier page entry by scan, not by adjacency
A key such as k.bak sorts between k and k.versions, so the entry a .versions
sibling replaces or retracts is not always the last one on the page. Scan
back through the page for the key, and insert a late resolution in sorted
position instead of at the end.
* s3: settle trailing null objects by lookup when a page fills
The quota can run out while keys still sit between a null object and its
.versions sibling, and the sibling-adjacent page-boundary exception never
fires for those. Track the trailing null objects whose sibling has not been
ruled out and look each one up before declaring the page full; a retraction
reopens the quota.
* s3: do not resolve a .versions sibling its page has already moved past
A page resuming from a marker inside the base key's extension region has
already listed and settled the base null object on an earlier page, so
resolving the .versions directory again re-emitted the key.
* s3: test listing with keys between a null object and its .versions sibling
* s3: pick the newer of the null object and the scanned versions
Making the null object win outright whenever the pointer is absent misread
multi-filer pointer lag: version files replicate ahead of the pointer, and a
key overwritten or delete-marked after pre-versioning days would list its
stale null again. The suspended-versioning write that legitimately makes the
null current is also the newer entry, so mtime tells the two apart.
* s3: a delete-marked null object no longer keeps its prefix alive
The hidden-entries probe took any plain file as proof of a listable key, but
a null object shadowed by its .versions sibling's delete marker is not one.
Hold plain files pending until the sibling settles them either way.
* s3: settle an evicted pending null instead of dropping it
Nested keys like k, k!, k!! can hold more pending nulls than the cap. A
silently evicted one could close the page unsettled, and the resume skip
would then keep the stale entry for good.
* s3: test deleted-prefix hiding and the pending-null cap
* s3: cover the reported '!' intervening key with a live version
* s3: an unstamped same-second version outranks the null object
Second-resolution mtimes cannot order same-second writes, so the tie went to
the stale null when the pointer lagged. The suspended write that makes a null
current stamps the version it displaces before clearing the pointer, so the
stamp is the authoritative signal and a tie without it goes to the version.
* s3: a pointer-less versions listing still checks what replicated
ListObjectVersions took a missing pointer as proof the null object is latest,
but under pointer lag the sibling can already hold newer replicated versions
or markers. Apply the same nullObjectWins rule as the listing recovery.
* s3: a failed null-object settlement fails the listing
Every getEntry error read as a missing sibling, so a transient filer error at
a page boundary committed the unsettled null and the next page skipped its
sibling for good. Only a definitive not-found means the null is live; other
failures are retained on eviction and fail the request at page close.
* s3: retract a CommonPrefix whose only backers were delete-marked nulls
The directory probe settles this for the / delimiter, but any other delimiter
derives prefixes from base-path keys directly, and a prefix built solely from
null objects survived their delete markers. Count the unsettled null backers
behind the newest prefix and retract it when the last one settles as a marker;
a live resolution or any listable contributor confirms the prefix instead.
* s3: test custom-delimiter prefix retraction
* s3: an explicit signal marks the null object current, not the demotion stamp
The NoncurrentSinceNs stamp survives promotion: delete the version that
demoted another and the promoted one is current yet still stamped, so a
lagging replica would resurrect the stale null. A suspended-versioning write
now records Seaweed-X-Amz-Null-Version-Is-Latest on the .versions directory
when it clears the pointer, every pointer update removes it, and the
recovery paths trust the signal instead of the stamp.
* s3: a filer failover retry rebuilds the listing page from scratch
The failover wrapper reruns the callback on another filer after a transport
error, and the partially built page, spent quota, and advanced marker leaked
into the retry, which could then return a stale or duplicated page as
success.
* s3: only a prefix's own backers can debit it
A delete marker for a version-only key (no base object) derived the same
prefix as its neighbors and decremented backing it never contributed,
retracting a prefix that a live null object still backed. Track backers by
key so settlement is idempotent and only debits what was counted.
* s3: test a version-only marker against a null-backed prefix
* s3: a pointer recompute clears the null-current signal
The routed finalize for delete markers, COPY, and multipart rewrites the
.versions pointer through RECOMPUTE_LATEST, which left a suspended-era
null-current signal in place. Version files never carry the signal, so
mapping it in CopyExtended deletes it whenever the pointer recomputes.
* s3: the pointer outranks the null-current signal in the versions listing
The signal check guarded the pointer check, so a stale signal a recompute
had not cleared yet would have let the null claim IsLatest alongside the
pointed-at version.
* topology: name a node's volumes without copying them
ToVolumeLocations reads a volume id off every volume in the cluster, and got
there through GetVolumes, which copies a whole storage.VolumeInfo per volume to
be read for four bytes of it. Every client that connects asks for this.
At 800k volumes the walk goes from 94.6MB to 16.0MB, which is the ids
themselves.
* master: log why a client send failed, not what was sent
The message names every volume on a newly connected node, so a client going
away had the master format a protobuf that size into text -- through the one
log level that is always on. The error is the part worth having.
Go stores a map value inline once it fits in 128 bytes, and indirectly above
that. Shrinking storage.VolumeInfo to 120 bytes therefore moved 1.6M of them
into the map's own slots, which are allocated to capacity rather than to
occupancy, and the map grew by 149MB even though each volume got smaller.
Hold pointers, so the size of a volume record stops deciding how it is stored.
Updates are written through the pointer already there, so a heartbeat that
changes a volume allocates nothing, as it did when the struct was above the
threshold and Go was doing this itself.
At 800k volume ids across 3 servers the resident topology goes from 705.3MB to
546.6MB, with heartbeat cost unchanged at 111.6MB.
* master: stream volume listings
A listing of 800k volumes is 36MB on the wire but 305MB as messages, and the
master built all of it, then held it while grpc encoded it. Two of those at
once is most of a small master's heap, and the maintenance scanner asks every
30 minutes.
The topology goes out first, listing nothing, then its volumes in batches, so
the master holds a batch rather than a cluster: 341MB of live heap for one
listing becomes 4.4MB. It allocates much the same either way -- what changes is
how much of it has to be live at once, which is what sets the heap ceiling.
Batches are built under their disk's lock and sent outside it, so a slow reader
stalls the stream rather than the topology. They therefore do not share one
instant, which a single listing did not either: it takes each disk's lock in
turn, so a volume moving during either can be seen twice or not at all.
The client helper hides which kind of master answered: one too old for the
stream is asked the old way and its reply cut into the same batches. Either way
the topology handed over lists no volumes, so a caller cannot come to depend on
finding them there.
* admin: stream the listing the maintenance scan reads
It asks for every volume in the cluster every 30 minutes. Reassembling it
client-side keeps the scan identical -- ActiveTopology splits disks by the
disk ids on the volumes, so it needs them in the topology -- while the master
no longer builds the whole reply to send it.
* topology: report a disk id that does not depend on map order
A topology disk that fronts several physical disks took its reported id from
whichever volume the map yielded first, so two listings of an unchanged disk
could disagree. Take the smallest instead.
* topology: test that a streamed listing rebuilds to the whole one
The callers that stream now rebuild the listing from a topology sent without
volumes plus the batches after it, so that has to come out the same as being
sent it whole, at every batch size and under a filter.
* clients: stream the volume listings that ask for everything
The dashboard's list and export pages, the collection and ec shard pages, the
topology view, the worker metrics and two shell commands each asked the master
to build all 800k volumes into one reply. They read the same listing as before,
rebuilt on their side, so the master no longer holds it.
The three that already ask for one volume or one collection stay as they are:
their replies are small, and streaming one costs a round trip to say so.
* master: stream volume listings
A listing of 800k volumes is 36MB on the wire but 305MB as messages, and the
master built all of it, then held it while grpc encoded it. Two of those at
once is most of a small master's heap, and the maintenance scanner asks every
30 minutes.
The topology goes out first, listing nothing, then its volumes in batches, so
the master holds a batch rather than a cluster: 341MB of live heap for one
listing becomes 4.4MB. It allocates much the same either way -- what changes is
how much of it has to be live at once, which is what sets the heap ceiling.
Batches are built under their disk's lock and sent outside it, so a slow reader
stalls the stream rather than the topology. They therefore do not share one
instant, which a single listing did not either: it takes each disk's lock in
turn, so a volume moving during either can be seen twice or not at all.
The client helper hides which kind of master answered: one too old for the
stream is asked the old way and its reply cut into the same batches. Either way
the topology handed over lists no volumes, so a caller cannot come to depend on
finding them there.
* admin: stream the listing the maintenance scan reads
It asks for every volume in the cluster every 30 minutes. Reassembling it
client-side keeps the scan identical -- ActiveTopology splits disks by the
disk ids on the volumes, so it needs them in the topology -- while the master
no longer builds the whole reply to send it.
Volume growth registers a provisional record before it can know which
directory the server chose, while the server's own report -- pushed
during the AllocateVolume RPC -- carries the real disk id. The merge is
last-writer-wins, so whichever lands second sticks, and fresh volumes
nondeterministically show disk 0 on multi-dir servers. Keep the reported
disk id when the provisional update carries none, before the report
digest is computed so the stored record stays consistent with what the
server keeps reporting.
Claude-Session: https://claude.ai/code/session_01QdTEEPbg4MtcoEGwqbgtZC
The volume server names the directory index in every
VolumeShortInformationMessage, but NewVolumeInfoFromShort dropped it, so
volumes registered through the incremental new-volume path showed
disk_id 0 at the master until a full report -- misreporting multi-dir
servers in volume.list and the per-physical-disk topology views.
Claude-Session: https://claude.ai/code/session_01QdTEEPbg4MtcoEGwqbgtZC
* volume: decode IPv6 transition addresses in the remote-endpoint guard
checkBlockedIP normalized only ::ffff: mapped IPv4, so NAT64 (64:ff9b::/96),
6to4 (2002::/16), Teredo (2001:0000::/32), and IPv4-compatible (::/96) addresses
that embed an internal IPv4 (loopback, 169.254.169.254, RFC 1918) passed the
endpoint guard even though the plain IPv4 forms are refused. Extract the
embedded IPv4 from those forms and re-check it against the deny list, which
covers both the up-front validation and the dial-time guard. Mirrored in the
Rust volume server.
* volume: require the full NAT64 well-known prefix before decoding
Only 64:ff9b::/96 carries the embedded IPv4 in the low 32 bits, so also require
bytes 4-11 to be zero before treating an address as NAT64; other 64:ff9b:
prefixes place the IPv4 elsewhere and are left untouched. Add public-target
coverage for 6to4, Teredo, and IPv4-compatible so every decoder is exercised on
both a blocked and an allowed destination. Mirrored in the Rust volume server.
* master: let VolumeList ask for the volumes it wants
The request carried nothing, so every caller was answered with the whole
cluster. A dashboard opening one volume's page, or a capacity probe adding up
one bucket, was served all 800k of them and threw away the rest -- and the
master built every one of those messages first.
The topology, its disks and their counters are still reported in full: a caller
reading free space or replica placement needs the cluster whichever volumes it
asked about. Only what is listed under a disk is selected, ec shards included.
An empty collection and a zero volume id take everything, the way volume.list
already reads its own -collectionPattern and -volumeId, so a caller that
forgets to narrow is answered too much rather than answered wrongly. That
leaves the default collection unnameable, since it is the one the empty string
names, so it gets a field of its own.
An older client sends none of it and is answered exactly as before.
* admin: ask the master for the volume the page is showing
A volume's detail page was pulling every volume in the cluster to find one and
its replicas, and discarding the rest.
* admin: ask the master for the ec volume the page is showing
Same as the volume detail page: one volume's shards were found by pulling every
ec shard in the cluster.
* s3: ask the master for the bucket's own collection
The SOSAPI capacity probe summed one collection's volumes out of a listing of
every volume in the cluster. Cluster capacity still comes out the same: it is
read from the disk counters, which a filtered listing reports in full.
* topology: read the disk usage counters atomically
They are written with atomic.AddInt64 from heartbeats but were read plainly by
the two listings and by FreeSpace, and the map they sit in was iterated without
the lock its neighbour takes. Under -race a listing concurrent with a heartbeat
trips on both.
A master decides nothing from it. Every caller that read it was asking whether
a volume is remote, which the backend name answers, and the value itself is
reported on demand by the server holding the volume, through the volume info in
ReadVolumeFileStatus.
It is also the one string here that cannot be shared: unique per volume, so
unlike the collection and backend names it carries its own characters for every
volume a master tracks.
VolumeInfo goes from 136 bytes to 120. 800k volumes registered from a heartbeat
that has been over the wire go from 214 to 163 B/volume when tiered.
The volume server's own status page keeps showing the key, now read from the
volume it holds rather than relayed through a master, which is also where the
other volume server implementation reads it.
The heartbeat digest drops it on the same grounds: a change to something the
master does not hold cannot make its copy stale. Both implementations and their
shared vectors move together, and the field-coverage test now names what is
deliberately not retained rather than being loosened.
* storage: order VolumeInfo by alignment
The struct is held for every volume replica in the cluster, so the padding the
compiler inserts is multiplied by however many volumes a master tracks. Two
one-byte fields each sat at the head of a word and left the rest of it empty,
which was ten of the eighteen wasted bytes.
Grouping by size rather than by meaning takes the struct from 152 bytes to 136,
and the map holding them shrinks with it, since a Go map's slack scales with
the size of the value.
800k volumes registered from a heartbeat that has been over the wire:
211 -> 195 B/volume, 214 -> 198 tiered.
* trim the comments on this change to the parts that are not evident
* topology: build the volume list without copying the volume map first
ToDiskInfo copied every VolumeInfo on the disk into a fresh slice, walked it to
build a protobuf message for each, and threw the copy away. The copy was as
large as the messages it produced.
Building them straight from the map holds the disk's read lock for the walk
rather than just the copy, so a heartbeat updating that disk waits for it. It
is a read lock on a call that is now infrequent, against an allocation of the
same size as the response.
ToTopologyInfo over 550k volumes 193617502 B/op -> 110011017 B/op, and faster
for not making the copy.
* trim the comments on this change to the parts that are not evident
* ec.balance: add a -volumeIds filter
Collection scope is often too broad for maintenance. -volumeIds narrows the
plan to the given ec volume ids by leaving every other volume out of the
topology handed to the planner, so no phase, dedup included, can plan against
them. Ids with no ec shard in the selected collection, dataCenter and disk type
are rejected rather than silently skipped.
* ec.encode: key the orphan sweep without narrowing the volume id
int is 32-bit on 32-bit builds, so int(vid) wraps for volume ids above
MaxInt32. Format the id as the uint32 it is.
2026-08-09 09:37:49 -07:00
1242 changed files with 118641 additions and 32807 deletions
org.opencontainers.image.description=SeaweedFS is a distributed storage system for blobs, objects, files, and data lake, to store and serve billions of files fast!
org.opencontainers.image.description=SeaweedFS is a distributed storage system for blobs, objects, files, and data lake, to store and serve billions of files fast!
foreach ($f in 'C:\seaweed-mount.log','C:\seaweed-mount.err.log','C:\seaweed-remount.log','C:\seaweed-remount.err.log','C:\seaweed-remount2.log','C:\seaweed-remount2.err.log','C:\seaweed-dirmount.log','C:\seaweed-dirmount.err.log','C:\seaweed-mini.log','C:\seaweed-mini.err.log') {
echo "::warning::RELEASE_PAT is not set. The tag will be pushed with GITHUB_TOKEN, so the binary, container and helm workflows will not start on their own."
fi
- name:Compute new version
id:compute
@@ -99,17 +134,144 @@ jobs:
sed -i -E "s/^version:.*/version: ${CHART_VERSION}/" "$CHART"
# The suite expects RGW's 400 InvalidPart for a partNumber past the last
# part; AWS answers 416 InvalidPartNumber, which is what we return.
sed -i "/# request PartNumber out of range/,+4{s/assert status == 400/assert status == 416/; s/assert error_code == 'InvalidPart'/assert error_code == 'InvalidPartNumber'/}" s3tests/functional/test_s3.py
# The suite expects RGW's 400 InvalidPart for a partNumber past the last
# part; AWS answers 416 InvalidPartNumber, which is what we return.
sed -i "/# request PartNumber out of range/,+4{s/assert status == 400/assert status == 416/; s/assert error_code == 'InvalidPart'/assert error_code == 'InvalidPartNumber'/}" s3tests/functional/test_s3.py
One `weed` binary serves an S3 object store, a POSIX file system, and a lakehouse with S3 Tables, all over the same data. Each blob is one disk read away, capacity grows by starting another volume server, and cloud storage can be cached or tiered transparently.
- [Download Binaries for different platforms](https://github.com/seaweedfs/seaweedfs/releases/latest)
- [SeaweedFS on Slack](https://join.slack.com/t/seaweedfs/shared_invite/enQtMzI4MTMwMjU2MzA3LTEyYzZmZWYzOGQ3MDJlZWMzYmI0OTE4OTJiZjJjODBmMzUxNmYwODg0YjY3MTNlMjBmZDQ1NzQ5NDJhZWI2ZmY)
- [SeaweedFS on Twitter](https://twitter.com/SeaweedFS)
- [SeaweedFS on Telegram](https://t.me/Seaweedfs)
- [SeaweedFS on Reddit](https://www.reddit.com/r/SeaweedFS/)
- [SeaweedFS White Paper](https://github.com/seaweedfs/seaweedfs/wiki/SeaweedFS_Architecture.pdf) and introduction slides: [2025.5](https://docs.google.com/presentation/d/1tdkp45J01oRV68dIm4yoTXKJDof-EhainlA0LMXexQE/edit?usp=sharing), [2021.5](https://docs.google.com/presentation/d/1DcxKWlINc-HNCjhYeERkpGXXm6nTCES8mi2W5G0Z4Ts/edit?usp=sharing), [2019.3](https://www.slideshare.net/chrislusf/seaweedfs-introduction)
Table of Contents
=================
* [Quick Start](#quick-start)
* [Quick Start with weed mini](#quick-start-with-weed-mini)
* [Quick Start for S3 API on Docker](#quick-start-for-s3-api-on-docker)
* [Introduction](#introduction)
* [Features](#features)
* [Additional Features](#additional-features)
* [Filer Features](#filer-features)
* [Example: Using Seaweed Object Store](#example-using-seaweed-object-store)
* [Architecture](#object-store-architecture)
* [Compared to Other File Systems](#compared-to-other-file-systems)
* [One command](#one-command)
* [Docker](#docker)
* [Docker Compose](#docker-compose)
* [Kubernetes with Helm](#kubernetes-with-helm)
* [Build from source](#build-from-source)
* [Scale out](#scale-out)
* [Why SeaweedFS](#why-seaweedfs)
* [Fast](#fast)
* [Scalable](#scalable)
* [The most complete S3 API](#the-most-complete-s3-api)
* [A data warehouse with S3 Tables](#a-data-warehouse-with-s3-tables)
* [A fast cache for cloud storage](#a-fast-cache-for-cloud-storage)
* [Active-active replication and more](#active-active-replication-and-more)
* [Architecture](#architecture)
* [Compared to Other Systems](#compared-to-other-systems)
* [Compared to HDFS](#compared-to-hdfs)
* [Compared to GlusterFS, Ceph](#compared-to-glusterfs-ceph)
* [Compared to GlusterFS](#compared-to-glusterfs)
* [Compared to MooseFS](#compared-to-moosefs)
* [Compared to Ceph](#compared-to-ceph)
* [Compared to Minio](#compared-to-minio)
* [Dev Plan](#dev-plan)
* [Installation Guide](#installation-guide)
* [Disk Related Topics](#disk-related-topics)
* [Compared to MinIO, RustFS](#compared-to-minio-rustfs)
* [Benchmark](#benchmark)
* [Enterprise](#enterprise)
* [License](#license)
* [Sponsors](#sponsors)
# Quick Start #
## One command ##
## Quick Start with weed mini ##
Download the latest binary from the [releases](https://github.com/seaweedfs/seaweedfs/releases/latest) page and unzip the single `weed` (or `weed.exe`) file, or let the install script put it in `/usr/local/bin`:
Download the latest binary from https://github.com/seaweedfs/seaweedfs/releases and unzip the single `weed` (or `weed.exe`) file, or run `go install github.com/seaweedfs/seaweedfs/weed@latest`. Then start a ready-to-use S3 object store with credentials and a pre-created bucket in one command:
That's it — the S3 endpoint is at http://localhost:8333, `my-bucket` already exists, and `admin`/`secret` are valid credentials. `S3_BUCKET` accepts a comma-separated list (e.g. `raw,processed`); use `S3_TABLE_BUCKET` for S3 Tables (Iceberg) buckets. Drop any of the env vars to skip that piece (no AWS keys → S3 runs in unauthenticated "Allow All" mode for development).
That's it. The S3 endpoint is at http://localhost:8333, `my-bucket` exists, and `admin`/`secret` are valid credentials:
The same process also runs the master, a volume server, the filer, WebDAV, the Iceberg REST catalog, and the Admin UI. Add `S3_TABLE_BUCKET=warehouse` to also create an Iceberg table bucket, or `warehouse:LANCE` for a Lance one. Drop the AWS keys to run without authentication for development.
> macOS: if the binary is quarantined, run `xattr -d com.apple.quarantine ./weed` first.
Perfect for development, testing, learning SeaweedFS, and single-node deployments. To scale out, add more volume servers by running `weed volume -dir="/some/data/dir2" -master="<master_host>:9333" -port=8081` locally, on another machine, or on thousands of machines.
`weed mini` is auto-tuned for one node and is fine for single-node production, such as an S3 gateway that issues presigned URLs. See [Quick Start with weed mini][WeedMini].
## Quick Start for S3 API on Docker ##
## Docker ##
```bash
docker run -p 8333:8333 \
docker run -p 8333:8333 -v weed-data:/data \
-e AWS_ACCESS_KEY_ID=admin \
-e AWS_SECRET_ACCESS_KEY=secret \
-e S3_BUCKET=my-bucket \
chrislusf/seaweedfs
```
Same behavior as the `weed mini` command above — the S3 endpoint is at http://localhost:8333 with `my-bucket` pre-created. Drop the env vars to run anonymously for development.
Same behavior as the `weed mini` command above.
# Introduction #
## Docker Compose ##
SeaweedFS is a simple and highly scalable distributed file system. There are two objectives:
To run master, volume server, filer, S3, and WebDAV as separate services:
docker compose -f seaweedfs-compose.yml -p seaweedfs up
```
SeaweedFS started as a blob store to handle small files efficiently.
Instead of managing all file metadata in a central master,
the central master only manages volumes on volume servers,
and these volume servers manage files and their metadata.
This relieves concurrency pressure from the central master and spreads file metadata into volume servers,
allowing faster file access (O(1), usually just one disk read operation).
[Docker Compose for S3][DockerComposeS3] adds credentials, and the [docker/compose](docker/compose) folder has variants for replication, mounts, message queues, and more.
There is only 40 bytes of disk storage overhead for each file's metadata.
It is so simple with O(1) disk reads that you are welcome to challenge the performance with your actual use cases.
## Kubernetes with Helm ##
SeaweedFS started by implementing [Facebook's Haystack design paper](http://www.usenix.org/event/osdi10/tech/full_papers/Beaver.pdf).
Also, SeaweedFS implements erasure coding with ideas from
[f4: Facebook’s Warm BLOB Storage System](https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-muralidhar.pdf), and has a lot of similarities with [Facebook’s Tectonic Filesystem](https://www.usenix.org/system/files/fast21-pan.pdf) and [Google's Colossus File System](https://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system)
A production-shaped `values.yaml` for a three-node cluster: two copies of every write, three masters, and an S3 endpoint with credentials and a bucket.
SeaweedFS can transparently integrate with the cloud.
With hot data on local cluster, and warm data on the cloud with O(1) access time,
SeaweedFS can achieve both fast local access time and elastic cloud storage capacity.
What's more, the cloud storage access API cost is minimized.
Faster and cheaper than direct cloud storage!
```yaml
global:
seaweedfs:
enableReplication:true
replicationPlacement:"001"# one extra copy on another server; "002" for two
SeaweedFS also ships a built-in **Iceberg REST Catalog**, turning the same cluster into a self-contained lakehouse.
Spark, Trino, Dremio, DuckDB, and RisingWave can query Iceberg tables directly — no Hive Metastore, Glue, or
external catalog service required. Storage and table metadata live in one system, simplifying on-prem and
small-team analytics stacks.
master:
replicas:3
data:
type:persistentVolumeClaim # the cluster's default storage class; add storageClass to pick one
size:1Gi
volume:
replicas:3# at least 1 + the sum of the replication digits
dataDirs:
- name:data
type:persistentVolumeClaim
size:500Gi
maxVolumes:0# size the volume count from the disk
filer:
replicas:2
data:
type:persistentVolumeClaim
size:20Gi
s3:
enabled:true
replicas:2
enableAuth:true
credentials:
admin:
accessKey:admin
secretKey:change-me
createBuckets:
- name:app-storage
```
The S3 endpoint is the `seaweedfs-s3` service on port 8333. [Helm Chart Recipes][HelmRecipes] has values for a development cluster, a lakehouse with the Iceberg catalog exposed, filer metadata on PostgreSQL, and node-local disks. The [SeaweedFS Operator][Operator] and the [CSI driver][SeaweedFsCsiDriver] are the other Kubernetes paths.
`weed` lands in `$GOPATH/bin`. [Getting Started][GettingStarted] covers running master, volume, filer, and S3 as separate processes.
## Scale out ##
Capacity is a volume server. Start one on any machine with disk and point it at the master:
```bash
weed volume -dir=/data -master=<master_host>:9333
```
Nothing rebalances until you ask it to. Throughput is a filer or S3 gateway; they are stateless, so run as many as you need behind a load balancer. [Production Setup][ProductionSetup] walks through a multi-node cluster.
[Back to TOC](#table-of-contents)
# Features #
## Additional Blob Store Features ##
* Support different replication levels, with rack and data center aware.
* Automatic master servers failover - no single point of failure (SPOF).
*Automatic compression depending on file MIME type.
*Automatic compaction to reclaim disk space after deletion or update.
*Flexible Capacity Expansion: Any server with some disk space can add to the total storage space.
*Adding/Removing servers does **not** cause any data re-balancing unless triggered by admin commands.
* Optional picture resizing.
* Support ETag, Accept-Range, Last-Modified, etc.
* Support in-memory/leveldb/readonly mode tuning for memory/performance balance.
* Support rebalancing the writable and readonly volumes.
* [Customizable Multiple Storage Tiers][TieredStorage]: Customizable storage disk types to balance performance and cost.
*[Transparent cloud integration][CloudTier]: unlimited capacity via tiered cloud storage for warm data.
*[Erasure Coding for warm storage][ErasureCoding] Rack-Aware 10.4 erasure coding reduces storage cost and increases availability. Enterprise version can customize EC ratio.
# Why SeaweedFS #
## Fast ##
*One disk read per blob. A small file is one blob; a large file is split into chunks of a few MB, each its own blob. A volume server keeps a 16-byte index entry per blob in memory and reads it in a single seek, also for erasure-coded data.
*The master is not in the read path. Clients cache the volume-to-server mapping and talk to volume servers directly.
*40 bytes of metadata per file on disk. Small files are packed into append-only volume files, so there is no per-file inode, no per-file metadata file, no fragmentation, and writes are SSD friendly.
*Hot data is replicated; [erasure coding][ErasureCoding] is applied to warm data in the background, so writes never pay the encoding cost.
*The [Rust volume server][RustVolume] is a drop-in for higher throughput and lower tail latency on the same on-disk format.
On one laptop, [`weed benchmark`][Benchmarks] writes 1KB files at 15,700 per second and reads them back at 47,000 per second, and a mixed S3 [warp][S3Benchmark] run totals 3.2 GiB/s. Numbers are in the [Benchmark](#benchmark) section; throughput grows with volume servers and gateways.
## Scalable ##
*The master tracks volumes, not files. A cluster with billions of files has a few thousand volumes, so the master stays small. One master is enough for most clusters; run three for [Raft failover][FailoverMaster].
*Adding a server adds capacity with no data reshuffle. Balancing, vacuum, erasure coding, and repair run on demand from [`weed shell`][WeedShell] or the [maintenance worker][Worker].
* Filer and S3 gateways are stateless and scale linearly. Directory metadata lives in a [store you already run][FilerStores]: LevelDB, RocksDB, SQLite, MySQL, PostgreSQL, Cassandra, HBase, MongoDB, Redis, Elasticsearch, etcd, TiKV, FoundationDB, YDB, ArangoDB, Tarantool, and MySQL or PostgreSQL compatible databases such as TiDB, CockroachDB, and MemSQL.
* Rack and data center aware [replication][Replication], [tiered storage][TieredStorage] across disk types, and [transparent cloud tiering][CloudTier] for unlimited capacity.
* Files from a byte to [tens of TB][SuperLargeFiles]. Volumes up to 8TB with the large-disk build.
## The most complete S3 API ##
The S3 gateway implements the object, bucket, S3 Tables, IAM, and STS APIs on one endpoint, so the AWS SDKs and CLI, rclone, restic, Spark, and Trino work unchanged.
| API | Operations |
| --- | --- |
| S3 bucket and object | 73 |
| S3 Tables | 36 |
| IAM | 39 |
| STS | 5 |
* [Versioning][Versioning], [Object Lock][ObjectLock] with retention and legal hold, [lifecycle][Lifecycle] rules, tagging, [CORS][CORS], [conditional reads and writes][ConditionalOps], checksums, presigned URLs, browser POST uploads, multipart uploads, and an atomic [RenameObject][RenameObject].
* [Bucket policies][BucketPolicies] with [conditions][PolicyConditions] and [variables][PolicyVariables]; IAM users, groups, and policies; STS with [OIDC][OIDC], LDAP, and [Kubernetes service accounts][K8sSA].
* [SSE-S3, SSE-KMS, and SSE-C][SSE] server-side encryption, with OpenBao and Vault, AWS KMS, Azure Key Vault, and GCP KMS as key providers.
* [Audit log][AuditLog], [bucket quota][BucketQuota], and [rate limiting][RateLimiting].
* Each bucket is its own collection, so deleting a bucket is instant.
The full operation list is in [Amazon S3 API][AmazonS3API], and [Supported APIs vs MinIO][S3vsMinio] compares. The S3 compatibility suite and the SDK, IAM, SSE, policy, and Spark integration tests run in CI on every change.
## A data warehouse with S3 Tables ##
SeaweedFS is a lakehouse in one system. [S3 Table Buckets][S3TableBucket] hold Apache Iceberg tables by default, or [Lance][LanceCatalog] tables for vectors and multimodal data, and the built-in [Iceberg REST Catalog][IcebergCatalog] and Lance namespace serve them directly. There is no Hive Metastore, Glue, or separate catalog service to deploy, secure, and back up.
* Query engines operate on the same tables at the same time: [Spark][SparkIceberg], [Trino][TrinoIceberg], [Dremio][DremioIceberg], [DuckDB][DuckDBIceberg], [Apache Doris][DorisIceberg], [RisingWave][RisingWaveIceberg], ClickHouse, and [LanceDB][LanceDB]. Catalog commits are atomic compare-and-swap, so concurrent writers are safe. [Lakekeeper][Lakekeeper] can front the same storage with STS-vended credentials.
* [Automated table maintenance][IcebergMaintenance]: compaction, snapshot expiration, orphan file removal, and manifest rewriting, configured per bucket or table through the S3 Tables maintenance APIs, and the same for [Lance][LanceMaintenance].
* IAM at the bucket, namespace, and table level with standard bucket policies, see [S3 Tables Security][S3TablesSecurity].
* A [Hadoop compatible file system][Hadoop] for Spark, Flink, and HBase.
`S3_TABLE_BUCKET=warehouse ./weed mini -dir=./data` brings the whole stack up on a laptop.
## A fast cache for cloud storage ##
[Cloud Drive][CloudDrive] mounts a bucket from S3, Google Cloud Storage, Azure, Backblaze B2, Wasabi, Storj, or any S3-compatible store into SeaweedFS and serves it at local speed:
* Metadata is pulled once, so listing, stat, and directory walks cost no cloud API calls.
* File content is downloaded once, on first read or [warmed][CacheRemote] by folder, name pattern, size, or age, and cached with the capacity of the whole cluster: cache everything, no churn.
* Local writes complete at local latency and are written back to the cloud asynchronously in the cloud's native layout, so other tools keep reading the bucket directly.
* Uncache by the same rules to free local disk while keeping the metadata.
[Cloud Tier][CloudTier] goes the other direction, moving whole warm volumes to cloud storage while keeping one-read access, and the [Gateway to Remote Object Storage][GatewayToRemoteObjectStore] mirrors every bucket to a remote store. Faster and cheaper than reading the cloud directly.
## Active-active replication and more ##
* [Active-active or active-passive replication][ActiveActiveAsyncReplication] between clusters, continuous and resumable, for the whole tree or chosen folders, across data centers.
* [Filer store replication][FilerStoreReplication] for metadata HA, [async backup][AsyncBackup] to cloud storage, [metadata backup][MetaBackup], and [change data capture][CDC] with [webhooks][Webhook] on every metadata event.
* The same data as a [FUSE mount][Mount] on Linux, macOS, and [Windows][MountWindows], over [WebDAV][WebDAV], [SFTP][SFTP], HDFS, HTTP, and [TUS resumable uploads][TUS]; on Kubernetes through the [CSI driver][SeaweedFsCsiDriver] and [Operator][Operator].
* [AES256-GCM encryption at rest][FilerDataEncryption], TLS and mTLS between components, JWT-signed volume access, and [FIPS][FIPS] builds.
* [Admin UI][AdminUI], Prometheus [metrics][Metrics], [TTL][VolumeServerTTL] per file or volume, automatic compression and compaction, and [seaweed-up][SeaweedUp] for bare-metal clusters.
[Back to TOC](#table-of-contents)
## Filer Features ##
* [Filer server][Filer] provides "normal" directories and files via HTTP.
* [File TTL][FilerTTL] automatically expires file metadata and actual file data.
* [Mount filer][Mount] reads and writes files directly as a local directory via FUSE.
* [Filer Store Replication][FilerStoreReplication] enables HA for filer meta data stores.
* [Amazon S3 compatible API][AmazonS3API] accesses files with S3 tooling.
* [Hadoop Compatible File System][Hadoop] accesses files from Hadoop/Spark/Flink/etc or even runs HBase.
* [Async Replication To Cloud][BackupToCloud] has extremely fast local access and backups to Amazon S3, Google Cloud Storage, Azure, BackBlaze.
* [WebDAV] accesses as a mapped drive on Mac and Windows, or from mobile devices.
* [AES256-GCM Encrypted Storage][FilerDataEncryption] safely stores the encrypted data.
* [Super Large Files][SuperLargeFiles] stores large or super large files in tens of TB.
* [Cloud Drive][CloudDrive] mounts cloud storage to local cluster, cached for fast read and write with asynchronous write back.
* [Gateway to Remote Object Store][GatewayToRemoteObjectStore] mirrors bucket operations to remote object storage, in addition to [Cloud Drive][CloudDrive]
# Architecture#
## Data Lakehouse Features ##
* [S3 Table Buckets][S3TableBucket] expose a dedicated namespace for Iceberg tables with strict layout validation.
* Built-in [Iceberg REST Catalog][IcebergCatalog] runs alongside the S3 endpoint — no external metastore needed.
* Native integrations with [Apache Spark][SparkIceberg], [Trino][TrinoIceberg], [Dremio][DremioIceberg], [DuckDB][DuckDBIceberg], and [RisingWave][RisingWaveIceberg].
* **Master** servers, one or a Raft group of three, track which volume lives on which volume server and hand out file ids. They are not in the read path.
***Volume** servers store blobs in append-only volume files, keep a 16-byte in-memory index per blob, and replicate or erasure-code at the volume level.
***Filer** servers add directories and files on top, with metadata in a store of your choice, and expose HTTP, S3, WebDAV, SFTP, FUSE, and the table catalogs.
The blob store started from [Facebook's Haystack](http://www.usenix.org/event/osdi10/tech/full_papers/Beaver.pdf), erasure coding takes ideas from [f4](https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-muralidhar.pdf), and the whole has a lot in common with [Tectonic](https://www.usenix.org/system/files/fast21-pan.pdf) and [Colossus](https://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system). How file ids are assigned, written, and looked up, and why a master that tracks volumes scales, is in [Blob Store Architecture][BlobStoreArchitecture]; the services are in [Components][Components] and the [white paper][WhitePaper].
[Back to TOC](#table-of-contents)
## Example: Using Seaweed Blob Store ##
By default, the master node runs on port 9333, and the volume nodes run on port 8080.
Let's start one master node, and two volume nodes on port 8080 and 8081. Ideally, they should be started from different machines. We'll use localhost as an example.
SeaweedFS uses HTTP REST operations to read, write, and delete. The responses are in JSON or JSONP format.
A blob, also referred as a needle, a chunk, or mistakenly as a file, is just a byte array. It can have attributes, such as name, mime type, create or update time, etc. But basically it is just a byte array of a relatively small size, such as 2 MB ~ 64 MB. The size is not fixed.
To upload a blob: first, send a HTTP POST, PUT, or GET request to `/dir/assign` to get an `fid` and a volume server URL:
Now, you can save the `fid`, 3,01637037d6 in this case, to a database field.
The number 3 at the start represents a volume id. After the comma, it's one file key, 01, and a file cookie, 637037d6.
The volume id is an unsigned 32-bit integer. The file key is an unsigned 64-bit integer. The file cookie is an unsigned 32-bit integer, used to prevent URL guessing.
The file key and file cookie are both coded in hex. You can store the <volume id, file key, file cookie> tuple in your own format, or simply store the `fid` as a string.
If stored as a string, in theory, you would need 8+1+16+8=33 bytes. A char(33) would be enough, if not more than enough, since most uses will not need 2^32 volumes.
If space is really a concern, you can store the file id in the binary format. You would need one 4-byte integer for volume id, 8-byte long number for file key, and a 4-byte integer for the file cookie. So 16 bytes are more than enough.
### Read a Blob ###
Here is an example of how to render the URL.
First look up the volume server's URLs by the file's volumeId:
Since (usually) there are not too many volume servers, and volumes don't move often, you can cache the results most of the time. Depending on the replication type, one volume can have multiple replica locations. Just randomly pick one location to read.
Now you can take the public URL, render the URL or directly read from the volume server via URL:
```
http://localhost:8080/3,01637037d6.jpg
```
Notice we add a file extension ".jpg" here. It's optional and just one way for the client to specify the file content type.
If you want a nicer URL, you can use one of these alternative URL formats:
### Rack-Aware and Data Center-Aware Replication ###
SeaweedFS applies the replication strategy at a volume level. So, when you are getting a blob id, you can specify the replication strategy. For example:
When requesting a blob key, an optional "dataCenter" parameter can limit the assigned volume to the specific data center. For example, this specifies that the assigned volume should be limited to 'dc1':
Usually distributed file systems split each file into chunks. A central server keeps a mapping of filenames to chunks, and also which chunks each chunk server has.
The main drawback is that the central server can't handle many small files efficiently, and since all read requests need to go through the central master, so it might not scale well for many concurrent users.
Instead of managing chunks, SeaweedFS manages data volumes in the master server. Each data volume is 32GB in size, and can hold a lot of blobs. And each storage node can have many data volumes. So the master node only needs to store the metadata about the volumes, which is a fairly small amount of data and is generally stable.
The actual blob metadata, which are the blob volume, offset, and size, is stored in each volume on volume servers. Since each volume server only manages metadata of blobs on its own disk, with only 16 bytes for each blob, all access can read the metadata just from memory and only needs one disk operation to actually read file data.
For comparison, consider that an xfs inode structure in Linux is 536 bytes.
### Master Server and Volume Server ###
The architecture is fairly simple. The actual data is stored in volumes on storage nodes. One volume server can have multiple volumes, and can both support read and write access with basic authentication.
All volumes are managed by a master server. The master server contains the volume id to volume server mapping. This is fairly static information, and can be easily cached.
On each write request, the master server also generates a file key, which is a growing 64-bit unsigned integer. Since write requests are not generally as frequent as read requests, one master server should be able to handle the concurrency well.
### Write and Read files ###
When a client sends a write request, the master server returns (volume id, file key, file cookie, volume node URL) for the blob. The client then contacts the volume node and POSTs the blob content.
When a client needs to read a blob based on (volume id, file key, file cookie), it asks the master server by the volume id for the (volume node URL, volume node public URL), or retrieves this from a cache. Then the client can GET the content, or just render the URL on web pages and let browsers fetch the content.
### Saving memory ###
All blob metadata stored on a volume server is readable from memory without disk access. Each file takes just a 16-byte map entry of <64bit key, 32bit offset, 32bit size>. Of course, each map entry has its own space cost for the map. But usually the disk space runs out before the memory does.
### Tiered Storage to the cloud ###
The local volume servers are much faster, while cloud storages have elastic capacity and are actually more cost-efficient if not accessed often (usually free to upload, but relatively costly to access). With the append-only structure and O(1) access time, SeaweedFS can take advantage of both local and cloud storage by offloading the warm data to the cloud.
Usually hot data are fresh and warm data are old. SeaweedFS puts the newly created volumes on local servers, and optionally upload the older volumes on the cloud. If the older data are accessed less often, this literally gives you unlimited capacity with limited local servers, and still fast for new data.
With the O(1) access time, the network latency cost is kept at minimum.
If the hot/warm data is split as 20/80, with 20 servers, you can achieve storage capacity of 100 servers. That's a cost saving of 80%! Or you can repurpose the 80 servers to store new data also, and get 5X storage throughput.
[Back to TOC](#table-of-contents)
## SeaweedFS Filer ##
Built on top of the blob store, SeaweedFS Filer adds directory structure to create a file system. The directory structure is an interface that is implemented in many key-value stores or databases.
The content of a file is mapped to one or many blobs, distributed to multiple volumes on multiple volume servers.
## Compared to Other File Systems ##
# Compared to Other Systems #
Most other distributed file systems seem more complicated than necessary.
@@ -438,9 +270,7 @@ SeaweedFS is meant to be fast and simple, in both setup and operation. If you do
SeaweedFS is constantly moving forward. Same with other systems. These comparisons can be outdated quickly. Please help to keep them updated.
[Back to TOC](#table-of-contents)
### Compared to HDFS ###
## Compared to HDFS ##
HDFS uses the chunk approach for each file, and is ideal for storing large files.
@@ -448,9 +278,7 @@ SeaweedFS is ideal for serving relatively smaller files quickly and concurrently
SeaweedFS can also store extra large files by splitting them into manageable data chunks, and store the file ids of the data chunks into a meta chunk. This is managed by "weed upload/download" tool, and the weed master or volume servers are agnostic about it.
[Back to TOC](#table-of-contents)
### Compared to GlusterFS, Ceph ###
## Compared to GlusterFS, Ceph ##
The architectures are mostly the same. SeaweedFS aims to store and read files fast, with a simple and flat architecture. The main differences are
@@ -466,27 +294,18 @@ The architectures are mostly the same. SeaweedFS aims to store and read files fa
| GlusterFS | hashing | | FUSE, NFS | | |
| Ceph | hashing + rules | | FUSE | Yes | |
| MooseFS | in memory | | FUSE | | No |
| MinIO | separate meta file for each file | | | Yes | No |
| MinIO | separate meta file per drive for each file | | | Yes | No |
| RustFS | separate meta file per drive for each file | | | Yes | No |
[Back to TOC](#table-of-contents)
GlusterFS stores files, both directories and content, in configurable volumes called "bricks". It hashes the path and filename into ids, and assigned to virtual volumes, and then mapped to "bricks".
### Compared to GlusterFS ###
GlusterFS stores files, both directories and content, in configurable volumes called "bricks".
GlusterFS hashes the path and filename into ids, and assigned to virtual volumes, and then mapped to "bricks".
[Back to TOC](#table-of-contents)
### Compared to MooseFS ###
## Compared to MooseFS ##
MooseFS chooses to neglect small file issue. From moosefs 3.0 manual, "even a small file will occupy 64KiB plus additionally 4KiB of checksums and 1KiB for the header", because it "was initially designed for keeping large amounts (like several thousands) of very big files"
MooseFS Master Server keeps all meta data in memory. Same issue as HDFS namenode.
MooseFS Master Server keeps all meta data in memory. Same issue as HDFS namenode.
[Back to TOC](#table-of-contents)
### Compared to Ceph ###
## Compared to Ceph ##
Ceph can be setup similar to SeaweedFS as a key->blob store. It is much more complicated, with the need to support layers on top of it. [Here is a more detailed comparison](https://github.com/seaweedfs/seaweedfs/issues/120)
@@ -506,134 +325,39 @@ SeaweedFS Filer uses off-the-shelf stores, such as MySql, Postgres, Sqlite, Mong
Please note, as Apr 25, 2026 MinIO ceased development. It's strongly discouraged to use that unmaintained software with multiple security bugs. RustFS is a MinIO reimplementation in Rust, Apache 2.0 licensed and still developed, keeping MinIO's storage model down to a byte-compatible on-disk format. So the points below apply to both.
Please note, as Apr 25, 2026 MinIO ceased development. It's strongly discouraged to use that unmaintained software with multiple security bugs.
MinIO followed AWS S3 closely and was ideal for testing for S3 API. It had good UI, policies, versionings, etc. SeaweedFS is trying to catch up here.
MinIO followed AWS S3 closely and was ideal for testing for S3 API. It had good UI, policies, versionings, etc. SeaweedFS is trying to catch up here.
The metadata are in simple files. Each file write incurs extra writes to the corresponding meta file, on every drive of the erasure set. Changing only tags or retention rewrites that meta file on all of them, so the write amplification does not shrink with object size.
MinIO metadata were in simple files. Each file write will incur extra writes to corresponding meta file.
MinIO did not have optimization for lots of small files. The files were simply stored as is to local disks.
There is no optimization for lots of small files. The files are simply stored as is to local disks.
Plus the extra meta file and shards for erasure coding, it only amplifies the LOSF problem.
MinIO had multiple disk IO to read one file. SeaweedFS has O(1) disk reads, even for erasure coded files.
Multiple disk IO are needed to read one file. SeaweedFS has O(1) disk reads, even for erasure coded files.
MinIO had full-time erasure coding. SeaweedFS uses replication on hot data for faster speed and optionally applies erasure coding on warm data.
Erasure coding is full-time. SeaweedFS uses replication on hot data for faster speed and optionally applies erasure coding on warm data.
MinIO did not have POSIX-like API support.
No POSIX-like API support.
MinIO had specific requirements on storage layout. It is not flexible to adjust capacity. In SeaweedFS, just start one volume server pointing to the master. That's all.
## Dev Plan ##
* More tools and documentation, on how to manage and scale the system.
* Read and write stream data.
* Support structured data.
This is a super exciting project! And we need helpers and [support](https://www.patreon.com/seaweedfs)!
There are specific requirements on storage layout, which makes it hard to scale out and to maintain. An erasure set must be 2 to 16 drives and must divide the drive list symmetrically, and capacity grows or shrinks a whole pool at a time. In SeaweedFS, just start one volume server pointing to the master. That's all.
[Back to TOC](#table-of-contents)
## Installation Guide ##
# Benchmark #
> Installation guide for users who are not familiar with golang
Unscientific single-machine numbers from a MacBook with an SSD. [`weed benchmark`][Benchmarks], 1 million 1KB files, concurrency 16:
Step 1: install go on your machine and setup the environment by following the instructions at:
Step 3: download, compile, and install the project by executing the following command
```bash
cd seaweedfs/weed && make install
```
Once this is done, you will find the executable "weed" in your `$GOPATH/bin` directory
For more installation options, including how to run with Docker, see the [Getting Started guide](https://github.com/seaweedfs/seaweedfs/wiki/Getting-Started).
[Back to TOC](#table-of-contents)
## Disk Related Topics ##
### Hard Drive Performance ###
When testing read performance on SeaweedFS, it basically becomes a performance test of your hard drive's random read speed. Hard drives usually get 100MB/s~200MB/s.
### Solid State Disk ###
To modify or delete small files, SSD must delete a whole block at a time, and move content in existing blocks to a new block. SSD is fast when brand new, but will get fragmented over time and you have to garbage collect, compacting blocks. SeaweedFS is friendly to SSD since it is append-only. Deletion and compaction are done on volume level in the background, not slowing reading and not causing fragmentation.
[Back to TOC](#table-of-contents)
## Benchmark ##
My Own Unscientific Single Machine Results on Mac Book with Solid State Disk, CPU: 1 Intel Core i7 2.6GHz.
Write 1 million 1KB file:
```
Concurrency Level: 16
Time taken for tests: 66.753 seconds
Completed requests: 1048576
Failed requests: 0
Total transferred: 1106789009 bytes
Requests per second: 15708.23 [#/sec]
Transfer rate: 16191.69 [Kbytes/sec]
Connection Times (ms)
min avg max std
Total: 0.3 1.0 84.3 0.9
Percentage of the requests served within a certain time (ms)
50% 0.8 ms
66% 1.0 ms
75% 1.1 ms
80% 1.2 ms
90% 1.4 ms
95% 1.7 ms
98% 2.1 ms
99% 2.6 ms
100% 84.3 ms
```
Randomly read 1 million files:
```
Concurrency Level: 16
Time taken for tests: 22.301 seconds
Completed requests: 1048576
Failed requests: 0
Total transferred: 1106812873 bytes
Requests per second: 47019.38 [#/sec]
Transfer rate: 48467.57 [Kbytes/sec]
Connection Times (ms)
min avg max std
Total: 0.0 0.3 54.1 0.2
Percentage of the requests served within a certain time (ms)
50% 0.3 ms
90% 0.4 ms
98% 0.6 ms
99% 0.7 ms
100% 54.1 ms
```
### Run WARP and launch a mixed benchmark. ###
`make benchmark` runs [warp][S3Benchmark] mixed S3 traffic against a local `weed server`:
```
make benchmark
warp: Benchmark data written to "warp-mixed-2025-12-05[194844]-kBpU.csv.zst"
Cluster Total: 3302.88 MiB/s, 550.51 obj/s over 43s.
```
Read throughput is bounded by the random read speed of the disks, and grows with every volume server added. More numbers, including multi-node, FUSE, and Hadoop, are in [Benchmarks][Benchmarks], [S3 API Benchmark][S3Benchmark], [FIO benchmark][FIO], and [Independent Benchmarks][IndependentBenchmarks].
[Back to TOC](#table-of-contents)
## Enterprise ##
# Enterprise #
For enterprise users, please visit [seaweedfs.com](https://seaweedfs.com) for the SeaweedFS Enterprise Edition,
which has advanced features, including data recovery, self-healing storage,
For enterprise users, please visit [seaweedfs.com](https://seaweedfs.com) for the SeaweedFS Enterprise Edition,
which has advanced features, including data recovery, self-healing storage,
customizable erasure coding, EC vacuum and repair, etc.
[Back to TOC](#table-of-contents)
## License ##
# License #
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -678,5 +404,112 @@ The text of this page is available for modification and reuse under the terms of
[Back to TOC](#table-of-contents)
## Stargazers over time
[](https://starchart.cc/seaweedfs/seaweedfs)
# Sponsors #
<h3 align="center"><a href="https://www.patreon.com/seaweedfs">Sponsor SeaweedFS via Patreon</a></h3>
SeaweedFS is an independent Apache-licensed open source project with its ongoing development made
possible entirely thanks to the support of these awesome [backers](https://github.com/seaweedfs/seaweedfs/blob/master/backers.md).
If you'd like to grow SeaweedFS even stronger, please consider joining our
<a href="https://www.patreon.com/seaweedfs">sponsors on Patreon</a>.
Your support will be really appreciated by me and other supporters!
# Use Azure's Ubuntu mirror — much faster than archive.ubuntu.com from GitHub-hosted runners,
# which have been hanging long enough on Ign:/retry to trip the 10-min step timeout.
# Note: This e2e test image intentionally runs as root for simplicity and compatibility.
# Production images (Dockerfile.go_build) use proper user isolation with su-exec.
# For testing purposes, running as root avoids permission complexities and dependency
# on Alpine-specific tools like su-exec (not available in Ubuntu repos).
RUN sed -i 's|http://archive.ubuntu.com/ubuntu|http://azure.archive.ubuntu.com/ubuntu|g; s|http://security.ubuntu.com/ubuntu|http://azure.archive.ubuntu.com/ubuntu|g' /etc/apt/sources.list &&\
Every image CI pushes to `chrislusf/seaweedfs` and `ghcr.io/chrislusf/seaweedfs` is signed with [cosign](https://docs.sigstore.dev/cosign/verifying/verify/), keyless, by the GitHub Actions workflow that built it, so there is no key to fetch or pin. The signature is attached to the image digest and covers the multi-arch index and each platform image in it; `latest` is the release image under another tag and verifies the same way. Images published before September 2026 predate signing.
cosign prints the digest it verified. Deploy by that digest, or let an admission controller resolve the tag, so what runs is what was checked.
The identity is `https://github.com/seaweedfs/seaweedfs/.github/workflows/<workflow>@<ref>`. The ref is `refs/tags/<version>` for a release and `refs/heads/master` when a variant was republished by hand. The workflow is `container_release_unified.yml` for the release images, `container_dev.yml` for `dev`, `container_latest.yml` for a `latest` rebuilt by hand, `container_release_foundationdb.yml` for the `_large_disk_foundationdb` release image, and `container_foundationdb_version.yml` or `container_rocksdb_version.yml` for the per-version builds. The regexp above accepts release images only; `container_[a-z_]+\.yml@` accepts everything this repository publishes, `dev` included.
The same check as a Kyverno policy, release images only:
* master/filer/volume are stateful sets with anti-affinity on the hostname,
so your deployment will be spread/HA.
*chart is using memsql(mysql) as the filer backend to enable HA (multiple filer instances) and backup/HA memsql can provide.
* mysql user/password are created in a k8s secret (default: `<release>-seaweedfs-db-secret`) and injected to the filer with ENV.
*leveldb2 is the default filer backend; a mysql-compatible database (memsql, ...) enables HA (multiple filer instances) and the backup/HA it can provide.
* with `filer.extraEnvironmentVars.WEED_MYSQL_ENABLED` set to `"true"`, mysql user/password are created in a k8s secret (default: `<release>-seaweedfs-db-secret`) and injected to the filer with ENV. On any other store neither the secret nor the `WEED_MYSQL_*` env, plain or secret-backed, is rendered.
* cert config exists and can be enabled, but not been tested, requires cert-manager to be installed.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
github-actions[bot]
Chris LuandGitHub
Junker der ProvinzandGitHub
yanglongweiandGitHub
Alex KandGitHub
Lorentz KindeandGitHub
Mathieu ArnoldandGitHub
孙超andGitHub
Lars LehtonenandGitHub
Александр МелентьевandGitHub
Cavus700andGitHub
