* filer: keep the TUS sub-chunks that already landed when a write fails
A PATCH is split into 4MB sub-chunks, and each one is recorded in the
session as soon as it is stored. The session listing is what HEAD reports
as Upload-Offset and what the final entry is assembled from, so a record
is a promise that the data behind it exists.
When a later sub-chunk failed - a read-only volume, or a client that hung
up mid-body - the error path deleted the needles of every sub-chunk the
same PATCH had written but left their records in place. The resuming
client was then told to continue past bytes the filer had just queued for
deletion, and the upload completed into a gapless manifest pointing at
needles that were gone: HEAD returned the right size, GET died mid-body
once a vacuum reclaimed them.
Recorded sub-chunks now stay, which is what resumption expects: the
client picks up at the offset the session reports, and an upload that is
abandoned frees its chunks with the session.
* filer: drop a TUS chunk's record before freeing its data
filer.CreateEntry can return an error with the entry already inserted -
the parent-directory pass runs after the insert and keeps the entry when
it fails. A failed saveTusChunk therefore does not mean the record is
absent, and deleting the needle outright left the same corruption the
resume path used to cause: a session record pointing at data that is gone.
Remove the record first and only free the needle once it is gone. A
record lost with its data still stored merely leaks, which the vacuum and
fsck paths already account for.
* test: cover a TUS PATCH that is cut off mid-body
Resets the connection after one 4MB sub-chunk has landed, resumes from the
offset the session reports, and vacuums before reading the file back, so
anything the filer deleted behind a kept record shows up as a short read.
* test: drive the Lance namespace with Spark
The counterpart of catalog_spark, which does this for the Iceberg REST
catalog. Spark is the engine most likely to be pointed at a lakehouse,
and it reaches the Lance catalog through the connector's DSV2 catalog -
org.lance.spark.LanceNamespaceSparkCatalog with impl=rest - over the same
routes every other client uses.
SHOW NAMESPACES -> ['`sparklance-lcephd80`.ml']
SHOW TABLES -> ['sparklance-lcephd80$ml$embeddings']
count -> 3
filtered -> [(2, 'two'), (3, 'three')]
count after a second commit -> 4
The second insert is there on purpose: a store that cannot order commits
fails on the second one, not the first.
Two things the run settled that were guesses beforehand. CREATE TABLE
works, because the connector declares through the namespace and writes the
data itself rather than pushing Arrow at the server. And SHOW TABLES
returns the namespace's own identifiers - bucket, namespace and name
joined by the delimiter - not bare Spark table names.
Credentials go under the catalog's storage.* prefix, which is handed to
lance as object_store options; a gateway without STS vends none, the same
trap the LanceDB suite documents.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: verify the Lance table bucket was actually created
weed shell prints a command's own failure and still exits 0, so the harness
would go on to blame Spark for a bucket that was never made.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: bound the Docker probe
An unhealthy daemon makes docker version hang, and the probe runs before the
test has a timeout of its own.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: keep the ivy cache under the user's cache directory
It is mounted into a container running as root, so a shared temp path lets
another local user pre-create it and choose what Spark loads.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: assert the vector column's type, not only its name
A column that came back as array<double> or array<string> would still be
called vector and still pass.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: read the dataset off its location for real
The catalog being optional is the property that lets duckdb and pandas read
these tables; it was asserted in a comment and printed, never exercised.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: do not persist credentials in the Spark Lance checkout
The job only uploads a log on failure; nothing in it pushes.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: say the hosts in the README are placeholders
The suite passes dynamically allocated host.docker.internal ports.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
The db credential secret and the filer's WEED_MYSQL_* env were gated on
filer.enabled alone, so a filer on mongodb, redis, postgres or leveldb2 got a
generated mysql secret it never reads - kept forever by resource-policy: keep -
plus a mysql-db-host pointing nowhere.
Gate all of it on WEED_MYSQL_ENABLED, which is how the store is selected, plain
keys and secret-backed ones alike. An enable flag the chart cannot read - a
valueFrom, or one in secretExtraEnvironmentVars - counts as selected, so nothing
is dropped from a filer that is actually on mysql.
* admin: add bucket lifecycle rule editing
* address greptile's comments
* more small fixes
* coderabbit's comments
* more comment fixes
* more fixes
* more
* maybe last
* last ?
* 14850
* 14851
* filer: stamp the content MD5 on every SaveInsideFiler write
An entry's ETag falls back to Attributes.Md5, so conditional writers key
IF_ETAG_MATCH off it. SaveInsideFiler carried the looked-up attributes
forward without refreshing the hash, leaving it describing whatever the
previous writer stored: a later conditional write matched the stale hash
and overwrote content that had already changed.
* s3api: give the bucket lifecycle constants and the write route key one definition each
The extended-attribute keys, the XML size cap and the object-write ring key
prefix were each spelled out in two places, so the admin dashboard's copies
could drift from the gateway's. Move them to the packages both sides already
import and alias them where the short local name reads better.
* admin: patch the bucket entry's lifecycle keys instead of rewriting the entry
The save read the bucket entry, edited its extended map and wrote the whole
entry back, guarded by IF_UNMODIFIED_SINCE. Nothing that writes a bucket
entry advances its mtime - not the S3 gateway's patchBucketEntry, not
SetBucketOwner, not SetBucketQuota - so the guard never fired and the stale
snapshot reverted whatever else had changed since the lookup.
Send the PATCH_EXTENDED mutation the S3 gateway already uses for these keys:
the filer re-reads and merges under the bucket path lock, so only the two
lifecycle keys move. That removes the reason for the mtime snapshot, the
verification retry loop and the compensating restore of the cleared day-TTL
rules, which the migration now logs instead.
* s3api: run the delete-lifecycle day-TTL migration through the shared helper
DeleteBucketLifecycleHandler kept its own copy of the read-strip-write
sequence the put handler now shares, including a missing return that let a
ToText failure persist a truncated filer.conf and write a second response.
It also wrote the whole file back unconditionally, reverting any concurrent
edit; the shared helper writes conditionally.
* admin: answer 404 when a lifecycle request names a bucket that does not exist
Every SetBucketLifecycle failure came back as 500, including the lookup miss
for an unknown bucket, so a client or monitor read a caller error as a server
fault and retried it.
* s3api: emit lifecycle XML a client would recognize
Two changes to what MarshalCanonical writes, both visible through
GetBucketLifecycleConfiguration, which replays the stored bytes verbatim:
stamp the S3 namespace on the root, and put a size range under <And>. A
<Filter> carries one predicate, so two size bounds side by side is a shape
AWS does not document. Parsing still accepts either.
* admin: fix the lifecycle editor's handling of stored status, deletes and empty saves
Four things the editor got wrong:
A stored <Status> the S3 API never validated, say 'enabled', left both radio
buttons unchecked, so reading the form threw on a null querySelector result
and Save did nothing. Collapse anything but an exact 'Enabled' to 'Disabled',
which is what the engine already does with it.
Deleting a rule re-rendered an open edit form from the snapshot taken when
editing began, discarding what had been typed; every other transition folds
the form in first.
The Transition warning only matched a bare <Transition>, missing the form
with attributes, self-closed or namespace-prefixed.
Saving an emptied rule list clears the configuration through a path with no
prompt, next to a Delete-all-rules button that asks.
Also collapses the three divergent copies of formatBytes on this page to one.
* filer: stop the day-TTL migration from deleting an operator's path rule
The migration removed every rule under the bucket's path that carried a day
TTL in the bucket's collection. The add path it is retiring used
AddLocationConf, which merged its TTL onto whatever already sat at the
prefix, so a rule can hold operator settings the lifecycle path never wrote -
a disk type, WORM retention, a read-only flag, a placement pin. Deleting the
whole rule to retire its TTL took those with it, leaving objects under that
prefix on defaults nobody asked for.
Delete only rules shaped like ones the add path created from scratch;
anything else keeps its settings and loses just the TTL.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
The failover harness treated an open volume server port as readiness, but the
master only learns of a volume server from its heartbeat. A lone master refuses
heartbeats until its bootstrap check elects it, and the servers back off and
retry, so registration lands seconds after the ports answer. Tests that started
writing in that window assigned against an empty topology, which fails with
"no free volumes left" and reaches the mount as ENOSPC.
* volume: clear per-collection metrics when a collection leaves a server
The read-only and disk size gauges are only ever set for collections the
heartbeat still finds here, and nothing zeroes the rest. volume.balance marks a
volume read-only to move it, so the last heartbeat that saw it counts it
read-only - and if it was the collection's last volume on that server, that
count stands until the process restarts. The dashboard then shows read-only
volumes that volume.list -readonly cannot find anywhere.
Remember what each heartbeat set, and drop what is gone on the next one.
* volume: stop the read-only volume count from wrapping at 256
The per-collection counters were uint8, so a server holding 256 read-only
volumes of one collection reported zero of them.
* volume: read the read-only flags once when counting them
The heartbeat asked IsReadOnly for the verdict and then read noWriteOrDelete
and noWriteCanDelete straight off the volume, unlocked, so the reasons could
disagree with the verdict they were explaining. Take them together, under one
lock. The location is now nil-checked rather than skipped by short-circuit
evaluation, so a volume that has not joined a disk location yet stays safe.
* volume: let only a surviving volume keep its collection reported
A volume being deleted for expiry still made an entry in the read-only counts,
which is what the cleanup reads as "this collection is still here". The
collection's last volume could go and its series would stand for one more
heartbeat. Count the survivors only.
* volume: size a collection from the volumes it still has
The size totals are rebuilt from scratch every heartbeat, so subtracting a
volume that is about to be deleted took the surviving volumes' sizes down with
it: a collection keeping a small volume and losing a larger one reported the
difference, or lost its entry and kept the previous heartbeat's number.
* volume: cover the deleted bytes total in the surviving volume test
Deleted bytes are totalled the same way as sizes and were going unchecked, so
the test now leaves deleted needles on both volumes and pins that gauge too.
* s3: resolve ${VAR} in static config credentials from the environment
A deployment that keeps its S3 keys in a secret store had no way to hand
them to the gateway: -config takes a file, so the keys had to be written
into that file. Let a key in the static config name an environment
variable instead, and drop any credential whose reference stays unset so
the placeholder never becomes a usable key.
* helm: source the generated s3 identities from an existing secret
The only way to reuse credentials that already live in a Secret was to
hand-author the whole seaweedfs_s3_config JSON, since the literal keys in
values.yaml end up in git and a lookup-based keyRef renders empty under
helm template and Argo CD. Let s3.credentials.admin/read name a Secret and
its keys instead: the generated config references them as ${VAR} and the
gateway resolves them from the environment, so nothing is read from the
cluster at render time.
* s3: treat an empty environment value as an unresolved credential reference
A secret store can hand over a key that exists but is blank. Resolving it
would leave an access key whose signing secret is empty, so count it as
unresolved and drop the credential.
* helm: render the s3 secret when only the all-in-one auth flag is set
The all-in-one deployment mounts the s3 secret whenever any of the three
enableAuth flags is set, but the secret itself only rendered for the s3 and
filer flags, so allInOne.s3.enableAuth on its own left the pod waiting on a
secret nothing creates.
* helm ci: check the credential wiring on every workload that mounts it
The render check only looked at the standalone s3 deployment and only at
one of the four variables, so a helper that bound a variable to the wrong
secret key would still pass.
* helm: create the all-in-one s3 secret for every flag that mounts it
The all-in-one pod mounts the secret on any of the three enableAuth flags,
so keying its creation off allInOne.s3.enableAuth alone still left
filer.s3.enableAuth without filer.s3.enabled pointing at a secret nothing
creates. Mirror the deployment's own condition instead, and check each
flag renders both the mount and the secret.
* s3: reject a malformed credential reference instead of keying on it
A typo such as ${MY-VAR} matches no substitution, so it survived expansion
and the placeholder itself became the access key the gateway accepted.
Require every ${ in a static credential to open a well-formed reference.
* release: cut the whole release from the version bump workflow
The bump workflow stopped after pushing the version commit, and the rest was
manual: create the release, then run "Prepare release" in the csi-driver and
the operator. It now pushes the tag itself, which is what starts the binary,
container and helm workflows, creates the release with generated notes, and
dispatches the other two repositories, waiting for both.
Pushing the tag and reaching the other repositories both need RELEASE_PAT;
GITHUB_TOKEN raises no events that start workflows.
* release: tighten the release workflow after review
Check out master explicitly: a dispatch can select any branch, and the tag,
the commit and the release would then come off that branch while the
downstream job dispatches master.
Scope contents:write to the job that pushes; the downstream job talks to the
other repositories with RELEASE_PAT and needs nothing here.
Wait for the module proxy to serve the release commit as the tip before
dispatching, instead of priming it and hoping. The dispatched workflows pin
seaweedfs with `go get -u ...@latest`, so a stale tip means they release
against a pre-release commit, silently.
Identify the dispatched run by diffing the run list against the snapshot
taken before dispatching, rather than assuming the newest run is ours.
* release: wait on the downstream release, not on the run that makes it
A dispatched run cannot be told apart from a concurrent one: the API does not
report the inputs a run was dispatched with, so watching "the run that appeared
after mine" can watch someone else's and report their result as ours.
Wait for a release to appear in the downstream repository instead. That is the
thing being waited for, and it holds however many runs are in flight.
* test: read Lance tables from DuckDB
The LanceDB and Spark suites go through the catalog. DuckDB does not: its
lance extension reaches the data over S3 with no namespace involved, which
exercises the other half of the design - a table bucket's layout is a
valid Lance dataset directory, so a table stays readable when the catalog
is not in the path.
scan_rows=128
scan_columns=id,title,vector
filtered_rows=5
nearest=1,0,2
It also pins the one place the layout costs us. DuckDB's replacement scan
recognises a dataset by a .lance path suffix, and tables created through
this catalog deliberately have none: the catalog entry is the dataset
directory, a table name may not contain a dot, and a suffix would leak
into ARNs and policies. So __lance_scan is the way in, and the bare
SELECT ... FROM 's3://...' form does not see these tables.
The test asserts both halves - a suffixed path is read, a suffix-less one
is not - so if the extension ever recognises a bare directory, it fails
and says to update the documentation rather than leaving it wrong.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: require the catalog error from the suffix-less read
Any failure satisfied the old check - a missing extension, bad credentials,
an unreachable endpoint - so the assertion could pass without the
replacement scan ever classifying the path.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: verify the Lance table bucket was actually created
weed shell prints a command's own failure and still exits 0, so the harness
would go on to blame DuckDB for a bucket that was never made.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: bound the Docker probe
An unhealthy daemon makes docker version hang, and the probe runs before the
test has a timeout of its own.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: order the aggregates the assertions read
string_agg over an unordered relation may return the names, and the vector
search's ids, in any order, so the expectations could fail on a run where
nothing changed.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: do not persist credentials in the DuckDB Lance checkout
The job only uploads a log on failure; nothing in it pushes.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* master: answer with the leader raft already knows
Topo.Leader() backs off for up to 20 seconds waiting for an election.
Callers that a health probe or a client is blocked on cannot afford that:
/cluster/status, /cluster/healthz and /readyz all sit past the probe
timeout of both the helm chart and the operator, so a master that is
still joining looks dead rather than joining, and the kubelet restarts
it. informNewLeader and SendHeartbeat hold the client on a master that
cannot serve it, exactly when it should move on to find the one that can.
Answer these from MaybeLeader instead, which reports what raft knows
right now. MaybeLeader takes over the "am I the leader myself" fallback
that Leader() used to apply on top of it, so one non-blocking call is
still correct; Leader() keeps the backoff for callers that must wait.
* master: let the leader admit a master that starts with no raft state
Neither raft implementation lets a server outside the configuration
campaign: goraft's promotable() requires a non-empty log, and hashicorp
rejects vote requests from a candidate that is not in its configuration.
A master that comes up with fresh state therefore cannot elect itself in
— the leader has to pull it in. Nothing did.
The peer list is static, rendered from the replica count, so scaling it
up leaves the sitting leader running the old list with no idea the new
masters exist. Under goraft they wait forever. Under hashicorp they are
worse off: each bootstraps a cluster of its own from the new list, and
two of them form a quorum next to the live leader, with their own
TopologyId. That is the split brain SetTopologyId kills a master over.
Admit the peer where it registers instead. Only the leader gets past the
IsLeader check in KeepConnected, and a joining master's client lands
there, so that is the moment it joins. The broadcast OnPeerUpdate rides
on is not enough on its own: it only reaches masters already connected,
which is why a leader that came up first missed both newcomers.
RaftAddServer grew a goraft branch on the way, so cluster.raft.add stops
silently doing nothing on the default raft, and RaftRemoveServer with it.
Bootstrapping is now one call for both implementations, made only after
the peers confirm nobody has a leader, and retried until this master is
in rather than checked once and dropped.
* master: do not evict a peer that is still in -peers
The hashicorp leader drops a master from the raft configuration as soon
as it stops answering pings. A master that is merely restarting answers
nothing, so an ordinary bounce shrinks the quorum behind the operator's
back — and then races its own return: the master comes back, registers,
gets re-admitted, and the eviction lands after it.
A randomized start/stop walk lands on it. Two of three masters running,
the leader evicts the one that just went down, the restart re-adds it,
the removal commits late and takes the leader's own leadership with it.
What is left is a two-server configuration whose other half is down, and
a running master that nobody will ask for a vote — no quorum, no way
back until the third master returns.
-peers is what declares membership. updatePeers already reconciles the
configuration against it on every leadership change, and an operator who
really means to drop a master can say so with cluster.raft.remove, so
keep the eviction for masters that are no longer listed at all.
* test: bounce masters at random and hold the election to it
Twelve rounds of stopping or starting a random master, on both raft
implementations, checking the two things an election must never get
wrong: two masters claiming leadership at once, and a quorum that comes
back without agreeing on one. The cluster's identity has to survive the
whole walk, since a master that re-mints a TopologyId is the split brain
SetTopologyId kills its peers over. The seed is random and logged, so a
failure names the walk that reproduces it.
Below a quorum the walk moves straight on. A master that has lost its
quorum cannot commit anything, and goraft only checks whether it still
has one on an election-timeout ticker, after its peers have been quiet
for a full timeout — measured taking over 30 seconds to step down. That
direction belongs to TestTwoMastersDownAndRestart, which was giving it
ten seconds and would have started failing on a slower machine; it now
waits on that behaviour explicitly rather than sleeping twice and hoping.
WaitForTopologyId returns the id it waited for. Reading it separately
raced the leader applying the raft entry that carries it, which shows up
as an empty id right after an election rather than as a wrong one.
* fix(filer): stop logging a held aggregated read as an error
An aggregated subscriber may not read past the peers' low-watermark, and
it stops at the first entry beyond it by returning a sentinel from the
read callback. LoopProcessLogData logs every callback error, so on a
cluster that keeps writing - where there is almost always an entry newer
than the watermark - every read wrote an ERROR line naming the entry it
stopped at, thousands per minute per filer.
Mark the stop as control flow: an error wrapping StopReadingError is
handed back to the caller unlogged, and the held-read sentinel wraps it.
* fix(filer): release an aggregated watermark hold on peer progress
A held read waited on the aggregated buffer's data channel, which the
next write signalled - but a write cannot release a hold, only a peer
reporting further progress can. On a cluster that keeps writing the loop
therefore re-ran a whole pass per arriving event, log file listing and
all, and held again on the same entry every time.
Signal held readers from the meta aggregator instead, whenever a
low-watermark rises: a peer reporting, or one dropped past its removal
grace. The retry interval stays as the backstop for what no watermark
covers. Count the holds so a parked subscriber stays visible.
* fix(filer): floor how often an aggregated watermark hold releases
Peers advance their delivery watermark on every event they stream, so
releasing a hold on every advance is the same pass-per-event storm as
releasing on every write, just without the log lines - and each pass
lists a day of log files.
Floor the release at 20ms. Advances inside the floor collapse into one
release, which then delivers everything they covered.
* fix(filer): pace a peer's delivery claim by what its subscribers hold at
A filer's local metadata stream carries an idle heartbeat to its peer
aggregators, and each peer turns it into that filer's delivery
low-watermark. Aggregated subscribers hold at the minimum across peers,
so a filer quiet enough to fall back on the heartbeat parked every
subscriber in the cluster up to a keepalive interval - 5 seconds -
behind live writes. With nine filers, most of them quiet at any moment,
the minimum sat there permanently.
Pace that heartbeat at 200ms once the filer has peers. It stays a
keepalive, at the keepalive interval, for a filer with none.
* fix(filer): wake each aggregated hold on its own watermark
A persisted-log read is held by what the peers have flushed, an
in-memory read by what they have delivered, but both parked on one
channel closed whenever either minimum rose. Peers advance their
delivery watermark on every event they stream, so a flush-held reader
woke at the coalescing floor to re-list a day of log files and park
again on the same entry - the storm this set out to fix, in the one
place asymmetric peer progress still reached.
Signal the two separately and park each read on the one that bounds it.
* lance worker: share the integration tests' scaffolding
The recorder that keeps what a handler sent, the config builder and the
storage-option fallback all lived inside compaction.rs, so a second test
binary would have had to copy them. They move to tests/common.
The fallback now reads AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and
AWS_ENDPOINT_URL from the environment, defaulting to what it used before.
A harness can then point these tests at a gateway that checks what it is
given rather than one that accepts anything.
* lance worker: maintain one named table, for a harness to drive
Compacts and cleans up whatever WEED_LANCE_TABLE names, through the
handlers' own detect-then-execute path: a proposal the worker would not
have made is not one worth running.
The existing tests seed the tables they check. This one deliberately does
not, so a harness that has already written a table and knows what is in it
can have the real handlers maintain it and then read it back.
* test: take a table through its whole life, for Iceberg and Lance
Created in the catalog, filled by a real client, maintained by the worker,
read again, dropped. The step nothing was checking is the read after
maintenance: compaction once rewrote every dictionary-encoded column onto
a single value and shipped, because the maintenance tests were thorough
about sequence numbers, manifest entries and metadata versions and none of
them opened the parquet file the worker had just written.
So the assertion is a tally - row count, the cardinality of each
dictionary-encoded column, and an md5 over whole rows - taken before
maintenance and again after, required to be equal. The cardinalities name
the failure that happened; the digest catches a rewrite that keeps every
column's cardinality and hands the values to the wrong rows. A compaction
that merged nothing fails rather than passes, or the read afterwards is
checking a file the worker never wrote.
The Iceberg half runs two clients. DuckDB is the one the corruption was
reported against and the only one here that writes the deprecated
PLAIN_DICTIONARY encoding, which parquet-go normalizes away on write, so a
Go writer cannot produce it. PyIceberg writes the modern spelling. Pinning
parquet-go back to v0.30.1 fails the DuckDB half and passes the PyIceberg
one, which is why both are here.
Lance maintenance lives in the Rust worker, so it runs there where cargo
is installed and through the two lance calls those handlers wrap where it
is not. WEED_LANCE_MAINTENANCE picks one instead of letting the test guess.
* ci: run the table lifecycle tests
CI maintains the Lance table through the lance library rather than the
worker: a cold build of the lance crate costs more than the glue it would
be checking, and the worker's own tests cover its handlers.
The suite drives the Iceberg maintenance worker, so a change to it now
triggers this workflow too.
* test: let the lifecycle harness fail instead of skipping
Setup failures all exited zero, so a cluster that would not come up, or a
port allocation that lost, reported a green run for code nothing had
executed. That is the failure mode this whole directory exists to close,
and it was in the harness itself.
Only a checkout without a weed binary skips now, and it runs the tests so
each one says so rather than the package quietly passing. Everything else
fails.
The filer existence probe gets a deadline while I am here: it ran without
one, so an unresponsive filer would hang the suite past every timeout the
clients have.
* test: make the lifecycle checks check what they claim to
Three of them could pass without having looked.
The DuckDB skip matched "syntax error", "not implemented" and "Failed to
load" anywhere in the output, in any phase. A parse error in the SQL this
test generates, or a refusal from our own catalog, would have taken the
only coverage of the PLAIN_DICTIONARY encoding out of CI and left it
green. It now matches the extension failing to install, and only in the
phase that installs it. Everything past LOAD is ours and fails.
The digests covered id, category and value. Compaction rewrites the whole
row, so a defect confined to ts, or to a Lance vector, changed nothing
either side of maintenance. Every persisted column goes in now, ts as
microseconds so no timezone sits between the two runs.
The Lance drop check caught every exception as proof the dataset was
gone. pylance turns credential and transport failures into the same
ValueError, so it only accepts the message that means not found.
* docs: say up front which maintenance path the Lance half takes
The opening summary said the worker maintains both tables. It maintains
the Iceberg one always and the Lance one only where cargo is installed,
which is not what CI does.
* volume: start a volume's batch write worker on first use
Mounting a volume started a goroutine parked on a 128-slot channel, plus
the 128-entry batch slice it had already allocated. That is around 6.7KB
per volume the server pays whether or not the volume ever takes a write:
7231 bytes per mounted volume, of which 4101 is goroutine stack.
Only a write that asks for fsync ever reaches the worker, and a
remote-tiered or read-only volume never can. Create the channel and its
goroutine on the first such request instead, and let a write arriving
after Destroy fall back to the inline path rather than queue onto a
worker that has gone.
Measured over 20000 mounted volumes: 7231 -> 1269 bytes each.
* volume: update the heartbeat report state in place
Every heartbeat built a second map of what it was about to tell the
master, holding a freshly allocated short information message per volume,
then swapped it in over the old one -- and computed departures through a
third map of the live volume ids. A server holding 2M volumes rebuilt all
three every VolumePulsePeriod for a report that usually says nothing.
Number the heartbeats instead and mark the entry already held with the
pass that found the copy, so a quiet volume costs a map lookup and no
allocation. Departures are the entries a pass did not mark; the live-id
map is now built only when there are some, sized to them.
Measured over 10000 mounted volumes: 436 -> 196 bytes allocated per
volume per heartbeat.
* volume: fill one volume information message per heartbeat, not per volume
The heartbeat built a message for every volume held so it could hash it,
then dropped all but the few it had something to say about. At 2M volumes
that is 2M messages allocated every VolumePulsePeriod to send almost none
of them.
Fill a message the caller supplies instead, and replace it only when the
heartbeat keeps it, so a server with nothing to report fills the same one
all the way through.
Measured over 10000 mounted volumes: 196 -> 4 bytes allocated per volume
per heartbeat, and a heartbeat runs a third faster.
* volume: drop the per-volume trace from the heartbeat's status read
glog.V(4).Infof evaluates its arguments whether or not the verbosity is
on, so every volume boxed its id into a fresh interface slice on every
heartbeat: 759 of the 773 allocations a 1000-volume heartbeat made, for a
line that at this scale would print millions of unreadable rows.
Measured over 1000 mounted volumes: 4776 -> 1792 bytes and 759 -> 14
allocations per heartbeat, which no longer grows with the volume count.
* seaweed-volume: mirror the in-place heartbeat report state
Same change as the Go volume server: number the heartbeats and mark the
entry already held with the pass that found the copy, instead of building
a second map of hashes and swapping it in.
The volume snapshot must leave the reporting state as it found it, so it
keeps asking through changed() while a real heartbeat marks through
record().
* volume: refuse writes to a closed volume instead of dereferencing nil
Close and Destroy leave the needle map and data backend nil, but a caller
that already holds the volume can still reach the write path, where both
are used unguarded: a write racing a volume deletion took the server down.
syncDelete has always checked; syncWrite and the batch worker had not.
Reachable before this series and now also from the inline fallback a
durable write takes when the worker has gone.
* seaweed-volume: guard the report state with one mutex, as Go does
The full-list flag and the generation that answers it have to move
together. Split across separate atomics they cannot: a request landing
between begin's two reads returns full == false with the generation it
just raised, and one landing between commit's read and its clear is
marked answered by a heartbeat that carried no list. Either way the
resend is dropped.
Neither is reachable today -- every caller reaches this through the
store's RwLock, the flag setters under a read lock and the heartbeat
build under a write lock, so they cannot interleave. The type should not
depend on that being true two files away, and Go holds a single mutex
over exactly these fields.
* test: build the servers under test to match the harness's offset size
The mixed Go/Rust suites run both servers against one dataset, so both
have to agree on the offset width. They did not: the harness built Go
with no tags, 4-byte offsets, while the Rust crate defaults to its 5bytes
feature, and the Rust server then refused the .vif the Go server had just
written -- "bytes_offset mismatch: found 4, expected 5".
Build each side to match the offset size the test binary itself was
compiled with, so a plain `go test` and one with -tags 5BytesOffset both
get a matched pair.
* feat(s3): serve from remote on local read failure
When a locally-cached chunk of a remote-mounted object becomes unreadable
(volume server down/restarting, or an evicted needle 404ing under
retry-backoff), fall back to serving the object from its mounted remote
instead of erroring. A bounded pre-flight probe makes a stuck volume trip
the timeout rather than stalling the request.
Gated by -localReadFallbackToRemote (default off) with
-localReadFallbackTimeout (2s default), so existing deployments are
unaffected until they opt in.
* fix(s3): register local-read-fallback flags for mini/server/filer
The mini, server and filer launchers build S3Options directly and only
populate the flag pointers they register. Without registering the two new
flags there, startS3Server dereferenced nil pointers and crashed at boot,
failing every integration suite that runs `weed mini`.
* fix(s3): treat a zero-byte probe read as unreadable
A read that returns no byte -- whether it reports io.EOF or no error at all
-- means the offset is not locally readable, so the probe must fall back to
the remote rather than proceeding to stream a truncated response. Only a
returned byte (including the object's final byte with a trailing io.EOF)
counts as readable.
* s3: finish a mid-stream local read failure from the remote mount
The pre-flight probe only proves the byte at the requested offset readable.
A multi-chunk object can still lose a later chunk after the 200/206 and its
Content-Length are committed, which truncated the body with no fallback.
Resume from the mounted remote at the byte the local copy stopped at, so the
response still carries the declared length. A short local read that surfaces
as a clean EOF is treated the same way instead of silently truncating.
* s3: fall back to the remote mount without a CLI switch
Serving a remote-mounted object from its authoritative remote is what the
read should have done all along -- the alternative is a 500 on an object the
cluster can still reach -- so make it the behavior instead of two new flags,
with the probe bounded by a constant.
* s3: trim the comments on the fallback path
* filer: report only the contiguous prefix when a parallel chunk read fails
The parallel branch of doReadAt fans the chunk reads straight into their own
windows of the output buffer, then sums every task's bytesRead. A middle chunk
failing while a later one succeeds therefore returned a length covering a hole
the reader never filled, handing the caller zeros in the middle of otherwise
valid data.
* s3: only splice the remote onto a local prefix while it is the cached generation
Eligibility establishes a size match, not byte identity: a remote key
overwritten with same-size content between the cache fill and the fallback
would have finished the response with bytes from a second generation, under
the first one's ETag. Stat the remote before resuming and keep the local
error when it no longer matches -- a truncated body is a visible failure,
a spliced one is not.
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* deps: upgrade parquet-go to v0.32.0
Iceberg compaction writes the merged file with the schema of its first
input, encodings included. parquet-go before v0.31.0 took the deprecated
PLAIN_DICTIONARY encoding that DuckDB writes at face value and encoded
those pages as plain int32 indices, but the spec gives PLAIN_DICTIONARY
the same bit-width-prefixed RLE layout as RLE_DICTIONARY. Every
dictionary-encoded column in a compacted file then decoded onto a single
dictionary entry, and anything past one page failed to decode at all.
* iceberg: cover compaction of dictionary-encoded input
The fixture is a DuckDB-written file, so it carries the PLAIN_DICTIONARY
encoding a Go writer will not produce.
* iceberg: tally whole rows in the dictionary merge test
Counting each column on its own passes a merge that remaps names while
leaving their cardinality intact.
The toast and modal changes edited static/js/ without rerunning
gen_static_gz.go, so the embedded assets served to browsers still carry
the old scripts and TestStaticGzMirror fails on master.
* admin: raise nested modals above the ones already open
Bootstrap gives every modal and every backdrop the same z-index, so a
modal opened while another is showing paints behind it and its buttons
cannot be clicked. Viewing an access key secret and then copying a field
left the confirmation stuck behind the details modal with no way to
dismiss it.
Give each nested modal, and the backdrop Bootstrap creates for it, a
z-index above what is already on screen, and put back the scroll lock
that Bootstrap drops as soon as any one of them closes.
* admin: confirm clipboard copies with a toast
The access key details modal offers three copy buttons, and each one
raised a modal that had to be dismissed before the next copy. Confirm
with a toast instead, so the credentials stay in view and nothing has to
be clicked away.
* mini: state the format of a -tableBucket, do not infer it
A table bucket holds one format and that format decides which catalog can
serve it, but the flag only took names. The format came from
miniTableBucketFormat(): Iceberg whenever its port was up, Lance only when
it was not. So -tableBucket=vectors on a default mini quietly made an
ICEBERG bucket that the Lance namespace then refused every table in, and
the only way to get a Lance one was -s3.port.iceberg=0, which buys it by
deleting the other catalog. One flag, two meanings, decided by an unrelated
port.
Each entry is now name[:FORMAT], unsuffixed meaning ICEBERG as before:
weed mini -tableBucket=warehouse,vectors:LANCE
Both catalogs stay up and both buckets are reachable. A name whose format
has no endpoint here is skipped with a warning rather than created out of
reach, and the Iceberg-only S3_TABLE_BUCKET default-routing hint gets the
Iceberg names alone, without their suffixes.
* mini: do not reuse a table bucket that holds another format
CreateTableBucket answers BucketAlreadyExists on the name alone, so
-tableBucket=vectors against a bucket created as LANCE logged "already
exists" and moved on, and the Iceberg default-warehouse hint then pointed
at it. Every table create against that catalog fails with "table bucket
vectors holds LANCE tables", far from the flag that chose it.
ensureMiniTableBuckets now reads the format of a bucket it did not create,
warns when it is not the one asked for, and returns only the buckets that
hold what was requested. S3_TABLE_BUCKET is seeded from that list, so an
unprefixed Iceberg request falls back to its own default rather than
committing into a Lance bucket. A bucket predating declared formats reports
an empty one and still accepts either.
* mini: normalize S3_TABLE_BUCKET whichever way the spec arrived
The rewrite that keeps Lance names out of the Iceberg default warehouse only
ran when the flag supplied the spec. Set the variable directly, as the docker
quickstart does, and it reached the catalog untouched: S3_TABLE_BUCKET=
vectors:LANCE,warehouse made the unprefixed default the literal string
"vectors:LANCE", a bucket no lookup finds, while warehouse sat behind it.
The variable is both mini's input and the catalog's routing hint, so it is
now always rewritten from the buckets that came back holding Iceberg tables,
and unset when there are none rather than left pointing somewhere stale.
* mini: reuse a table bucket only when its format reads back
An ordinary S3 bucket wearing the name answers CreateTableBucket with the
same BucketAlreadyExists as a table bucket does, and the format lookup that
follows returned "" for a failed read exactly as it does for a bucket
predating declared formats. So -bucket=data -tableBucket=data reported
nothing and published data as the Iceberg default warehouse, where every
unprefixed request 404s on a bucket that is not a catalog.
The lookup now returns its error, and only a bucket that reads back as the
format asked for is reused. Anything else is left alone with a warning
naming why, rather than routed to and discovered later.
* test: drive the Lance namespace with LanceDB
The Iceberg catalog is checked against Spark, Trino, ClickHouse, Doris,
Dremio and RisingWave. The Lance one had only its own reference client,
which is the same thing as checking it against ourselves.
LanceDB connects with connect_namespace("rest", ...), which speaks the
routes this catalog implements, so the suite exercises the protocol rather
than our idea of it: list the catalog, open a table through it, read the
schema, run a vector search and a filtered scan, create a table, and read
the same dataset straight off its URI with no catalog at all.
table_names -> ['lancedb-p0guidmm$ml$embeddings']
open_table -> 64 rows
search -> [1, 0, 2]
create_table -> 4 rows, listed by the catalog
direct read without the catalog -> 64 rows
Seeding is pylance, because the namespace records where a table lives and
does not carry its data. That split is the design rather than a limit of
the test.
One interop note the test encodes: a gateway without STS vends
storage_options carrying an endpoint and a region but no credentials, and
LanceDB uses what the namespace vends on some paths. The container gets
credentials in its environment as well, which is what a deployment without
STS would do.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: pin the LanceDB client, and index before searching
Three from review.
The client's dependencies were unpinned, so an unrelated upstream release
could change what an old commit reproduces. Pinned to the versions this
suite was verified against; the client is as much the thing under test as
the server.
The search was called ANN and was not: without an index LanceDB scans.
The test now builds an IVF_PQ index over 1024 rows first, which is worth
more than the wording fix - an index writes into a directory of the table
that the S3 door has to admit, and that guard has refused a Lance
directory before. It builds, covers all 1024 rows, and searches.
The assertion moved with it. Demanding the exact nearest neighbour was
right for a brute-force scan and wrong for a quantized index, which
answered 0 as readily as 1; both are correct, so the check is now the
neighbourhood.
And the pushdown check accepted any failure. It now requires the refusal
to be the catalog's Unsupported and requires that nothing was left behind,
or, when the client falls back, that the table is complete.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
Nothing displayed the worker metrics, and the panels that look like they
did are about something else: Workers Connected, Worker Slots and Worker
Events in the Admin / Maintenance row read SeaweedFS_admin_*, which the
older maintenance queue feeds. A cluster running plugin workers - Go or
Rust - reads zero there while they are connected and busy.
A Plugin Workers row graphs what the workers themselves publish: how many
are connected, jobs and their failures, detection and proposal rates, job
duration, slot usage, stream events, and what the Lance jobs reclaimed.
The panel worth having is Objects Seen vs Skipped, since a sweep with
nothing to do and a sweep that could read nothing report the same number
of proposals.
Also a commented scrape target in the sample Prometheus config. It is 9328
rather than 9327: the sample compose already gives 9327 to the S3 gateway,
so the port the worker's own usage text suggests collides with it on a
single host.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* seaweed-worker: serve health, readiness and metrics
A Rust worker had no surface of its own. If it wedged, the only signals
were its stdout and whatever admin could infer from a stream that had gone
quiet; nothing could be scraped and nothing could be alerted on.
--metrics-port serves /health, /ready and /metrics, the same three the Go
worker serves under -metricsPort, so one scrape config covers workers in
either language. Off by default, loopback unless --metrics-ip says
otherwise, since the endpoint is unauthenticated. Names follow the Go
convention, SeaweedFS_worker_*.
The counters live in core and are raised where the stream already knows
what happened - connect, close, detection, execution, preview - so a
worker for another format gets them without writing any of this. Slots are
published from the heartbeat that already computes them, so a scrape and
the admin UI cannot disagree.
The pair worth having is objects_seen_total and objects_skipped_total. A
sweep that proposed nothing because there was nothing to do and a sweep
that proposed nothing because it could not read anything are the same
number of proposals; they are not the same event, and until now only a log
line told them apart.
The Lance jobs add what they reclaimed - fragments, rows brought under an
index, versions, bytes - on the same registry, so one endpoint serves both.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* seaweed-worker: fix the metrics address, the count, and a dead field
Three from review.
--metrics-ip ::1 failed at startup: the address was built by joining host
and port with a colon, and "::1:9327" is not an address. It is parsed as a
host and combined with SocketAddr::new now, so an IPv6 literal works, with
or without the brackets an operator will reasonably type after seeing one
in a URL.
proposals_total counted before the send rather than after, so a stream
that closed mid-sweep left the counter claiming proposals admin never
received.
And MeteredSender carried a Metrics clone and a job type it never read,
kept alive by two statements that existed only to silence the warning
about them. Everything is recorded by the caller, so both are gone.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
Every directory-state lookup went through path2inode, the map that holds one
full path per inode in the table, and then through dirStates. Directories now
carry their own path and are indexed by it directly.
There are orders of magnitude fewer directories than files, so this map stays
small whatever the mount holds, and it is what a file needs before it can stop
carrying a full path of its own: a child's path is its parent's plus its name.
No behavior change - the two indexes are asserted to agree.
* topology: refresh oversized mark on every heartbeat
The oversized flag on a volume location was only set when the volume was
registered (RegisterVolume). A volume that later grew past the size limit
kept its stale "not oversized" mark, so the heartbeat path
(ensureCorrectWritables) kept re-adding it to the writable list while
RecordAssign removed it on every assign - a writable/unwritable flip loop
that let writes continue past the limit and made vacuum race in-flight
writes.
Refresh the mark from each heartbeat's reported size in both heartbeat
paths (ApplyVolumeChanges and SyncDataNodeRegistration), mirroring what
RegisterVolume already did at registration time. A volume that grew past
the limit now stays unwritable, and one that shrank back clears the mark
and can recover.
* topology: order heartbeat writable correction after decay and honor cooldown
Review feedback (Greptile, CodeRabbit) on the oversized-mark refresh:
1. Greptile: clearing the oversized mark before EnsureCorrectWritables let
the delay-unaware helper re-add a just-compacted volume to writables,
bypassing capacityRecoveryDelay. ensureCorrectWritables now checks
fullSince and skips the re-add while the cooldown is pending, so a
volume removed for capacity only recovers through UpdateVolumeSize's
heartbeat recovery path.
2. CodeRabbit: in the full-heartbeat path the mark was refreshed after
the writable correction, so a newly oversized volume stayed writable
for an extra heartbeat cycle. The standalone changedVolumes loop is
merged into the volumeInfos loop and EnsureCorrectWritables now runs
after UpdateOversizedState + UpdateVolumeSize in both heartbeat paths,
using the freshly refreshed mark.
3. TestHandlingVolumeServerHeartbeat used a size (254320) that is past
the test's volumeSizeLimit (32768); it only passed because the stale
mark hid the oversized state. Sized down to 30000 to keep testing the
add/remove flow, and added TestEnsureCorrectWritablesHonorsRecoveryCooldown
covering the cooldown window and the recovery after it.
* topology: do not restore a still-crowded volume after the cooldown
Greptile review: after capacityRecoveryDelay elapses, ensureCorrectWritables
could restore a volume whose effective size is still past the crowded
threshold. UpdateVolumeSize refuses the recovery (effectiveSize > crowded
threshold -> setVolumeCrowded + return false), but the cooldown check in
ensureCorrectWritables only looked at fullSince, so once the delay passed
it re-added the volume even though capacity tracking still considers it
crowded.
Check the crowded mark before re-adding: a volume UpdateVolumeSize just
marked crowded must not be restored here, otherwise assignments resume
while the volume is still flagged for growth.
Adds TestEnsureCorrectWritablesDoesNotRestoreCrowdedVolume: effectiveSize
decays to 10500 (past the 9000 crowded threshold) after a report of 8000,
and ensureCorrectWritables keeps the volume unwritable past the cooldown.
* ci: trigger re-run of flaky FUSE jobs
* topology: gate the writable restore on the limit, not on crowded
A crowded volume is above the growth threshold, not full, and is normally
writable. Refusing to restore one locks it out for good: nothing writes to
a volume that is not writable, so its size can never fall back under the
threshold. Gate on the same size the assign path uses to remove it.
* topology: let only the heartbeat refresh set the oversized mark
Registration also set it, from whatever VolumeInfo it was handed. The
incremental path builds that from a short heartbeat message, which carries
no size, so every arrival announcement cleared the mark and handed the
volume back to the writable list until the next full report.
* topology: use the re-resolved layout after a dropped one is replaced
A layout dropped with its collection makes RegisterVolume refuse, and the
full heartbeat then re-registered against a fresh layout but kept applying
the size, oversized and writable updates to the dropped one.
---------
Co-authored-by: hzsunchao <hzsunchao@corp.netease.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* iceberg: skip tables the maintenance worker does not own
A Lance dataset registered through the Lance namespace's Iceberg REST adapter
arrives as an Iceberg table with a placeholder schema and table_type=lance, and
keeps its fragments under data/ - the same subdirectory the orphan cleaner
walks. Every fragment is unreferenced by the Iceberg metadata, so a maintenance
pass deletes the dataset. Views share the entry shape and were only skipped
because parsing their metadata happened to fail first.
Gate the scan and the execution path on the entry actually being an Iceberg
table. Maintenance is off by default, so this was latent rather than live.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* s3tables: let a table declare a format the catalog does not interpret
CreateTable accepted ICEBERG and nothing else. A Lance table has no metadata
file for the catalog to maintain - the entry records a name and the dataset
root, and the client owns everything under it - so accept LANCE, and carry the
declared format on the entry instead of hardcoding it back on the way out.
ListTables now reports format and metadataLocation, so listing a catalog that
holds both kinds takes one pass rather than a GetTable per row. AWS omits both
fields; adding them is additive.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* s3tables: move the in-memory filer into its own package
The Lance namespace tests need the same harness, and copying it would leave two
of them to keep in step. Extracted as it was, plus the two fidelity gaps that
only surface once a paginating caller uses it: ListEntries ignored
startFromFileName and limit, so a caller that paginates re-read the first page
until it hit its own cap and reported the same entry over and over, and
GetFilerConfiguration was missing, which CreateTableBucket needs to resolve the
buckets directory.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* lance: serve the Lance Namespace REST spec
A second catalog surface beside the Iceberg one, over the same table buckets:
the namespace and table metadata operations, the $-delimited identifier codec,
the spec's numeric error model, the directory-catalog marker files, and
storage_options vending through the STS path the Iceberg catalog already uses.
Listens on -port.lance, 9101 by default, and inherits ARNs, policies and tags
from the storage layer, so a Lance table needs no second permission model.
Identifiers map bucket / namespace / table onto the three levels Lance clients
already use, which is why there is no warehouse selector to invent. The data
plane needs Lance format support that does not exist in Go and answers with the
spec's Unsupported code rather than a bare 404.
Two things it deliberately will not do: create a table bucket as a side effect
of creating a namespace inside one, since a bucket carries its own policy and
lifecycle, and resolve an Iceberg table's location for a Lance client, which
would hand it a table another engine owns.
The design note this follows is in design-lance-catalog.md, including the
.lance directory suffix it proposed and this does not implement.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* mini: give the Lance port the same treatment as the Iceberg one
The flag was registered but nothing else knew about it, so mini would start the
server without reserving its port, waiting for it, or saying where it is. Adds
it to the startup service list, the conflict resolver, the gRPC allocator's
reserved set, the readiness wait, the stop reporting and the banner.
The admin server still takes only the Iceberg port, because there is no Lance
page for it to link to.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* lance: stop deregister and repoint from deleting the dataset
Deregistering preserves data by definition, and this did the opposite: the
catalog entry is the dataset directory, so DeleteTable took the files with it.
Registering over an existing name had the same shape, destroying the dataset
the name used to hold. Found by driving the running server rather than the
in-memory filer, where both looked like success because the table did stop
being listed.
Deregistering is now a state on the entry - the marker file hides it, and
declaring or registering the name again brings it back. Repointing a name at
another dataset is an UpdateTable against the version token, so neither dataset
loses files. Drop is left alone; it is the operation that does remove data.
The storage endpoint now falls back to the advertised -ip where the Iceberg
derivation gives up. An Iceberg client brings its own s3.endpoint and
advertising the wrong one hijacks it, but storage_options is the only place a
Lance client learns where the store is, and without it object_store quietly
talks to real AWS.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* s3tables: refuse to create a table over one of another format
Creating a table that already exists is idempotent, and that path returned the
existing table without looking at its format. A Lance declare over an Iceberg
table answered 200 and handed back a directory Iceberg owns, so the client
would write its dataset on top. The view check immediately above it already
guards the same class of collision.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* s3tables: let a table bucket hold a format other than Iceberg
The S3 door validated every object written into a table bucket against
Iceberg's file layout, so a Lance client could not write its dataset at all: it
got 403 on data/*.lance, on _versions/, and on the _transactions/ directory it
turned out to write as well. Table buckets were only neutral containers by
intention; in practice they were Iceberg-shaped and enforced as such.
The allowed set is now the union of what the supported formats write, because
the validator runs where the table's format is not in hand. Underscore-prefixed
directories are treated as belonging to the format, since enumerating them
means guessing at the next one - _transactions is exactly the one this missed -
and their contents are checked only for traversal. Iceberg writes none of them,
so it loses nothing. Marker files at the table root are admitted too, which the
namespace/table/dir/file shape had rejected as too shallow.
Describe also honours the request-body spellings of with_table_uri,
load_detailed_metadata and check_declared. The spec puts them in the query
string, but real clients send both.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* design: record what the implementation found
The table bucket being an Iceberg-shaped container, enforced at the S3 door,
was the premise this design never questioned and the one that had to change
before anything worked end to end.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* iceberg: prove the data loss the foreign-format guard prevents
The guard landed with a unit test for the predicate and nothing showing what it
saves. These seed what the Lance namespace's Iceberg REST adapter actually
leaves behind - an Iceberg table with a placeholder schema and table_type=lance
whose directory holds a Lance dataset - and assert both halves: orphan
collection does flag the dataset's fragments, because the Iceberg metadata
beside them references nothing, and the scan never reaches the table. An
ordinary Iceberg table in the same shape is still scanned, so the guard is not
just skipping everything.
Confirmed against a running gateway first: our Iceberg catalog accepts the
adapter's registration, and a real Lance client then writes a dataset into that
table's location.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* s3tablestest: make the in-memory filer safe to race against
Two gaps that only matter once a test drives concurrent writers, which is what
an exclusive create has to be tested with: the entry map had no lock, and
CreateEntry ignored O_EXCL entirely, so both writers of the same name would
have won and the test would have passed while proving nothing.
The BeforeUpdate hook runs before the lock is taken. Its whole purpose is to
land a competing write in a handler's read-to-write window, and that write
needs the lock the hook would otherwise be holding.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* lance: make the namespace an external manifest store
Lance commits a version by writing _versions/{v}.manifest with
put-if-not-exists. The S3 layer in front of this same filer evaluates
If-None-Match by looking the entry up and then writing without a precondition,
so two writers can both pass the check and one commit is lost. The filer itself
has the primitive: CreateEntry with o_excl.
Adds the four version operations a Lance client actually calls - create, list,
describe and batch-delete - recording one entry per version under
_lance_versions/, and advertises managed_versioning so the client routes its
commits here. Reserving a version is the exclusive create, so exactly one of
several racing writers wins and the rest rebase.
Off by default, behind -lance.managedVersioning. Turning it on moves where a
table's version history lives, and a reader that does not come through this
namespace no longer sees all of it; that is the operator's call, not a default.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* design: record what managed versioning does and does not reach
The first commit through a namespace-backed store works and is recorded the way
the protocol specifies. Later commits do not, because lance 4.0.0 refuses
put_if_exists on that path in its own code, so the feature is capped upstream
rather than here.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: integration tests for the Lance namespace
Everything this surface got wrong so far - a deregister that deleted the
dataset, an S3 door that refused every Lance file, a version reservation that
could not actually be exclusive - passed against an in-memory filer first. So
these run against a live gateway, and where the claim is about data they check
storage rather than visibility.
Five Go tests on the shared harness: namespace and table lifecycle including
that deregister keeps the bytes and drop removes them, that a Lance client
cannot resolve or declare over an Iceberg table, that a Lance dataset's files
get past the table-bucket layout guard while junk still does not, and that
eight writers racing for one version produce exactly one winner.
One Docker-gated test drives the real Lance client, which is the only way to
check that the location and storage_options the namespace vends are between
them enough to write and read a dataset. It overrides the endpoint with the
container's view of the same gateway, because the shared harness binds a
wildcard address and so vends none.
The harness gains a Lance port and turns managed versioning on; the flag
touches nothing outside that surface.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* s3tables: a directory with no namespace metadata is a missing namespace
Three callers resolved a namespace by reading its metadata attribute and each
tested only for a missing entry, so a directory that carried no metadata came
back as an internal error saying "attribute not found". Creating a table under
a namespace that does not exist answered 500.
Collapses the three copies into one helper that reports both conditions as
absent, which is what they are: a directory without namespace metadata is not a
namespace.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* iceberg: stop reporting storage-layer refusals as server faults
writeManagerError recognised a missing table bucket and sent everything else to
500, so a missing namespace, a duplicate name and a commit conflict all reached
the client as InternalServerError with nothing to act on. Creating a table in a
namespace that does not exist is the case that turned up: 500 where the spec
wants 404 NoSuchNamespaceException.
Maps the storage error types onto the exception names this package already
uses, and keeps the existing bucket message, which explains how to select a
table bucket.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* iceberg: skip a foreign-format table by name, not by failing to parse it
A table the namespace created as LANCE carries no Iceberg metadata, so the
worker skipped it only because the parse failed, and logged that as damaged
metadata. The catalog records the format on the entry and this never read it.
Reading it turns an accident into a decision, and separates a mixed catalog
from a corrupt one in the logs. The property check beside it still covers the
other shape: a real Iceberg table wearing table_type=lance, which is what the
Lance namespace's Iceberg REST adapter writes.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* design: answer whether a Lance table needs maintenance
It does, and index optimization has no Iceberg equivalent: rows written after
an index was built are not covered by it, so a vector search quietly misses
them. None of the three jobs can run in the Go worker, and there is no useful
subset, because deciding what an old version still references means parsing
Lance manifests. Version cleanup at least has an answer that needs nothing from
us - Lance can enable it on the dataset itself.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* design: the Lance maintenance worker is a plugin worker, in Rust
Framing it as a sidecar was wrong. plugin.proto already defines a
language-agnostic gRPC contract for external maintenance workers, and
"weed worker -admin=..." is the Go reference implementation of it from outside
the admin process. seaweed-volume already compiles protos out of weed/pb with
tonic_build, so a Lance worker is that build plus plugin.proto and the lance
crate.
Scheduling, retries, dedupe, progress and the admin settings page all come from
the protocol: a worker that answers RequestConfigSchema with a descriptor gets
its configuration form rendered without a line of Go.
The data plane is the part that genuinely does need a process answering HTTP,
and this had the two conflated.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* seaweed-worker: Rust plugin worker workspace, with Lance as the first one
plugin.proto is language-agnostic and the Rust toolchain was already in the
tree, so a Lance maintenance worker needs no new integration surface: core is
the contract and nothing else, and a worker crate beside it supplies handlers
and a binary. A second worker is a new member here rather than a fork of the
protocol, which is why this is seaweed-worker and not seaweed-lance-worker.
Verified against a running admin: it connects, is accepted, and admin prefetches
descriptors for lance_compact, lance_optimize_indices and lance_cleanup_versions,
so their settings pages render from the Rust side without a line of Go. The
stream stays up across heartbeats.
The job bodies are stubs that report failure. Doing the work means adding the
lance crate and opening the dataset, and claiming success before that would be
worse than saying so.
Two things running it caught that reading the proto did not: the admin address
has to be converted to the gRPC port the way pb.ServerToGrpcAddress does, or the
dial fails as an h2 frame error; and the generated field names differ from the
Go ones in several places, so JobCompleted carries success rather than a state
enum.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* lance worker: implement compaction
Detection lists tables from the namespace, opens each one, and proposes a job
for any with more fragments than the policy allows; opening a dataset reads its
manifest and not its data, so a sweep stays cheap. Execution re-resolves the
table rather than trusting what detection saw - it may have been repointed, and
the vended credentials expire - then compacts and reports the fragment counts
either side.
Verified against a live gateway: a twelve-fragment dataset became one fragment
with all twelve rows intact. The test drives the handler directly and skips
unless WEED_LANCE_NAMESPACE names a namespace, the way the Go integration tests
skip without Docker.
Running it turned up a gap the design had not: a gateway without STS vends no
credentials at all, so the worker could not open anything and detection quietly
proposed nothing. --access-key/--secret-key are the fallback, and whatever the
namespace vends still wins over them.
Two API assumptions did not survive contact either. Datasets open through
DatasetBuilder::with_storage_options, not ReadParams, and lance 10's
ObjectStoreParams has no storage_options field at all.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* lance worker: implement index optimization and version cleanup
Index optimization is the job with no Iceberg equivalent: rows appended after
an index was built are invisible to a search of it until this runs. Detection
reads num_unindexed_rows from each index's statistics and proposes a table once
more rows sit outside its indices than the budget allows; a table with no
indices is skipped, which is different from one whose indices have fallen
behind.
Cleanup applies a retention window, refusing rather than silently dropping a
tagged version, and leaving unverified files alone because they may belong to a
commit still in flight.
Both verified against a live gateway: 512 uncovered rows became 0, and a
fourteen-version table lost its old ones. Each test now seeds what it needs,
including building an IVF_PQ index and appending rows outside it. The first
version of these depended on state a script had left, so the second run found
the work already done and asserted nothing - a test that passes by doing
nothing is worse than no test.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* lance: answer an empty catalog with an empty list, not null
ListAllTables built its result from a nil slice, so a namespace holding no
tables answered {"tables":null} on a field the spec marks required. A generated
client may decode that differently from an empty list. Found running the
namespace on a dev box, where the catalog was empty.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* admin: give Lance maintenance its own scheduler lane
Lane assignment is a hardcoded map, so the three lance_* job types fell through
to the default lane. That lane serialises its work under the cluster admin lock
because volume management shares global state, which would queue a table's
compaction behind volume balancing for no reason - Iceberg has its own
lock-free lane for exactly this.
Adds the lane, maps the three job types to it, and puts it in the sidebar
beside Iceberg and Lifecycle. The lane routes were already generic, so only the
nav was hand-written.
The lane-coverage test spelled out the three known lanes, so a fourth failed
it. It now checks against AllLanes(), which is the property it was reaching for
and does not need editing next time.
Found by connecting the Rust worker to a real admin: it registered fine and its
job types were known, but they were filed under "default" and had no page.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* lance worker: log what detection saw
"Detection proposed nothing" and "the worker could not read the table" look
identical from the admin side, and the second is what a missing credential
produces. One line per table separates them.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* lance worker: fix a leaked heartbeat and a silent reconnect loop
spawn_heartbeat returned a handle to an empty task rather than the ticker it
had just spawned, so aborting it aborted nothing and every reconnect left
another heartbeat running against a dead channel.
A stream that admin closes cleanly is not an error, but reconnecting in silence
hides why. Two workers sharing an id evict each other forever and the log shows
nothing but a login every five seconds - which is exactly how this presented on
a dev box, and it took a look at the admin's own log to see it. The message now
names the id to check.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* lance: a namespace cannot be created without its parent
Storage keeps a namespace's parts flattened, so creating "a.b" with no "a"
was accepted and left an intermediate that only existed inside a name. Listing
derives child names by slicing those parts, so it reported "a", while describe
and exists on "a" both answered 404 - a client walking the tree got a 404 on
something the listing had just handed it.
The spec asks for NamespaceNotFound when the parent is missing, which is also
what keeps listing and describe telling the same story.
Namespaces created through the S3 Tables API still bypass this, so listing
keeps deriving intermediates rather than hiding whatever is already there.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* admin: say why a non-Iceberg table shows no schema
The table pages read Iceberg metadata for schema and snapshots, and a Lance
table has none, so both panels rendered "No schema available" - which reads as
an empty table rather than a table this page cannot describe. The dataset
behind the one that prompted this holds 1024 rows.
The format is already on the entry and shown two rows above, so the empty
states now use it: the catalog records where a LANCE table lives, not what is
in it.
Reading the schema for real needs Lance format code, which is the same wall as
the data plane.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* seaweed-worker: run rustfmt over the workspace
Committed the crates unformatted, so `cargo fmt --all --check` failed on
files nothing had touched since.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* plugin: let a worker report what it saw about an object
Admin cannot read a Lance table: it knows where the dataset lives and
nothing else, so the details page had a location and two empty panels.
The worker already opens every dataset during detection to decide whether
it needs compacting, so it knows the schema, the row count and the
fragment count at that moment. It just had no way to say so.
Add a WorkerObservations body to the worker stream. Admin caches the last
observation per object and serves it back, timestamped, for display;
nothing schedules from it. The Lance compaction sweep reports what it
opened, and the S3 Tables details page fills its schema panel from the
cache when it has no metadata of its own, badged with when the worker
looked and which worker it was.
Nothing about this is Lance-specific past the reporting side, which is
the point: any format admin cannot parse can describe itself the same way.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* design: record the observation channel
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* plugin: ask a worker for sample rows of a table admin cannot read
Browse Data reads an Iceberg table's Parquet files directly, so it shows
real rows. For a Lance table it showed "Table has no Iceberg metadata"
and an empty grid, because there is no Go Lance reader and never will be
one worth maintaining.
The worker has the reader. Add RequestObjectPreview / ObjectPreviewResponse
to the stream, mirroring the config-schema round trip that already exists,
and give the Rust worker a PreviewProvider that scans the dataset and
formats the rows with Arrow's own formatter, so a vector column reads as a
vector. Admin picks the worker from the observation store: whichever one
last described this table is the one that can read it.
Unlike an observation the rows are not cached. They are the table's data
rather than a description of it, and a copy sitting in admin would be both
stale and nobody's business. The page fetches on load, bounded at 200 rows
and a 15 second round trip, and drops the snapshot and data-file panels
that only mean something for Iceberg.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* design: record the preview channel
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: disable the lance listener when two gateways share a host
* test: keep AllocatePorts away from the lance default port
* s3tables: let a table bucket declare the format it holds
A bucket is a catalog, and a catalog serves one protocol. Format was
recorded per table, so nothing could answer "where do I point a client at
this bucket" without opening a table first, and an empty bucket had no
answer at all.
CreateTableBucket takes an optional format, stored with the rest of the
bucket metadata and returned by Get and List. Empty means ICEBERG, which
is what AWS S3 Tables serves and therefore what an SDK that has never
heard of the field means. CreateTable refuses a table of another format,
and CreateView refuses outright in a bucket that is not Iceberg, since a
view is Iceberg metadata.
Buckets that already exist carry no declaration and keep accepting
anything, so nothing is migrated and nothing that worked stops working.
The Lance namespace declares LANCE for the buckets it creates, which is
what stops one of them being described to a client as an Iceberg catalog.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* admin: take the Lance port the way it takes the Iceberg one
The UI cannot name the endpoint that serves a Lance bucket without it,
and every format-aware page below needs to.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* admin: show which format a table bucket holds
The bucket list printed an Iceberg endpoint for every bucket, including
ones holding Lance datasets, where that endpoint serves nothing. It was
the most visible place the UI assumed one format.
The list gains a Format column and its endpoint column follows the
bucket's declaration. The banner names both endpoints rather than
asserting everything is Iceberg, and says so only for the servers that
are actually running. Create Bucket picks a format with two cards rather
than a dropdown, since what matters is not the name but which clients can
read the result, and the endpoint under them updates as you choose so the
operator leaves the modal knowing where to point one.
A bucket from before the declaration existed shows "unset" in an outline
badge, explained on hover. It is a fact about the bucket's age, not a
fault, so nothing nags about it.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* admin: carry the bucket's format into the pages inside it
Namespaces and tables are reached through a bucket, so both now say which
catalog they belong to rather than making you go back up to find out. The
tables list gains a Format column and a Rows column filled from what a
worker last observed, since for a format admin cannot read that is the
only row count there is; a table nothing has looked at shows a dash, not
a zero.
Create Table stops offering a choice the bucket has already made: in a
declared bucket the format is fixed and says why, and only an undeclared
one still offers both. Before this the select had exactly one option,
hardcoded, which made a Lance table impossible to create from the UI at
all.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* admin: let the table page speak the table's own format
Partitions and Snapshot History are Iceberg's shape. Rendering them empty
for a Lance table reads as a fault; a Lance table has neither, and says
so by not showing them. In their place is a Versions panel, which is what
that format calls its history, carrying the worker's timestamp so it is
clear the numbers are a cached look rather than something read live.
The breadcrumb carries the format badge, so the page names what it is
looking at before you read a panel and wonder why it is empty.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* admin: show how to connect to either catalog, and group the two format workers
The client examples on the buckets page were Iceberg's alone, so the one
thing an operator wants after creating a Lance bucket - what to type to
reach it - was not written down anywhere in the UI. Both formats now get
a pair of snippets, and only for a server that is running.
In the Workers menu, Iceberg moves below Lifecycle so it sits next to
Lance: the two table-format workers together, the two cluster-wide ones
above them.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* shell: create a table bucket of either format
s3tables.bucket -create takes -format, so a Lance bucket can be made
without going through the UI. The integration harness passes it too: its
Lance tests were creating Iceberg buckets and getting away with it only
because nothing checked.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* design: record that a bucket declares its format
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* lance: drop managed versioning; the store already orders commits
The namespace offered itself as an external manifest store, so that a
commit could reserve a version through a real put-if-not-exists. That was
designed around a gateway that no longer exists: If-None-Match: * is
reduced to a filer WriteCondition and evaluated at the object's owner
under its per-path lock, or under the object write lock on the fallback
path. Sixteen writers racing one fresh key get a single 200 and fifteen
412s, every time.
Lance needs nothing else. commit_handler_from_url hands every s3:// dataset
a ConditionalPutCommitHandler, which puts with PutMode::Create, which
object_store sends as If-None-Match: *. So the feature solved a problem
this store does not have, while moving a table's version history out of
the dataset and into the catalog - and lance could not use it past the
first commit anyway, since its own namespace-backed store answers
"put_if_not_exists is not supported" to the second.
The version operations answer Unsupported with the rest, managed_versioning
is false, and the flag is gone. In place of the reserve-once test there is
one that races eight writers at the manifest key through S3, which is the
path a commit actually takes.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* lance worker: honour the version floor, the slot limits, and a shutdown
Five findings from review, all of them things the worker claimed to do and
did not.
The version floor was checked when a cleanup job was proposed and ignored
when it ran, so a table whose versions had aged past the retention window
in between could be taken below the count the operator asked to keep.
Execution now computes the floor itself and passes it as before_version;
CleanupPolicy ANDs its clauses, so a version has to be both too old and
below the floor to go. Both settings are clamped to the range the form
offers, since Duration::hours panics on a large enough value and a
negative min-versions wraps to a huge usize.
Admin's shutdown was answered by returning from the stream, which the
reconnect loop read as a healthy close and logged straight back in: the
worker could not be stopped. serve_once now says which of the two
happened.
The advertised concurrency limits bounded nothing - every request spawned
a task - and the heartbeat reported zero slots in use whatever was
running. Both now go through semaphores sized from the limits, with the
permits held for the life of the request and reported in the heartbeat.
A namespace call had no timeout, so a gateway that accepted the connection
and went quiet held a detection slot forever. And one table whose stats
could not be read failed the whole sweep, losing the proposals for every
table already scanned; it is now skipped and warned about, like a table
that cannot be opened.
The tests drove one shared catalog concurrently, which is why one of them
asserted "no proposals at all" and passed by luck. They now take a lock
and judge only their own tables.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* admin: fix the review findings on the format-aware pages
The endpoint hint in Create Bucket built its HTML by concatenating the
bucket name the operator is typing, so a name like <img onerror=...> ran
in the admin origin as they typed it. It is built from DOM nodes now.
A preview reply looked its channel up under the lock and then sent outside
it, which Shutdown can close in between: a Gosched in that gap panics with
"send on closed channel" every time. The send now happens under the lock.
Observations were looked up by path alone, so a table dropped and remade
in another format at the same path was described by the observation left
behind. Lookups now have to agree on the format.
Also: the Lance namespace caps a request body rather than reading whatever
arrives; the details action no longer says "Iceberg" over a Lance table;
mini stops advertising a catalog port when it is not running S3; a format
whose server this cluster does not run cannot be picked in the modal or
accepted by the API, since a bucket nothing can reach is not worth
creating; and the unused catalogPortFor helper is gone.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* lance worker: let the control stream use mTLS
The channel was hardcoded to http://, so off loopback the stream carried
preview rows and execution commands in the clear - and a cluster with grpc
TLS turned on would refuse the worker outright.
--tls-ca, --tls-cert and --tls-key take the same certificates the Go
worker reads from the [grpc.worker] section of security.toml, and must be
given together: a CA on its own would quietly mean one-way TLS, which a
mutual setup rejects anyway. Without them the stream stays plaintext,
which is what the Go worker also does when nothing is configured.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* lance: answer null properties rather than an empty map
The catalog does not keep a table's properties. Declare echoed the
request's back and describe answered {}, both of which claim they were
stored and are empty. Null says the catalog does not keep them, which is
what the spec distinguishes and what is true here.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* lance worker: test the slot accounting
The heartbeat reporting and the waiting are the two things the semaphores
are for, and neither is observable from outside without catching a sweep
mid-flight.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: fix the mixed-format catalog test, and name the binary it drives
The integration suite passed locally and failed in CI on
TestLanceRefusesIcebergTables. Both were right: CI builds the binary
first, my tree had one from the day before, so locally the test drove a
gateway with no format enforcement at all.
The test itself no longer holds as written. It made a bucket, put an
Iceberg table in it, and checked the Lance surface hid it - but a bucket
that declares LANCE now refuses the Iceberg table outright. The invariant
still matters from the other side, so it starts from an Iceberg bucket
instead: Lance must not describe or list a table whose format it does not
serve, and must refuse to declare one beside it.
The harness now prints which weed binary it is about to run and when that
was built. `make test` rebuilds first; a plain `go test` will happily
drive a weeks-old binary and report a pass for code it never ran, which is
exactly what happened here.
Also make the row-limit conversion in the preview request explicitly
bounded: CodeQL flagged the int-to-int32 conversion, and clamping by
reassignment beforehand is not a form it recognises.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* lance: prove concurrent commits are kept, and preselect the only format on offer
Two more from review.
The commit test asserted that exactly one writer wins the conditional PUT,
which is the mechanism, not the claim. The claim is that nothing is lost:
the losers see the conflict, rebase and commit again. So there is now a
test that has eight writers append to one dataset at once and counts the
rows afterwards - all eight batches survive. That is also the sequence
managed versioning could not finish, since its store refuses the second
commit outright.
And when Iceberg's endpoint is not running, the format picker offered two
options with neither selected, so Create Bucket submitted no format at
all, fell back to ICEBERG, and was refused by the guard added last round.
Lance is preselected when it is the only format this cluster serves.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* Clamp the remaining worker settings, and bootstrap buckets in a served format
Compaction and index optimization read their thresholds and cast straight
to usize and u64, so a negative arrives as an enormous number and turns
the threshold into "never": compaction and reindexing both go quiet with
nothing to say. The cleanup job was fixed last round; these are the same
bug. Clamped to the values that stay meaningful rather than to what the
form offers - zero uncovered rows is a real setting, meaning reindex as
soon as anything is not covered, so the floor there is zero and not the
form's thousand.
mini pre-creates the buckets named by -tableBucket, and did so without a
format, which now means Iceberg. Started with the Iceberg endpoint off
and the Lance one on, that left buckets nothing could reach and which
refused every Lance table. It takes the format from the endpoint that is
actually running, and creates nothing when neither is.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* s3: allow-unordered is a listing parameter, not an unimplemented subresource
The guard that stops a bucket GET with an unknown subresource from being
answered with a listing does not know about allow-unordered, so it answers
501 NotImplemented - to a parameter the listing handlers already read and
already validate against delimiter.
This is why test_bucket_list_unordered and test_bucket_listv2_unordered
fail in the Ceph s3-tests suite. They fail on master too; this is not a
Lance change and can be taken on its own.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
The guard that stops a bucket GET with an unknown subresource from being
answered with a listing does not know about allow-unordered, so it answers
501 NotImplemented - to a parameter the listing handlers already read and
already validate against delimiter.
This is why test_bucket_list_unordered and test_bucket_listv2_unordered
fail in the Ceph s3-tests suite. They fail on master too; this is not a
Lance change and can be taken on its own.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
Multipart parts stage under /buckets/<bucket>/.uploads/<id>/, so the filer
resolved filer.conf storage rules against that path when the gateway assigned
volumes for them. A rule scoped to a key prefix - fs.configure
-locationPrefix=/buckets/b/data/ -ttl=30d - then matched a small object but not
the parts of a large one, so an object whose entry carried the rule's TTL had
its bytes spread over TTL-less volumes.
Assign part chunks against the destination object's filer path instead, the way
the x-seaweedfs-destination header made the filer resolve it before the S3 write
path moved off the filer proxy. Covers PutObjectPart and both UploadPartCopy
paths. The part entry itself is still written under .uploads, so a read-only
rule there still rejects it.
The lifecycle XML Expiration.Days TTL keeps passing 0 for parts: that rule
targets the user-visible object key and would start its clock before
CompleteMultipartUpload.
* fix(s3): stamp the gateway's own ids on single-shot PutObject entries
putToFiler builds the entry in the gateway now instead of proxying a PUT to
the filer, and it hardcoded Uid/Gid 0 while every sibling write path stamps
filer_pb.OS_UID/OS_GID. On a non-root deployment that leaves single-shot PUTs
and multipart parts owned by root while directories and completed multipart
objects keep the real ids, so a mount reader can list the tree but gets EACCES
on every open once objects are not world-readable.
* fix(s3): stamp mode and ownership on copy destinations
CopyObject and UploadPartCopy build the destination attributes themselves and
then assign them over the entry filer_pb.MkFile just stamped, so the copy
landed with mode 0000 and uid/gid 0 - unreadable on a mount even by the filer's
own user. Build the destination with the same mode PutObject resolves for the
request and the gateway's own ids.
* fix(filer): watermark-bound aggregated metadata subscription against multi-source merge races
The aggregated metadata subscription (SubscribeMetadata) merges per-filer
sources that become readable at independent paces, but tracks its progress
with a single scalar cursor. Once the cursor passes a timestamp T, anything
a source materializes below T afterwards is silently skipped: a peer
recovering from a stall re-inserts its backlog late (late ring merge), and
a source's flush can land a log file, or a later chunk of the same file,
after a subscriber's disk pass listed the files (late persisted-log
landing). This is the residual documented in #10501.
Bound the subscriber's two read paths by what every source has provably
made visible, each with its own watermark:
- Delivery low-watermark -> in-memory reads. The meta aggregator tracks,
per subscribed peer (self included), the newest timestamp received on
that peer's stream - real events, or idle heartbeats (peer streams now
opt into ClientSupportsIdleHeartbeat). The aggregated ring is complete
up to the minimum across peers; in-memory reads hold at it.
- Flush low-watermark -> persisted-log reads. Each filer reports its local
log-buffer flush watermark on its stream: a new flushed_ts_ns response
field, carried on idle heartbeats and on periodic flush reports (gated
on ClientSupportsIdleHeartbeat). Disk passes freeze the minimum across
peers before listing the log files and hold at it; the day-boundary
cursor jump and the metadata-chunks ref listing are bounded the same
way, the latter at minute-file granularity.
- Held reads keep the cursor at the last entry actually delivered and
retry; the retry re-lists the log files, which is what picks up a
late-landing file. Both watermarks are relaxed by the settled horizon
(2 x LogFlushInterval) as a liveness escape, so a peer stalled beyond it
delays subscribers by at most the horizon instead of forever - any loss
that escape allows was unconditional before.
With reads held at the flush watermark, a disk advance below it is proven
complete on every peer's disk, so the unproven-crossing counter now only
counts crossings the horizon escape allowed past a stalled peer.
Live delivery on the aggregated stream may lag by up to the idle-heartbeat
interval when some peers are quiet; SubscribeLocalMetadata consumers are
unaffected.
* fix(filer): resume evicted aggregated readers from an original-space disk anchor
The aggregated ring rewrites out-of-order peer arrivals to its head, so a
subscriber tailing it advances its cursor in bumped (arrival) timestamps,
while persisted logs keep original timestamps. When a slow reader's unread
window is evicted (e.g. a peer backlog flooding in after a stall) and the
reader falls back to disk, resuming from the bumped cursor skips every
original-space entry below it that memory never delivered - reproduced as
a ~66% silent loss on a 3-filer cluster with one peer's stream frozen for
~70s while the subscriber lagged.
Track a disk anchor: the newest original-space position the stream is
proven complete through. Disk passes advance it directly; contiguous
memory reads advance it to the peers' delivery low-watermark observed
before the read (per-peer streams are ordered, so everything with an
original timestamp at or below that watermark had already arrived and was
delivered). A reader kicked off the ring resumes the disk pass from the
anchor instead of the bumped cursor - redelivering what memory already
sent is within the subscription's at-least-once contract, skipping what
it never sent is not.
* fix(filer): close review findings on the peer-watermark subscription bounds
Four correctness holes found in review, one generated-file cleanup:
- The flush-through claim could assert durability for events still on
their way into the buffer: an event is timestamped before notification
work that can block, and only then appended. Track stamped-but-unappended
events on the Filer (the stamp shares a lock with the reader, and appends
are bumped monotonically past the buffer head), and cap the reported
flush watermark just below the oldest in-flight stamp.
- Removing a peer deleted its watermark entries while its stream kept
running: its next signal recreated the deleted entry, which then pinned
the low-watermark forever once the stream died. Watermarks now advance
only for tracked peers, and peer removal cancels the subscription
context so the stream stops feeding the aggregated buffer promptly.
- The pipelined sender folded flush reports (TsNs 0 reads as far behind)
into batch Events tails, where the aggregator's nil-notification guard
dropped them - a busy backlog replay could starve the flush watermark
until the settled-horizon escape opened a loss window. Control messages
are now unbatchable on the sender, and the receiver also reads watermark
state off nested batch entries as belt and braces.
- A give-up skip's cursor was not anchored, so the next eviction rewind
undid the counted decision and re-entered the same park forever when the
evicted window carried bumped timestamps. The anchor now follows give-up
skips; an anchored cursor makes the rewind a no-op and keeps the gap
machinery's re-arm onto the retained window reachable.
- Regenerated-file churn from a different protoc-gen-go-vtproto version is
dropped: the vtproto file is upstream's, plus only the flushed_ts_ns
marshal/size/unmarshal cases in the same generator style.
New tests pin the in-flight floor, the no-resurrection rule for removed
peers, and that control messages are never nested in batches.
* fix(filer): keep a removed peer's watermarks through a grace period
Deleting a peer's watermark entries the moment the master removes it
reopened the loss the watermarks exist to prevent: a filer frozen or
partitioned long enough to miss master heartbeats is removed from the
cluster, its unflushed events still exist, and with its entries gone the
low-watermarks snap forward to the healthy peers - subscribers advance
past the absent peer's window and its late-landing log files are silently
skipped. Reproduced on a 3-filer cluster: freezing two filers for ~70s got
them removed ~28s in, and a catching-up subscriber lost their entire
overlapping window.
Removal now only marks the peer; its watermarks keep participating in the
low-watermarks for a grace period (2 x LogFlushInterval, matching the
subscribe loops' settled horizon, which already bounds a stale watermark's
influence meanwhile). A re-added peer clears the mark and continues its
values monotonically - the flap case costs nothing. A peer that stays gone
is dropped when the grace expires, so a decommission cannot pin the
low-watermarks, and a dropped peer's straggling signals cannot resurrect
its entry.
* fix(filer): cap delivery heartbeats by the in-flight floor; harden stamps
Second review pass on the watermark bounds:
- Idle heartbeats on the local stream claimed delivery-completeness
through "now" while an event could still sit stamped-but-unappended
behind blocking notification work. A peer aggregator turns that claim
into its delivery low-watermark, so it could advance (and anchor
credits with it) past an event that had not been streamed yet. The
heartbeat timestamp is now capped just below the oldest in-flight
stamp, like the flush claim already was.
- In-flight stamps are forced monotonic against the registry's own
history, so a wall-clock step backwards cannot slip a new stamp under
an already-sampled floor. The cross-goroutine ordering still shares
the meta log's global forward-clock assumption; the comments now say
so instead of overclaiming.
- Duplicate removal notifications no longer refresh a removed peer's
grace deadline: the first removal time wins, so a decommissioned peer
cannot sit in the watermark sets forever on repeated updates.
- A failed buffer append clears the event's in-flight stamp on purpose:
the event is dropped from the change stream entirely (a pre-existing
defect of the append path, loudly logged), and a watermark waiting for
it would pin this filer's claims forever. The comments now state the
decision instead of implying the failure cannot happen.
* docs(filer): tighten the watermark comments
Comment-only: compress the narrative comments added on this branch down
to their load-bearing invariants, and fix one stale sentence (peer
removal no longer deletes the watermark entries immediately). No code
changes.
* fix(filer): subscribe to the local filer before remote peers
Self's events reach the aggregated buffer only through the aggregator's
own subscription to it, but bootstrap only seeded the peers the master
already listed - and self's master registration races that listing, so
the watermark set could hold remote peers without self. Once the remotes
signalled, the low-watermarks would claim completeness for a stream that
was still missing a merge source, letting aggregated subscribers advance
past the local filer's events before its subscription started.
Seed self first, unconditionally: before that the watermark set is empty
(a documented safe state - reads hold at the settled horizon), and after
it the set can never be remotes-only. The later master update for self,
or a duplicate in the listed peers, is a no-op via the already-followed
check in OnPeerUpdate.
* fix(filer): fence watermark claims against wall-clock regression
Record issued heartbeat/flush claims in the in-flight registry and stamp
later events above them, so a backward clock step cannot land an event
under a watermark a peer has already advanced to.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(filer): re-check the buffer head after fencing heartbeat claims
An event appended between the caught-up check and the delivery claim
was covered by the claim but not yet sent on the stream. The claims
fence later stamps, so re-checking the head after them proves every
covered event was already sent before the heartbeat.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(filer): cross the aggregated ring's pre-subscription range only on proof
The eviction gate and the gap proofs read "nothing evicted yet" as "memory
holds everything after the cursor". That is false for the merge-fed
aggregated ring, which is born empty while every peer's history sits on
disk: before the ring's first real eviction, a subscriber whose cursor was
still below the bounded chunk pass's listing stop was served the ring's
earliest entry inclusively, silently skipping the withheld pre-restart
files - and the idle-wait callback credited the delivery low-watermark to
the disk anchor in the same disconnected state.
Mark everything at or below the subscriptions' start as evicted when the
aggregator is built, credit the anchor only once the run is connected to
the ring, and give the aggregated gap pass a real proof to cross the
marked boundary with: each disk pass's proven coverage (the peer flush
low-watermark capped by the pass's listing bound). An empty pass whose
proof reaches the eviction watermark crosses to it silently - no park, no
loss counter - so the mark costs a bounded catch-up delay instead of the
15-minute give-up.
* fix(filer): keep shipped chunk tails at or below the hold point
A log file spans past its named minute (window start plus up to a flush
interval), and chunk-mode clients apply a shipped file whole - so a file
tail past the hold point can become a persisted client checkpoint beyond
what every peer has proven, and a crash inside that window resumes past
another peer's late-but-in-contract flush. Stop the ref listing a minute
plus a flush interval below the hold; the withheld band is served by the
memory pass (ring retention far exceeds it) or by later passes as the
hold advances, so freshness is unchanged. A frozen peer flushing one
window that spans its whole freeze can still overshoot; that residual is
bounded by the freeze and needs a crash inside it.
* docs(filer): trim the review-fix comments
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* mount: discard a path-cache insert that raced a purge
The Windows adapter's walk resolves a component with a Lookup RPC and inserts
the result holding no lock, so a purge can land in between - and what the walk
just resolved is then the very name the purge removed. Anything opening the
old path concurrently with a rename repopulates the cache with the vacated
name, which the next stat is served from for up to a second. The release path
already guards its equivalent insert; the walk had nothing.
The cache counts purges now. A resolve snapshots the generation before its
lookups and insert discards the entry when any purge ran in between, parking
the reference in the graveyard so the in-flight caller keeps a valid inode
either way.
Seen once in CI as TestRenameOverExisting failing with 'source survived the
rename': every SeaweedFS layer is synchronous with the rename, but a
background open of the source - an antivirus scan of the just-written file
fits - can requalify the stale name through this window. The assertion also
reports what stat returned now, and whether it persisted, so a recurrence
indicts a specific layer instead of reading as a mystery.
* mount: cover the path-cache discard by key, and let a discard rest
Review follow-ups. The generation was global, so any purge between a walk's
snapshot and its insert discarded the entry whatever its name - and an open
retries resolve-then-steal only four times before failing with EIO, so
sustained unrelated churn could fail opens of untouched paths. Purges are
remembered by key now and only one that covers the inserted name discards it;
past the remembered window the insert is discarded without a check, which only
costs a retry.
A discard that itself tripped the sweep also handed its own reference straight
to forget while the walker was still using the inode. The graveyard holds two
generations now, so an appended reference always survives the sweep of the
call that appended it - which the displaced-entry and purge paths needed too.
Also restores the original path-cache test suite this branch had overwritten
instead of extended, and rewords the semantics-test failure so it no longer
claims the source survived when stat returned a transient error.
* mount: take an open's reference directly instead of stealing it back
resolveAndSteal cached the final component only to steal it back, so an open
depended on that insert surviving whatever purges raced it - four attempts and
then EIO. The keyed purge window narrowed how often an insert is discarded,
but past the window the discard is blind again, so the cliff had only moved.
A cached entry is still stolen; anything else is now looked up directly, with
the caller owning the reference from the start. No retry loop, and no way for
churn - covered, unrelated or overflowing the window - to fail an open.
Also covers the whole-cache purge: purge of the root with prefix set clears
every entry, but the covers check tested for a '/'-prefixed key that a
normalised key never has, so it covered no in-flight insert at all.
* mount: leave the target alone when a move has no source
MovePath cleared whatever sat at the target before it checked that the source
was still there, so a move it then declined to make had already taken the
target's mapping apart. The same rename reaching the table twice - once for an
open handle, once for the invalidation behind it - was enough to leave the
moved inode with no path at all.
* mount: move the inode table when a rename arrives from the cluster
A rename made by another client reaches this mount only as a metadata event,
and the only table update on that path sat inside the open-file-handle branch.
Every other inode the kernel still addresses by nodeid kept resolving to its
pre-rename path, so the next operation on it went to a path the filer no
longer has.
Move the entry for every rename invalidation. The filer sends one event per
moved entry, so a renamed directory's children follow their parent without a
descendant walk.
* mount: move the inode table exactly once per rename event
Making MovePath return early on a missing source was the wrong half to fix.
The source is also missing when the rename came from a client that never
visited it, and there the destination really was replaced and has to be
unlinked - the early return kept the name resolving to a file the rename
destroyed, so a dirty handle on it could still flush over what took its place.
The two cases are indistinguishable from inside MovePath, so leave it alone and
stop calling it twice: invalidateOpenFileHandle reports whether it moved, and
the handler moves only when it did not. Both paths mark a replaced file's
handle deleted, which the no-handle path previously did not do at all.
* mount: leave a rename alone unless the source is still ours to move
Two ways the fallback move could act on state it did not own. A handle whose
version guard skipped the event never reached RememberPath, so moving the table
under it left the handle flushing to the pre-rename path; the handle path owns
its inode's rename, so the fallback now runs only for an inode without one.
And the subscription can redeliver a rename once it falls out of the 4096-entry
dedup ring. A replay found no source and a live target, unlinked the mapping
the first delivery had just made, and marked the moved file's handle deleted -
worse than the stale path this set out to fix. Only a source still in the table
is moved now.
That gives up unlinking a destination the rename replaced when the source was
never visited here, which is where this started. It is what the mount already
did before this branch, and it is the safer of the two: retaining a stale name
costs a wrong lookup, while unlinking the wrong one costs a file's dirty data.
* mount: decide a rename move inside the table's lock
The source-presence check sat outside MovePath, so two invalidations for one
rename could both see the source and the loser would unlink what the winner had
just placed - the same damage the check was added to prevent. MovePath makes
the decision under its own lock now and reports that nothing moved.
The handle a rename destroyed is also marked from the caller rather than from
inside invalidateOpenFileHandle, which was setting isDeleted bare on a second
handle while holding the first one's lock. markHandleDeleted already takes the
lock the flush reads that flag under, and marking from the caller keeps it to
one handle lock at a time.
The invalidation test harness wires onEntryInvalidation now, the way the mount
does, rather than reaching past it.
* mount: apply a rename ahead of the handle's version fence
The fence exists so an old event cannot roll a handle's entry back to stale
content. A rename carries no content: it says the name the inode answered to is
gone. Skipping one on the strength of the fence left the inode and the handle
both pointing at a name the filer had vacated, and no fallback ran either,
since a handle owns its inode's rename.
Applied before the fence now, and only when MovePath reports the source was
still ours to move - which is what keeps a replayed rename from remembering a
path the handle has already moved past.
* Add monitoring label to filer servicemonitor
* helm: label the headless filer service, not the client one
The ServiceMonitor takes its job label from the service name, and the
bundled dashboard queries job="seaweedfs-filer". Selecting the client
service would have renamed the job and blanked those panels. The
headless service also publishes not-ready addresses, so it keeps
reporting while a filer is starting up or shutting down.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Four workflows ran apt to install libfuse3-dev before every FUSE job. Nothing
needs it: go-fuse implements the protocol in pure Go, no cgo in the tree
references fuse, and the package does not even provide the fusermount3 the
mount actually execs - fuse3 does, and it is already on the runner image, which
is why the setuid-repair step finds it.
So the step downloaded a dev package to build against headers no compiler ever
opened, and it is the step that has been hanging whenever the Ubuntu mirror
goes slow. Configuring /etc/fuse.conf is all that is left.
* rust: stop writing when a durable write's index flush fails
A durable write flushes the .dat, publishes the needle map row, then
flushes the .idx. If that last flush failed we returned the error and
carried on: the row stayed live, the volume stayed writable, and the
handler answered 500 without replicating. The primary then served a needle
its replicas never saw, for a write the client was told had failed - and
if the unflushed row was lost on restart, the durable .dat tail took the
volume read only anyway.
Taking the row back out is not an option: it means undoing published state
on a disk that is already failing, and a truncate afterwards would leave
an .idx row pointing past the end. So the volume stops taking writes
instead, the same as when the truncate after a failed .dat flush cannot be
done. Nothing more gets appended past a record whose index is in doubt,
and the master routes writes elsewhere once the volume heartbeats read
only. The divergence against the replicas is still there, but it is
bounded and it is visible.
A failed nm.put after the .dat is down leaves the same durable but
unindexed record, so it takes the same route.
* rust: drop the import the rollback removal left behind
NeedleValue came in with rollback_unflushed_write, which went away when
the durable path moved to flushing before it publishes. Nothing has used
the type since.
* rust: mark the test-only heartbeat helper as such
collect_heartbeat has only ever been called from the tests - the send loop
uses collect_heartbeat_with_snapshot, which it wraps - so a lib build
rightly called it dead code.
* rust: flush the index on a durable write that dedups
A durable write matching content already in the volume flushed the .dat
and returned before reaching the index flush. So a fsync=true write that
deduped against an earlier non-durable one was acked with the row that
indexes it still in the page cache - the same false promise the index
flush exists to rule out, and the same read-only volume on restart if the
row is lost.
The dedup path now flushes both files, and the quarantine on a failed
index flush moved into flush_idx so it applies wherever the flush is
reached rather than only at the one call site that had it inline.
The e2e workflow sets defaults.run.working-directory: docker, so the call I
added resolved to docker/docker/apt-install and every FUSE Mount run has
failed with 'sudo: docker/apt-install: command not found' since it merged.
Every workflow that builds the Rust volume server first installed protoc
from a package manager - twelve steps across apt, brew and choco. That is
37s per job on a good day, and this week archive.ubuntu.com stalled long
enough for four jobs to burn their whole timeout without reaching a build.
protoc-bin-vendored ships the compiler as a build-dependency, so it now
arrives through the cargo registry the workflows already cache and there
is nothing left to install. cargo build works on a machine with no protoc
at all, which is worth as much locally as it is in CI.
It also pins the version. The apt protoc on ubuntu-22.04 is 3.12, old
enough to reject proto3 optional, which is why build.rs passes
--experimental_allow_proto3_optional; the vendored one is 31.1. The flag
stays, since it costs nothing and keeps a build against an older PROTOC
working, and an explicit PROTOC still overrides the vendored binary for
packagers who supply their own.
The e2e job overwrote the runner's sources.list with two azure-only lines and
installed fuse from it, so the same mirror outage that took out the image
builds failed the step outright - this time on the runner rather than inside
the container, where the image-side fallback cannot reach.
Install through the same helper, and widen its rewrite to match any archive
host so it works whether the pristine list came from the base image
(archive.ubuntu.com) or from a CI runner (azure.archive.ubuntu.com). Keeping
the runner's original list also restores the security and backports pockets,
which the hand-written two-line replacement dropped.
Verified against the outage itself: with the pristine list pointed at Azure,
the build logged the skip after Azure timed out for real and installed from
archive.ubuntu.com.
The e2e image pointed both archive and security at azure.archive.ubuntu.com and
nothing else, and the samba and pjdfstest images inherit that list. When Azure
is unreachable the build has nowhere to go: Acquire::Retries just retries a dead
host, every package fails, and apt exits 100 before a single test runs. Two
different workflows lost runs to it tonight.
Install through a helper that starts from the pristine sources.list each time
and walks a list of mirrors, so Azure stays the preferred one - the reason it
was pinned in the first place - without being the only one.
Verified both paths against a real build: the normal one installs from Azure,
and with the first entry pointed at an unroutable host the fallback logs the
skip and installs from archive.ubuntu.com.
* Give the local Unix socket gRPC transport room to breathe
Unix socket buffers default small and never autotune: 208KB on Linux, 8KB on
macOS. Once the buffer cannot absorb what gRPC's loopyWriter emits for the
in-flight streams the writer blocks on Write, and since v1.82.1 grpc-go counts
per-RPC bookkeeping toward its control-buffer throttle, so both peers stop
reading and the connection deadlocks for good. weed mini wedged at roughly 320
concurrent S3 PUTs with every filer RPC parked in waitOnHeader and no handler
running.
Force 8MB on both ends of the sockets we open. Best effort, since a kernel may
clamp it lower; that only lowers the concurrency this survives. TCP loopback
never hit this because its buffers start large and grow.
* Set the buffer on accepted connections too
Linux does not carry the listener's SO_SNDBUF onto sockets returned by accept,
so only the dialing half was getting the headroom: measured 8388608 on the
dialed side against the 212992 default on the accepted side. Wrap the listener
and re-apply per connection. macOS inherits either way, which is why this did
not show up locally.
InodeEntry held its one path in a slice, so every inode the kernel references
cost a 16-byte backing array and a second heap object on top of the 32-byte
entry. The extra links of a hard-linked file now hang off a pointer instead,
which keeps the struct in the same 32-byte size class and leaves the ordinary
single-link file with nothing to allocate.
Populating the table with 1M children: 237.5 -> 221.5 B/inode at 85-character
paths, 301.3 -> 285.6 at 148.
readdir built dirPath.Child(name) for every child while entry.FullPath was
already that exact string, from NewFullPath in the meta cache store or from
FromPbEntry on the read-through path. One allocation per entry, and on a wide
tree with long paths that is most of what a listing allocates.
BenchmarkReadDirectory/kernel_readdirplus over 200k entries: 2,039,656 ->
1,839,318 allocs/op, 174.5 -> 167.8 MB/op, 152.6 -> 135.1 ms/op.
The Rust volume server ignored the fsync parameter completely: nothing
parsed it, and write_volume_needle -> write_needle -> append_needle never
flushed. So a ?fsync=true upload was acked out of the page cache, and
since ReplicatedWrite forwards the parameter, a Go primary handing a
durable write to a Rust replica got the same empty promise.
The upload handler now reads fsync the way Go's r.FormValue does, off the
decoded query fields, and threads it down to the volume. A durable write
appends, flushes the .dat, publishes the needle map entry, then flushes
the .idx, and only then is it acked. Nothing points at bytes that are not
down yet, so a failed flush only has to take its own append back off the
end - the index never moved and the volume's counters never saw the
rejected write. If that truncate cannot be done the volume stops taking
writes, rather than letting a later append bury the rejected record
mid-file where the tail integrity check cannot see it.
The .idx flush is what keeps the ack honest: load() rebuilds the map from
.idx, so an acked write whose row was lost comes back as a .dat tail the
integrity check cannot account for, and the volume loads read only.
A dedup hit flushes too: there is nothing to append, but the write it
matched may have been non-durable, and the caller is asking for the
content to be on disk.
Batched writes carry the flag per request rather than one flush per
batch, so the write queue's module doc no longer claims otherwise.
A single Read on the response body can return the last bytes together
with io.EOF, so asserting NoError on it fails even though the body is
complete. Use io.ReadAll, like every other test in this package.
* volume: forward fsync=true to replicas in ReplicatedWrite
When a write request carries fsync=true, only the primary volume server
flushed to disk: the replica fan-out URL in ReplicatedWrite only carried
type/ttl/ts/cm, so replicas always wrote without fsync even when the
client explicitly requested a durable write.
Forward the fsync request parameter to the replica volume servers so a
durable write means every replica has flushed to disk, not just the
primary. Replicas without fsync are untouched (zero behavior change).
* storage: flush a durable write inline while stopping
The fsync flag on the write path really selects the async batch worker,
and it was switched off once the store is stopping. So a fsync=true write
landing during the pre-stop drain got acked without ever being flushed -
and now that ReplicatedWrite forwards fsync, that covers replicas too.
Flush it inline instead of queueing it. The drain keeps accepting writes,
which is the whole point of preStopSeconds, and the ack still means the
.dat is on disk. If the fsync fails, the append comes back off the .dat
and the needle map goes back to what it pointed at before, so nothing
resolves to an offset past the truncated end.
* storage: make the store's stopping flag atomic
SetStopping runs on the signal handler goroutine while the write and
vacuum paths read the flag, so every read of it was racy. Nothing about
the shutdown ordering changes; only the flag itself is now safe to read.
* topology: check the errors the replication test was dropping
The mock replica ignored its response write and the mock master ignored
whatever Serve returned, so a broken mock would have shown up as a
confusing timeout rather than a failure. Also drops the explicit listener
close: grpc.Server.Stop already closes the listener it was given.
---------
Co-authored-by: hzsunchao <hzsunchao@corp.netease.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3: answer GetBucketReplication, GetBucketWebsite and GetBucketNotificationConfiguration
None of the three had a route, so they reached the unconstrained ListObjectsV1
catch-all and a client asking for a bucket's replication config got 200 and a
<ListBucketResult> back. Replication and website report their configuration as
absent the way AWS does; notification returns the empty configuration AWS
returns for a bucket with no events wired up.
* s3: stop an unrouted bucket subresource from being answered with a listing
ListObjectsV1 is the catch-all GET on a bucket, so every subresource without a
route of its own - ?torrent today, whatever AWS adds next - came back 200 with a
<ListBucketResult>. A client that asked for a configuration and got a listing
either fails its XML decode in a way that reads like corruption, or worse,
tolerantly parses it. Refuse the request instead.
The allow-list is the ListObjects parameters rather than the subresources,
so a new one fails closed. Presigned URLs sign their credentials into the
query string, so X-Amz-* and the SigV2 trio have to stay listable.
* filer: add filer.options.disable_remote_storage_deletion for cache-only deletes
Deleting a filer entry under a remote.mount path also deletes the backing
object from the remote store (maybeDeleteFromRemote). Deployments that use a
remote mount as a read-through cache in front of an authoritative,
externally-managed object store cannot allow this: the filer typically holds
read-only credentials, so the remote delete fails and the entire delete
errors out; and even where it would succeed, it destroys data the filer does
not own.
Add filer.options.disable_remote_storage_deletion (default false, so existing
behaviour is unchanged). When enabled, maybeDeleteFromRemote is skipped for
both single-entry and recursive folder deletes: local metadata and cached
chunks are still removed, but the remote object is left intact.
* filer: assert local removal in cache-only recursive delete test
The recursive cache-only delete test only checked that no remote delete
happened; it did not verify the local child and directory entries were
removed. Add FindEntry assertions so a regression that skips local
recursive deletion is caught.
* filer: reload the remote mount mapping when /etc/remote changes
The mapping was only read at startup, so remote.unmount left the mount live
in the filer: the purge that follows the mapping delete then went to the
remote store and wiped every object under the mount.
Rebuild the rules trie and the conf map from scratch on each load, since
ptrie cannot drop a key, and swap them under a lock.
* filer: drop the filer-wide remote deletion switch
With the mapping reloaded on unmount, the purge no longer reaches the remote
store, so there is nothing left for the switch to protect against.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
The cached readdir trims the head of the handle's entry stream as the
client walks past it; the read-through path never did, so a directory
too large to cache -- the only kind that takes that path -- was held
whole in the handle for the length of the walk. Hoist the trim to cover
both paths.
* mount: re-resolve volume locations after a failed chunk read
NewChunkGroup passed nil as the ReaderCache's CacheInvalidator, so
retryFetchAfterCacheInvalidation was dead code on the FUSE read path. A
mount that cached a volume's locations while one server was down kept
retrying that server after it died, then returned EIO, even though the
master and filer both resolved the live replica. The S3 gateway already
passes its filerClient; do the same for the mount.
* test: FUSE integration tests for volume server failover
One mount appends while a second tails, and a volume server is killed,
started or restarted mid-stream against a 001-replicated cluster of three
volume servers. Automates the scenario matrix reported for Docker Swarm
mounts, including the large-file variant and a no-chaos control.
* test: report the filer's own view when append content mismatches
A mismatch between what the writer wrote and what the reader sees can come
from either side's cache. Read the file back through the filer's HTTP
handler as well, and let the mount verbosity be raised from the
environment, so a failing run says which layer lost the data.
* test: wait for the reader mount to converge before comparing
A mount caches metadata for about a second, so reading the file the instant
the writer's last close returned can legitimately come back short. Poll the
reader until it matches or the timeout expires; content that is wrong rather
than merely late never converges and still fails, now with the writer's
mount and the filer's own view alongside it.
* test: detect a failover cluster child that exited at startup
Signal(0) succeeds for a zombie and nothing reaped these children until
shutdown, so a process that died on startup looked alive until the readiness
timeout expired. Reap each child as it is started and consult the result.
* test: read a file the killed volume server actually holds
Placement decides which two of three servers back each volume, so killing
volume N and reading readfile-N could pass without the victim ever holding a
replica of it. Resolve each file's volumes through the filer and the master,
and pick one the victim backs, preferring a file the reader has not cached.
* ci: stop persisting checkout credentials in the failover workflow
The job does not use the token after cloning. Also tag the README's command
block as bash and match the timeout the workflow actually uses.
* test: discard the ignored errors errcheck flags in the failover harness
* test: resolve manifests when mapping a file to its volumes
A manifest chunk's own fid names the volume holding the manifest, not the
volumes holding the data, so a large enough file would point the failover
victim at the wrong server.
* test: pin the stale-location recovery path with a primed reader
Reading a file for the first time after a server dies proves nothing: the
lookup is fresh and returns the survivor. Kill one holder and wait for the
master to drop it, read a file on that volume so the reader caches the lone
survivor, restart the first server, then kill the survivor. The reader's only
cached location is now dead while the data is live elsewhere, which is the
case the invalidator exists for: EIO without it, recovery with it.
* filer: re-look-up a chunk's locations as soon as they all fail
A read that fails against every location it was given is far more likely to be
holding a stale list than to be hitting a cluster that is briefly slow, but the
retry loops spent the whole backoff ladder, about 13 s, before the caller got a
chance to invalidate and look the chunk up again. Give the loops a refresh hook
and let the reader cache invalidate on the first fully failed pass, so recovery
starts in milliseconds. Clients without an invalidator keep the old behavior.
The filer's streaming read path has its own fetch loop and is not covered.
* webdav: give the chunk reader a bounded, invalidatable location cache
WebDav resolved chunk locations through filer.LookupFn, whose own doc asks
long-running processes to prefer wdclient.FilerClient: its cache is unbounded,
and it has no way to invalidate an entry, so the reader cache was constructed
with a nil invalidator and a WebDav server that had cached a location kept
reading from it after the volume moved or died. Use FilerClient, as the mount
and the S3 gateway already do.
* filer: refresh locations on the random-read path too
readChunkSliceAt bypasses the chunk cacher in random-access mode and fetches
the range directly, which left it without the invalidation the cacher does:
a random reader parked on a stale location had no way back at all. Hoist the
refresh hook onto the reader cache so both paths share it.
* filer: compare chunk locations as a set, not in order
Lookups shuffle the locations they return, so comparing positionally reads a
reshuffle of the very same replicas as a fresh set and spends an immediate
retry on locations that just failed. weed/filer already had an
order-independent comparison for this; move it next to the retry loops so
both callers share one helper.
* mount: re-resolve volume locations after a failed chunk read
NewChunkGroup passed nil as the ReaderCache's CacheInvalidator, so
retryFetchAfterCacheInvalidation was dead code on the FUSE read path. A
mount that cached a volume's locations while one server was down kept
retrying that server after it died, then returned EIO, even though the
master and filer both resolved the live replica. The S3 gateway already
passes its filerClient; do the same for the mount.
* test: FUSE integration tests for volume server failover
One mount appends while a second tails, and a volume server is killed,
started or restarted mid-stream against a 001-replicated cluster of three
volume servers. Automates the scenario matrix reported for Docker Swarm
mounts, including the large-file variant and a no-chaos control.
* test: report the filer's own view when append content mismatches
A mismatch between what the writer wrote and what the reader sees can come
from either side's cache. Read the file back through the filer's HTTP
handler as well, and let the mount verbosity be raised from the
environment, so a failing run says which layer lost the data.
* test: wait for the reader mount to converge before comparing
A mount caches metadata for about a second, so reading the file the instant
the writer's last close returned can legitimately come back short. Poll the
reader until it matches or the timeout expires; content that is wrong rather
than merely late never converges and still fails, now with the writer's
mount and the filer's own view alongside it.
* test: detect a failover cluster child that exited at startup
Signal(0) succeeds for a zombie and nothing reaped these children until
shutdown, so a process that died on startup looked alive until the readiness
timeout expired. Reap each child as it is started and consult the result.
* test: read a file the killed volume server actually holds
Placement decides which two of three servers back each volume, so killing
volume N and reading readfile-N could pass without the victim ever holding a
replica of it. Resolve each file's volumes through the filer and the master,
and pick one the victim backs, preferring a file the reader has not cached.
* ci: stop persisting checkout credentials in the failover workflow
The job does not use the token after cloning. Also tag the README's command
block as bash and match the timeout the workflow actually uses.
* test: discard the ignored errors errcheck flags in the failover harness
* test: resolve manifests when mapping a file to its volumes
A manifest chunk's own fid names the volume holding the manifest, not the
volumes holding the data, so a large enough file would point the failover
victim at the wrong server.
* test: pin the stale-location recovery path with a primed reader
Reading a file for the first time after a server dies proves nothing: the
lookup is fresh and returns the survivor. Kill one holder and wait for the
master to drop it, read a file on that volume so the reader caches the lone
survivor, restart the first server, then kill the survivor. The reader's only
cached location is now dead while the data is live elsewhere, which is the
case the invalidator exists for: EIO without it, recovery with it.
* filer: re-look-up a chunk's locations as soon as they all fail
A read that fails against every location it was given is far more likely to be
holding a stale list than to be hitting a cluster that is briefly slow, but the
retry loops spent the whole backoff ladder, about 13 s, before the caller got a
chance to invalidate and look the chunk up again. Give the loops a refresh hook
and let the reader cache invalidate on the first fully failed pass, so recovery
starts in milliseconds. Clients without an invalidator keep the old behavior.
The filer's streaming read path has its own fetch loop and is not covered.
* filer: refresh locations on the random-read path too
readChunkSliceAt bypasses the chunk cacher in random-access mode and fetches
the range directly, which left it without the invalidation the cacher does:
a random reader parked on a stale location had no way back at all. Hoist the
refresh hook onto the reader cache so both paths share it.
* filer: compare chunk locations as a set, not in order
Lookups shuffle the locations they return, so comparing positionally reads a
reshuffle of the very same replicas as a fresh set and spends an immediate
retry on locations that just failed. weed/filer already had an
order-independent comparison for this; move it next to the retry loops so
both callers share one helper.
* filer: re-list a folder after deleting it, and put it back if it is not empty
The emptiness check inside the delete and the removal of the folder entry are
not atomic, so an entry can land between them and be left reachable by its own
path but out of every listing. Looking again after the delete catches the ones
whose create event has not arrived yet, and does not depend on the event stream
or on the observation window holding.
* filer: create the directories holding an entry after the entry
A parent checked before the insert can be taken by the empty-folder cleaner
before the entry lands, which leaves the entry reachable by its own path but out
of every listing. Creating the parents afterwards cannot be undone by a delete
that was authorised before the insert, and pairs with the cleaner re-listing
after its own delete: whichever of the two acts second sees what the other did.
Going second means the entry is already stored when the parent fails, so it is
taken back out and the caller still sees the error it used to get.
* filer: narrow a directory that came back wider than the one it replaced
A writer recreating its own missing parent has only the entry it is inserting to
go on, so the directory it mints can grant access the deleted one denied - a
0700 folder comes back 0751. The cleaner read the real attributes before
deleting, so its restore now puts the original mode back instead of leaving the
inferred one in place. It only ever narrows, so a directory deliberately
tightened since is left as it is.
* mount: re-resolve volume locations after a failed chunk read
NewChunkGroup passed nil as the ReaderCache's CacheInvalidator, so
retryFetchAfterCacheInvalidation was dead code on the FUSE read path. A
mount that cached a volume's locations while one server was down kept
retrying that server after it died, then returned EIO, even though the
master and filer both resolved the live replica. The S3 gateway already
passes its filerClient; do the same for the mount.
* test: FUSE integration tests for volume server failover
One mount appends while a second tails, and a volume server is killed,
started or restarted mid-stream against a 001-replicated cluster of three
volume servers. Automates the scenario matrix reported for Docker Swarm
mounts, including the large-file variant and a no-chaos control.
* test: report the filer's own view when append content mismatches
A mismatch between what the writer wrote and what the reader sees can come
from either side's cache. Read the file back through the filer's HTTP
handler as well, and let the mount verbosity be raised from the
environment, so a failing run says which layer lost the data.
* test: wait for the reader mount to converge before comparing
A mount caches metadata for about a second, so reading the file the instant
the writer's last close returned can legitimately come back short. Poll the
reader until it matches or the timeout expires; content that is wrong rather
than merely late never converges and still fails, now with the writer's
mount and the filer's own view alongside it.
* test: detect a failover cluster child that exited at startup
Signal(0) succeeds for a zombie and nothing reaped these children until
shutdown, so a process that died on startup looked alive until the readiness
timeout expired. Reap each child as it is started and consult the result.
* test: read a file the killed volume server actually holds
Placement decides which two of three servers back each volume, so killing
volume N and reading readfile-N could pass without the victim ever holding a
replica of it. Resolve each file's volumes through the filer and the master,
and pick one the victim backs, preferring a file the reader has not cached.
* ci: stop persisting checkout credentials in the failover workflow
The job does not use the token after cloning. Also tag the README's command
block as bash and match the timeout the workflow actually uses.
* test: discard the ignored errors errcheck flags in the failover harness
* test: resolve manifests when mapping a file to its volumes
A manifest chunk's own fid names the volume holding the manifest, not the
volumes holding the data, so a large enough file would point the failover
victim at the wrong server.
* test: pin the stale-location recovery path with a primed reader
Reading a file for the first time after a server dies proves nothing: the
lookup is fresh and returns the survivor. Kill one holder and wait for the
master to drop it, read a file on that volume so the reader caches the lone
survivor, restart the first server, then kill the survivor. The reader's only
cached location is now dead while the data is live elsewhere, which is the
case the invalidator exists for: EIO without it, recovery with it.
* volume: accept only static-key gcs credentials on the fetch request
An inline credentials document of a federated type points the SDK at a url,
file or executable of the caller's choosing for the token exchange, so the
request-supplied value is no longer just a key.
* volume: guard the gcs token endpoint like the other remote endpoints
Inline credentials pick where the token request goes, so route the gcs client
through the same deny-list and rebinding-safe dialer used for S3 and azure.
* rust volume: pin that gcs has no credential-driven dial path
* volume: only check gcs credentials on a gcs remote conf
Only the gcs backend reads that field, so another backend carrying a stale
value should not fail the request.
* gcs: load credentials with the type the caller expects
The untyped loader is deprecated because it reads whatever the document
claims to be; callers handling credentials they do not control now name the
types they accept.
* arangodb: bind list prefix, start file name and collection into the AQL query
Concatenating them into the query text let a caller-supplied prefix or
start name close the string literal and append arbitrary AQL, which runs
with the filer's ArangoDB credentials against any collection.
* arangodb: bind the folder path and collection into the recursive delete query
A trailing-slash S3 key reaches DeleteFolderChildren through the
directory-marker cleanup, so quotes in the path could turn the filter
into a match-everything REMOVE over the whole bucket collection.
* arangodb: match the real directory prefix in the recursive delete
The prefix was built by re-joining the path segments with commas, so it
never matched a stored directory and the subtree sweep did nothing.
* worker: log the balance move stage through a constant format string
Go 1.26's printf analyzer now follows printf wrappers reached through an
interface, so passing the stage straight to Logger.Info is a vet failure.
* s3api: bracket the IPv6 host in the signature test URL
A bare IPv6 literal is legal in a Host header but never in a URL. Go 1.26
stopped parsing it leniently, so carry the two forms separately and set
r.Host to the value the client would actually have signed.
* mini: bracket IPv6 addresses in the readiness probe URLs
An IPv6-only host hands mini a bare literal, and %s:%d pasted it into a URL
unbracketed. Under Go 1.26 that URL no longer parses, so waiting for the
admin server never succeeds and mini refuses to start.
* readme: add RustFS to the file system comparison
* readme: note RustFS write amplification and rigid layout
* readme: correct RustFS version, parity and protocol details
* readme: merge the RustFS comparison into the MinIO section
* filer: restore a folder that received an entry while it was deleted
The empty-folder cleaner checks that a folder is empty and then deletes it,
and those two steps are not atomic. An entry created in between survives the
delete but loses the directory holding it: still readable by its own path, yet
absent from every listing until a later write happens to recreate the parent.
Record the folders deleted in each pass and re-check them on the next one,
putting back any that turned out to hold entries. The check waits a pass on
purpose - a writer looks up the parent before inserting the child, so checking
straight after the delete can still run ahead of the insert and see nothing.
Restoring a directory that holds entries is always correct, and restoring one
whose entry went away again just leaves an empty folder for a later pass to
collect, so the repair needs no locking or coordination.
Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r
* filer: keep failed restores queued and inherit the ancestor's ownership
Two gaps in the restore pass.
A folder whose count or restore hit a transient store error was dropped from
the tracking list and never looked at again, leaving its entries out of
listings until some later write recreated the folder - the very thing the pass
exists to avoid. Put those back for the next pass, still under the cap.
A restored folder was minted with a fixed mode and no owner, so a directory
that had been private came back world-readable and owned by root. Take the
mode and ownership from the nearest ancestor still present instead.
Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r
* filer: let the redis stores keep a directory listing that still has entries
On the redis stores the listing is not derived from the entries, it is the only
record that they sit under that directory. DeleteEntry opened by dropping it
outright, so an entry that arrived after the caller judged the directory empty
lost its membership and became unreachable: readable by exact path, absent from
every listing, and invisible to any later check, since counting the directory
reads the listing that was just destroyed. Nothing could detect or repair it.
Drop the listing in DeleteFolderChildren instead, alongside the children it
describes, and leave it alone in DeleteEntry. redis3 needs it explicitly, since
removeChildren clears the skip list nodes but not the list itself, and the plain
redis store was leaking the key entirely.
Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r
* filer: restore folders with their own attributes, and observe them for a window
Five gaps in the restore pass.
The restored directory was reconstructed from whatever ancestor happened to
still be present, and the mode was ORed with 0111 on the way. A private
directory under a world-traversable parent came back granting traversal it had
denied. Read the folder's own attributes before deleting it and put exactly
those back. That also removes the ancestor walk, which treated a transient
store error as "not found" and silently fell through to a broader ancestor.
A single check a pass later was not a delay at all. Ticker sends coalesce, so
when a pass runs long the next one starts immediately, and a writer already
past its parent lookup can insert after the check has read zero - after which
the folder was discarded for good. Keep each folder under observation for a
bounded wall-clock window and re-check it on every pass until it expires. This
narrows the exposure rather than closing it; only making the emptiness check
and the delete atomic would do that.
A delete that returned an error was never observed at all, though the redis
stores drop the folder before its parent-list member, so a failure return is
not proof the folder survived. Record the folder before the delete instead.
Restores now run shallowest first, so a folder taken by the parent cascade is
rebuilt with its own attributes before anything below it needs it as a parent.
Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r
* filer: recover a deleted folder from the create event for the entry that raced it
Checking each deleted folder on a timer was the wrong instrument. It cost a
listing per folder per pass, and it could only ever be a guess about when the
racing write would land.
The metadata stream already carries the answer. A folder is recorded before it
is deleted, so any entry that can be orphaned is created after that record and
its create event names that exact directory. Match the event against the
recently deleted folders and the folder is known to need putting back, rather
than inferred to.
The window stops being a guess at the race and becomes what it should be: how
far behind the event stream is allowed to run before a folder stops being
watched. Listing is now done once, for a folder an event has already named, to
skip the restore when the entry has since gone away again.
Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r
* filer: bound how long a folder is watched, and rebuild ancestors from themselves
Four gaps found reviewing the restore pass.
A folder whose restore kept failing was never let go: the written-to check ran
before the age check, so it was picked up, retried, put back, and counted again
on every pass for the life of the process. Apply the window first, whatever
state the folder is in.
At the cap, the folder being recorded was the one turned away, though it is the
one whose race is still live - the older entries are already close to ageing
out. Give up one of those instead, picked as the oldest of a small sample so
the cost stays flat under heavy deletion rates.
An ancestor taken by the same cascade was left to the descendant's restore to
recreate, which minted it from the descendant's attributes and handed back
access the ancestor never granted. Rebuild those from what they were, ahead of
anything below them.
Reading a directory's attributes assumed an entry came back. Some stores return
nothing with no error, so treat that as not found. The mode is also taken whole
rather than through Perm(), which was dropping setgid, setuid and sticky.
Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r
* redis3: take a directory listing left behind by a failed delete
Removing the last name deletes the list, and if that delete fails the header
survives pointing at a name that is gone. The retry finds nothing to remove,
reports no changes, and returns before reaching the delete, so the key stays
for good. Take it on that path too.
Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r
* filer: do not sweep children when deleting a folder non-recursively
doBatchDeleteFolderMetaAndData lists a folder and bails out if it has any
children, then calls Store.DeleteFolderChildren unconditionally. On the
non-recursive path that bulk sweep has nothing legitimate to remove: it only
runs once the listing came back empty, so the sole rows it can delete are
ones inserted after the check.
The S3 empty-folder cleaner deletes through this path, so a PUT landing
between the listing and the sweep loses its entry after the write was already
acknowledged. Neither side sees an error - the client has its 200 and the
cleaner logs an ordinary empty-folder deletion - and the chunks leak, since
the cleaner passes shouldDeleteChunks=false and nothing was enumerated to
collect. Workloads that scatter objects over many shallow prefixes empty and
refill those folders constantly, which is what makes the window reachable.
Sweep only when the delete is recursive, or when the whole-bucket shortcut
skipped the listing and depends on it.
Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r
* filer: pin the folder entry removal left by the racing-child test
The surviving entry is reachable by path but drops out of listings until the
folder comes back, and nothing in the test said so. Assert it, so the exposure
that remains after this change is visible rather than implied.
Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r
* iceberg: vend table-scoped credentials to clients that ask for delegation
The catalog recognised X-Iceberg-Access-Delegation: vended-credentials
and then deliberately said nothing, because it had nothing to vend: it
withheld even the S3 endpoint so the client would keep the credentials it
was configured with. That left every engine expecting the catalog to hand
out access - Snowflake, Databricks, Trino with vending, any multi-tenant
setup - needing static S3 keys distributed out of band.
Mint an STS session per request instead, scoped by a session policy to
the table's own prefix plus the bucket listing needed to resolve it, and
return it in the load response config and storage-credentials. The role
to assume is named by -s3.iceberg.credentialRole; its trust policy is
what decides whether a caller may assume it, and vending stays off until
it is set. A failed mint falls back to the old silence rather than
handing back an endpoint the client cannot sign for.
* iceberg: keep vended credentials inside the table prefix
Review follow-ups on credential vending:
Listing was granted on the bucket ARN with no condition, so a credential
vended for one table could enumerate every other table's object names.
Constrain s3:prefix to the table's own prefix, which the S3 gateway
already populates for list requests.
A table location carrying * or ? would have gone into the policy's
resource pattern unescaped and widened the session to sibling prefixes.
Refuse to vend for such a location rather than escaping it; nothing the
catalog generates contains those characters.
DurationSeconds skipped the 900..43200 bounds the other assume-role paths
enforce, so -s3.iceberg.credentialDurationSeconds could ask for a session
outside them. The check is now shared by all three entry points.
* iceberg: return the vended credentials from buildFileIOConfig itself
buildStorageConfig was a second name for what buildFileIOConfig already
did; it now returns the storage credentials alongside the properties, and
callers that only want the properties drop them.
* iceberg: split the vended bucket grants, and refuse a whole-bucket scope
The prefix condition sat on a statement that also granted
GetBucketLocation and ListBucketMultipartUploads, neither of which carries
an s3:prefix to satisfy it, so both were denied for every vended
credential. GetBucketLocation moves to its own unconditioned statement.
ListBucketMultipartUploads is dropped: Iceberg writers complete and abort
by upload id, and granting it either leaks in-flight keys bucket-wide or
breaks on the same missing prefix.
A table whose location has no prefix - one registered at the bucket root -
would have been vended read and write over every other table in the
bucket. Refuse, the way a location with wildcards is refused.
* iceberg: add view rename, scan-report and snapshots=refs to the catalog
Three gaps against the REST spec that clients hit in normal use:
Views had no rename, though tables did and views are stored the same way,
so the move is the same catalog-only pointer move. Tables and views share
a namespace directory, so both renames now refuse the other kind instead
of moving it.
Engines POST a scan or commit report after planning; a 404 there turns
into an error line per query. Accept the report and discard it - the
catalog keeps no metrics store.
LoadTable ignored ?snapshots=refs and always returned the whole snapshot
history, which is what clients use the parameter to avoid on long-lived
tables.
* iceberg: authorize view rename against the view ARN, tighten the metrics endpoint
Review follow-ups:
The shared rename checked the source against a table ARN whatever the
kind, so a policy scoped to a view's own ARN never matched and one
written for a table ARN was evaluated for a view. The entry kind now
carries the ARN builder.
The metrics endpoint truncated a report at 1 MiB and then failed to parse
it, answering 400 for a query that had actually succeeded. Read one byte
past the limit to tell "fits" from "cut short", and discard an oversized
report instead of rejecting it. Empty bodies and reports without a
report-type are now rejected, which the REST schema requires.
?snapshots= is defined for LoadTable, so it no longer filters what
CreateTable echoes back.
* iceberg: make a table commit a compare-and-swap
The catalog validated the caller's version token, ran its authorization
checks, and only then wrote the new metadata xattr. Two engines
committing against the same base both passed that check and both wrote,
so the second silently dropped the first one's snapshot. Both also derive
the same v{N}.metadata.json name and the file write overwrote, leaving
the surviving pointer aimed at the loser's metadata - and the loser's
conflict cleanup then deleted the winner's file.
Write the metadata file with an exclusive create and update the xattr
conditionally on the bytes the handler read, the way the maintenance
worker already commits. A writer that lost the race re-reads and retries,
and reports 409 CommitFailedException once out of attempts.
* iceberg: stage a commit under a unique name when the versioned one is taken
Two follow-ups from review of the commit compare-and-swap:
Refusing to overwrite v{N}.metadata.json also refused to get past a file
left behind by a commit that died between staging and updating the
pointer. Every later commit derived the same name, saw the collision, and
reported a conflict, so the table stayed uncommittable until an orphan
sweep removed the file. Stage under v{N}-{uuid} instead: neither writer's
file is overwritten and the catalog pointer still decides who won, which
is how the maintenance worker has always staged its own metadata.
metadataVersionFromLocation learned to read the version back out of that
name.
The conditional update guarded only the metadata attribute while the
write replaced the whole entry, so a policy or tag written in the same
window was silently reverted. Guard every catalog attribute, which turns
that into a conflict the caller retries on fresh state.
* iceberg: give saveMetadataFile the exclusive flag instead of a second name
saveNewMetadataFile, saveMetadataBlobExclusive and uniqueMetadataFileName
were three new names around one existing helper. The flag now rides on
saveMetadataFile and saveMetadataBlob, and the unique-name construction
sits where it is used.
* iceberg: reuse the filer CAS helpers #10773 added, and stage transactions exclusively
#10773 landed mutateEntryExtended, which already writes an entry back under a
whole-entry precondition and retries. Drop the helper this branch added and
route the table commit through it: the check that the metadata is still the
one this request read now lives in the mutation, where it sees current state.
The policy the request was authorized against is asserted too, so an
administrator restricting it mid-commit sends the caller back through
authorization instead of having a stale decision applied. Bucket and
namespace policies live on other entries and a single-entry precondition
cannot cover them.
Multi-table transactions stage their metadata exclusively for the same
reason single-table commits do, and carry the name they landed on into the
pointer flip.
#10774 gave buildTestMetadata its refs and age parameters while #10773
added a caller with the old arity. Each was green against a master that
did not yet have the other, and the merge of both does not compile, so
vet and the unit tests fail on master.
* iceberg maintenance: keep the snapshots that branches and tags pin
expireSnapshots only ever protected the current snapshot, so a snapshot
held by a tag or a non-main branch was expired once it aged out of the
retention window. iceberg-go's RemoveSnapshots drops any ref whose
snapshot is gone without complaint, so the tag disappeared and the files
behind it were deleted as unreferenced.
Protect every ref target, and honour a branch's own
min-snapshots-to-keep / max-snapshot-age-ms over the ancestors behind its
head. Detection skips pinned snapshots for the same reason: proposing a
job whose only outcome is a no-op keeps the worker busy forever.
* iceberg maintenance: re-plan when a ref appears mid-commit, and stop proposing no-op expiry
Three follow-ups from review of the ref-aware expiry:
The commit guard only compared the table head, so a tag created between
planning and commit could pin a snapshot the plan was about to expire.
Re-check the refs against the metadata the commit actually reads.
Detection now asks snapshotsToExpire what execution would remove instead
of approximating with its own count-and-age rules. Expiry always requires
a snapshot past the retention window, so a table over the quota whose
snapshots are all young was being proposed for a job that could only
no-op.
The branch retention test could not tell "retained the whole lineage"
from "honoured min-snapshots-to-keep", because the branch had exactly as
many ancestors as the count. Give it one more, and cover
max-snapshot-age-ms too. Both need snapshots genuinely older than a
retention window, which iceberg-go will not accept at build time, so the
fixture backdates the metadata after building it.
* iceberg maintenance: fold the metadata test builders back into one
buildTestMetadata, buildTestMetadataWithRefs, buildTestMetadataAged and
buildTestMetadataNow were four names for one thing. Keep the original and
give it the refs and age it needs.
* s3tables: add the maintenance configuration APIs
Stores the configuration verbatim as the wire shape under a new
s3tables.maintenance extended attribute, so Get hands back what Put took
and no translation layer can drift from the AWS model.
Nothing reads the configuration yet.
Put merges a single type into the stored map so configuring compaction
does not drop snapshot management, and asserts the attribute's prior value
so two concurrent Puts cannot silently clobber each other.
* iceberg: apply the maintenance configuration in the worker
The worker now reads the per-table and per-bucket maintenance
configuration written by the control plane, so the wildcard plugin config
is a default rather than the only setting a table can have.
Table properties still win by default, since a table declaring its own
layout is what every engine honours and the compactor has to agree with
whoever writes the files. Clearing table_properties_override makes the
maintenance configuration authoritative instead.
Status is not part of that contest: a disabled type drops its operations
and no property can re-enable them, so the operator's kill switch always
holds. Manifest and delete-file rewrites have no AWS equivalent and ride
with compaction.
Detection reads both attributes from entries it already lists.
* s3tables: report maintenance job status
The worker records the outcome of each run in its own extended attribute,
separate from the configuration so operator and worker writes do not
contend, and GetTableMaintenanceJobStatus reads it back.
Only the types a run touched are written, so a partial run cannot erase
what an earlier one recorded. The reader fills in the rest: Disabled when
the configuration switched a type off, Not_Yet_Run otherwise.
Status is advisory, so a lost race is logged rather than failing a job
whose work already committed.
* s3tables: route the maintenance APIs over REST
The five actions were only reachable by X-Amz-Target dispatch, which the
AWS CLI and SDK do not use for this service. They address the operations
by path, so the APIs were unreachable from any official client.
* s3tables: fix the table bucket ARN field name
GetTableBucketMaintenanceConfiguration emitted tableBucketArn where the
wire field is tableBucketARN, as every other response in this package
already spells it. Official SDK deserializers ignore the unknown key, so
the required field came back unset.
* s3tables: carry the compaction strategy through to the worker
IcebergCompactionSettings modelled only targetFileSizeMB, so a request
naming a strategy was accepted and then dropped on the way to storage.
The worker now maps binpack and sort onto its own rewrite strategy and
lets auto defer to the worker configuration.
z-order is rejected rather than accepted and quietly binpacked.
* s3tables: report bucket-level maintenance status
GetTableMaintenanceJobStatus read only the table's configuration, so
unreferenced file removal — which is configured on the bucket — reported
Not_Yet_Run or a stale success after an operator disabled it.
The merge helper now lives in this package and the worker shares it.
* iceberg: delete orphans only after the non-current window
AWS marks a file non-current once it has been unreferenced for
unreferencedDays, then deletes it a further nonCurrentDays later.
The cutoff was taken from unreferencedDays alone, so a 3/10 configuration
hard-deleted on day three and threw away the ten day recovery window.
remove_orphans deletes in one step rather than marking, so the cutoff is
now the sum of the two.
* s3tables: assert every attribute when rewriting an entry
UpdateEntry writes the whole entry back from the snapshot the caller
read, and its precondition only covers the keys the caller names. Both
maintenance writers named one key, so a job status write could revert a
maintenance configuration an operator had just disabled, turning an
advisory write into a silent re-enable.
Both now assert the entry's full attribute set, including the target key
when absent so a concurrent create also fails the precondition.
* s3tables: assert absent attributes when rewriting an entry
The precondition covered the attributes present when the writer read the
entry, so an attribute created between that read and the write was absent
from it. A first-time PutTableMaintenanceConfiguration disabling a type
therefore lands, passes the per-key checks, and is then deleted by the
stale whole-entry write.
Every attribute this package stores is now asserted, absent ones
included. The metadata commit and planning index writers rewrite the same
entries and had the same exposure, so both use the shared snapshot too.
* iceberg: implement the auto compaction strategy
auto was accepted, stored and read back, but left the worker on its own
default, so a sorted table configured as auto was compacted with binpack.
AWS defines auto as sorting tables that declare a sort order and
bin-packing the rest. That needs the table metadata, so the choice is made
where the rewrite plan is resolved: an unsorted table falls back to
binpack rather than failing the way an explicit sort request does.
* s3tables: validate the maintenance setting ranges
PUT accepted zero, negative and oversized values for every numeric
setting. The worker then ignores a non-positive value and saturates an
oversized one, so the configuration read back was not the one that ran.
AWS bounds all five to 1..2147483647, which is now enforced. The fields
are pointers so an explicit zero is distinguishable from an omitted one
and can be rejected rather than silently ignored.
* s3tables: give every entry writer the same compare-and-swap
updateExtendedAttribute asserted the entry's attributes, but the helpers
behind the metadata, policy and tag handlers still wrote the whole entry
unconditionally. Any of them could land on a stale snapshot and delete a
maintenance configuration an operator had just written.
They all share one read-modify-write loop now, so the precondition and
the bounded retry apply wherever an entry is rewritten.
* s3tables: move the maintenance configuration with a renamed table
RenameTable carried the metadata, version, policy and tags to the new
name but left the maintenance configuration and job status behind. A
table with snapshot management disabled came back enabled under its new
name, and the stale configuration stayed on the old name where a table
created there would inherit it.
The decoupled-delete cleanup left the same two attributes behind.
* s3tables: accept every AWS partition in ARNs
The route regexes and the ARN patterns both hardcoded arn:aws, so valid
aws-cn and aws-us-gov ARNs never reached a handler. The router now shares
the partition-tolerant prefix with the parser, and a generated ARN uses
the partition its region belongs to so it parses back.
* s3tables: generate ARNs in the region's partition
The handler's own ARN generators still formatted arn:aws directly rather
than going through the partition-aware builder, so a China or GovCloud
deployment routed the request but then returned a commercial ARN and
matched IAM policies against it.
The round-trip test missed this because parsing accepts any partition, so
it now asserts the prefix the region implies.
* s3tables: complete the ARN partition table
aws-iso-e, aws-iso-f and aws-eusc were missing, so eu-isoe-*, us-isof-*
and eusc-* regions fell through to the commercial partition.
* s3tables: do not let a rename swallow a concurrent maintenance write
Rename copied the source attributes early and cleared the source at the
end, so a Put landing in between missed the copy to the destination and
was then deleted by the cleanup. It succeeded and vanished.
The cleanup now clears the source only while it still holds exactly what
was copied, and returns a conflict otherwise. Put checks the catalog
identity inside the same conditional mutation, so it also cannot write to
a name that a rename or delete has already soft-deleted.
* iceberg: carry snapshot retention in milliseconds
Config stored retention as hours, so any sub-hour value would have to be
truncated to 0 and then clamped back up to the 168 hour default. Keep the
plugin config key in hours and convert once at parse time.
* iceberg: let table properties override the worker config
Every other Iceberg implementation lets a table's own properties win over
engine defaults; the worker ignored them entirely. A writer honouring
write.target-file-size-bytes and a compactor rewriting to the plugin
config's size would rewrite each other's output forever.
Resolved once per job rather than per operation, so compaction committing
new metadata mid-job cannot change the settings underneath it.
* iceberg: clamp the orphan cutoff so it cannot overflow
collectOrphanCandidates converts the cutoff to a time.Duration. Past
roughly 2.5 million hours that multiplication wraps negative, putting the
cutoff in the future so every file walked looks like an orphan and gets
deleted, including data a concurrent writer has not yet committed.
Reachable today through orphan_older_than_hours.
* ec.encode: require the shards to agree on size before deleting the volume
Before an encode deletes the volume it just encoded, it asks whether
enough shards exist and whether they are spread across nodes. Both are
questions about presence: nothing asks whether those shards are whole.
Every shard takes one piece of each block row, so they are all written to
the same length. One that disagrees was truncated, half copied, or landed
on a disk that filled up -- and counting cannot see it, so the source
volume is deleted on the strength of a set that cannot rebuild it.
Compare the sizes the cluster already reports (shard_sizes travels in the
heartbeat) and hold the deletion back when they disagree, naming the odd
shard and its holder. Sizes reported as zero are skipped rather than read
as a disagreement: a volume server that predates shard-size reporting, or
one that has not heartbeated them yet, must not strand every encode in
the volume-plus-shards state this check exists to avoid.
* ec.encode: judge shard sizes on the newest encode generation only
The size check collected every shard the master reports for the volume,
while the recoverability check beside it counts only the newest encode
generation. A re-encode can change the ratio, so an orphaned older
generation -- one the pre-encode sweep could not reach, but the master
still hears about -- has shards of a different length by nature. Merging
those into the comparison makes a healthy current set look inconsistent,
and because the orphan keeps being reported, every retry fails and the
encode is left holding the volume and its shards for good.
Collect sizes the way CollectEcShardBitsByNode collects bits: fenced to
the newest EncodeTsNs, with unstamped entries forming the one legacy
generation.
The harness kills shells mid-command, and the master releases the dead
session's lock only when it notices the connection is gone. That cleanup
lands after the harness has already re-acquired the lock, so it can clear
the lock this run holds and the next command refuses with
need to run "lock" first to continue
recoverInterruptedBalance answered that the way an operator would -- run
lock again and retry -- but the encode and decode recoveries called
shellCommand once and required success, so the same reap failed the run
outright. Move the retry into shellCommand: the reap can land during any
command that follows a kill, not only a balance.
A decode ends by deleting the shards it read, and the only thing standing
between that and a bad reconstruction is verifyDecodedVolumeBeforeDelete,
which asks whether .dat and .idx are non-empty. A .dat truncated to a
single byte passes, and the shards -- the only other copy of everything
past the cut -- are deleted on the strength of it.
The server already knows the answer it never checks: FindDatFileSize
returns the extent the EC index references, and WriteDatFile rebuilds to
it. Compare the two once the file is written and fail the decode instead
of reporting a short volume as a good one.
Longer than the extent still verifies -- padding is not missing data --
so only a genuinely short rebuild is rejected.
Needle counts cannot answer this: .idx is written from .ecx, so the count
matches by construction and a truncated .dat still reports every needle.
A decode deletes the shards only after the regenerated volume is mounted
and verified, so a run interrupted in that last phase leaves the volume
in place with its shards partway through deletion. The re-run then finds
both, tries to collect the shards again to rebuild a volume that already
exists, and fails on the first shard the interrupted run had removed:
generate normal volume 3 ...: ec volume 3 missing shard 6
Nothing recovers from there: the shard set is deliberately being
destroyed, so every retry fails the same way while the decoded volume
sits there, already complete.
Finish that cleanup instead. A volume beside the shards is not enough to
act on -- an encode interrupted before it deleted the original leaves the
same shape, as does a decode killed while generating, whose volume may be
half written -- so require a data shard to be gone. Only the deletion
phase removes one, and it is also exactly the state no decode can
recover from, so finishing is the only move left rather than a choice
between two. The deletion still runs behind
verifyDecodedVolumeBeforeDelete, the check that guards it in a normal
run.
* admin: show capacity per storage tier and stop counting remote-tiered bytes as local disk usage
A remote-tiered volume reports its cloud object's size, so summing volume
sizes inflated the dashboard's used-vs-capacity numbers (the local .dat is
gone after volume.tier.move). Split the accounting: DiskUsage now only
counts bytes on local disks, with the cloud bytes surfaced separately per
server and per remote storage name.
The dashboard gains a Storage Tiers table breaking volumes and EC shards
down by tier (each local disk type plus each remote storage), using the
per-disk-type statfs numbers already in the VolumeList response. The
volumes page badges remote-tiered volumes with their storage name, and
the EC shards page fills in real per-shard sizes instead of hardcoding 0.
* admin: review fixes for the tier capacity display
- A disk that predates disk_total_bytes now contributes its logical
bytes to the tier's DiskUsed, so a tier mixing old and new volume
servers doesn't underreport usage; the usage bar always reflects the
displayed Disk Used value (the DataSize fallback in UsagePercent is
gone, and the percent math is overflow-safe).
- getTopologyViaGRPC defaults a zero VolumeSizeLimitMb to 30000 MB like
GetClusterVolumeServers, keeping slot-based capacities consistent.
- The dashboard volume-servers column reads Usage / Capacity to match
its cell content, and the hdd disk-type default is shared between the
volumes-page badge and countUniqueDiskTypes.
* ec: bounded-exhaustive model check of the volume lifecycle
The randomized chaos harness samples the state space; this enumerates
it. The lifecycle is a state machine whose steps mirror the pipelines in
this package, and the checker explores every schedule within the bound:
a crash at every step boundary, an error return running the rollback
(itself crashable at every step), a volume-server restart applying the
startup reconciliation rules in every quiescent state, and the
prescribed restart-based recovery from every crashed state.
Checked in every reachable state: durability (a readable copy always
exists), at most one generation mounted, and — a property the sweep
discipline turns out to guarantee — at most one generation's files on
disk. From every quiescent state the recovery must converge to a clean
volume. Runs in well under a second.
* test: deterministic EC interruption matrix
Enumerate every phase of every interruptible EC operation and kill a
real weed shell exactly when the phase announces itself on the command
output, instead of at a random moment: four encode phases, four decode
phases, and the balance's move phase (set up with -rebalance=false so a
move is guaranteed). Each scenario prepares its precondition, kills at
the marker, runs the prescribed recovery, and verifies every stored byte
still reads back identical.
The interruption recoveries move out of the randomized ops into shared
chaosRun helpers both drivers use.
* test: make the randomized EC chaos walk opt-in
The systematic layers — the interruption matrix and the lifecycle model
check — carry the CI coverage deterministically; the randomized walk
stays for exploratory runs, behind EC_CHAOS_SEED.
* ci: bound the EC integration suite by the job budget, not go test's default
The suite with the interruption matrix runs close to the default 10m
binary timeout on slower runners.
* test: require every interruption-matrix marker to appear
A marker that never prints means a pipeline refactor renamed or dropped
the progress line; silently degenerating into a no-interruption run
would let CI pass without exercising the boundary the scenario names.
Also recheck the marker channel after the wait: a shell that prints and
exits at once makes both channels ready, and select picking the exit
case must not report a printed marker as missed.
* ec: let the encode's balance see a migrating volume's shards across disk-type buckets
Shard generation writes beside the source .dat, so a cross-tier encode
(source on hdd, -diskType=ssd) leaves the fresh shards in the source
disk-type bucket. The encode's internal balance ingested only the target
bucket, saw no shards, and planned no moves; the spread guard then
correctly aborted the encode (and before that guard existed, the shards
silently stayed clumped on the generation host in the wrong tier).
EcBalance now takes the encode batch as migratingVolumeIds and ingests
those volumes' shards from every bucket, while everything else keeps the
bucket filter so a plain ec.balance never drags deliberately tiered
shards onto another disk type. The in-memory model delete also becomes
bucket-agnostic: a node holds a given shard in exactly one bucket, and a
bucket-scoped delete missed cross-bucket moves in the dry-run model.
* volume: decode reads shard 0 from its resolved path, not the EC volume's base dir
On a multi-disk server a volume's shards can sit on several disks; the
store registers each shard with its own path and CollectEcShards resolves
them, but FindDatFileSize derived the .ec00 path from the EcVolume's base
directory. When shard 0 lived on a sibling disk, VolumeEcShardsToVolume
failed with 'open ...ec00: no such file or directory' and ec.decode
aborted.
* ec: decode re-copies shards the topology claims but the target does not hold
An interrupted earlier decode or balance can leave the master believing
the decode target holds a shard whose file never landed: the mount
registered but the partial copy was cleaned, or the file was swept. The
collect step took the topology's word for it, excluded the shard from
the copy set, and the decode failed with 'missing shard'. Probe the
target's live inventory (VolumeEcShardsInfo) and treat anything it
cannot serve as still-to-copy.
* ec: decode discovers shards across disk-type buckets
Shards sit wherever encode generation and balance left them: a
cross-tier encode leaves them in the source disk-type bucket, a partial
migration straddles buckets. ec.decode scoped its shard discovery to the
-diskType bucket and reported a decodable volume as having no shards at
all. Union across buckets, the way the encode's shard verification
already does.
* test: EC chaos lifecycle harness
Randomized, seeded sequences of the EC lifecycle against a live cluster
in the production-shaped layout: multiple data disks per server, a
separate -dir.idx directory so .ecx/.ecj sidecars are shared across
disks, and a tagged ssd tier. Operations cover encode (hdd and ssd
targets), balance, shard damage plus rebuild, decode, re-encode,
deletes, scrub, tier moves, crash-restarts, sidecar fault injections
(a data-dir .vif pushed into the shared idx dir; a stale-generation
shard planted beside a newer encode), and interruptions: a real weed
shell subprocess killed mid-encode, mid-decode, and mid-balance, with
the recovery re-run required to converge.
One invariant holds after every step: every stored byte reads back
identical and every deleted needle stays deleted. EC_CHAOS_SEED and
EC_CHAOS_STEPS make runs reproducible and scalable.
A known gap is tolerated and logged rather than fixed here: a shard
mounted on two disks of one node (orphan adoption after an interrupted
copy) is invisible to ec.balance's dedup and unaddressable by
ec.shard.unmount's shard@address form, so no cleanup path exists yet.
* test: fail payload-corruption checks on the test goroutine
t.Fatalf inside require.Eventually's condition runs on the poller's
goroutine, where Goexit kills only that goroutine and the corruption
message can be lost behind a generic timeout. Record the mismatch, end
the polling, and fail on the test goroutine. Also assert the full shard
count in the cross-bucket decode-discovery test.
* shell: move ErrorWaitGroup to weed/util
* shell: remove unused CandidateEcNode and EcRack types
* ec: extract EC orchestration logic from weed/shell into weed/ec
Move the EC node/topology model, balance engine, encode pipeline, decode
pipeline, and rebuild engine into a new weed/ec package so shell commands
and maintenance workers can share the logic. Shell commands keep flag
parsing and delegate through a small ec.Env (dial option, topology fetch,
volume locations, lock check). Tests move along with the code.
* shell: remove unused proportional-rebalance type stubs
* ec: move scrub, replication check, and shard unmount engines into weed/ec
* worker: share the EC generation-aware shard counter from weed/ec
* ec: gofmt
* shell: drop EC aliases with no remaining callers
* ec: guard a missing topology hook and nil disk entries in topology helpers
* ec: drop trailing newlines from decode error strings
* ec: re-check the shell lock before applying shard unmounts
* shell: trim -node entries in ec.scrub
* s3: add option to disable bucket auto-creation on upload
* command: expose -autoCreateBucket in s3, filer, server, and mini
* s3: apply the bucket auto-create policy to directory marker uploads
* s3: validate the bucket name before the auto-create disabled check
* s3: cover the disabled auto-create gate at all three upload entry points
sftp: url-encode the upload path so filenames can't inject filer query commands
The SFTP put handler concatenated the user-controlled filename straight into
the filer upload URL, so a name containing "?" was parsed as a query string.
Build the URL via url.URL{Path: ...} so "?" becomes %3F and stays a literal
path character.
* volume: validate replica upload targets in FetchAndWriteNeedle
The replica leg forwarded the fetched needle to a caller-supplied address
without checking it, so a malformed target could redirect the upload to an
unintended host or path. Require each replica target to be a bare host:port
whose host is not loopback / link-local / unspecified, reusing the address
deny-list; cluster peers legitimately sit on private networks, so RFC 1918 /
CGNAT stay allowed and -volume.allowUntrustedRemoteEndpoints still opts out.
Validate every target up front so a bad one fails the request before the local
write, and upload through a client that re-checks the resolved address at
connect time so a replica hostname cannot rebind to a blocked address after
validation. Mirrored in Rust (validation moved ahead of the local write; the
Rust S3 path's connect-time re-check is still a follow-up there).
* volume: only accept inline gcs credentials in FetchAndWriteNeedle
The gcs credentials value on this request could name a local filesystem path,
which the SDK reads from disk. Accept only inline JSON here; the server-side
GOOGLE_APPLICATION_CREDENTIALS env var still supplies a path. The Rust volume
server has no gcs backend, so there is nothing to mirror.
* remote_storage/azure: allow a per-request HTTP client
Thread an optional *http.Client through NewAzBlobClient and add
azure.MakeWithHTTPClient, mirroring the S3 backend. When set, the client
overrides the azblob transport so a caller can pin the dial path. The
existing makers pass nil, so behavior is unchanged.
* volume: extend the remote-endpoint guard to the azure backend
The endpoint validation and rebinding-safe dialer in FetchAndWriteNeedle
covered the S3-SDK backends. The azure backend also dials a caller-supplied
AzureEndpoint, so route both families through a single guardedRemoteClient
helper that returns the endpoint each backend dials and a constructor bound
to the guarded HTTP client. azure is guarded only when AzureEndpoint is set;
an empty endpoint derives the public host from the account.
-volume.allowUntrustedRemoteEndpoints still opts out.
* rust volume: assert the azure endpoint has no remote-client path
The Rust volume server has no azure backend, so make_remote_storage_client
rejects the type before any client is built. Add a regression test pinning
that invariant.
* volume_move: treat zero-sized EC shards as absent in move verification
A zero-sized shard file is residue of a failed operation (issue 10730),
not a shard - but VerifyEcShards only checked presence, so a copy that
landed as an empty file passed verification and the source was deleted
behind it. Size zero now reads as absent, with a distinct error naming
the zero-sized shard so the operator can tell a broken copy from a
missing one.
* storage: exclude zero-sized EC shards from rebuilds and clean up stale ones
The reproducer in issue 10730: a zero-sized shard file left by a failed
operation was selected as a Reed-Solomon input and failed the whole
rebuild with an input size mismatch, because input discovery checked
existence, not substance.
- RebuildEcFiles treats a zero-sized shard file as missing and
regenerates over it in place (the reclassified-corrupt path: temp
file beside the residue, atomic rename).
- The startup/rescan shard loader, which always skipped zero-sized
files, now deletes them once they are older than an hour - young
enough files can be an in-flight copy's just-created file, since the
same scan runs from LoadNewVolumes while serving.
Regression tests: a rebuild with one emptied shard regenerates it
byte-identical; the loader deletes a stale zero-sized shard and leaves
a fresh one alone.
* storage: age-check each zero-shard cleanup candidate individually
The shard scan merges the data and idx directory listings, so the
age-checked entry and a deletion candidate can be different files
sharing one name - a stale zero-sized file in one directory next to a
fresh same-named file in the other (possibly an in-flight copy's
just-created one) could get the fresh file deleted. Each candidate's
own modification time now decides, both directories are handled in one
pass, and the split-directory case is pinned by a test.
* redis2: route the orphan cleanup existence checks to the master
* scaffold: the redis_cluster2 read routing key is useReadOnly
* ci: run the gated redis store tests
* redis2: poll for the redis expiry instead of a fixed sleep
* redis2: assert the value key exists before testing its expiry
The Go volume server has VolumeConsolidateIndex, which moves a volume's
.idx out of the data directory into the configured -dir.idx directory
(where an EC decode/reconstruct can leave it co-located) and reloads the
volume in place. The Rust port's proto omitted the RPC entirely, so its
generated VolumeServer trait was one method short of Go's.
Add the proto message and rpc, the gated grpc handler, and
Store::consolidate_volume_index / Volume::relocate_index_to, mirroring
Go's Store.ConsolidateVolumeIndex and Volume.RelocateIndexTo -- including
the cross-device copy fallback and the reopen-against-the-old-dir path
when the move fails.
Integration tests cover the real move (index relocated, volume still
serves reads and the move is idempotent), the no-op paths (index already
in place, no separate idx dir) and the not-found error, plus the grpc
handler end to end.
A bounded LoopProcessLogData (stopTsNs set) on a buffer that never took a
write since process start fell into the ResumeFromDiskError branch, which
never checks stopTsNs when ReadFromDiskFn is nil and HasData() is false.
The read parked on the notification loop forever while the subscription's
idle heartbeats kept the stream looking alive, so a bounded
SubscribeMetadata pass on a freshly restarted idle filer never completed.
Terminate like the caught-up path does, returning a nil error: leaking
the pending ResumeFromDiskError would latch the filer's outer loop into
its gap machinery, which parks the bounded subscriber all over again.
* feat: throughput limits for replicate, EC shard, and worker-driven moves
VolumeCopy was the only rate-limitable transfer; EC shard copies,
replica creation, and worker-driven moves all ran at whatever the
receiving server's maintenance rate allowed, with no per-operation
control.
- proto: VolumeEcShardsCopyRequest and the balance / ec_balance task
params and configs gain io_byte_per_second; 0 keeps today's behavior
(the volume server's own maintenance rate governs).
- volume server: VolumeEcShardsCopy throttles with one WriteThrottler
per request, shared across the shard, .ecx, .ecj, .vif, and .ecsum
copies so the limit caps the transfer as a whole - the same shape as
VolumeCopy.
- volume_move: ReplicateVolume accepts the limit; EcMoveOptions carries
it through MoveEcShards/CopyAndMountEcShards into the copy request,
with fake-client tests asserting propagation.
- shell: ec.balance gains -ioBytePerSecond; volume.tier.move's
replication top-up honors the command's existing -ioBytePerSecond
instead of running unthrottled.
- worker: balance and ec_balance configs gain io_byte_per_second
(surfaced in the admin config schema), carried through detection and
plugin job parameters into task params and handed to the shared
mover; batch balance jobs inherit the limit from their detection
results.
The limit is per copy stream, so maxParallelization multiplies the
aggregate ceiling.
* worker plugins: expose io_byte_per_second in the plugin config and derive it
The plugin-driven detection path derives its task Config from the
plugin configuration values, and both balance and ec_balance left
IoBytePerSecond at zero there - a configured limit silently reverted
to the server maintenance rate. Both derive functions now read the
field (clamped at zero), and the plugin descriptors expose it with
defaults so the configuration form carries it.
rust volume: gate the remaining admin RPCs behind check_grpc_admin_auth
The Go volume server gates 29 destructive VolumeServer RPCs on the
-whiteList admin check; the Rust port only gated 14. Add the gate to the
other 15 -- batch_delete, read_all_needles, fetch_and_write_needle, the
EC-shard generate/rebuild/copy/unmount/to-volume RPCs, both tier-move RPCs,
volume_copy, volume_tail_receiver, set_state, scrub_ec_volume and
volume_needle_status -- so a configured whitelist restricts them the same
way it already does on the Go side.
check_grpc_admin_auth also required peer info before checking whether any
control was configured, unlike Go's `if vs.guard == nil { return nil }`.
Short-circuit when no whitelist and no signing key are set, so in-process
callers keep working with security inactive and only the gate ordering
changes for configured servers.
tests/admin_auth_coverage.rs mirrors the Go coverage test: every handler
must either gate or be listed as intentionally open with a reason, so the
two implementations can't silently drift apart again.
* redis2: derive the orphan cleanup keys inside the helper
* redis2: skip orphan cleanup in super large directories
* redis2: detach orphan cleanup from the request context and log a failed restore
* redis2: keep a directory member whose child index is still live
* redis2: run restore-path tests under both key prefixes and fix the test harness
* redis2: check cleanup errors in tests
* fix(redis2): remove orphaned directory index members on listing
ListDirectoryEntries skipped index members whose value key was gone and
left them in the ZSET, so the per-directory child index grew without
bound under any TTL workload. Mirror the ZRem the logical-expiry branch
already performs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(redis2): keep the index member when a concurrent insert recreates the value
The orphan cleanup removed the member unconditionally, so an InsertEntry
landing between FindEntry and the ZRem left a live value with no index
member, invisible to listings until another InsertEntry on that path.
UpdateEntry does not re-add it, so the loss persisted.
Restore the member when the value is present again after the removal.
The value key and the directory index key hash to different slots, so a
Lua script or MULTI over both is not available to the cluster store.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The unmount+full-teardown of EC shards was duplicated: the plugin-worker EC
task had unmountAndDeleteEcShards and the shell had unmountAndDeleteEcShardsQuiet,
byte-identical apart from a fence parameter and a sentinel error. That
duplication is how the teardown fence semantics drifted between the two paths.
Distribute, mount and verify already live in weed/storage/erasure_coding and are
shared by both callers; move the teardown there too, as UnmountAndDeleteEcShards
plus the shared ErrFullTeardownNotAcked sentinel. Both paths now call the one
function, so the fence semantics cannot diverge again. The shell keeps a thin
type-converting wrapper and aliases the sentinel; behavior is unchanged.
* EC worker: clear stale/interrupted shards at task start and on failure
The EC encode task cleared stale shards from a prior interrupted encode only
at 55% progress (after mark-readonly, copy, and generate), and used a
generation-fenced teardown. Two gaps left orphan shards behind:
- a retried encode's prior attempt carries the same admin-issued encodeTsNs,
and the server's teardown fence preserves same-or-newer generations, so the
prior attempt's shards were never cleared;
- shards left by an interrupted distribute often have an unreadable .vif
generation (the sidecar never landed), which the fence also preserves.
Both survive the next volume-server restart as orphans and make detection
refuse the volume (Manual intervention required).
Move the cleanup to a Step 0 preflight that runs before any destructive step,
and switch it to the server's blanket (generation-independent) teardown -- the
same wipe the shell ec.encode pre-cleanup uses. The admin dedupe key already
prevents a concurrent newer encode of the volume, and the blanket path aborts
rather than clobber a live newer mount.
Add rollbackDistribute: a failure after distribute begins but before verify
commits the EC copy now tears down the shards it wrote and restores the sources
to writable, so a terminally-failed encode (a single-attempt job, or the last
of a retry series, which has no successor preflight) leaves nothing behind.
The preflight also rejects a plan with no targets or no source before marking
the source readonly.
* EC worker: reject malformed targets and keep source readonly on incomplete teardown
Address review feedback:
- ensureCleanEcStart only rejected an empty target slice; a target with an
empty Node (or no shard ids) passed the length check, was then silently
skipped by cleanupStaleEcShards, and let Execute mark the source readonly
with nothing to distribute to. Validate each target before the first
destructive step. Add regression cases.
- rollbackDistribute marked the source writable even when the shard teardown
returned an error, exposing a writable source beside stale (possibly mounted)
shards -- reads/writes could diverge and orphan cleanup will not remove a
writable source. On an incomplete teardown, leave the source readonly for the
next preflight or an operator to reconcile.
* shell: roll back a failed ec.encode instead of leaving readonly volumes and orphan shards
ec.encode marks the source volumes readonly and generates EC shards before it
verifies the shards and deletes the originals. If any step in between failed,
the command just returned the error: the volumes were left readonly and the
partially-produced EC shards survived as orphans, cleaned up only by the next
ec.encode run (via clearPreexistingEcShards) if the operator retried.
Add a deferred rollback that runs when the batch fails before the originals are
deleted: it tears down the EC shards produced this run and restores the sources
to writable, reusing the existing clearPreexistingEcShards and
markVolumeReplicaWritable helpers. Once the shards are verified recoverable the
batch is committed to the EC copy and does not roll back. Both rollback steps
are idempotent, so a failure before the volumes were marked readonly is safe.
* shell: re-read volume locations when restoring writable in ec.encode rollback
Address review: rollbackFailedEcEncode restored writable using the location
snapshot taken before doEcEncode, but doEcEncode re-reads locations and marks
every replica of that later snapshot readonly. A replica added or moved in
between would be left readonly. Re-read locations in the rollback and fall back
to the pre-encode snapshot only if the re-read fails.
* refactor: extract remote mount resolution into shared helpers
* refactor: share the adaptive remote cache wait policy
* filer: stream cold remote reads from the origin while caching
* s3: stream cold remote reads from the origin instead of 503 retries
* test: cover the S3 origin stream-through path
* remote mounts: match on path components and prefer the longest mount
* fail short origin streams instead of silently truncating
* s3: try the origin before failing a cold read on a local cache error
* s3: gate origin streaming on the entry's resolved version
* return the cache RPC's NotFound as a canonical status and classify it everywhere
* filer: keep multipart-range cold reads on the retry path
test: assert EC shard identity and empty-view, not just counts, in lifecycle
Follow-up to the multi-disk EC lifecycle tests (#10721), addressing review
feedback.
The phase checks compared shard counts. A reconcile that put a shard on the
wrong disk, or loaded a different shard than the file on disk, keeps 6/5/3
right while corrupting the mapping. Compare the exact registered shard set per
disk at every phase instead, via a shared assertRegistered helper. The
cross-disk mount phase now also pins that shard 0 landed on disk2 with the
existing shards, not merely that it is findable.
The sidecar-disk-lost scenario only logged the registered view, so a change
that registered shards without reachable sidecars would pass despite the
documented expectation that the view stays empty. It now asserts
countRegistered == 0: a registered-but-unreadable shard is worse than an
unregistered one, because the master advertises it.
The first store's closer is now deferred as a closure the moment the store is
created, so a Fatalf in an early phase no longer leaks it and its
notification-drainer goroutine; the closure reads the reassigned variable so it
also covers the post-restart store.
* plugin scheduler: drain started jobs past the window close instead of killing them
* plugin scheduler: never drain-cap an attempt below its declared estimated runtime
* plugin scheduler: cap estimated_runtime_seconds before the Duration conversion
* operation: add shared volume_move package for volume and EC shard moves
The shell commands (volume.move, volume.balance, ec.balance, tier moves)
and the maintenance workers (balance, ec_balance) each carried their own
copy of the move RPC sequences, and the copies had drifted: the worker
verified the target before deleting the source but dropped the disk
type and IO throttle; the shell passed those but deleted the source
unverified.
volume_move.Mover carries the merged sequences, keeping the stricter
behavior from each side:
- LiveMoveVolume: check-then-hard-freeze the source (VolumeStatus's
IsReadOnly also covers low-disk and readonly-but-can-delete states,
which still accept needle deletes), copy with disk type and IO
throttle, tail, verify the target is not behind the source before the
destructive source delete (a target that is ahead holds writes it
accepted during the tail and the move commits to keep them), and
restore the source's writability when a failure precedes the delete
and this move did the freezing. Aborts clean up the incomplete target
copy; a failed cleanup or an ambiguous source delete keeps the source
readonly (ErrSourceKeptReadonly) so callers do not thaw a source next
to a possibly-authoritative copy. With a readonly source, an existing
or unknown-state target refuses the move outright: no client-side
observation can prove such a copy is a stale remnant rather than the
authoritative copy of an unfinished move.
- MoveEcShards: copy with the .ecx/.ecj/.vif/.ecsum sidecars, mount,
verify the target registered every shard before unmount+delete on the
source, and reject same-server moves (the EC delete is server-wide).
Server identity is the grpc endpoint (SameServer), so node:8080 and
node:8080.18080 compare equal while test servers sharing a degenerate
HTTP address stay distinct; addresses are validated non-fatally before
dialing and before being embedded in copy/tail requests, since both the
client dialer and the receiving server normalize them through a parser
that aborts the process on a malformed port. The Rust volume server's
codes.NotFound counts as a definitively absent probe answer alongside
the Go server's plain-error code Unknown.
All RPCs go through an injectable ClientFunc, so the sequences are unit
tested against a fake volume server client: RPC order, request fields,
and that verification failures keep the source intact.
* shell, worker: delegate volume and EC shard moves to operation/volume_move
LiveMoveVolume and the copy/tail/delete/mark-writable helpers become
thin wrappers over the shared mover, keeping their signatures; the EC
helpers keep their per-step output and delegate the RPCs. BalanceTask
and ECBalanceTask keep their parameter validation, progress reporting,
and guards (same-node cross-disk rejection, dedup keep-node
verification, shard ids range-checked before the uint8 narrowing) and
hand the RPC sequences to the mover. volume.tier.move skips its
thaw-on-failure when the mover deliberately kept the source readonly,
since reopening the replicas beside a possibly-authoritative target
copy would fork the volume.
The tail-failure tolerance moves inside the mover: a failed tail is
tolerated only when the volume was already readonly before the move
began, backstopped by a stability re-read across the idle window, so
volume.balance's -skipTailError-by-readonly heuristic and tier-move's
unconditional skip both become the same authoritative rule.
* volume_move: keep the source readonly when a failed copy leaves a target of unknown origin
A failed copy can leave a complete, mounted copy on the target (the
server finishes after the client loses the stream). The abort probed
the target only when its pre-copy state was known-absent; an unknown
prior state skipped both the probe and the cleanup and then reopened
the source - two writable replicas of one volume, diverging from the
next write on.
The abort now probes the target on every failed copy and restores the
source only when the target provably holds nothing. A copy whose
provenance cannot be proven (unknown prior state, a pre-existing
replica, or an unreachable target) is never deleted, and the source
stays readonly with ErrSourceKeptReadonly naming the recovery.
* test: teach the plugin worker harness the shared move sequence
The fake volume server lacked VolumeStatus, which the shared mover now
issues before freezing the source, and the batch execution test's
status-read accounting predates the pre-copy target probe and the
verification reads. Mirrors the harness the enterprise tree already
carries.
The existing multi-disk EC integration test asserts on shard counts. Counting
cannot tell a healthy volume from one a repair reassembled out of the wrong
inputs — both have fourteen shards. This drives the real shell commands
(ec.encode, ec.balance, ec.rebuild) against a live three-node, four-disk
cluster and reads the stored bytes back after every step, so a rebuild that
produced fourteen plausible-but-wrong shards fails here.
An 8 KB random payload is stored, then encoded, balanced, damaged (two shard
files removed and the servers restarted so the master relearns the reduced
set from disk), and rebuilt. The rebuild output matches the shape of the
support case that motivated this — "rebuildOneEcVolume", "missing shard N.0",
"copied N.1 from ..." — and the payload is verified identical after each of
upload, encode, balance, shard loss, and rebuild.
Two ordering facts the test pins, both of which cost real debugging time:
ec.rebuild is driven by the master's topology, not disk truth, so shards must
be relearned (via restart) before a repair can target the right set; and the
shell lock is dropped when the restart disconnects the master, so it has to be
retaken before the rebuild.
A multi-disk volume server keeps one .ecx / .ecj / .vif set per volume on a
single disk while ec.balance scatters the shards across the others. Every EC
operation on such a node crosses that split: startup registration, balancing
the sidecar disk's shards away, rebooting in that state, and mounting a shard
delivered to a disk that has no local sidecars.
Each of those transitions is handled by a different mechanism (per-disk scan,
cross-disk reconcile, mount-time .ecx lookup), individually tested but never
as the sequence a production node actually lives through — where the output
state of one transition is the input of the next. A regression in any hop
shows up as shards that exist on disk while the master's view says otherwise,
and every topology-driven repair then works against the wrong shard set.
The layout, volume id and collection mirror a support case. The second test
pins the failure floor when the sidecar disk itself dies: shards on the
surviving disks may drop out of the registered view, since nothing can read
them without the .ecx, but their files must survive so restoring the sidecars
restores the volume.
* mount: invalidate hot directory listings by section
A cached directory used to be dropped whole when it saw 64 changes in
2s: with a continuous writer the listing cycled through wipe, direct
listing and full rebuild for as long as the writer kept going, and
every sibling lookup fell through to the filer in between.
Split each cached listing into name-range sections of 1024 entries. A
burst of foreign changes invalidates just the section it lands in;
entries stay served and events keep applying, and the next readdir
re-lists only that range from the filer, reconciled through the version
gate so it cannot roll back newer applied events. Lookups in an
invalidated section read through until then. The mount's own writes no
longer invalidate anything: they are ground truth for its cache.
* meta_cache: drop the version floor with a deleted or moved directory
The other teardown paths already clear both maps; a floor left behind
here would fence the listing of a directory re-created at the same
path.
* mount: harden section refresh
An unversioned listing (pre-upgrade filer) now only fills gaps instead
of reconciling: without a snapshot to order against, an overwrite or
the deletion sweep could roll back an event applied after the listing.
The section table can be rebuilt or re-split between the listing and
its apply, so the refresh only marks fresh or splits when the section
still covers the range it read. Splicing bounds from a stale range
into a rebuilt table could leave them unsorted.
Bound the wait: a readdir gives a refresh five seconds before serving
the maintained-but-unverified cache. Bound the size: a range grown
past four sections aborts the refresh and drops the directory cache,
re-tiling it with a full rebuild, with that request served direct.
Cover the filer-facing path with a listing server: paging with the
snapshot pinned across pages, the section cutoff, no calls for a
fresh section, and the overgrown-range abort.
* meta_cache: make the section table a self-contained state machine
Churn counting, freshness, stale-range scanning and the refresh
completion with its guard and split now live on dirSections itself,
free of the lock, the store and the apply loop, so they test directly
with synthetic clocks and tables. MetaCache keeps thin wrappers that
hold its mutex and find the directory's table.
* meta_cache: keep section internals out of the apply request
The request now carries the completed build's table and one refresh as
opaque values built by section code, and the boundary-derivation rule
moves out of the build loop into a collector next to the rest of the
section logic.
* mount: fence refreshed sections with a snapshot floor
A refresh versioned the entries it fetched and tombstoned the ones it
swept, but a name absent from both cache and listing kept the old
directory floor, so a delayed event between the two snapshots could
resurrect it into a section already marked fresh. The section now
carries its own floor, consulted next to the directory floor, covering
every name in the range, present or absent — which also retires the
refresh's per-entry version stamps and sweep tombstones.
An unversioned listing sets no floor and vouches for nothing: it may
still fill gaps, but the section stays stale and reads through until a
filer that stamps snapshots re-validates it.
A listing's reach is unknowable up front — a resumed handle can skip
far ahead, and shrunken sections let one batch span many — so a
readdir now re-validates every stale section from its start name to
the end of the directory instead of the next two.
* mount: fence tombstoned names with floors and gate the reconcile
A tombstone answered for its name before the floors were consulted, so
one at an old position let through events the newer listing floor
should have fenced; a build never hit this because it prunes
superseded tombstones, which a section refresh does not. The version
gate now raises a tombstone to the floors like any other record.
With no per-entry versions, only the section floor fences a
reconcile's work, so a range the rebuilt or re-split table no longer
has must not touch the store either: the range check moves ahead of
the mutations, under the same lock the floor install holds.
An unversioned refresh no longer retries: the section is remembered as
unverifiable and skipped by the stale scan, or every batch of every
readdir would re-list the same ranges against a filer that cannot
vouch for them.
* mount: clear beaten unversioned markers and skip refresh mid-build
An unversioned marker outliving the snapshot write that replaced its
content bypassed the section floor the same way an old tombstone did,
letting a delayed pre-snapshot event roll the entry back. The refresh
now clears the marker when its write wins; pinned local-only entries
are not replaced at all, keeping their content and marker.
A rebuild wipes and repopulates the store off the apply loop, so a
refresh reconciling meanwhile could sweep children the build had
already inserted and let it publish the directory incomplete. The
refresh now skips a building directory, as events (buffered) and
purges (skipped) already do; its staleness dies with the build's
fresh table.
* mount: clear the unversioned marker only after its replacement lands
Clearing before the insert meant a failed write left the old local
content claiming the listing floors, fencing the very events that were
still entitled to correct it.
* meta_cache: rename the section state machine to sectionList
dirSections named both the type and the map of them.
* mount: raise the default cacheDirMaxEntries to 100000
The low ceiling guarded against whole-listing rebuild churn: a big
cached directory under writes kept re-streaming everything. Sectioned
invalidation ended that — a burst now costs one range listing — so the
remaining cost of caching a large directory is its one-time build,
comparable to the single direct listing that read-through mode pays on
every enumeration instead.
* meta_cache: cover section border and edge cases
A bound-named entry belongs to the section starting at the bound: the
neighboring refresh's sweep stops before it, its own section's covers
it. Churn past everything the build saw lands in the tail section, a
rename spanning two sections invalidates both, and a listed entry at
the section's end name is cut off with the ones beyond it.
* ec: confirm a surviving copy before deleting a duplicate EC shard
The dedup phase of EC balancing removes a shard it believes exists elsewhere.
It copies nothing first, so the shard surviving on another node is the only
thing that makes the delete safe -- and it took the plan's word for that.
The plan is built from the master's topology, which can name a location that
holds nothing: such a server answers "CopyFile not found ec volume id N" when
something later tries to read the shard there. A shard listed on a phantom
location and on a real one looks duplicated, so dedup deletes one of them. When
it picks the real one the last copy is gone, and the job reports success -- the
loss only surfaces later, as a rebuild that cannot assemble enough shards.
The move phase already refuses to work on trust: it verifies the shard
registered on the destination before removing the source. Dedup now holds to
the same standard. The planner records which node it chose to keep, and both
executors -- the worker task and the shell's ec.balance -- confirm that node
really holds the shard before deleting. A keep node that cannot be queried is
unknown rather than confirmed, and blocks the delete.
Tests drive the destructive path against an in-process volume server that
tracks what is actually on disk separately from what the plan claims, which is
the distinction the bug turns on. Without the guard, two of them fail by
deleting the only copy and returning success.
* ec: check the collection and bound the wait when confirming a survivor
Two gaps in the dedup survivor check.
The inventory RPC is keyed by volume id alone, so a server holding the same
number for a different collection answers "yes, I have that shard" to a
question about this one. Accepting that deletes the last real copy on the
strength of an unrelated volume. The response already carries the collection,
so verify against it rather than widening the RPC.
The shell path also queried on a background context, so a keep node that
accepts the connection but never answers would hang the whole balance run
instead of reporting that the survivor could not be confirmed. Bound it.
The check moves into VerifyShardsOnServer next to the existing helper, shared
by both executors, so the two paths cannot drift.
* remote_storage: build S3-compatible clients through one constructor
The eight non-s3 S3-SDK providers each duplicated the AWS session setup
and only the s3 maker could take a custom *http.Client. Route every
S3-compatible type (s3, wasabi, b2, aliyun, tencent, baidu, filebase,
storj, contabo) through MakeWithHTTPClient with a single options table,
and add S3CompatibleEndpoint so callers can resolve the endpoint a given
type dials. No behavior change.
* volume: apply the remote-endpoint check to all S3-compatible providers
FetchAndWriteNeedle validated the endpoint and used the pinned dialer only
for type "s3". Every S3-SDK backend (wasabi, b2, aliyun, tencent, baidu,
filebase, storj, contabo) dials a caller-supplied endpoint through the same
client, so gate on S3CompatibleEndpoint to apply the same check uniformly.
-volume.allowUntrustedRemoteEndpoints still opts out.
* volume: don't route the guarded remote-endpoint client through a proxy
The guarded client exists to dial the validated endpoint directly and
re-check the resolved IP at connect time. With http.ProxyFromEnvironment
set, the dialer only validates the proxy's address while the proxy
re-resolves the endpoint host, which reopens the rebinding window. Drop
the proxy on this path; operators that need one can opt out with
-volume.allowUntrustedRemoteEndpoints.
FileCount and DeleteCount were int, so each cost a word on every replica the
master holds. A volume caps at 30GB on a 4-byte-offset build and 8TB on a
5-byte one, and neither holds 4.29 billion needles.
That takes VolumeInfo from 120 bytes to 112, which is its own size class rather
than rounding up into the 128 one, so a replica costs 135.7 bytes in the map
instead of 151.7 -- about 25MB across the 1.6M replicas in a cluster the size
of the one this came from.
Counts are narrowed where they are read rather than assigned across, so a
report claiming more than a volume can hold pins at the ceiling instead of
wrapping to a small number.
* filer: stop silently dropping metadata replay failures from peers
When two filers do not share a store (e.g. one leveldb3 per pod), each
subscribes to its peers' metadata streams and replays their events
locally (meta_aggregator.go's maybeReplicateMetadataChange, wired into
doSubscribeToOneFiler). A failed Replay() was logged and then treated
as done anyway: processEventFn always returned nil regardless of the
replay outcome, and processOne advanced lastTsNs unconditionally. The
offset is the only record of subscription progress, so a dropped event
was gone for good - no retry, and nothing else ever observed it.
An entry that fails to replay this way diverges from its peer
permanently. This is how a bucket's quota (entry.Quota, carried on
peer events like everything else - see entry_codec.go's EqualEntry
comparing Quota, and FromPbEntry copying it in entry.go) can end up
different across filers indefinitely: one replay hiccup on one filer,
and its enforcement and any metric reading its own store diverges from
the others' with no signal anything went wrong.
Fix: replicateMetadataChange now retries a failure with util.Retry,
which already distinguishes transient errors (timeouts, connection
resets, throttling, ...) from everything else and bounds the backoff.
That covers the common case - a busy store, a blip talking to a
remote-backed backend - without changing behavior when replay
succeeds. An error that is not transient, or outlives the retry
budget, is not retried further: propagating it so the offset never
advances would stall this peer's entire stream behind one event that
may never replay, which is worse than the one entry staying stale.
Instead it is skipped, loudly - counted in a new
stats.FilerMetaAggregatorReplayFailures metric and logged at error
level - so the divergence is discoverable instead of silent.
Tested: go build ./... and go test ./weed/filer/... ./weed/server/...
Added meta_aggregator_replay_test.go: one test fails against the old
one-shot Replay call (a single transient failure is never retried, so
the store never converges) and passes with the fix; a second covers a
permanently-failing event completing quickly and being counted instead
of retried forever.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* filer: keep the test quota constant int64 for 32-bit builds
An untyped shift constant passed to t.Fatalf's ...any defaults to int and
overflows on 32-bit, failing go vet there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* filer: name the diverged entry in the give-up log line
event.Directory is only the parent (typically /buckets), so for any
directory with more than one child the previous log line could not say
which entry failed to replay - the exact thing the change exists to
make discoverable. Name comes from NewEntry, falling back to OldEntry
for deletes; both getters are nil-safe.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* chore(filer): trim metadata replay comments to the non-obvious why
Compress the added comments on replicateMetadataChange and its tests down to
the reasoning a maintainer cannot get from the code: why a retry-exhausted
failure is skipped rather than propagated, what the old one-shot Replay body
did that the test pins, and why the quota constant is typed int64. Drops
deployment-specific narration and restatement of the code. No behaviour change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* chore: restore load-bearing clauses trimmed in the comment pass
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* filer: document and test the multi-step DeleteEntry replay hazard
CodeRabbit flagged that FilerStoreWrapper.DeleteEntry skips the delete
once FindEntry reports the path already gone, and that the redis store
families remove the primary key before parent-directory membership.
Chained together, a delete that fails between those two steps is
retried as a no-op: the stale membership is never revisited, and
replicateMetadataChange now reports overall success for it without
incrementing FilerMetaAggregatorReplayFailures, whereas before this PR
every such failure was unconditionally logged. The underlying store
inconsistency is pre-existing (a single non-retried Replay already
leaves the same stale membership behind); what retry adds is that this
one case no longer surfaces it.
Making Replay atomic or teaching every store to repair secondary
mutations on retry is out of scope here. Instead: document the hazard
at Replay, filerstore_wrapper.go's DeleteEntry, and
replicateMetadataChange, and add a test against the real
FilerStoreWrapper (not a strawman) that pins down the current,
documented behavior.
* filer: trim replay retry comments and tests
Drop the comment-only hunks documenting the pre-existing DeleteEntry
partial-failure hazard, the test that asserted that hazard still exists,
and the second hand-rolled fake store. Reuse stubFilerStore for the two
retry tests.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* admin: make paths relative
* admin: make filer browser link and nav path checks prefix-relative
* admin: add isCurrentPath and currentPathStartsWith helpers
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* benchmark tool for mounted filesystems
* ci: on-demand mount benchmark, native WinFsp vs rclone plus a Linux reference
* windows mount: let the Windows cache manager cache file data
WinFsp only turns the cache manager on for a file when FileInfoTimeout
is infinite; at any finite value every application read and write is a
synchronous trip into the mount process at whatever size the application
issued. Metadata events already reach FspFileSystemNotify, which purges
a changed file's cached pages and attributes, so an infinite timeout
stays coherent. The dir listing, volume info and EA timeouts are pinned
to one second so they do not silently inherit the infinity.
* windows mount: cache resolved paths and attributes in the adapter
WinFsp addresses every operation by path and has no FORGET, so the
adapter walked the whole path through Lookup on each one, and in a
directory the filer has not listed yet every walk was a filer round
trip; nothing played the part of the kernel's dentry and attribute
caches. The path cache owns one lookup reference per entry the way the
kernel holds one until FORGET, serves attribute reads for files without
an open handle, and is purged by the mount's own mutations and by
metadata events, with the timeout as backstop.
* windows mount: keep a closed file's attributes cached
Open steals the path's cache entry for its handle and Release returned
the reference with a purge, so the stat that follows every copied file
walked to the filer again. Reading the handle's final attributes before
it goes away and moving the reference back into the cache serves that
stat locally, the way the kernel's attribute cache does after a close.
Only if the path still names that inode, though: WinFsp reports the
path the handle opened with, and after a delete-on-close or a rename
caching it would resurrect an entry that is gone.
* windows mount: persist entries at create, and let the flush stay at close
WinFsp posts the cleanup and close that carry the flush after
CloseHandle has returned, so deferring the filer entry to the flush let
everything that reads through the filer race an unflushed close: a
listing missed just-written files, and a directory rename moved a
directory on the filer before its newest child existed there, leaving
the straggler flush to recreate the child under the dead path.
Flush-at-cleanup is not the answer either: it makes every handle's
cleanup flush, and those flushes race the unlinks of delete-on-close,
re-inserting the entry the unlink just removed. Persisting the entry at
create takes the ordering question away.
* mount: flush written pages before a truncate shrinks past them
The shrink trims chunks, but written pages that have not become chunks
yet are invisible to it, so the next flush wrote them back and the file
grew again, resurrecting the truncated bytes. Windows hits this on
every write-then-shrink because its flush runs after CloseHandle, but
the gap is platform-neutral.
* mount: order a file's unlink against its in-flight flush
Unlink set the handle's deleted flag bare, so a flush already past its
own check of that flag wrote the entry back right after the delete
removed it, and a delete-on-close file outlived its last handle. The
flag is now set under the handle's flush lock and re-checked under it,
so a flush either completes before the delete or sees the flag and
skips. An eagerly created handle also starts clean: the dirty mark
existed to make the deferred filer create happen at flush, and eager
creates have nothing to flush.
* shell: volume.move restores source writability when aborted after the copy phase
* shell: volume.move removes the incomplete target copy when aborted before the source delete
* shell: give each abort cleanup RPC its own timeout
* filer: TUS creation accepts Upload-Concat partial uploads
* filer: TUS final uploads concatenate completed partials
* filer: TUS concatenation tests
* filer: consumed marker pins TUS chunk ownership on completion
* filer: TUS session delete decides chunk ownership after removing the session info
* filer: TUS completion persists the consumed marker before creating the entry
* filer: TUS completion re-verifies the session after persisting the consumed marker
* filer: serialize TUS session ownership transitions per filer
* filer: surface failed TUS consumed-marker rollbacks
* s3: a null object wins over a rescan when the latest-version pointer is absent
The read path already resolves an absent pointer this way; the listing-path
counterpart scanned .versions/ first and could surface an old version or
delete marker over the current suspended-versioning null object.
* s3: dedup a key against its .versions sibling in suspended buckets too
A suspended bucket keeps its .versions directories, so a suspended-versioning
null object and its .versions sibling emitted the same key twice.
* s3: retract a null object from the listing when a delete marker shadows it
Deleting a key whose null version predates versioning leaves the base-path
entry in place and records the delete marker under <key>.versions. The
listing appended the base-path entry and relied on the .versions sibling to
replace it, but a delete-marker current version emitted nothing, so the
deleted key stayed visible to ListObjects while GET and HEAD returned 404.
* s3: keep a key's .versions sibling on the same page as the key
When the page quota ran out between a base-path entry and its .versions
directory, the page ended with the stale entry and the next page skipped the
directory as a marker echo, so the replacement or retraction never happened.
* s3: the null version is not latest when the .versions pointer names a newer one
ListObjectVersions stamped IsLatest on every base-path null object, so a key
deleted after enabling versioning reported IsLatest on both the delete marker
and the null version.
* s3: test listing after a pre-versioning null object is delete-marked
* s3: find a key's earlier page entry by scan, not by adjacency
A key such as k.bak sorts between k and k.versions, so the entry a .versions
sibling replaces or retracts is not always the last one on the page. Scan
back through the page for the key, and insert a late resolution in sorted
position instead of at the end.
* s3: settle trailing null objects by lookup when a page fills
The quota can run out while keys still sit between a null object and its
.versions sibling, and the sibling-adjacent page-boundary exception never
fires for those. Track the trailing null objects whose sibling has not been
ruled out and look each one up before declaring the page full; a retraction
reopens the quota.
* s3: do not resolve a .versions sibling its page has already moved past
A page resuming from a marker inside the base key's extension region has
already listed and settled the base null object on an earlier page, so
resolving the .versions directory again re-emitted the key.
* s3: test listing with keys between a null object and its .versions sibling
* s3: pick the newer of the null object and the scanned versions
Making the null object win outright whenever the pointer is absent misread
multi-filer pointer lag: version files replicate ahead of the pointer, and a
key overwritten or delete-marked after pre-versioning days would list its
stale null again. The suspended-versioning write that legitimately makes the
null current is also the newer entry, so mtime tells the two apart.
* s3: a delete-marked null object no longer keeps its prefix alive
The hidden-entries probe took any plain file as proof of a listable key, but
a null object shadowed by its .versions sibling's delete marker is not one.
Hold plain files pending until the sibling settles them either way.
* s3: settle an evicted pending null instead of dropping it
Nested keys like k, k!, k!! can hold more pending nulls than the cap. A
silently evicted one could close the page unsettled, and the resume skip
would then keep the stale entry for good.
* s3: test deleted-prefix hiding and the pending-null cap
* s3: cover the reported '!' intervening key with a live version
* s3: an unstamped same-second version outranks the null object
Second-resolution mtimes cannot order same-second writes, so the tie went to
the stale null when the pointer lagged. The suspended write that makes a null
current stamps the version it displaces before clearing the pointer, so the
stamp is the authoritative signal and a tie without it goes to the version.
* s3: a pointer-less versions listing still checks what replicated
ListObjectVersions took a missing pointer as proof the null object is latest,
but under pointer lag the sibling can already hold newer replicated versions
or markers. Apply the same nullObjectWins rule as the listing recovery.
* s3: a failed null-object settlement fails the listing
Every getEntry error read as a missing sibling, so a transient filer error at
a page boundary committed the unsettled null and the next page skipped its
sibling for good. Only a definitive not-found means the null is live; other
failures are retained on eviction and fail the request at page close.
* s3: retract a CommonPrefix whose only backers were delete-marked nulls
The directory probe settles this for the / delimiter, but any other delimiter
derives prefixes from base-path keys directly, and a prefix built solely from
null objects survived their delete markers. Count the unsettled null backers
behind the newest prefix and retract it when the last one settles as a marker;
a live resolution or any listable contributor confirms the prefix instead.
* s3: test custom-delimiter prefix retraction
* s3: an explicit signal marks the null object current, not the demotion stamp
The NoncurrentSinceNs stamp survives promotion: delete the version that
demoted another and the promoted one is current yet still stamped, so a
lagging replica would resurrect the stale null. A suspended-versioning write
now records Seaweed-X-Amz-Null-Version-Is-Latest on the .versions directory
when it clears the pointer, every pointer update removes it, and the
recovery paths trust the signal instead of the stamp.
* s3: a filer failover retry rebuilds the listing page from scratch
The failover wrapper reruns the callback on another filer after a transport
error, and the partially built page, spent quota, and advanced marker leaked
into the retry, which could then return a stale or duplicated page as
success.
* s3: only a prefix's own backers can debit it
A delete marker for a version-only key (no base object) derived the same
prefix as its neighbors and decremented backing it never contributed,
retracting a prefix that a live null object still backed. Track backers by
key so settlement is idempotent and only debits what was counted.
* s3: test a version-only marker against a null-backed prefix
* s3: a pointer recompute clears the null-current signal
The routed finalize for delete markers, COPY, and multipart rewrites the
.versions pointer through RECOMPUTE_LATEST, which left a suspended-era
null-current signal in place. Version files never carry the signal, so
mapping it in CopyExtended deletes it whenever the pointer recomputes.
* s3: the pointer outranks the null-current signal in the versions listing
The signal check guarded the pointer check, so a stale signal a recompute
had not cleared yet would have let the null claim IsLatest alongside the
pointed-at version.
* topology: name a node's volumes without copying them
ToVolumeLocations reads a volume id off every volume in the cluster, and got
there through GetVolumes, which copies a whole storage.VolumeInfo per volume to
be read for four bytes of it. Every client that connects asks for this.
At 800k volumes the walk goes from 94.6MB to 16.0MB, which is the ids
themselves.
* master: log why a client send failed, not what was sent
The message names every volume on a newly connected node, so a client going
away had the master format a protobuf that size into text -- through the one
log level that is always on. The error is the part worth having.
Go stores a map value inline once it fits in 128 bytes, and indirectly above
that. Shrinking storage.VolumeInfo to 120 bytes therefore moved 1.6M of them
into the map's own slots, which are allocated to capacity rather than to
occupancy, and the map grew by 149MB even though each volume got smaller.
Hold pointers, so the size of a volume record stops deciding how it is stored.
Updates are written through the pointer already there, so a heartbeat that
changes a volume allocates nothing, as it did when the struct was above the
threshold and Go was doing this itself.
At 800k volume ids across 3 servers the resident topology goes from 705.3MB to
546.6MB, with heartbeat cost unchanged at 111.6MB.
* master: stream volume listings
A listing of 800k volumes is 36MB on the wire but 305MB as messages, and the
master built all of it, then held it while grpc encoded it. Two of those at
once is most of a small master's heap, and the maintenance scanner asks every
30 minutes.
The topology goes out first, listing nothing, then its volumes in batches, so
the master holds a batch rather than a cluster: 341MB of live heap for one
listing becomes 4.4MB. It allocates much the same either way -- what changes is
how much of it has to be live at once, which is what sets the heap ceiling.
Batches are built under their disk's lock and sent outside it, so a slow reader
stalls the stream rather than the topology. They therefore do not share one
instant, which a single listing did not either: it takes each disk's lock in
turn, so a volume moving during either can be seen twice or not at all.
The client helper hides which kind of master answered: one too old for the
stream is asked the old way and its reply cut into the same batches. Either way
the topology handed over lists no volumes, so a caller cannot come to depend on
finding them there.
* admin: stream the listing the maintenance scan reads
It asks for every volume in the cluster every 30 minutes. Reassembling it
client-side keeps the scan identical -- ActiveTopology splits disks by the
disk ids on the volumes, so it needs them in the topology -- while the master
no longer builds the whole reply to send it.
* topology: report a disk id that does not depend on map order
A topology disk that fronts several physical disks took its reported id from
whichever volume the map yielded first, so two listings of an unchanged disk
could disagree. Take the smallest instead.
* topology: test that a streamed listing rebuilds to the whole one
The callers that stream now rebuild the listing from a topology sent without
volumes plus the batches after it, so that has to come out the same as being
sent it whole, at every batch size and under a filter.
* clients: stream the volume listings that ask for everything
The dashboard's list and export pages, the collection and ec shard pages, the
topology view, the worker metrics and two shell commands each asked the master
to build all 800k volumes into one reply. They read the same listing as before,
rebuilt on their side, so the master no longer holds it.
The three that already ask for one volume or one collection stay as they are:
their replies are small, and streaming one costs a round trip to say so.
* master: stream volume listings
A listing of 800k volumes is 36MB on the wire but 305MB as messages, and the
master built all of it, then held it while grpc encoded it. Two of those at
once is most of a small master's heap, and the maintenance scanner asks every
30 minutes.
The topology goes out first, listing nothing, then its volumes in batches, so
the master holds a batch rather than a cluster: 341MB of live heap for one
listing becomes 4.4MB. It allocates much the same either way -- what changes is
how much of it has to be live at once, which is what sets the heap ceiling.
Batches are built under their disk's lock and sent outside it, so a slow reader
stalls the stream rather than the topology. They therefore do not share one
instant, which a single listing did not either: it takes each disk's lock in
turn, so a volume moving during either can be seen twice or not at all.
The client helper hides which kind of master answered: one too old for the
stream is asked the old way and its reply cut into the same batches. Either way
the topology handed over lists no volumes, so a caller cannot come to depend on
finding them there.
* admin: stream the listing the maintenance scan reads
It asks for every volume in the cluster every 30 minutes. Reassembling it
client-side keeps the scan identical -- ActiveTopology splits disks by the
disk ids on the volumes, so it needs them in the topology -- while the master
no longer builds the whole reply to send it.
Volume growth registers a provisional record before it can know which
directory the server chose, while the server's own report -- pushed
during the AllocateVolume RPC -- carries the real disk id. The merge is
last-writer-wins, so whichever lands second sticks, and fresh volumes
nondeterministically show disk 0 on multi-dir servers. Keep the reported
disk id when the provisional update carries none, before the report
digest is computed so the stored record stays consistent with what the
server keeps reporting.
Claude-Session: https://claude.ai/code/session_01QdTEEPbg4MtcoEGwqbgtZC
The volume server names the directory index in every
VolumeShortInformationMessage, but NewVolumeInfoFromShort dropped it, so
volumes registered through the incremental new-volume path showed
disk_id 0 at the master until a full report -- misreporting multi-dir
servers in volume.list and the per-physical-disk topology views.
Claude-Session: https://claude.ai/code/session_01QdTEEPbg4MtcoEGwqbgtZC
* volume: decode IPv6 transition addresses in the remote-endpoint guard
checkBlockedIP normalized only ::ffff: mapped IPv4, so NAT64 (64:ff9b::/96),
6to4 (2002::/16), Teredo (2001:0000::/32), and IPv4-compatible (::/96) addresses
that embed an internal IPv4 (loopback, 169.254.169.254, RFC 1918) passed the
endpoint guard even though the plain IPv4 forms are refused. Extract the
embedded IPv4 from those forms and re-check it against the deny list, which
covers both the up-front validation and the dial-time guard. Mirrored in the
Rust volume server.
* volume: require the full NAT64 well-known prefix before decoding
Only 64:ff9b::/96 carries the embedded IPv4 in the low 32 bits, so also require
bytes 4-11 to be zero before treating an address as NAT64; other 64:ff9b:
prefixes place the IPv4 elsewhere and are left untouched. Add public-target
coverage for 6to4, Teredo, and IPv4-compatible so every decoder is exercised on
both a blocked and an allowed destination. Mirrored in the Rust volume server.
* master: let VolumeList ask for the volumes it wants
The request carried nothing, so every caller was answered with the whole
cluster. A dashboard opening one volume's page, or a capacity probe adding up
one bucket, was served all 800k of them and threw away the rest -- and the
master built every one of those messages first.
The topology, its disks and their counters are still reported in full: a caller
reading free space or replica placement needs the cluster whichever volumes it
asked about. Only what is listed under a disk is selected, ec shards included.
An empty collection and a zero volume id take everything, the way volume.list
already reads its own -collectionPattern and -volumeId, so a caller that
forgets to narrow is answered too much rather than answered wrongly. That
leaves the default collection unnameable, since it is the one the empty string
names, so it gets a field of its own.
An older client sends none of it and is answered exactly as before.
* admin: ask the master for the volume the page is showing
A volume's detail page was pulling every volume in the cluster to find one and
its replicas, and discarding the rest.
* admin: ask the master for the ec volume the page is showing
Same as the volume detail page: one volume's shards were found by pulling every
ec shard in the cluster.
* s3: ask the master for the bucket's own collection
The SOSAPI capacity probe summed one collection's volumes out of a listing of
every volume in the cluster. Cluster capacity still comes out the same: it is
read from the disk counters, which a filtered listing reports in full.
* topology: read the disk usage counters atomically
They are written with atomic.AddInt64 from heartbeats but were read plainly by
the two listings and by FreeSpace, and the map they sit in was iterated without
the lock its neighbour takes. Under -race a listing concurrent with a heartbeat
trips on both.
A master decides nothing from it. Every caller that read it was asking whether
a volume is remote, which the backend name answers, and the value itself is
reported on demand by the server holding the volume, through the volume info in
ReadVolumeFileStatus.
It is also the one string here that cannot be shared: unique per volume, so
unlike the collection and backend names it carries its own characters for every
volume a master tracks.
VolumeInfo goes from 136 bytes to 120. 800k volumes registered from a heartbeat
that has been over the wire go from 214 to 163 B/volume when tiered.
The volume server's own status page keeps showing the key, now read from the
volume it holds rather than relayed through a master, which is also where the
other volume server implementation reads it.
The heartbeat digest drops it on the same grounds: a change to something the
master does not hold cannot make its copy stale. Both implementations and their
shared vectors move together, and the field-coverage test now names what is
deliberately not retained rather than being loosened.
* storage: order VolumeInfo by alignment
The struct is held for every volume replica in the cluster, so the padding the
compiler inserts is multiplied by however many volumes a master tracks. Two
one-byte fields each sat at the head of a word and left the rest of it empty,
which was ten of the eighteen wasted bytes.
Grouping by size rather than by meaning takes the struct from 152 bytes to 136,
and the map holding them shrinks with it, since a Go map's slack scales with
the size of the value.
800k volumes registered from a heartbeat that has been over the wire:
211 -> 195 B/volume, 214 -> 198 tiered.
* trim the comments on this change to the parts that are not evident
* topology: build the volume list without copying the volume map first
ToDiskInfo copied every VolumeInfo on the disk into a fresh slice, walked it to
build a protobuf message for each, and threw the copy away. The copy was as
large as the messages it produced.
Building them straight from the map holds the disk's read lock for the walk
rather than just the copy, so a heartbeat updating that disk waits for it. It
is a read lock on a call that is now infrequent, against an allocation of the
same size as the response.
ToTopologyInfo over 550k volumes 193617502 B/op -> 110011017 B/op, and faster
for not making the copy.
* trim the comments on this change to the parts that are not evident
* ec.balance: add a -volumeIds filter
Collection scope is often too broad for maintenance. -volumeIds narrows the
plan to the given ec volume ids by leaving every other volume out of the
topology handed to the planner, so no phase, dedup included, can plan against
them. Ids with no ec shard in the selected collection, dataCenter and disk type
are rejected rather than silently skipped.
* ec.encode: key the orphan sweep without narrowing the volume id
int is 32-bit on 32-bit builds, so int(vid) wraps for volume ids above
MaxInt32. Format the id as the uint32 it is.
2026-08-09 09:37:49 -07:00
652 changed files with 65302 additions and 10137 deletions
echo "::warning::RELEASE_PAT is not set. The tag will be pushed with GITHUB_TOKEN, so the binary, container and helm workflows will not start on their own."
fi
- name:Compute new version
id:compute
@@ -99,17 +134,103 @@ jobs:
sed -i -E "s/^version:.*/version: ${CHART_VERSION}/" "$CHART"
* [Example: Using Seaweed Object Store](#example-using-seaweed-object-store)
* [Example: Using Seaweed Blob Store](#example-using-seaweed-blob-store)
* [Architecture](#object-store-architecture)
* [Compared to Other File Systems](#compared-to-other-file-systems)
* [Compared to HDFS](#compared-to-hdfs)
* [Compared to GlusterFS, Ceph](#compared-to-glusterfs-ceph)
* [Compared to GlusterFS](#compared-to-glusterfs)
* [Compared to Ceph](#compared-to-ceph)
* [Compared to Minio](#compared-to-minio)
* [Compared to MinIO, RustFS](#compared-to-minio-rustfs)
* [Dev Plan](#dev-plan)
* [Installation Guide](#installation-guide)
* [Disk Related Topics](#disk-related-topics)
@@ -90,7 +90,7 @@ S3_BUCKET=my-bucket \
./weed mini -dir=/data
```
That's it — the S3 endpoint is at http://localhost:8333, `my-bucket` already exists, and `admin`/`secret` are valid credentials. `S3_BUCKET` accepts a comma-separated list (e.g. `raw,processed`); use `S3_TABLE_BUCKET` for S3 Tables (Iceberg) buckets. Drop any of the env vars to skip that piece (no AWS keys → S3 runs in unauthenticated "Allow All" mode for development).
That's it — the S3 endpoint is at http://localhost:8333, `my-bucket` already exists, and `admin`/`secret` are valid credentials. `S3_BUCKET` accepts a comma-separated list (e.g. `raw,processed`); use `S3_TABLE_BUCKET` for S3 Tables buckets, each `name` or `name:FORMAT` where the format is `ICEBERG` (the default) or `LANCE`. Drop any of the env vars to skip that piece (no AWS keys → S3 runs in unauthenticated "Allow All" mode for development).
The same command starts everything else too:
- **S3 Endpoint**: http://localhost:8333
@@ -466,7 +466,8 @@ The architectures are mostly the same. SeaweedFS aims to store and read files fa
| GlusterFS | hashing | | FUSE, NFS | | |
| Ceph | hashing + rules | | FUSE | Yes | |
| MooseFS | in memory | | FUSE | | No |
| MinIO | separate meta file for each file | | | Yes | No |
| MinIO | separate meta file per drive for each file | | | Yes | No |
| RustFS | separate meta file per drive for each file | | | Yes | No |
[Back to TOC](#table-of-contents)
@@ -508,24 +509,26 @@ SeaweedFS Filer uses off-the-shelf stores, such as MySql, Postgres, Sqlite, Mong
[Back to TOC](#table-of-contents)
### Compared to MinIO ###
### Compared to MinIO, RustFS ###
Please note, as Apr 25, 2026 MinIO ceased development. It's strongly discouraged to use that unmaintained software with multiple security bugs.
Please note, as Apr 25, 2026 MinIO ceased development. It's strongly discouraged to use that unmaintained software with multiple security bugs. RustFS is a MinIO reimplementation in Rust, Apache 2.0 licensed and still developed, keeping MinIO's storage model down to a byte-compatible on-disk format. So the points below apply to both.
MinIO followed AWS S3 closely and was ideal for testing for S3 API. It had good UI, policies, versionings, etc. SeaweedFS is trying to catch up here.
MinIO metadata were in simple files. Each file write will incur extra writes to corresponding meta file.
The metadata are in simple files. Each file write incurs extra writes to the corresponding meta file, on every drive of the erasure set. Changing only tags or retention rewrites that meta file on all of them, so the write amplification does not shrink with object size.
MinIO did not have optimization for lots of small files. The files were simply stored as is to local disks.
There is no optimization for lots of small files. The files are simply stored as is to local disks.
Plus the extra meta file and shards for erasure coding, it only amplifies the LOSF problem.
MinIO had multiple disk IO to read one file. SeaweedFS has O(1) disk reads, even for erasure coded files.
Multiple disk IO are needed to read one file. SeaweedFS has O(1) disk reads, even for erasure coded files.
MinIO had full-time erasure coding. SeaweedFS uses replication on hot data for faster speed and optionally applies erasure coding on warm data.
Erasure coding is full-time. SeaweedFS uses replication on hot data for faster speed and optionally applies erasure coding on warm data.
MinIO did not have POSIX-like API support.
No POSIX-like API support.
MinIO had specific requirements on storage layout. It is not flexible to adjust capacity. In SeaweedFS, just start one volume server pointing to the master. That's all.
There are specific requirements on storage layout, which makes it hard to scale out and to maintain. An erasure set must be 2 to 16 drives and must divide the drive list symmetrically, and capacity grows or shrinks a whole pool at a time. In SeaweedFS, just start one volume server pointing to the master. That's all.
# Use Azure's Ubuntu mirror — much faster than archive.ubuntu.com from GitHub-hosted runners,
# which have been hanging long enough on Ign:/retry to trip the 10-min step timeout.
# Note: This e2e test image intentionally runs as root for simplicity and compatibility.
# Production images (Dockerfile.go_build) use proper user isolation with su-exec.
# For testing purposes, running as root avoids permission complexities and dependency
# on Alpine-specific tools like su-exec (not available in Ubuntu repos).
RUN sed -i 's|http://archive.ubuntu.com/ubuntu|http://azure.archive.ubuntu.com/ubuntu|g; s|http://security.ubuntu.com/ubuntu|http://azure.archive.ubuntu.com/ubuntu|g' /etc/apt/sources.list &&\
* master/filer/volume are stateful sets with anti-affinity on the hostname,
so your deployment will be spread/HA.
* chart is using memsql(mysql) as the filer backend to enable HA (multiple filer instances) and backup/HA memsql can provide.
* mysql user/password are created in a k8s secret (default: `<release>-seaweedfs-db-secret`) and injected to the filer with ENV.
* leveldb2 is the default filer backend; a mysql-compatible database (memsql, ...) enables HA (multiple filer instances) and the backup/HA it can provide.
* with `filer.extraEnvironmentVars.WEED_MYSQL_ENABLED` set to `"true"`, mysql user/password are created in a k8s secret (default: `<release>-seaweedfs-db-secret`) and injected to the filer with ENV. On any other store neither the secret nor the `WEED_MYSQL_*` env, plain or secret-backed, is rendered.
* cert config exists and can be enabled, but not been tested, requires cert-manager to be installed.
"description":"Workers with a live control stream. Unlike the Admin/Maintenance row, this counts plugin workers, which is what a Rust or Go plugin worker actually is.",
"type":"stat",
"id":219,
"gridPos":{
"h":4,
"w":6,
"x":0,
"y":28
},
"fieldConfig":{
"defaults":{
"color":{
"mode":"thresholds"
},
"mappings":[],
"unit":"short",
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
}
},
"overrides":[]
},
"options":{
"colorMode":"value",
"graphMode":"area",
"justifyMode":"auto",
"orientation":"auto",
"reduceOptions":{
"calcs":[
"lastNotNull"
],
"fields":"",
"values":false
},
"textMode":"auto"
},
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"editorMode":"code",
"expr":"count(SeaweedFS_worker_connected{cluster=~\"$cluster\"} == 1) or vector(0)",
"instant":true,
"range":false,
"refId":"A"
}
]
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"title":"Worker Job Failures (1h)",
"description":"Jobs that ended in failure across all workers in the last hour.",
"type":"stat",
"id":220,
"gridPos":{
"h":4,
"w":6,
"x":6,
"y":28
},
"fieldConfig":{
"defaults":{
"color":{
"mode":"thresholds"
},
"mappings":[],
"unit":"short",
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
}
},
"overrides":[]
},
"options":{
"colorMode":"value",
"graphMode":"area",
"justifyMode":"auto",
"orientation":"auto",
"reduceOptions":{
"calcs":[
"lastNotNull"
],
"fields":"",
"values":false
},
"textMode":"auto"
},
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"editorMode":"code",
"expr":"sum(increase(SeaweedFS_worker_jobs_total{cluster=~\"$cluster\",result=\"failed\"}[1h])) or vector(0)",
"instant":true,
"range":false,
"refId":"A"
}
]
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"title":"Objects Seen vs Skipped",
"description":"A sweep that proposes nothing because there was nothing to do and one that proposes nothing because it could read nothing produce the same proposal count. Skips are the difference, and are the thing to alert on.",
"type":"timeseries",
"id":221,
"gridPos":{
"h":8,
"w":12,
"x":12,
"y":32
},
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"drawStyle":"line",
"fillOpacity":10,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":4,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"never",
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"mappings":[],
"unit":"ops",
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
}
},
"overrides":[]
},
"options":{
"legend":{
"calcs":[
"lastNotNull",
"max"
],
"displayMode":"table",
"placement":"bottom",
"showLegend":true
},
"tooltip":{
"mode":"multi",
"sort":"desc"
}
},
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"editorMode":"code",
"expr":"sum by (job_type) (rate(SeaweedFS_worker_objects_seen_total{cluster=~\"$cluster\"}[$__rate_interval]))",
"range":true,
"refId":"A",
"legendFormat":"seen {{job_type}}"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"editorMode":"code",
"expr":"sum by (job_type, reason) (rate(SeaweedFS_worker_objects_skipped_total{cluster=~\"$cluster\"}[$__rate_interval]))",
"expr":"sum by (job_type) (rate(SeaweedFS_worker_proposals_total{cluster=~\"$cluster\"}[$__rate_interval]))",
"range":true,
"refId":"B",
"legendFormat":"proposals {{job_type}}"
}
]
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"title":"Jobs Executed",
"description":"Jobs run by these workers, by outcome.",
"type":"timeseries",
"id":223,
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":40
},
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"drawStyle":"line",
"fillOpacity":10,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":4,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"never",
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"mappings":[],
"unit":"ops",
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
}
},
"overrides":[]
},
"options":{
"legend":{
"calcs":[
"lastNotNull",
"max"
],
"displayMode":"table",
"placement":"bottom",
"showLegend":true
},
"tooltip":{
"mode":"multi",
"sort":"desc"
}
},
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"editorMode":"code",
"expr":"sum by (job_type, result) (rate(SeaweedFS_worker_jobs_total{cluster=~\"$cluster\"}[$__rate_interval]))",
"range":true,
"refId":"A",
"legendFormat":"{{job_type}} ({{result}})"
}
]
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"title":"Job Duration p99",
"description":"How long a job takes, per job type.",
"type":"timeseries",
"id":224,
"gridPos":{
"h":8,
"w":12,
"x":12,
"y":40
},
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"drawStyle":"line",
"fillOpacity":10,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":4,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"never",
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"mappings":[],
"unit":"s",
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
}
},
"overrides":[]
},
"options":{
"legend":{
"calcs":[
"lastNotNull",
"max"
],
"displayMode":"table",
"placement":"bottom",
"showLegend":true
},
"tooltip":{
"mode":"multi",
"sort":"desc"
}
},
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"editorMode":"code",
"expr":"histogram_quantile(0.99, sum(rate(SeaweedFS_worker_job_seconds_bucket{cluster=~\"$cluster\"}[$__rate_interval])) by (le, job_type))",
"range":true,
"refId":"A",
"legendFormat":"{{job_type}}"
}
]
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"title":"Worker Slots",
"description":"Capacity a worker advertised and how much of it is in use. Held slots are what admin schedules against.",
"type":"timeseries",
"id":225,
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":48
},
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"drawStyle":"line",
"fillOpacity":10,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":4,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"never",
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"mappings":[],
"unit":"short",
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
}
},
"overrides":[]
},
"options":{
"legend":{
"calcs":[
"lastNotNull",
"max"
],
"displayMode":"table",
"placement":"bottom",
"showLegend":true
},
"tooltip":{
"mode":"multi",
"sort":"desc"
}
},
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"editorMode":"code",
"expr":"sum by (lane) (SeaweedFS_worker_slots_used{cluster=~\"$cluster\"})",
"range":true,
"refId":"A",
"legendFormat":"used {{lane}}"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"editorMode":"code",
"expr":"sum by (lane) (SeaweedFS_worker_slots_total{cluster=~\"$cluster\"})",
"range":true,
"refId":"B",
"legendFormat":"total {{lane}}"
}
]
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"title":"Control Stream Events",
"description":"Connects, closes and failures. A worker that reconnects steadily is usually two workers sharing one id, evicting each other.",
"type":"timeseries",
"id":226,
"gridPos":{
"h":8,
"w":12,
"x":12,
"y":48
},
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"drawStyle":"line",
"fillOpacity":10,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":4,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"never",
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"mappings":[],
"unit":"ops",
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
}
},
"overrides":[]
},
"options":{
"legend":{
"calcs":[
"lastNotNull",
"max"
],
"displayMode":"table",
"placement":"bottom",
"showLegend":true
},
"tooltip":{
"mode":"multi",
"sort":"desc"
}
},
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"editorMode":"code",
"expr":"sum by (event) (rate(SeaweedFS_worker_stream_events_total{cluster=~\"$cluster\"}[$__rate_interval]))",
"range":true,
"refId":"A",
"legendFormat":"{{event}}"
}
]
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"title":"Lance Maintenance Reclaimed",
"description":"What the Lance jobs actually removed. A job that runs every minute and reclaims nothing is a different thing from a job that never runs.",
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
github-actions[bot]
Chris LuandGitHub
Lars LehtonenandGitHub
Александр МелентьевandGitHub
Cavus700andGitHub
