* arangodb: bind list prefix, start file name and collection into the AQL query
Concatenating them into the query text let a caller-supplied prefix or
start name close the string literal and append arbitrary AQL, which runs
with the filer's ArangoDB credentials against any collection.
* arangodb: bind the folder path and collection into the recursive delete query
A trailing-slash S3 key reaches DeleteFolderChildren through the
directory-marker cleanup, so quotes in the path could turn the filter
into a match-everything REMOVE over the whole bucket collection.
* arangodb: match the real directory prefix in the recursive delete
The prefix was built by re-joining the path segments with commas, so it
never matched a stored directory and the subtree sweep did nothing.
* worker: log the balance move stage through a constant format string
Go 1.26's printf analyzer now follows printf wrappers reached through an
interface, so passing the stage straight to Logger.Info is a vet failure.
* s3api: bracket the IPv6 host in the signature test URL
A bare IPv6 literal is legal in a Host header but never in a URL. Go 1.26
stopped parsing it leniently, so carry the two forms separately and set
r.Host to the value the client would actually have signed.
* mini: bracket IPv6 addresses in the readiness probe URLs
An IPv6-only host hands mini a bare literal, and %s:%d pasted it into a URL
unbracketed. Under Go 1.26 that URL no longer parses, so waiting for the
admin server never succeeds and mini refuses to start.
* readme: add RustFS to the file system comparison
* readme: note RustFS write amplification and rigid layout
* readme: correct RustFS version, parity and protocol details
* readme: merge the RustFS comparison into the MinIO section
* filer: restore a folder that received an entry while it was deleted
The empty-folder cleaner checks that a folder is empty and then deletes it,
and those two steps are not atomic. An entry created in between survives the
delete but loses the directory holding it: still readable by its own path, yet
absent from every listing until a later write happens to recreate the parent.
Record the folders deleted in each pass and re-check them on the next one,
putting back any that turned out to hold entries. The check waits a pass on
purpose - a writer looks up the parent before inserting the child, so checking
straight after the delete can still run ahead of the insert and see nothing.
Restoring a directory that holds entries is always correct, and restoring one
whose entry went away again just leaves an empty folder for a later pass to
collect, so the repair needs no locking or coordination.
Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r
* filer: keep failed restores queued and inherit the ancestor's ownership
Two gaps in the restore pass.
A folder whose count or restore hit a transient store error was dropped from
the tracking list and never looked at again, leaving its entries out of
listings until some later write recreated the folder - the very thing the pass
exists to avoid. Put those back for the next pass, still under the cap.
A restored folder was minted with a fixed mode and no owner, so a directory
that had been private came back world-readable and owned by root. Take the
mode and ownership from the nearest ancestor still present instead.
Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r
* filer: let the redis stores keep a directory listing that still has entries
On the redis stores the listing is not derived from the entries, it is the only
record that they sit under that directory. DeleteEntry opened by dropping it
outright, so an entry that arrived after the caller judged the directory empty
lost its membership and became unreachable: readable by exact path, absent from
every listing, and invisible to any later check, since counting the directory
reads the listing that was just destroyed. Nothing could detect or repair it.
Drop the listing in DeleteFolderChildren instead, alongside the children it
describes, and leave it alone in DeleteEntry. redis3 needs it explicitly, since
removeChildren clears the skip list nodes but not the list itself, and the plain
redis store was leaking the key entirely.
Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r
* filer: restore folders with their own attributes, and observe them for a window
Five gaps in the restore pass.
The restored directory was reconstructed from whatever ancestor happened to
still be present, and the mode was ORed with 0111 on the way. A private
directory under a world-traversable parent came back granting traversal it had
denied. Read the folder's own attributes before deleting it and put exactly
those back. That also removes the ancestor walk, which treated a transient
store error as "not found" and silently fell through to a broader ancestor.
A single check a pass later was not a delay at all. Ticker sends coalesce, so
when a pass runs long the next one starts immediately, and a writer already
past its parent lookup can insert after the check has read zero - after which
the folder was discarded for good. Keep each folder under observation for a
bounded wall-clock window and re-check it on every pass until it expires. This
narrows the exposure rather than closing it; only making the emptiness check
and the delete atomic would do that.
A delete that returned an error was never observed at all, though the redis
stores drop the folder before its parent-list member, so a failure return is
not proof the folder survived. Record the folder before the delete instead.
Restores now run shallowest first, so a folder taken by the parent cascade is
rebuilt with its own attributes before anything below it needs it as a parent.
Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r
* filer: recover a deleted folder from the create event for the entry that raced it
Checking each deleted folder on a timer was the wrong instrument. It cost a
listing per folder per pass, and it could only ever be a guess about when the
racing write would land.
The metadata stream already carries the answer. A folder is recorded before it
is deleted, so any entry that can be orphaned is created after that record and
its create event names that exact directory. Match the event against the
recently deleted folders and the folder is known to need putting back, rather
than inferred to.
The window stops being a guess at the race and becomes what it should be: how
far behind the event stream is allowed to run before a folder stops being
watched. Listing is now done once, for a folder an event has already named, to
skip the restore when the entry has since gone away again.
Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r
* filer: bound how long a folder is watched, and rebuild ancestors from themselves
Four gaps found reviewing the restore pass.
A folder whose restore kept failing was never let go: the written-to check ran
before the age check, so it was picked up, retried, put back, and counted again
on every pass for the life of the process. Apply the window first, whatever
state the folder is in.
At the cap, the folder being recorded was the one turned away, though it is the
one whose race is still live - the older entries are already close to ageing
out. Give up one of those instead, picked as the oldest of a small sample so
the cost stays flat under heavy deletion rates.
An ancestor taken by the same cascade was left to the descendant's restore to
recreate, which minted it from the descendant's attributes and handed back
access the ancestor never granted. Rebuild those from what they were, ahead of
anything below them.
Reading a directory's attributes assumed an entry came back. Some stores return
nothing with no error, so treat that as not found. The mode is also taken whole
rather than through Perm(), which was dropping setgid, setuid and sticky.
Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r
* redis3: take a directory listing left behind by a failed delete
Removing the last name deletes the list, and if that delete fails the header
survives pointing at a name that is gone. The retry finds nothing to remove,
reports no changes, and returns before reaching the delete, so the key stays
for good. Take it on that path too.
Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r
* filer: do not sweep children when deleting a folder non-recursively
doBatchDeleteFolderMetaAndData lists a folder and bails out if it has any
children, then calls Store.DeleteFolderChildren unconditionally. On the
non-recursive path that bulk sweep has nothing legitimate to remove: it only
runs once the listing came back empty, so the sole rows it can delete are
ones inserted after the check.
The S3 empty-folder cleaner deletes through this path, so a PUT landing
between the listing and the sweep loses its entry after the write was already
acknowledged. Neither side sees an error - the client has its 200 and the
cleaner logs an ordinary empty-folder deletion - and the chunks leak, since
the cleaner passes shouldDeleteChunks=false and nothing was enumerated to
collect. Workloads that scatter objects over many shallow prefixes empty and
refill those folders constantly, which is what makes the window reachable.
Sweep only when the delete is recursive, or when the whole-bucket shortcut
skipped the listing and depends on it.
Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r
* filer: pin the folder entry removal left by the racing-child test
The surviving entry is reachable by path but drops out of listings until the
folder comes back, and nothing in the test said so. Assert it, so the exposure
that remains after this change is visible rather than implied.
Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r
* iceberg: vend table-scoped credentials to clients that ask for delegation
The catalog recognised X-Iceberg-Access-Delegation: vended-credentials
and then deliberately said nothing, because it had nothing to vend: it
withheld even the S3 endpoint so the client would keep the credentials it
was configured with. That left every engine expecting the catalog to hand
out access - Snowflake, Databricks, Trino with vending, any multi-tenant
setup - needing static S3 keys distributed out of band.
Mint an STS session per request instead, scoped by a session policy to
the table's own prefix plus the bucket listing needed to resolve it, and
return it in the load response config and storage-credentials. The role
to assume is named by -s3.iceberg.credentialRole; its trust policy is
what decides whether a caller may assume it, and vending stays off until
it is set. A failed mint falls back to the old silence rather than
handing back an endpoint the client cannot sign for.
* iceberg: keep vended credentials inside the table prefix
Review follow-ups on credential vending:
Listing was granted on the bucket ARN with no condition, so a credential
vended for one table could enumerate every other table's object names.
Constrain s3:prefix to the table's own prefix, which the S3 gateway
already populates for list requests.
A table location carrying * or ? would have gone into the policy's
resource pattern unescaped and widened the session to sibling prefixes.
Refuse to vend for such a location rather than escaping it; nothing the
catalog generates contains those characters.
DurationSeconds skipped the 900..43200 bounds the other assume-role paths
enforce, so -s3.iceberg.credentialDurationSeconds could ask for a session
outside them. The check is now shared by all three entry points.
* iceberg: return the vended credentials from buildFileIOConfig itself
buildStorageConfig was a second name for what buildFileIOConfig already
did; it now returns the storage credentials alongside the properties, and
callers that only want the properties drop them.
* iceberg: split the vended bucket grants, and refuse a whole-bucket scope
The prefix condition sat on a statement that also granted
GetBucketLocation and ListBucketMultipartUploads, neither of which carries
an s3:prefix to satisfy it, so both were denied for every vended
credential. GetBucketLocation moves to its own unconditioned statement.
ListBucketMultipartUploads is dropped: Iceberg writers complete and abort
by upload id, and granting it either leaks in-flight keys bucket-wide or
breaks on the same missing prefix.
A table whose location has no prefix - one registered at the bucket root -
would have been vended read and write over every other table in the
bucket. Refuse, the way a location with wildcards is refused.
* iceberg: add view rename, scan-report and snapshots=refs to the catalog
Three gaps against the REST spec that clients hit in normal use:
Views had no rename, though tables did and views are stored the same way,
so the move is the same catalog-only pointer move. Tables and views share
a namespace directory, so both renames now refuse the other kind instead
of moving it.
Engines POST a scan or commit report after planning; a 404 there turns
into an error line per query. Accept the report and discard it - the
catalog keeps no metrics store.
LoadTable ignored ?snapshots=refs and always returned the whole snapshot
history, which is what clients use the parameter to avoid on long-lived
tables.
* iceberg: authorize view rename against the view ARN, tighten the metrics endpoint
Review follow-ups:
The shared rename checked the source against a table ARN whatever the
kind, so a policy scoped to a view's own ARN never matched and one
written for a table ARN was evaluated for a view. The entry kind now
carries the ARN builder.
The metrics endpoint truncated a report at 1 MiB and then failed to parse
it, answering 400 for a query that had actually succeeded. Read one byte
past the limit to tell "fits" from "cut short", and discard an oversized
report instead of rejecting it. Empty bodies and reports without a
report-type are now rejected, which the REST schema requires.
?snapshots= is defined for LoadTable, so it no longer filters what
CreateTable echoes back.
* iceberg: make a table commit a compare-and-swap
The catalog validated the caller's version token, ran its authorization
checks, and only then wrote the new metadata xattr. Two engines
committing against the same base both passed that check and both wrote,
so the second silently dropped the first one's snapshot. Both also derive
the same v{N}.metadata.json name and the file write overwrote, leaving
the surviving pointer aimed at the loser's metadata - and the loser's
conflict cleanup then deleted the winner's file.
Write the metadata file with an exclusive create and update the xattr
conditionally on the bytes the handler read, the way the maintenance
worker already commits. A writer that lost the race re-reads and retries,
and reports 409 CommitFailedException once out of attempts.
* iceberg: stage a commit under a unique name when the versioned one is taken
Two follow-ups from review of the commit compare-and-swap:
Refusing to overwrite v{N}.metadata.json also refused to get past a file
left behind by a commit that died between staging and updating the
pointer. Every later commit derived the same name, saw the collision, and
reported a conflict, so the table stayed uncommittable until an orphan
sweep removed the file. Stage under v{N}-{uuid} instead: neither writer's
file is overwritten and the catalog pointer still decides who won, which
is how the maintenance worker has always staged its own metadata.
metadataVersionFromLocation learned to read the version back out of that
name.
The conditional update guarded only the metadata attribute while the
write replaced the whole entry, so a policy or tag written in the same
window was silently reverted. Guard every catalog attribute, which turns
that into a conflict the caller retries on fresh state.
* iceberg: give saveMetadataFile the exclusive flag instead of a second name
saveNewMetadataFile, saveMetadataBlobExclusive and uniqueMetadataFileName
were three new names around one existing helper. The flag now rides on
saveMetadataFile and saveMetadataBlob, and the unique-name construction
sits where it is used.
* iceberg: reuse the filer CAS helpers #10773 added, and stage transactions exclusively
#10773 landed mutateEntryExtended, which already writes an entry back under a
whole-entry precondition and retries. Drop the helper this branch added and
route the table commit through it: the check that the metadata is still the
one this request read now lives in the mutation, where it sees current state.
The policy the request was authorized against is asserted too, so an
administrator restricting it mid-commit sends the caller back through
authorization instead of having a stale decision applied. Bucket and
namespace policies live on other entries and a single-entry precondition
cannot cover them.
Multi-table transactions stage their metadata exclusively for the same
reason single-table commits do, and carry the name they landed on into the
pointer flip.
#10774 gave buildTestMetadata its refs and age parameters while #10773
added a caller with the old arity. Each was green against a master that
did not yet have the other, and the merge of both does not compile, so
vet and the unit tests fail on master.
* iceberg maintenance: keep the snapshots that branches and tags pin
expireSnapshots only ever protected the current snapshot, so a snapshot
held by a tag or a non-main branch was expired once it aged out of the
retention window. iceberg-go's RemoveSnapshots drops any ref whose
snapshot is gone without complaint, so the tag disappeared and the files
behind it were deleted as unreferenced.
Protect every ref target, and honour a branch's own
min-snapshots-to-keep / max-snapshot-age-ms over the ancestors behind its
head. Detection skips pinned snapshots for the same reason: proposing a
job whose only outcome is a no-op keeps the worker busy forever.
* iceberg maintenance: re-plan when a ref appears mid-commit, and stop proposing no-op expiry
Three follow-ups from review of the ref-aware expiry:
The commit guard only compared the table head, so a tag created between
planning and commit could pin a snapshot the plan was about to expire.
Re-check the refs against the metadata the commit actually reads.
Detection now asks snapshotsToExpire what execution would remove instead
of approximating with its own count-and-age rules. Expiry always requires
a snapshot past the retention window, so a table over the quota whose
snapshots are all young was being proposed for a job that could only
no-op.
The branch retention test could not tell "retained the whole lineage"
from "honoured min-snapshots-to-keep", because the branch had exactly as
many ancestors as the count. Give it one more, and cover
max-snapshot-age-ms too. Both need snapshots genuinely older than a
retention window, which iceberg-go will not accept at build time, so the
fixture backdates the metadata after building it.
* iceberg maintenance: fold the metadata test builders back into one
buildTestMetadata, buildTestMetadataWithRefs, buildTestMetadataAged and
buildTestMetadataNow were four names for one thing. Keep the original and
give it the refs and age it needs.
* s3tables: add the maintenance configuration APIs
Stores the configuration verbatim as the wire shape under a new
s3tables.maintenance extended attribute, so Get hands back what Put took
and no translation layer can drift from the AWS model.
Nothing reads the configuration yet.
Put merges a single type into the stored map so configuring compaction
does not drop snapshot management, and asserts the attribute's prior value
so two concurrent Puts cannot silently clobber each other.
* iceberg: apply the maintenance configuration in the worker
The worker now reads the per-table and per-bucket maintenance
configuration written by the control plane, so the wildcard plugin config
is a default rather than the only setting a table can have.
Table properties still win by default, since a table declaring its own
layout is what every engine honours and the compactor has to agree with
whoever writes the files. Clearing table_properties_override makes the
maintenance configuration authoritative instead.
Status is not part of that contest: a disabled type drops its operations
and no property can re-enable them, so the operator's kill switch always
holds. Manifest and delete-file rewrites have no AWS equivalent and ride
with compaction.
Detection reads both attributes from entries it already lists.
* s3tables: report maintenance job status
The worker records the outcome of each run in its own extended attribute,
separate from the configuration so operator and worker writes do not
contend, and GetTableMaintenanceJobStatus reads it back.
Only the types a run touched are written, so a partial run cannot erase
what an earlier one recorded. The reader fills in the rest: Disabled when
the configuration switched a type off, Not_Yet_Run otherwise.
Status is advisory, so a lost race is logged rather than failing a job
whose work already committed.
* s3tables: route the maintenance APIs over REST
The five actions were only reachable by X-Amz-Target dispatch, which the
AWS CLI and SDK do not use for this service. They address the operations
by path, so the APIs were unreachable from any official client.
* s3tables: fix the table bucket ARN field name
GetTableBucketMaintenanceConfiguration emitted tableBucketArn where the
wire field is tableBucketARN, as every other response in this package
already spells it. Official SDK deserializers ignore the unknown key, so
the required field came back unset.
* s3tables: carry the compaction strategy through to the worker
IcebergCompactionSettings modelled only targetFileSizeMB, so a request
naming a strategy was accepted and then dropped on the way to storage.
The worker now maps binpack and sort onto its own rewrite strategy and
lets auto defer to the worker configuration.
z-order is rejected rather than accepted and quietly binpacked.
* s3tables: report bucket-level maintenance status
GetTableMaintenanceJobStatus read only the table's configuration, so
unreferenced file removal — which is configured on the bucket — reported
Not_Yet_Run or a stale success after an operator disabled it.
The merge helper now lives in this package and the worker shares it.
* iceberg: delete orphans only after the non-current window
AWS marks a file non-current once it has been unreferenced for
unreferencedDays, then deletes it a further nonCurrentDays later.
The cutoff was taken from unreferencedDays alone, so a 3/10 configuration
hard-deleted on day three and threw away the ten day recovery window.
remove_orphans deletes in one step rather than marking, so the cutoff is
now the sum of the two.
* s3tables: assert every attribute when rewriting an entry
UpdateEntry writes the whole entry back from the snapshot the caller
read, and its precondition only covers the keys the caller names. Both
maintenance writers named one key, so a job status write could revert a
maintenance configuration an operator had just disabled, turning an
advisory write into a silent re-enable.
Both now assert the entry's full attribute set, including the target key
when absent so a concurrent create also fails the precondition.
* s3tables: assert absent attributes when rewriting an entry
The precondition covered the attributes present when the writer read the
entry, so an attribute created between that read and the write was absent
from it. A first-time PutTableMaintenanceConfiguration disabling a type
therefore lands, passes the per-key checks, and is then deleted by the
stale whole-entry write.
Every attribute this package stores is now asserted, absent ones
included. The metadata commit and planning index writers rewrite the same
entries and had the same exposure, so both use the shared snapshot too.
* iceberg: implement the auto compaction strategy
auto was accepted, stored and read back, but left the worker on its own
default, so a sorted table configured as auto was compacted with binpack.
AWS defines auto as sorting tables that declare a sort order and
bin-packing the rest. That needs the table metadata, so the choice is made
where the rewrite plan is resolved: an unsorted table falls back to
binpack rather than failing the way an explicit sort request does.
* s3tables: validate the maintenance setting ranges
PUT accepted zero, negative and oversized values for every numeric
setting. The worker then ignores a non-positive value and saturates an
oversized one, so the configuration read back was not the one that ran.
AWS bounds all five to 1..2147483647, which is now enforced. The fields
are pointers so an explicit zero is distinguishable from an omitted one
and can be rejected rather than silently ignored.
* s3tables: give every entry writer the same compare-and-swap
updateExtendedAttribute asserted the entry's attributes, but the helpers
behind the metadata, policy and tag handlers still wrote the whole entry
unconditionally. Any of them could land on a stale snapshot and delete a
maintenance configuration an operator had just written.
They all share one read-modify-write loop now, so the precondition and
the bounded retry apply wherever an entry is rewritten.
* s3tables: move the maintenance configuration with a renamed table
RenameTable carried the metadata, version, policy and tags to the new
name but left the maintenance configuration and job status behind. A
table with snapshot management disabled came back enabled under its new
name, and the stale configuration stayed on the old name where a table
created there would inherit it.
The decoupled-delete cleanup left the same two attributes behind.
* s3tables: accept every AWS partition in ARNs
The route regexes and the ARN patterns both hardcoded arn:aws, so valid
aws-cn and aws-us-gov ARNs never reached a handler. The router now shares
the partition-tolerant prefix with the parser, and a generated ARN uses
the partition its region belongs to so it parses back.
* s3tables: generate ARNs in the region's partition
The handler's own ARN generators still formatted arn:aws directly rather
than going through the partition-aware builder, so a China or GovCloud
deployment routed the request but then returned a commercial ARN and
matched IAM policies against it.
The round-trip test missed this because parsing accepts any partition, so
it now asserts the prefix the region implies.
* s3tables: complete the ARN partition table
aws-iso-e, aws-iso-f and aws-eusc were missing, so eu-isoe-*, us-isof-*
and eusc-* regions fell through to the commercial partition.
* s3tables: do not let a rename swallow a concurrent maintenance write
Rename copied the source attributes early and cleared the source at the
end, so a Put landing in between missed the copy to the destination and
was then deleted by the cleanup. It succeeded and vanished.
The cleanup now clears the source only while it still holds exactly what
was copied, and returns a conflict otherwise. Put checks the catalog
identity inside the same conditional mutation, so it also cannot write to
a name that a rename or delete has already soft-deleted.
* iceberg: carry snapshot retention in milliseconds
Config stored retention as hours, so any sub-hour value would have to be
truncated to 0 and then clamped back up to the 168 hour default. Keep the
plugin config key in hours and convert once at parse time.
* iceberg: let table properties override the worker config
Every other Iceberg implementation lets a table's own properties win over
engine defaults; the worker ignored them entirely. A writer honouring
write.target-file-size-bytes and a compactor rewriting to the plugin
config's size would rewrite each other's output forever.
Resolved once per job rather than per operation, so compaction committing
new metadata mid-job cannot change the settings underneath it.
* iceberg: clamp the orphan cutoff so it cannot overflow
collectOrphanCandidates converts the cutoff to a time.Duration. Past
roughly 2.5 million hours that multiplication wraps negative, putting the
cutoff in the future so every file walked looks like an orphan and gets
deleted, including data a concurrent writer has not yet committed.
Reachable today through orphan_older_than_hours.
* ec.encode: require the shards to agree on size before deleting the volume
Before an encode deletes the volume it just encoded, it asks whether
enough shards exist and whether they are spread across nodes. Both are
questions about presence: nothing asks whether those shards are whole.
Every shard takes one piece of each block row, so they are all written to
the same length. One that disagrees was truncated, half copied, or landed
on a disk that filled up -- and counting cannot see it, so the source
volume is deleted on the strength of a set that cannot rebuild it.
Compare the sizes the cluster already reports (shard_sizes travels in the
heartbeat) and hold the deletion back when they disagree, naming the odd
shard and its holder. Sizes reported as zero are skipped rather than read
as a disagreement: a volume server that predates shard-size reporting, or
one that has not heartbeated them yet, must not strand every encode in
the volume-plus-shards state this check exists to avoid.
* ec.encode: judge shard sizes on the newest encode generation only
The size check collected every shard the master reports for the volume,
while the recoverability check beside it counts only the newest encode
generation. A re-encode can change the ratio, so an orphaned older
generation -- one the pre-encode sweep could not reach, but the master
still hears about -- has shards of a different length by nature. Merging
those into the comparison makes a healthy current set look inconsistent,
and because the orphan keeps being reported, every retry fails and the
encode is left holding the volume and its shards for good.
Collect sizes the way CollectEcShardBitsByNode collects bits: fenced to
the newest EncodeTsNs, with unstamped entries forming the one legacy
generation.
The harness kills shells mid-command, and the master releases the dead
session's lock only when it notices the connection is gone. That cleanup
lands after the harness has already re-acquired the lock, so it can clear
the lock this run holds and the next command refuses with
need to run "lock" first to continue
recoverInterruptedBalance answered that the way an operator would -- run
lock again and retry -- but the encode and decode recoveries called
shellCommand once and required success, so the same reap failed the run
outright. Move the retry into shellCommand: the reap can land during any
command that follows a kill, not only a balance.
A decode ends by deleting the shards it read, and the only thing standing
between that and a bad reconstruction is verifyDecodedVolumeBeforeDelete,
which asks whether .dat and .idx are non-empty. A .dat truncated to a
single byte passes, and the shards -- the only other copy of everything
past the cut -- are deleted on the strength of it.
The server already knows the answer it never checks: FindDatFileSize
returns the extent the EC index references, and WriteDatFile rebuilds to
it. Compare the two once the file is written and fail the decode instead
of reporting a short volume as a good one.
Longer than the extent still verifies -- padding is not missing data --
so only a genuinely short rebuild is rejected.
Needle counts cannot answer this: .idx is written from .ecx, so the count
matches by construction and a truncated .dat still reports every needle.
A decode deletes the shards only after the regenerated volume is mounted
and verified, so a run interrupted in that last phase leaves the volume
in place with its shards partway through deletion. The re-run then finds
both, tries to collect the shards again to rebuild a volume that already
exists, and fails on the first shard the interrupted run had removed:
generate normal volume 3 ...: ec volume 3 missing shard 6
Nothing recovers from there: the shard set is deliberately being
destroyed, so every retry fails the same way while the decoded volume
sits there, already complete.
Finish that cleanup instead. A volume beside the shards is not enough to
act on -- an encode interrupted before it deleted the original leaves the
same shape, as does a decode killed while generating, whose volume may be
half written -- so require a data shard to be gone. Only the deletion
phase removes one, and it is also exactly the state no decode can
recover from, so finishing is the only move left rather than a choice
between two. The deletion still runs behind
verifyDecodedVolumeBeforeDelete, the check that guards it in a normal
run.
* admin: show capacity per storage tier and stop counting remote-tiered bytes as local disk usage
A remote-tiered volume reports its cloud object's size, so summing volume
sizes inflated the dashboard's used-vs-capacity numbers (the local .dat is
gone after volume.tier.move). Split the accounting: DiskUsage now only
counts bytes on local disks, with the cloud bytes surfaced separately per
server and per remote storage name.
The dashboard gains a Storage Tiers table breaking volumes and EC shards
down by tier (each local disk type plus each remote storage), using the
per-disk-type statfs numbers already in the VolumeList response. The
volumes page badges remote-tiered volumes with their storage name, and
the EC shards page fills in real per-shard sizes instead of hardcoding 0.
* admin: review fixes for the tier capacity display
- A disk that predates disk_total_bytes now contributes its logical
bytes to the tier's DiskUsed, so a tier mixing old and new volume
servers doesn't underreport usage; the usage bar always reflects the
displayed Disk Used value (the DataSize fallback in UsagePercent is
gone, and the percent math is overflow-safe).
- getTopologyViaGRPC defaults a zero VolumeSizeLimitMb to 30000 MB like
GetClusterVolumeServers, keeping slot-based capacities consistent.
- The dashboard volume-servers column reads Usage / Capacity to match
its cell content, and the hdd disk-type default is shared between the
volumes-page badge and countUniqueDiskTypes.
* ec: bounded-exhaustive model check of the volume lifecycle
The randomized chaos harness samples the state space; this enumerates
it. The lifecycle is a state machine whose steps mirror the pipelines in
this package, and the checker explores every schedule within the bound:
a crash at every step boundary, an error return running the rollback
(itself crashable at every step), a volume-server restart applying the
startup reconciliation rules in every quiescent state, and the
prescribed restart-based recovery from every crashed state.
Checked in every reachable state: durability (a readable copy always
exists), at most one generation mounted, and — a property the sweep
discipline turns out to guarantee — at most one generation's files on
disk. From every quiescent state the recovery must converge to a clean
volume. Runs in well under a second.
* test: deterministic EC interruption matrix
Enumerate every phase of every interruptible EC operation and kill a
real weed shell exactly when the phase announces itself on the command
output, instead of at a random moment: four encode phases, four decode
phases, and the balance's move phase (set up with -rebalance=false so a
move is guaranteed). Each scenario prepares its precondition, kills at
the marker, runs the prescribed recovery, and verifies every stored byte
still reads back identical.
The interruption recoveries move out of the randomized ops into shared
chaosRun helpers both drivers use.
* test: make the randomized EC chaos walk opt-in
The systematic layers — the interruption matrix and the lifecycle model
check — carry the CI coverage deterministically; the randomized walk
stays for exploratory runs, behind EC_CHAOS_SEED.
* ci: bound the EC integration suite by the job budget, not go test's default
The suite with the interruption matrix runs close to the default 10m
binary timeout on slower runners.
* test: require every interruption-matrix marker to appear
A marker that never prints means a pipeline refactor renamed or dropped
the progress line; silently degenerating into a no-interruption run
would let CI pass without exercising the boundary the scenario names.
Also recheck the marker channel after the wait: a shell that prints and
exits at once makes both channels ready, and select picking the exit
case must not report a printed marker as missed.
* ec: let the encode's balance see a migrating volume's shards across disk-type buckets
Shard generation writes beside the source .dat, so a cross-tier encode
(source on hdd, -diskType=ssd) leaves the fresh shards in the source
disk-type bucket. The encode's internal balance ingested only the target
bucket, saw no shards, and planned no moves; the spread guard then
correctly aborted the encode (and before that guard existed, the shards
silently stayed clumped on the generation host in the wrong tier).
EcBalance now takes the encode batch as migratingVolumeIds and ingests
those volumes' shards from every bucket, while everything else keeps the
bucket filter so a plain ec.balance never drags deliberately tiered
shards onto another disk type. The in-memory model delete also becomes
bucket-agnostic: a node holds a given shard in exactly one bucket, and a
bucket-scoped delete missed cross-bucket moves in the dry-run model.
* volume: decode reads shard 0 from its resolved path, not the EC volume's base dir
On a multi-disk server a volume's shards can sit on several disks; the
store registers each shard with its own path and CollectEcShards resolves
them, but FindDatFileSize derived the .ec00 path from the EcVolume's base
directory. When shard 0 lived on a sibling disk, VolumeEcShardsToVolume
failed with 'open ...ec00: no such file or directory' and ec.decode
aborted.
* ec: decode re-copies shards the topology claims but the target does not hold
An interrupted earlier decode or balance can leave the master believing
the decode target holds a shard whose file never landed: the mount
registered but the partial copy was cleaned, or the file was swept. The
collect step took the topology's word for it, excluded the shard from
the copy set, and the decode failed with 'missing shard'. Probe the
target's live inventory (VolumeEcShardsInfo) and treat anything it
cannot serve as still-to-copy.
* ec: decode discovers shards across disk-type buckets
Shards sit wherever encode generation and balance left them: a
cross-tier encode leaves them in the source disk-type bucket, a partial
migration straddles buckets. ec.decode scoped its shard discovery to the
-diskType bucket and reported a decodable volume as having no shards at
all. Union across buckets, the way the encode's shard verification
already does.
* test: EC chaos lifecycle harness
Randomized, seeded sequences of the EC lifecycle against a live cluster
in the production-shaped layout: multiple data disks per server, a
separate -dir.idx directory so .ecx/.ecj sidecars are shared across
disks, and a tagged ssd tier. Operations cover encode (hdd and ssd
targets), balance, shard damage plus rebuild, decode, re-encode,
deletes, scrub, tier moves, crash-restarts, sidecar fault injections
(a data-dir .vif pushed into the shared idx dir; a stale-generation
shard planted beside a newer encode), and interruptions: a real weed
shell subprocess killed mid-encode, mid-decode, and mid-balance, with
the recovery re-run required to converge.
One invariant holds after every step: every stored byte reads back
identical and every deleted needle stays deleted. EC_CHAOS_SEED and
EC_CHAOS_STEPS make runs reproducible and scalable.
A known gap is tolerated and logged rather than fixed here: a shard
mounted on two disks of one node (orphan adoption after an interrupted
copy) is invisible to ec.balance's dedup and unaddressable by
ec.shard.unmount's shard@address form, so no cleanup path exists yet.
* test: fail payload-corruption checks on the test goroutine
t.Fatalf inside require.Eventually's condition runs on the poller's
goroutine, where Goexit kills only that goroutine and the corruption
message can be lost behind a generic timeout. Record the mismatch, end
the polling, and fail on the test goroutine. Also assert the full shard
count in the cross-bucket decode-discovery test.
* shell: move ErrorWaitGroup to weed/util
* shell: remove unused CandidateEcNode and EcRack types
* ec: extract EC orchestration logic from weed/shell into weed/ec
Move the EC node/topology model, balance engine, encode pipeline, decode
pipeline, and rebuild engine into a new weed/ec package so shell commands
and maintenance workers can share the logic. Shell commands keep flag
parsing and delegate through a small ec.Env (dial option, topology fetch,
volume locations, lock check). Tests move along with the code.
* shell: remove unused proportional-rebalance type stubs
* ec: move scrub, replication check, and shard unmount engines into weed/ec
* worker: share the EC generation-aware shard counter from weed/ec
* ec: gofmt
* shell: drop EC aliases with no remaining callers
* ec: guard a missing topology hook and nil disk entries in topology helpers
* ec: drop trailing newlines from decode error strings
* ec: re-check the shell lock before applying shard unmounts
* shell: trim -node entries in ec.scrub
* s3: add option to disable bucket auto-creation on upload
* command: expose -autoCreateBucket in s3, filer, server, and mini
* s3: apply the bucket auto-create policy to directory marker uploads
* s3: validate the bucket name before the auto-create disabled check
* s3: cover the disabled auto-create gate at all three upload entry points
sftp: url-encode the upload path so filenames can't inject filer query commands
The SFTP put handler concatenated the user-controlled filename straight into
the filer upload URL, so a name containing "?" was parsed as a query string.
Build the URL via url.URL{Path: ...} so "?" becomes %3F and stays a literal
path character.
* volume: validate replica upload targets in FetchAndWriteNeedle
The replica leg forwarded the fetched needle to a caller-supplied address
without checking it, so a malformed target could redirect the upload to an
unintended host or path. Require each replica target to be a bare host:port
whose host is not loopback / link-local / unspecified, reusing the address
deny-list; cluster peers legitimately sit on private networks, so RFC 1918 /
CGNAT stay allowed and -volume.allowUntrustedRemoteEndpoints still opts out.
Validate every target up front so a bad one fails the request before the local
write, and upload through a client that re-checks the resolved address at
connect time so a replica hostname cannot rebind to a blocked address after
validation. Mirrored in Rust (validation moved ahead of the local write; the
Rust S3 path's connect-time re-check is still a follow-up there).
* volume: only accept inline gcs credentials in FetchAndWriteNeedle
The gcs credentials value on this request could name a local filesystem path,
which the SDK reads from disk. Accept only inline JSON here; the server-side
GOOGLE_APPLICATION_CREDENTIALS env var still supplies a path. The Rust volume
server has no gcs backend, so there is nothing to mirror.
* remote_storage/azure: allow a per-request HTTP client
Thread an optional *http.Client through NewAzBlobClient and add
azure.MakeWithHTTPClient, mirroring the S3 backend. When set, the client
overrides the azblob transport so a caller can pin the dial path. The
existing makers pass nil, so behavior is unchanged.
* volume: extend the remote-endpoint guard to the azure backend
The endpoint validation and rebinding-safe dialer in FetchAndWriteNeedle
covered the S3-SDK backends. The azure backend also dials a caller-supplied
AzureEndpoint, so route both families through a single guardedRemoteClient
helper that returns the endpoint each backend dials and a constructor bound
to the guarded HTTP client. azure is guarded only when AzureEndpoint is set;
an empty endpoint derives the public host from the account.
-volume.allowUntrustedRemoteEndpoints still opts out.
* rust volume: assert the azure endpoint has no remote-client path
The Rust volume server has no azure backend, so make_remote_storage_client
rejects the type before any client is built. Add a regression test pinning
that invariant.
* volume_move: treat zero-sized EC shards as absent in move verification
A zero-sized shard file is residue of a failed operation (issue 10730),
not a shard - but VerifyEcShards only checked presence, so a copy that
landed as an empty file passed verification and the source was deleted
behind it. Size zero now reads as absent, with a distinct error naming
the zero-sized shard so the operator can tell a broken copy from a
missing one.
* storage: exclude zero-sized EC shards from rebuilds and clean up stale ones
The reproducer in issue 10730: a zero-sized shard file left by a failed
operation was selected as a Reed-Solomon input and failed the whole
rebuild with an input size mismatch, because input discovery checked
existence, not substance.
- RebuildEcFiles treats a zero-sized shard file as missing and
regenerates over it in place (the reclassified-corrupt path: temp
file beside the residue, atomic rename).
- The startup/rescan shard loader, which always skipped zero-sized
files, now deletes them once they are older than an hour - young
enough files can be an in-flight copy's just-created file, since the
same scan runs from LoadNewVolumes while serving.
Regression tests: a rebuild with one emptied shard regenerates it
byte-identical; the loader deletes a stale zero-sized shard and leaves
a fresh one alone.
* storage: age-check each zero-shard cleanup candidate individually
The shard scan merges the data and idx directory listings, so the
age-checked entry and a deletion candidate can be different files
sharing one name - a stale zero-sized file in one directory next to a
fresh same-named file in the other (possibly an in-flight copy's
just-created one) could get the fresh file deleted. Each candidate's
own modification time now decides, both directories are handled in one
pass, and the split-directory case is pinned by a test.
* redis2: route the orphan cleanup existence checks to the master
* scaffold: the redis_cluster2 read routing key is useReadOnly
* ci: run the gated redis store tests
* redis2: poll for the redis expiry instead of a fixed sleep
* redis2: assert the value key exists before testing its expiry
The Go volume server has VolumeConsolidateIndex, which moves a volume's
.idx out of the data directory into the configured -dir.idx directory
(where an EC decode/reconstruct can leave it co-located) and reloads the
volume in place. The Rust port's proto omitted the RPC entirely, so its
generated VolumeServer trait was one method short of Go's.
Add the proto message and rpc, the gated grpc handler, and
Store::consolidate_volume_index / Volume::relocate_index_to, mirroring
Go's Store.ConsolidateVolumeIndex and Volume.RelocateIndexTo -- including
the cross-device copy fallback and the reopen-against-the-old-dir path
when the move fails.
Integration tests cover the real move (index relocated, volume still
serves reads and the move is idempotent), the no-op paths (index already
in place, no separate idx dir) and the not-found error, plus the grpc
handler end to end.
A bounded LoopProcessLogData (stopTsNs set) on a buffer that never took a
write since process start fell into the ResumeFromDiskError branch, which
never checks stopTsNs when ReadFromDiskFn is nil and HasData() is false.
The read parked on the notification loop forever while the subscription's
idle heartbeats kept the stream looking alive, so a bounded
SubscribeMetadata pass on a freshly restarted idle filer never completed.
Terminate like the caught-up path does, returning a nil error: leaking
the pending ResumeFromDiskError would latch the filer's outer loop into
its gap machinery, which parks the bounded subscriber all over again.
* feat: throughput limits for replicate, EC shard, and worker-driven moves
VolumeCopy was the only rate-limitable transfer; EC shard copies,
replica creation, and worker-driven moves all ran at whatever the
receiving server's maintenance rate allowed, with no per-operation
control.
- proto: VolumeEcShardsCopyRequest and the balance / ec_balance task
params and configs gain io_byte_per_second; 0 keeps today's behavior
(the volume server's own maintenance rate governs).
- volume server: VolumeEcShardsCopy throttles with one WriteThrottler
per request, shared across the shard, .ecx, .ecj, .vif, and .ecsum
copies so the limit caps the transfer as a whole - the same shape as
VolumeCopy.
- volume_move: ReplicateVolume accepts the limit; EcMoveOptions carries
it through MoveEcShards/CopyAndMountEcShards into the copy request,
with fake-client tests asserting propagation.
- shell: ec.balance gains -ioBytePerSecond; volume.tier.move's
replication top-up honors the command's existing -ioBytePerSecond
instead of running unthrottled.
- worker: balance and ec_balance configs gain io_byte_per_second
(surfaced in the admin config schema), carried through detection and
plugin job parameters into task params and handed to the shared
mover; batch balance jobs inherit the limit from their detection
results.
The limit is per copy stream, so maxParallelization multiplies the
aggregate ceiling.
* worker plugins: expose io_byte_per_second in the plugin config and derive it
The plugin-driven detection path derives its task Config from the
plugin configuration values, and both balance and ec_balance left
IoBytePerSecond at zero there - a configured limit silently reverted
to the server maintenance rate. Both derive functions now read the
field (clamped at zero), and the plugin descriptors expose it with
defaults so the configuration form carries it.
rust volume: gate the remaining admin RPCs behind check_grpc_admin_auth
The Go volume server gates 29 destructive VolumeServer RPCs on the
-whiteList admin check; the Rust port only gated 14. Add the gate to the
other 15 -- batch_delete, read_all_needles, fetch_and_write_needle, the
EC-shard generate/rebuild/copy/unmount/to-volume RPCs, both tier-move RPCs,
volume_copy, volume_tail_receiver, set_state, scrub_ec_volume and
volume_needle_status -- so a configured whitelist restricts them the same
way it already does on the Go side.
check_grpc_admin_auth also required peer info before checking whether any
control was configured, unlike Go's `if vs.guard == nil { return nil }`.
Short-circuit when no whitelist and no signing key are set, so in-process
callers keep working with security inactive and only the gate ordering
changes for configured servers.
tests/admin_auth_coverage.rs mirrors the Go coverage test: every handler
must either gate or be listed as intentionally open with a reason, so the
two implementations can't silently drift apart again.
* redis2: derive the orphan cleanup keys inside the helper
* redis2: skip orphan cleanup in super large directories
* redis2: detach orphan cleanup from the request context and log a failed restore
* redis2: keep a directory member whose child index is still live
* redis2: run restore-path tests under both key prefixes and fix the test harness
* redis2: check cleanup errors in tests
* fix(redis2): remove orphaned directory index members on listing
ListDirectoryEntries skipped index members whose value key was gone and
left them in the ZSET, so the per-directory child index grew without
bound under any TTL workload. Mirror the ZRem the logical-expiry branch
already performs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(redis2): keep the index member when a concurrent insert recreates the value
The orphan cleanup removed the member unconditionally, so an InsertEntry
landing between FindEntry and the ZRem left a live value with no index
member, invisible to listings until another InsertEntry on that path.
UpdateEntry does not re-add it, so the loss persisted.
Restore the member when the value is present again after the removal.
The value key and the directory index key hash to different slots, so a
Lua script or MULTI over both is not available to the cluster store.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The unmount+full-teardown of EC shards was duplicated: the plugin-worker EC
task had unmountAndDeleteEcShards and the shell had unmountAndDeleteEcShardsQuiet,
byte-identical apart from a fence parameter and a sentinel error. That
duplication is how the teardown fence semantics drifted between the two paths.
Distribute, mount and verify already live in weed/storage/erasure_coding and are
shared by both callers; move the teardown there too, as UnmountAndDeleteEcShards
plus the shared ErrFullTeardownNotAcked sentinel. Both paths now call the one
function, so the fence semantics cannot diverge again. The shell keeps a thin
type-converting wrapper and aliases the sentinel; behavior is unchanged.
* EC worker: clear stale/interrupted shards at task start and on failure
The EC encode task cleared stale shards from a prior interrupted encode only
at 55% progress (after mark-readonly, copy, and generate), and used a
generation-fenced teardown. Two gaps left orphan shards behind:
- a retried encode's prior attempt carries the same admin-issued encodeTsNs,
and the server's teardown fence preserves same-or-newer generations, so the
prior attempt's shards were never cleared;
- shards left by an interrupted distribute often have an unreadable .vif
generation (the sidecar never landed), which the fence also preserves.
Both survive the next volume-server restart as orphans and make detection
refuse the volume (Manual intervention required).
Move the cleanup to a Step 0 preflight that runs before any destructive step,
and switch it to the server's blanket (generation-independent) teardown -- the
same wipe the shell ec.encode pre-cleanup uses. The admin dedupe key already
prevents a concurrent newer encode of the volume, and the blanket path aborts
rather than clobber a live newer mount.
Add rollbackDistribute: a failure after distribute begins but before verify
commits the EC copy now tears down the shards it wrote and restores the sources
to writable, so a terminally-failed encode (a single-attempt job, or the last
of a retry series, which has no successor preflight) leaves nothing behind.
The preflight also rejects a plan with no targets or no source before marking
the source readonly.
* EC worker: reject malformed targets and keep source readonly on incomplete teardown
Address review feedback:
- ensureCleanEcStart only rejected an empty target slice; a target with an
empty Node (or no shard ids) passed the length check, was then silently
skipped by cleanupStaleEcShards, and let Execute mark the source readonly
with nothing to distribute to. Validate each target before the first
destructive step. Add regression cases.
- rollbackDistribute marked the source writable even when the shard teardown
returned an error, exposing a writable source beside stale (possibly mounted)
shards -- reads/writes could diverge and orphan cleanup will not remove a
writable source. On an incomplete teardown, leave the source readonly for the
next preflight or an operator to reconcile.
Chris Lu
Chris LuandGitHub
github-actions[bot]
