* s3api/iceberg: report the reason a table schema was rejected
newTableMetadata swallowed the iceberg-go error and returned nil, so every
schema the metadata builder refused came back as a bare 500 "Failed to build
table metadata". A v3-only column type is the common case: creating a table
with a variant field but no format-version 3 property leaves the client with
nothing, while "variant is not supported until v3" sits in the server log.
Return the error instead and classify it. Schema, spec and argument failures
are the caller's input, so they answer 400 with the underlying reason; the
rest stay 500. Paths that build placeholder metadata with no schema keep
their existing 500 via newEmptyTableMetadata.
* s3api/iceberg: fail LoadTable when placeholder metadata cannot be built
buildLoadTableResult dropped a nil from the placeholder path straight into
the response. That serializes as "metadata":null under HTTP 200, which no
Iceberg client can parse -- a worse outcome than the 500 the nil was meant
to signal.
Return an error instead and let the five callers answer 500. The nil-return
convention goes away with it, so the commit and transaction paths check an
error rather than a sentinel.
* s3api/iceberg: route rejected schemas through writeManagerError
The two helpers added here duplicated work the package already does.
writeManagerError is the canonical error-to-response mapper -- it already
downgrades client-input failures to 400 and defaults the rest to 500 -- so
teach it the iceberg-go schema and spec sentinels instead of standing up a
parallel classifier. The placeholder wrapper was a pure alias for
newTableMetadata with nil arguments; call that directly.
No behavior change beyond the 500 message, which now reads err.Error()
like every other manager error rather than carrying its own prefix.
curl (unlike wget --spider) exits 0 as soon as it gets any HTTP response, even 401/403 - required if the filer rejects unauthenticated requests when JWT auth is enabled
The staged-new-volume placement skipped a disk holding the vid's EC shards using only the in-memory ecVolumes map, missing a shard present on disk but not mounted. Also scan the candidate disk for <vid>.ecNN files, so the promise holds regardless of mount state.
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
volume: skip a shard-holding disk when staging a decoded volume
ReceiveFile staged-new-volume mode picked any free disk of the target
medium. Skip a disk that already holds the vid's EC shards (Go
DiskLocation.FindEcVolume / Rust ec_volumes), so a decoded .dat never
lands in the same directory as a shard. This lets a caller safely stage
onto a shard host that has a spare disk, instead of requiring a host with
no shard of the vid at all.
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
Decoding EC shards back to a normal volume in place reconstructs <vid>.dat
in the shards' own directory, so the vid is momentarily registered as both
an EC and a normal volume in one location — the load/scan path then sees it
as both, risking mount ambiguity and needle loss. VolumeEcShardsToVolume
still supports that in-place path; this adds the primitives to decode onto
a *clean* peer instead:
- ReceiveFile gains a staged-new-volume mode: when the volume does not
exist here and ReceiveFileInfo.disk_type is set, pick a free-slot disk
of that medium and write <base><ext>.copying (not a valid volume name,
so the scanner never half-loads a partial push).
- VolumeEcShardsToVolume gains from_staged: adopt the pushed .dat/.idx/
.vif — rename .copying into place under a .note in-progress marker,
then mount — so <vid> lands on the peer only as a normal volume.
The caller decodes the shards off-box and streams the finished volume to a
peer holding no shard of the vid on the target medium. Go and Rust volume
servers get identical handlers. Proto: ReceiveFileInfo.disk_type (12; 8-11
reserved for versioned-EC), VolumeEcShardsToVolumeRequest.from_staged (3) +
disk_type (4).
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
VolumeMarkReadonly mutates raft-replicated master topology, so it must
reach the leader. notify_master_volume_readonly targeted the static seed
(config.masters.first()), so after any master failover it hit a follower
and failed "not current leader". Prefer current_master_url (the live
leader the heartbeat tracks), fall back to the seed before the first
heartbeat, mirroring store_ec.rs and Go's vs.GetMaster().
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
* s3: list the buckets an attached IAM policy grants
ListBuckets served an identity authorized by an attached IAM policy only
the buckets it had created itself. A user granted s3:ListBucket on a
bucket someone else provisioned could GetObject and ListObjectsV2 against
it, but the bucket never showed up in the listing any S3 client uses to
build its bucket picker.
The owner-index fast path is only valid for an identity whose grants name
every bucket it can reach, and the routing check assumed a policy could
never be enumerated. Read the names out of the policy instead: statements
that allow s3:ListBucket on a concrete bucket ARN become candidates, and
the per-bucket permission re-check still decides what is listed. A policy
that can reach a bucket it does not name -- a wildcard resource, a policy
variable, a NotResource, an STS session policy -- falls back to the full
scan, which evaluates the policy per bucket.
* s3: share the attached policy name lookup
authorizeWithIAM and the ListBuckets enumeration both built an identity's
policy names the same way, its own plus the ones from its enabled groups.
Pull that into one helper so group eligibility is decided in a single place.
* s3: read policy actions the way the IAM authorizer matches them
The IAM authorizer matches action names case-insensitively, so a policy
granting "S3:LISTBUCKET" or "S3:*" authorizes a list. The ListBuckets
classifier read those actions with the local case-sensitive matcher and
found no grant, so once the owner index was ready the buckets that policy
allows dropped out of the listing.
Match the action the looser way in the classifier: case-insensitive, and
true for any pattern holding a policy variable. Over-matching only costs a
candidate the per-bucket permission check then rejects, while under-matching
hides a bucket the caller can read.
* s3: infer a multipart grant in any case
The classifier matches action patterns case-insensitively but looked the
requested action up in a canonical-cased set, so "S3:UPLOADPART" missed the
s3:PutObject inference that the authorizer makes. Key the set for lookup in
lower case, matching how the IAM authorizer holds it.
* azure: let the blob endpoint be configured
The service url was always derived as <account>.blob.core.windows.net,
which leaves out Azure Government, Azure China, and private endpoints.
Name the blob service url instead and those accounts become reachable.
The url has to be https, since the account key or the bearer token would
otherwise travel in the clear.
* azure: reject an endpoint that carries no hostname
A url like https://:443/ has a host of ":443", so the emptiness check on
Host let it through and the request only failed once it reached Azure.
The hostname is what has to be there.
df on a mount shows the space the cluster gives up to the data: every
replica of a regular volume, every shard of an ec one. That is the honest
answer for capacity planning, but it is not the question a user asks when
they want to know how much of their data is stored.
Add -df.logical. The master reports the logical sizes alongside the raw
ones: one replica per regular volume, the data shards of each ec volume
counted once. Free space is divided by the copies the requested
replication makes, so used plus available stays the amount of data the
mount can still write, and it comes off the cluster-wide usage rather
than one collection's, since capacity is cluster-wide too.
Statistics through a filer resolves an unset replication to the filer's
default rather than the master's, matching where the writes it is sizing
for actually land.
The flag governs the quota check too, so a mount has one notion of how
much it is using. A filer that predates the new fields sends zeros, and
the mount keeps reporting the raw sizes.
Statistics aggregates the volume layouts of a collection, but EC volumes
are tracked outside collectionMap, so they were reported as nothing. A
mount over a cluster whose volumes have mostly been encoded showed a df
used size of a few GiB against terabytes of EC data.
Walk the data nodes and add the EC volumes of the requested collection.
Every shard copy counts, parity included, the way a regular volume's used
size counts every replica, so used size stays the space the cluster
actually occupies.
File count comes from the volume-wide .ecx and .ecj counts, taking the
largest a holder reports rather than summing them: both files travel with
the shards on a move, so several nodes can report the same tombstones.
* azure: authenticate the blob sink with Entra ID
Shared account keys have to be distributed and rotated everywhere a sink
runs. Leaving account_key empty now falls back to the identity chain, so a
workload identity or managed identity carries the authorization instead.
* azure: authenticate remote storage with Entra ID
The remote storage client demanded an account key and refused to start
without one. Fall back to the identity chain when it is absent, and let
azure.client_id pin a user-assigned identity.
* azure: reject a malformed storage account name
The account name is interpolated into the service URL, so a name carrying
a "/", "?" or "@" moves the authority elsewhere and an authenticated
request follows it. Hold callers to Azure's own naming rule instead.
* azure: keep a leftover environment key off the identity path
A configured client id asks for Entra ID, but AZURE_STORAGE_ACCESS_KEY
still filled in the account key behind it. An old mounted secret would go
on authenticating until it rotated, and the failure then blamed the key.
* azure: say what the identity path reads from the environment
A pinned client id alone is not enough for workload identity: the tenant
and the projected token come from the environment, and missing them only
surfaces later, when a token is first requested.
* s3: apply filer identity changes despite a static config file
A -config file with inline identities disabled the metadata-subscription
reload entirely, leaving the best-effort filer->s3 push as the only way
s3.configure changes could reach a running gateway. Reload on IAM events
regardless: the merge keeps the file's identities protected, and a full
credential-manager snapshot now also drops dynamic identities the store
no longer has, so revocation works without a restart.
* s3: log identity propagation failures as warnings
* s3: retry failed IAM reloads and reconcile policies and groups
An event-driven reload that fails now hands off to a coalescing retry
loop, so a transient filer error cannot strand a revoked credential
until the next IAM event. Full-state merges also drop dynamic policies
the store no longer has, keeping the static file's, and treat the group
snapshot as authoritative even when empty.
* s3: serialize IAM configuration loads
The SIGHUP file reload, subscription reloads, the retry loop, and the
postgres poll run on different goroutines. Without an end-to-end lock a
load holding an older store snapshot can commit after a newer one and
revert it. Hold reloadMu from snapshot through commit in both load
entry points; partial merges from pushed updates stay lock-free and
self-heal through the next event-driven reload.
* s3: keep static-file groups through full-state reconciliation
Group names from the static config file are tracked like identities and
policies, and a full snapshot that does not carry them keeps the current
definition and its memberships instead of dropping them.
* credential: include groups in postgres configuration snapshots
Full-state reconciliation treats absent groups as deleted, so a
snapshot that never carries them would erase every dynamic group.
* s3: revoke static-file groups dropped from the config file
A file reload is authoritative for the file's group set while keeping
dynamic groups, mirroring how full snapshots are authoritative for
dynamic groups while keeping the file's.
* credential: fail filer snapshots on unreadable entries
A skipped identity or policy file made the load report success with an
incomplete snapshot, which reconciliation reads as deletion and the
retry loop never sees. Unparseable content is still skipped: it is
durable, matches boot behavior, and must not block reloads forever.
* s3: ignore groups in static config files
Groups are managed through the IAM API and the dynamic store; no
deployment defines them in a bootstrap config file. Ignoring them with
a warning removes the two-directional group merge: full snapshots are
plainly authoritative and file reloads never touch groups.
* s3: require a bucket-policy action to write a bucket policy
PutBucketPolicy and DeleteBucketPolicy were gated on ACTION_WRITE, the same
action that grants object writes. An explicit Allow in a bucket policy
short-circuits IAM entirely -- authRequestWithAuthType sets policyAllows and
skips VerifyActionPermission -- so anyone who could write an object could
author a policy granting itself, or anonymous, anything on the bucket.
That is what separates a bucket policy from the sibling bucket controls also
gated on ACTION_WRITE: rewriting cors or lifecycle can destroy data, but only
a policy hands out access.
Give the two verbs their own actions, mapped to the AWS names that were
already defined but unrouted. ACTION_ADMIN would also have closed it, but it
resolves to s3:* for IAM identities, forcing a blanket grant on a user holding
a precise s3:PutBucketPolicy. Admins are unaffected, since isAdmin
short-circuits CanDo, and an operator can delegate with PutBucketPolicy:bucket.
The route binding is asserted from the router source: checking the action
constants alone still passes when the route says ACTION_WRITE.
* s3: also read the action from a direct iam.Auth call in the route test
Routes read iam.Auth(cb.Limit(handler, ACTION)), a multi-value pass-through:
Limit returns (http.HandlerFunc, Action) and those become Auth's parameters, so
the action Auth authorizes on is Limit's second argument and the two cannot
disagree -- Auth(Limit(h, X), Y) does not compile.
A route that skipped Limit and called Auth with its own action would compile,
though, and the test reported that as a missing route rather than as the wrong
action. Recognise the two-argument Auth form so it names the action instead.
* s3: make the bucket-policy actions grantable through an IAM policy
The new actions close the escalation only if an operator can grant them, and
they were not reachable: MapToStatementAction had no entry for PutBucketPolicy,
so an IAM policy naming s3:PutBucketPolicy was rejected outright with "not a
valid action". GetBucketPolicy was unmapped the same way.
DeleteBucketPolicy was mapped, but to ACTION_ADMIN -- granting an identity
permission to delete a bucket policy handed it full administrative access.
Map all three to the actions the router now uses, and add the reverse
direction so an identity holding them renders back as a policy statement
instead of a bare "s3:".
* admin: offer the bucket-policy permissions in the user editor
The two new actions are otherwise only grantable by hand-editing identity JSON
or by calling the IAM API, so an operator using the UI cannot delegate bucket
policy management without granting Admin.
Regenerating this file also picks up codegen the repo has not taken yet: the
checked-in _templ.go files were produced by templ v0.3.1001 while go.mod pins
v0.3.1020, so the generator rewrites the attribute-value calls. That churn is
confined to this one file; running `make generate` in weed/admin reproduces it
across all 36.
* volume: gate the admin RPCs that only shell and workers call
checkGrpcAdminAuth covered 19 of the 48 VolumeServer RPCs, so an operator who
sets -whiteList expecting it to cover the gRPC surface gets partial coverage.
Extend it to ten that mutate state and are only ever called by the shell or a
worker: SetState, VolumeCopy, the EC generate/rebuild/copy/unmount/to-volume
pair, both tier moves, and VolumeTailReceiver. That is safe because the same
callers already reach gated RPCs today -- VolumeMarkReadonly, VacuumVolume*,
VolumeEcShardsDelete, VolumeDelete -- so a whitelist deployment already lists
those hosts. Nothing here is on a master or peer path, which is what made the
earlier fail-closed gate break multi-host clusters.
The split is by caller rather than by blast radius: the guard matches a peer IP
against the whitelist, and a whitelist holds masters, shell hosts and workers,
not every peer volume server. Gating a call one volume server makes to another
would break replication, EC and tiering, so those stay open.
Two test fakes embedded a nil grpc.ServerStream and only implemented Send;
they now implement Context, which the streaming RPCs read to authorize.
* volume: fail the build when a gRPC method skips the admin gate
The admin gate is an opt-in list in a 48-method service, which is how it
drifted down to covering 19 of them: nothing tied adding an RPC to deciding
whether it needed the gate.
Parse volume_server.proto, walk the AST of every *VolumeServer method, and
require each RPC to either call checkGrpcAdminAuth or appear in
ungatedVolumeServerRPCs with the reason it stays open. A stale entry naming an
RPC that no longer exists fails too, so the list can't quietly stop exempting
anything.
The exemptions are the cluster-internal calls -- replica sync, EC shard
distribution, vacuum reads, backup, tailing -- plus the read-only and liveness
RPCs. Closing the cluster-internal ones needs a peer identity rather than an
IP whitelist; recording them here makes that a visible decision instead of an
omission.
The AST walk also corrects the count: a line-window scan credits
VacuumVolumeCheck and VolumeServerStatus with a neighbouring function's guard.
* volume: fix EC decode/reconstruct index locality under -dir.idx
EC->replicated decode failed under -dir.idx and on multi-disk with "volume not
found on disk". The reconstruct rebuilds the .dat on the data disk but the
on-demand VolumeMount scans only the data directory, matching on .idx/.vif;
with the rebuilt .idx off in the index directory it matched the volume's
leftover EC .vif and skipped the volume as EC metadata.
- Resolve the EC .ecx local-first: prefer the copy co-located with the shards
over the shared -dir.idx copy, with a non-empty preference so a 0-byte local
stub still yields to a valid sibling (the cross-disk fallback).
- Co-locate the rebuilt .idx with the .dat at the end of the reconstruct so the
mount finds it; sweep .ecx/.ecj from both the data and index directories on
Destroy so a stale copy cannot re-mount as a phantom EC volume.
- Add VolumeConsolidateIndex: once the EC shards are deleted, unmount, move the
.idx/.sdx from the data disk back to the -dir.idx directory (copy fallback
across filesystems), and remount. A no-op without -dir.idx.
* volume: tests for EC index locality (local-first .ecx, sweep, consolidate)
- NewEcVolume prefers a non-empty local .ecx over the shared index dir, and a
0-byte local stub yields to a non-empty shared copy (the #9212 fallback).
- Destroy sweeps .ecx/.ecj from both the data and index directories.
- ConsolidateVolumeIndex moves a co-located index back to the -dir.idx dir and
keeps the volume mounted; no-op without a separate index dir.
- RenameOrCopyFile moves a file and drops the source.
* volume: relocate the decoded index in place, without a read gap
ConsolidateVolumeIndex previously unmounted the volume, moved the index, and
remounted it. Between the EC-shard delete and the remount the volume had neither
a normal nor an EC form mounted, so a read landing in that window got a
not-found (or was proxied away).
Move the index in place instead: RelocateIndexTo takes the data-file write lock,
closes the needle map and data backend, moves the .idx (and derived .sdx), then
retargets dirIdx and reloads — the same close-swap-load CommitCompact uses. The
volume never leaves the mounted set, so a concurrent read blocks briefly on the
lock rather than failing. The test now writes a needle before consolidating and
reads it back after, proving the in-place reload keeps the volume serving.
* volume: address review — maintenance guard, no orphan on copy failure
- VolumeConsolidateIndex now rejects the request under maintenance mode, like
VolumeConfigure and the other mutating volume RPCs.
- RenameOrCopyFile rolls the cross-device copy back when the source cannot be
removed, so a failed move never leaves two divergent copies (the loader would
keep the data-dir one while the idx-dir orphan goes stale).
- RelocateIndexTo logs a failed reopen-after-failed-move instead of swallowing
it, since that leaves the volume unusable until the next load.
The sentinel stores hardcoded a 30s read timeout and a 1m retry backoff.
After a sentinel failover every request that picked a pooled connection to
the old master sat there for 30s before the connection was retired, and the
pool timeout derived from it (read timeout + 1s) queued the rest behind
them. The other redis stores took the go-redis defaults with no way to tune
anything.
Read the dial, timeout and pool knobs from each redis store section instead,
keeping the go-redis default for every key left unset.
security.GetJwt reads the "jwt" query parameter before the Authorization
header, and the proxy forwarded the caller's whole query apart from
proxyChunkId. So on a read, where the filer mints a volume token and sets the
header itself, a caller-supplied ?jwt= silently outranked it: the volume
server validated a credential the caller chose rather than the one the filer
attached, and the read failed with a 401 the filer could not explain.
Drop it on the read path, where the filer owns the credential. Writes keep
theirs -- the proxy forwards a writer's own AssignVolume token either way, so
the query parameter is just a second channel for the same credential and
stripping it would break a caller that presents it that way.
Nothing in the tree passes a jwt by query; maybeAddAuth always sets the
header.
The S3 write path cut fixed 8MB chunks, so an object stored through S3
chunked differently from the same bytes stored through the filer, WebDAV
or a mount, and -maxMB had no effect on it. Read maxMB from the filer
configuration at startup and use it, falling back to 8MB when the filer
reports none.
* filer: add a placement overlay seam for the write path
New volumes take their disk type, replication, and data center from the
explicit request or the matched filer.conf rule. That leaves no way for a
feature to steer a whole collection onto a medium without an operator
writing an fs.configure rule by hand.
Add a generic PlacementOverlay hook on the filer: a func that maps a
collection to a placement override, installed by a factory the way the
plugin-worker handlers register. detectStorageOption consults it between
the explicit request value and the filer.conf rule, so it overrides the
rule but yields to a value the caller asked for.
The seam names no feature concepts, so it stays generic; a downstream
build registers the overlay it wants (e.g. a storage-class Landing tier).
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
* filer: address review on the placement overlay seam
Honor ResolvePlacement's ok flag explicitly rather than relying on empty
values falling through the util.Nvl chain, and log at V(4) when the
overlay steers a collection. Document that RegisterPlacementOverlay is
init-only, so the unsynchronized read in NewFiler cannot race the write.
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
* filer: claim the base fid when minting a volume read token
GenJwtForVolumeServer stamped the fid verbatim, but the volume server strips a
trailing _N delta suffix before comparing the claim, so a token minted for a
batch-assigned fid like 3,01637037d6_1 was checked against 3,01637037d6 and
never matched. Reading such a chunk through the filer returned 401 wherever
jwt.signing.read.key was configured.
Strip the suffix before minting, via a helper shared with the proxyChunkId
validation that was already doing the same thing inline.
* filer: don't mint a volume write token for an anonymous proxy caller
The ?proxyChunkId= branch dispatches and returns before the JWT gate, so
whatever credential the proxy attaches is reachable without authentication.
It attached a token from maybeGetVolumeReadJwtAuthorizationToken, which fell
back to the write signing key when jwt.signing.read.key was unset -- the
configuration scaffold/security.toml recommends for a filer, since read JWTs
are only supported in a master+volume setup. An anonymous
DELETE /?proxyChunkId=<fid> therefore arrived at the volume server holding a
write-key token scoped to that fid, and the volume server honored it.
Sign read tokens with the read key only. The fallback bought nothing on a
read anyway: a volume server enforces read JWTs solely when that same key is
set, so when the fallback fired the read was unchecked regardless.
Mint only for reads. Writers proxied through the filer carry their own volume
JWT from AssignVolume, forwarded with the rest of the caller's headers, so
weed mount -filerProxy uploads are unaffected. Moving the dispatch below the
JWT gate instead would have broken them, since that token is signed with
jwt.signing rather than jwt.filer_signing.
On a read with nothing to mint, drop the caller's Authorization rather than
relaying it: there it is a filer credential, and forwarding it would hand a
volume server a token it never used to see.
* filer: keep proxied writes out of the read concurrency semaphore
The semaphore is named and documented for reads -- it exists so replication
bursts can't open hundreds of connections to one volume server -- but it was
applied to every proxied method. A write queued behind sixteen in-flight reads
can wait past the 10s default expiry of the AssignVolume token it carries, and
the volume server then answers 401. shouldReassignUpload treats a 4xx as final,
so the uploader replays the same expired token instead of re-assigning and the
write fails up to the caller.
This only became reachable once the filer stopped re-minting a fresh token
after the wait.
LookupFileId only requires the fid to contain a single comma, and the value
is pasted straight into the volume server URL path, so ?proxyChunkId=3,x/../../status
resolves to volume 3 and then addresses an endpoint the caller never named:
Go sends the dot segments verbatim, the volume server's mux cleans the path
and redirects to /status, and the filer follows the redirect and relays the
body. That reaches any handler on the volume server -- status, stats, the UI --
past a filer that operators expect to be the only exposed surface.
Parse the fid before the lookup and answer 400 when it doesn't parse.
A trailing _N delta suffix from batch assigns is legal, so it is stripped
first, but only when it is a non-empty run of digits. The volume server strips
at the last "_" unconditionally, which is safe there because its fid came out
of a path the mux parsed and so cannot hold a "/"; here the value is raw query
input, and an unguarded strip would reduce "3,01637037d6_1/../../status" to a
valid fid and wave the traversal through.
The queue holds sixteen sealed windows, which is a memory bound only
while a window is BufferSize. An entry larger than that grows its window
to fit, and the depth then multiplies straight through: sixteen queued
copies of a 100 MB window is 1.6 GB of flush data alone.
Account the queued bytes and make producers wait once they pass the
ceiling the depth was chosen for. What is charged is the pooled slab
rather than the window length, since mem.Allocate rounds up to a size
class and the queue holds the whole slab. A window larger than the whole
budget still goes through on its own, so an oversized entry is never
stuck.
Windows are admitted in the order they were sealed. A producer can now
park here for seconds, and letting a later window overtake an earlier one
would persist them out of order and walk lastFlushedOffset and
lastFlushTsNs backwards.
A window is copied into its slab under the write lock, before the
reservation is taken, so a burst of concurrent oversized writers would
each hold a full copy in hand while queueing up -- memory the budget
never sees. Large writers wait for queue headroom before they take the
lock, which throttles the burst; it does not bound it, since a writer
that passes the check still seals unconditionally.
Take any room in the queue before the shutdown escape, too: the window is
already sealed by then, so dropping it loses records the caller was told
were accepted. A shutdown that races a full queue can still drop one --
that predates this change and needs the flush loop's lifetime reworked.
Size a grown window to the entry rather than to twice it: the extra room
only bought space for a second oversized record in the same window, which
doubles the flush copy and the snapshot taken of it. The overflow guard
halved its bound for that doubled allocation, so raise it to match what
is now allocated and what maxBufferSize documents.
* volume: reject needle blob writes to read-only volumes
WriteNeedleBlob appends the blob to .dat and only then calls nm.Put. On a
read-only volume the needle map is a SortedFileNeedleMap whose Put always
fails, so the append is never indexed and never rolled back.
Nothing upstream stops this: volume.check.disk picks its targets from the
master's cached topology, which goes stale the moment a volume server marks
a replica read-only itself — a failed data integrity check at load, or an
EIO quarantine. Each sync attempt then grows the .dat of a replica that is
supposed to be frozen by one unindexed needle, and reports it as "invalid
argument", the bare os.ErrInvalid the needle map returns.
Check IsReadOnly before touching .dat, same as the upload path does.
* volume: say which needle and volume failed to index
An index write that fails surfaced as a bare errno with no volume, no needle
and no file — "invalid argument" for a read-only needle map, or a plain
ENOSPC when .idx lives on its own filesystem via -dir.idx. Both were logged
at V(4), so by default the operator saw only the errno the client got back.
* filer: write the metadata log in pieces a volume server will accept
A single oversized metadata event grows the log buffer past the volume
server's fileSizeLimitMB, and the flush of that buffer is then rejected
forever: the retry loop has no exit, so the blob at the head of the queue
blocks every later flush and the metadata feed stalls until restart.
Split the flushed buffer into BufferSize pieces, on record boundaries
where possible so each piece still decodes on its own, and retry each
piece separately so a partial success is not replayed. Log files are
already read as a chunk stream, with a whole-file fallback when a chunk
does not decode standalone, so a record may cross a piece boundary.
* log_buffer: let go of a window array grown for an oversized entry
An entry larger than BufferSize grows the window array to 2*size+4, and
window arrays cycle through SealBuffer rather than being freed. One such
entry therefore leaves every later window carrying its size, and
currentSnapshotView allocates a snapshot as wide as the array on each
window, so a few KB of metadata keeps paying for it.
Drop the array when SealBuffer hands it back. Growth is on demand, so
the next oversized entry just reallocates.
* iceberg maintenance: store merged data files as chunks, not inline
saveFilerFile had no size threshold, so compaction wrote whole merged
parquet files -- hundreds of MB -- as Entry.Content. That puts the
parquet bytes verbatim in the filer store and sends them through the
metadata change log again as one event.
Keep manifests and metadata JSON inline, upload anything larger to
volume servers in chunks, assigning through the filer so the path's
storage rules apply.
* filer: follow the file size limit the volume servers report
The starting piece size is a constant, so a cluster whose
-fileSizeLimitMB is set below it would reject every piece and wedge just
the same. The rejection names the limit, so take it from there and
re-cut the rest of the flush to fit.
Piece the buffer one at a time rather than up front, since the size can
change partway through a flush.
plugin: stop the scheduler lock test racing its own background loops
TestRunLaneSchedulerIterationLockBehavior constructed the plugin with a
cluster-context provider, which makes New start a background scheduler
loop per lane. Those loops call runLaneSchedulerIteration on the same
lane the test then drives by hand, so a loop could consume the due job —
running detection and pushing the next-detection time forward — before
the manual call observed the lock. The Default case then saw the lock
acquired zero times and failed intermittently.
Construct without the provider so no loops start, and set the provider
afterward so the manual iteration can still detect. This is the pattern
scheduler_status_test.go already uses for the same reason.
Reproduced under -race -count=100 -p 4 before, green after.
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
* util: match transient error messages case-insensitively, expose the message form
The same condition reaches different layers capitalized differently -- a
volume server relays its idle timeout as "I/O timeout" inside a JSON string --
and the callers that grew their own substring lists all lower-case first.
Also split out IsTransientErrorMessage for the paths that carry only the text,
such as the per-file status strings in a batch delete response, and pick up
"no route to host" and "network is unreachable" from the gRPC classifier.
* filersink: classify transient network errors through util.IsTransientError
The local list caught i/o timeout, connection reset, and broken pipe but not
connection refused, no such host, unexpected EOF, the syscall errnos, or the
gRPC and S3 overload codes. Keep only the bare io.EOF case, which is transient
here -- a truncated chunk read -- but a clean stream end elsewhere.
* filer deletion: reuse util.IsTransientErrorMessage for the network patterns
Six of the sixteen patterns were already covered. Keep the ones specific to
this pipeline -- read-only volumes, lookup failures, backpressure -- and note
why context cancellation stays retryable here: it decides whether to requeue
the deletion, not whether to retry a call.
* wdclient: fold the shared classifier into the volume lookup retry check
The string tail duplicated the shared list and missed the syscall errnos and
net.Error timeouts. Keep "connection" and "timeout", which are broader than
the shared classifier on purpose: a volume lookup is a cheap read-only call.
* filer.replicate: commit the kafka offset after replicating, not on receipt
The partition consumer committed the offset as soon as it handed the message
to the channel, so a sink write that failed was logged and the message was
already behind the committed offset -- never redelivered, permanently missing
from the sink.
Commit in onSuccessFn instead, and hold the committed offset behind the
oldest offset that failed to replicate so a restart redelivers from there.
* filer.replicate: delete the sqs message after replicating, not on receipt
ReceiveMessage deleted the message before the replicator had a chance to run,
so a failed sink write dropped it for good. Move the delete into onSuccessFn
and leave the message in the queue otherwise, letting the visibility timeout
redeliver it.
* util: retry transient errors, not just the ones containing "transport"
util.Retry only retried when the error string contained "transport", so a
plain "read: connection reset by peer" from S3 got zero retries. Classify
the error instead: net timeouts, connection resets, and the throttling and
overload replies S3 and gRPC return are all worth another attempt, while a
cancelled or expired context is not.
* filer sync: hold the sync offset behind a failed event
A sync job that returned an error was logged and forgotten, and the
watermark advanced past it anyway. The offset is the durable resume point,
so the event was never replayed: for filer.remote.sync that left the file
present locally, absent on the remote, with no RemoteEntry and nothing to
retry it.
Pin the watermark at the oldest failed event. Later events keep flowing,
but the persisted offset stays behind the failure, so a restart replays it.
The scaffold advertises enable_tls, ca_cert_path, client_cert_path and
client_key_path under redis2, redis2_sentinel and redis_cluster2, but only the
plain redis2 and redis3 stores ever read them, and under a different name,
enable_mtls. Sentinel and cluster setups quietly connected in plaintext.
Build the TLS config in one place and use it from all six stores. enable_mtls
still works. The CA and the client key pair are optional now, so enable_tls
alone verifies against the system roots, and ServerName is left unset so
go-redis validates each address the sentinel and cluster clients dial.
redis2_sentinel and redis3_sentinel stores only passed Username/Password
into redis.FailoverOptions, which authenticates against the Redis
master/replica servers. When Sentinel itself requires auth (requirepass
set in sentinel.conf), go-redis had no credentials to send to it,
causing a NOAUTH error before ever reaching the master.
Add sentinel_username/sentinel_password config options that map to
go-redis's SentinelUsername/SentinelPassword fields, distinct from the
existing master auth credentials.
Deleting the only object under a prefix in a versioned bucket writes a
delete marker and keeps the version history, so the filer directory
survives with nothing a current-version listing would return. A delimited
ListObjects kept reporting that path in CommonPrefixes, because the
prefixes come from the directory tree rather than from the keys, while a
listing scoped inside the prefix correctly came back empty.
Probe a directory before reporting it: one that holds entries but no key
the listing returns is neither a CommonPrefix nor a path the
trailing-slash probe answers for. Empty directories keep the meaning they
have today, and the probe only runs for buckets with versioning
configured, the only ones that can reach this state.
A maintenance snapshot carried only its own labels — merged-files,
delete-groups and friends — and no summary counters, so every engine that
reads a table's size out of the current snapshot summary reported nothing
for it: PyIceberg's inspect.snapshots, Trino's $snapshots and Spark's
DESCRIBE all read total-records, total-data-files and total-files-size
verbatim, and a table lost them the moment compaction touched it.
Accumulate the files each operation adds and removes, and render them the
way the spec defines: the added-*/removed-* counters from the files
themselves, then the totals carried over from the parent snapshot.
Carry a total only when the parent recorded it. Iceberg treats a missing
total as zero, which turns a compaction replacing two files with one into a
negative total-data-files, or a table with millions of rows into
total-records: 0. Leaving the field out lets a reader fall back to the
manifests instead of believing a made-up number.
Compaction also accounts for the delete files it consumes, so a run that
folds every delete into the rewritten data reports them as removed.
The worker assumed every file of a table sits under its catalog path, so
loadFileByIcebergPath stripped the scheme off a recorded location and joined
the remainder onto /buckets/<bucket>/<ns>/<table>. A table the REST catalog
placed elsewhere in the bucket — which is what a client gets whenever the
catalog path is already occupied — then resolves to a doubled path:
lookup /buckets/lake/source/t/lake/source/t-0cd81bca-.../metadata/snap-.avro
so the very first manifest list read fails and the job fails again on every
scan interval, indefinitely.
Resolve absolute references (s3:// URIs and /buckets paths) from the bucket
root and keep relative ones under the table's own directory; the
bucket-relative form is now the canonical key everywhere references are
compared. That directory comes from the metadata location the catalog stores,
so reads, writes and deletes all land where the table's other files are
instead of splitting it across two trees. References outside the table's
bucket are rejected rather than silently misresolved.
Rewritten position-delete files now name their data file by absolute URI,
the way the table itself names it, instead of a path relative to the table.
The dashboard charted one summed disk-usage line, so a step in the total
gave no hint which cluster moved. A new panel stacks each cluster's daily
size as its own band: the top of the stack is the fleet total, each band
is one cluster, and the clusters past the twentieth are summed into an
"other" band so the stack still adds up to the total.
The series is built from the per-cluster daily histories and served by
/api/cluster-sizes. Clusters report roughly once a day at no fixed hour,
so a day with no report carries the previous value forward — dropping it
to zero would sag the total every day as the clusters that have not
reported yet fall out from under it. A cluster that stops reporting past
the active window ends at its last sample instead of holding capacity
forever. Ranking is by the most recent day, tie-broken on cluster id so
the colors do not shuffle between refreshes.
Hover and click resolve to the band under the pointer: Chart.js's builtin
interaction modes match the nearest line, which on a stack of thin bands
is rarely the band being pointed at. Clicking one fills the per-cluster
history lookup below it.
LiveMoveVolume and the copy, tail, delete, mark, replicate, and
configure helpers around it issued every RPC on context.Background(), so
a caller had no way to bound or abort a move once it started. They now
take a context, which the exported LiveMoveVolume in particular needs:
callers outside the shell drive long moves and want to stop them.
The deferred restore in copyVolume runs on a detached, bounded context
rather than the caller's. Marking the source writable again is cleanup,
and cancelling the copy must not skip it and leave the volume readonly —
the same guard balance_task.go already applies for the same reason.
Shell commands pass context.Background(): their Do signature carries no
context, and changing it would touch every command in the package.
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
Chris LuandGitHub
Lukas KalliesandGitHub
Kobi HikriandGitHub
Lars LehtonenandGitHub
TJDawson10andGitHub