If checkpoints keep failing (e.g. a persistent .dat flush failure whose
error is not EIO and so does not mark the volume read-only), the counter
increments on every write with no upper bound and wraps at ~4.3 billion.
Use saturating_add so it pins at u32::MAX instead, which keeps
checkpoint_due() true and retries on every subsequent write.
Co-Authored-By: Chris Lu <chris.lu@gmail.com>
On the fsync=true write path, flush_idx() already fsyncs the .idx before
maybe_checkpoint_index() runs, so the checkpoint's own sync() fsyncs the
same file a second time for nothing. Thread an idx_already_synced flag
from the volume through maybe_checkpoint_index into checkpoint(sync_idx):
when it is true the checkpoint skips its .idx fsync and only does the
durable redb commit. The delete path and close() still sync (they have
not flushed the .idx beforehand).
Co-Authored-By: Chris Lu <chris.lu@gmail.com>
Every put and delete on a redb-backed volume committed with
Durability::None and nothing ever committed durably, on the theory that
the .idx file is the source of truth. redb, however, keeps an entry in
its transaction tracker for every non-durable commit and cannot recycle
pages that were on disk at the last durable commit until a durable one
happens. With no durable commit for the life of the process, both grew
with every write, and .rdb files could bloat toward double size after a
restart (#11179, the hash-table rehash stacks in the memleak output).
The needle map now counts non-durable commits and reports when a
checkpoint is due; the volume takes it, data first: flush the .dat, then
the map fsyncs the .idx and commits redb durably, recording in the same
transaction how much of the .idx the table reflects. A checkpoint makes
the index durable, so the bytes it points at must be down before it, or
after a power loss the index would reference past the end of the .dat
and the volume would load read-only. A failed .dat flush skips the
checkpoint; it is retried on the next write.
Volume::close() now closes the needle map instead of only syncing it,
and the redb map's close() takes the same checkpoint. Before, a clean
shutdown left the table durable (redb flushes on drop) but the recorded
.idx size stale at its load-time value, so the next load replayed every
entry written since load on top of the counters.
On load, the redb map's counters now come from the whole .idx history,
the way Go's LevelDB map rebuilds them (newest entry first, with a bloom
filter of seen keys), instead of from the table's final state. Both the
reuse and the full-rebuild path use it, so overwritten and deleted bytes
keep counting as garbage across restarts, and the incremental replay of
the .idx tail only touches the table, which makes it idempotent whether
or not the table is ahead of the recorded .idx size.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019x36FiSeyePh77YXao15kK
* s3: warn when a lifecycle change leaves fast-path-stamped objects on their old TTL
The per-write TTL fast path (opt-in via s3.bucket.lifecycle.fastpath)
stamps a volume TTL at PutObject time that can't be taken back. When an
operator lengthens or removes an Expiration.Days rule (or deletes the
bucket lifecycle) on a fast-path-enabled bucket, objects already written
keep their baked-in TTL and won't be rescued by the change — unlike the
default worker-driven path, which re-evaluates the current rules each
pass. This is the data-loss direction described in #11183.
Surface it: Put/DeleteBucketLifecycle now emit a glog warning and set
X-Seaweed-Lifecycle-Fastpath-Warning on the response when the change
removes, disables, lengthens, or re-scopes a fast-path-eligible rule.
Shortening a rule does not warn (old objects simply expire later, not
data loss). Tag-only and overflow-day rules are never on the fast path
and never warn.
Addresses the warning half of option 2 in #11183.
* s3: address review — emit warning after mutation succeeds, fix ID-rename false positive
Two issues raised by CodeRabbit, Greptile, and Devin reviews:
1. Failed mutations retained the warning header. The warning was set on
the ResponseWriter before storeBucketLifecycleConfiguration /
clearStoredBucketLifecycleConfiguration was called; if that failed,
the error response carried a warning for a change that was never
applied. Now the reason is computed before the mutation but the log
and header are emitted only after it succeeds.
2. Rule renames produced false "removed" warnings. fastpathRuleKey used
Rule.ID as the sole identity when present, so renaming a rule (same
prefix/size/days, different ID) treated the old rule as removed.
Replaced with two-pass matching: first by ID, then by fast-path
predicates (prefix + size). An ID-only rename with unchanged
predicates and days no longer warns. Greedy matching ensures each
new rule is consumed by at most one old rule.
Added regression tests: ID-only rename (no warn), rename + lengthen
(warn), rename + shorten (no warn).
* feat(s3): make RenameObject idempotent for a retried request - #10661
A rename that succeeds but whose response is lost leaves the client with
no safe move: retrying returned 404, because the source is already gone,
so a retry was indistinguishable from a rename that never happened.
The destination now carries what the rename that created it was, under
x-seaweedfs-rename-token: the client's token, the source key and the
time. A retry that names the same token and the same source and
destination is answered 200 without touching anything. The same token
sent for a different rename is refused with 409 rather than silently
answered, and a token older than 24 hours is treated as unrelated so a
key cannot answer for a request indefinitely.
Requests without the header behave exactly as before.
* s3: answer a reused rename token with 409, not 400
The PR promised Conflict and the code returned Bad Request. 400 tells a
client its request was malformed and invites it to give up; this request
is well formed and resending it unchanged will not help, because what it
collides with is a rename the same token already stands for.
The status code is now asserted in a test, since it is the part of this
behaviour a client actually acts on.
* Update weed/s3api/s3err/s3api_errors.go
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: fix rename token review notes
- ErrIdempotentParameterMismatch returns 409 Conflict, not 400. The
comment and TestRenameTokenReuseAnswersConflict both expect 409; the
code regressed to 400 in a later commit.
- stampRenameToken: clarify that markRenameToken mutates srcEntry in
place, so the token reaches the destination via the move regardless
of whether the UpdateEntry succeeds. The precondition only guards the
pre-move write, not the move itself.
- Extract the handler retry branch into retryRenameDecision and add
TestRetryRenameDecision, covering the source-still-exists fallthrough
that was previously reasoned about but not tested.
* s3: IdempotentParameterMismatch returns 400, matching AWS docs
The AWS S3 RenameObject API documentation specifies HTTP Status Code: 400
for IdempotencyParameterMismatch. Revert the previous 409 change and align
the comment and test with the documented behavior.
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
The Rust volume server opens one redb database per volume and built
each with redb's defaults, which give every database a 1 GiB page
cache (0.9 GiB read cache + 0.1 GiB write buffer). With hundreds of
volumes behind one disk the process-wide ceiling was volumes x 1 GiB:
memory grew in proportion to the pages traffic touched, never shrank
when traffic stopped, and hosts running many instances were OOM-killed
under bulk ingest. redb, redbMedium and redbLarge were also treated
identically, so the "memory~performance" tiers did nothing.
Size the cache per tier instead: 4, 8 and 16 MiB per volume, mirroring
the Go server's 3/6/12 MiB LevelDB block cache + write buffer. Thread
the budget through RedbNeedleMap::new/load_from_idx so every open path
(create, reuse, full rebuild) uses Database::builder().set_cache_size.
Fixes#11179
Claude-Session: https://claude.ai/code/session_019x36FiSeyePh77YXao15kK
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
No source changes: the API surface the needle map uses (Database
create/open/builder, set_cache_size, set_durability, tables, iterators)
is unchanged and the on-disk format is still v3, so existing .rdb files
open as-is. The 4.0.0 breaking changes (Drop on AccessGuardMut, removal
of the Legacy type) do not touch this crate.
Relevant to the redb-backed index (#11179):
- 4.1.0: optimizes cache usage and memory usage; ~1.5x faster writes.
- 4.2.0: Durability::None commits ~2x faster; pages freed by a durable
transaction are reused by the very next one; a crash-recovery fix for
a crash during repair of an earlier crash.
Claude-Session: https://claude.ai/code/session_019x36FiSeyePh77YXao15kK
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Change the matplotlib figure size from (10, 4) to (10, 6) so the
star history chart renders vertically longer in the README. The
regenerated note/star_history.svg reflects the new 5:3 aspect ratio
(720x432pt) instead of the previous flat 2.5:1 (720x288pt).
The "Mount and exercise" step left the weed.exe mini server and the final
WinFsp mount running when it exited. The next step's pwsh.exe then failed
with STATUS_DLL_INIT_FAILED (0xC0000142), failing a job whose actual test
step had passed. The same code passed on both the PR branch and the next
master run, so this was a transient launch failure — but it was caused by
an unclean environment and made fatal by a diagnostic step.
Tear down all weed.exe processes at the end of the test step so subsequent
steps launch into a clean environment, and mark the Logs step
continue-on-error so a diagnostic step can never fail the job on its own.
2026-09-05 00:10:40 -07:00
Chris LuGitHubdevin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* add a per-mount cache_wait_ms to the remote storage mount mapping
A read of an uncached remote-only object waits on a hardcoded size tier
before it can fall back to the origin, so every ranged read of a large
remote-only object pays that wait. Carry the wait in the mount mapping so
it can be tuned, or set to zero, per mount.
* resolve the cache wait of an uncached remote-only read from its mount
The wait came only from the object size, so an operator could not trade
cache hits for time to first byte. Both read paths now resolve the mount
covering the object and let its cache_wait_ms replace the size tiers.
* read straight from the remote when a mount waits zero for its cache
A mount used as a streaming source pays the cache wait on every ranged
read of an object too large to finish caching, and the caching itself is
wasted work. A zero wait now skips the cache call, so both read paths go
to the origin immediately.
* let remote.mount set the cache wait of a mount
remote.mount -cacheWait=0 turns a mount into a streaming source, and any
other duration trades cache hits against time to first byte.
* keep the size based wait for a version-specific read
A read pinned to a version cannot fall back to the origin, since the
mounted remote only holds the current key, so a mount that opts out of
caching would leave it on the 503 retry loop forever.
* let the operator allow a remote-only read to dial an internal endpoint
The remote-mount read paths in the filer and the S3 gateway always refused
an endpoint resolving to a loopback or private host, so a mount backed by
an internal S3 could never be read from its origin, only through the local
cache. Both now take the allowance the volume server already has, still
off by default.
* skip the background cache of a mount that waits zero for its cache
GetObjectHandler kicks off caching for every remote-only read, so a mount
serving as a streaming source kept downloading whole objects even though no
read ever waited for them.
* cover a zero cache wait end to end
The read has to reach a real origin, so the harness also opts the filer and
the S3 gateway into dialing the loopback remote it already allows for the
volume server.
* resolve the S3 cache wait once so the background cache follows it too
The background cache that GetObjectHandler starts read the mount on its
own, so it skipped a version-specific read that the foreground path still
waits for. Both now ask the same resolver.
* answer 404 when the origin of a zero-wait read is gone
Metadata can outlive the object it points at, and with no cache to fill
the read would sit on the 503 retry path forever. The remote backends
already report a missing object as ErrRemoteObjectNotFound.
* open the origin at write time for a multipart range
Every part of a multipart Range is prepared before any is written, so
opening eagerly would hold one origin connection per part and leak the
ones already opened when a later part fails to open.
* reject a cache wait shorter than a millisecond
The mapping stores milliseconds, so -cacheWait=500us truncated to zero
and silently turned caching off instead of waiting.
* restore the doc comment of cacheRemoteObjectForStreamingWithShortTimeout
Extracting the wait resolver left its comment on the new function.
* stat the origin before committing a multipart range
Opening at write time keeps no connection through the preparation, but it
also moved a failure past the point where the multipart body picks the
response status, so a gone origin truncated a 206 instead of answering
404. One stat up front puts the status back.
* stat the origin once per request
Every part of a multipart Range is prepared on its own, so the preflight
ran once per range instead of once per read.
* map Azure and GCS stream not-found to ErrRemoteObjectNotFound
ReadFileAsStream on Azure and GCS returned provider-specific not-found
errors instead of ErrRemoteObjectNotFound, so a zero-wait read of a
deleted object was misclassified as a transient cache failure and
retried indefinitely. Map BlobNotFound and ErrObjectNotExist the same
way StatFile already does.
* Update weed/remote_storage/gcs/gcs_storage_client.go
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: count a TTL volume's age from its last write, not the .dat mtime
A delete appends a tombstone needle and vacuum rewrites the .dat wholesale,
so the file's mtime moves without any write ever landing. The loader read
lastModifiedTsSeconds back from that mtime, so every restart of a volume
taking delete traffic re-armed expired() for another full TTL: an
overwrite-heavy collection kept growing until it hit the max-volume cap.
Recover the clock from the newest .idx entry that is not a tombstone and
read that needle's append timestamp, falling back to the mtime when no
write is recoverable. Only TTL volumes pay for the scan.
Fixes#11160
* volume: count the .vif destroy time from the last write too
ExpireAtSec is what an EC volume is reclaimed on, and it was recomputed as
now+TTL every time the .vif was written. A read-only mark, a tier upload or
an EC encode therefore handed an already expiring volume another full TTL,
the same way the .dat mtime did.
Derive it from the volume's last write, falling back to now for a volume
that has not taken one yet so a fresh volume is not born expired.
* volume: mirror the last-write TTL clock in the Rust volume server
Same recovery as the Go loader: scan the .idx backwards for the newest
entry that is not a tombstone and take that needle's append timestamp,
leaving the clock on the .dat mtime when no write is recoverable.
* volume: mirror the last-write destroy time in the Rust volume server
Both .vif writers and the EC encode computed ExpireAtSec as now+TTL, the
same way Go did, so the destroy time moved every time the sidecar was
rewritten. Route all three through the volume's last write.
* volume: report the .dat mtime in the Rust heartbeat, like Go does
The Rust server reported its TTL clock as ModifiedAtSecond while Go
reports the .dat mtime. The shell's quiet-period gates (volume.tier.move,
volume.delete_empty) read that field as "last touched", which a delete
has to count towards even though the TTL clock deliberately ignores it --
and with the clock now recovered from the last write, the two drift
further apart.
* volume: take the newest write by timestamp on a vacuumed volume
The reverse .idx scan trusted position, which holds only while the .dat is
append ordered. Vacuum rewrites it in key order, and since an overwrite
keeps its original key, the highest-key survivor is not necessarily the
newest write -- the recovered clock could land up to a TTL early and take
the volume with data still inside its TTL.
A volume that has been vacuumed (CompactionRevision > 0) now takes the
maximum append timestamp over a bounded window of write entries instead.
An append-ordered volume still answers in one read.
* volume: never guess a vacuumed volume's last write, and resolve wrapped offsets
Two holes in the reverse scan, both from review:
A vacuumed volume's writes are ordered by key, so any of them can hold the
newest timestamp. Reading a capped window sampled the highest keys, which
could still miss a recently overwritten low-key needle and expire data
inside its TTL. The scan now covers every write a vacuumed volume indexes,
and a volume too large to scan keeps the .dat mtime rather than report a
partial maximum -- late is recoverable, early is not.
A .dat past MaxPossibleVolumeSize wraps the offsets in its .idx, so reading
a timestamp at the unwrapped offset picks up an unrelated needle. Resolve
the entry against the needle header first and retry one volume size in,
the way doCheckAndFixVolumeData already does.
* volume: drop GitHub issue references from TTL comments
* docs(readme): replace star-history.com with self-generated chart
The star-history.com SVG is a third-party dependency that can rate
limit or go down. Replace it with a GitHub Action that fetches
stargazers via the REST API and renders an SVG with matplotlib,
committing note/star_history.svg weekly. The README references the
committed file directly, so the chart has no runtime dependency on
any external service.
* ci(star-history): run daily instead of weekly
starchart.cc is rate-limiting the SVG endpoint, so the Stargazers
chart renders blank. Switch to star-history.com, which serves a live
SVG for this repo and links to the interactive chart.
The Patreon CTA and Gold Sponsors logos sat between the logo and the
project intro, pushing the actual description below the fold. Move the
whole block to a dedicated `# Sponsors #` section after `# License #`,
add it to the TOC, and give it a real markdown heading so the anchor
works on GitHub.
* mount: let volumeName take an explicit override
volumeName only ever derived the disk's label from -filer.path, -dir,
or the filer address, so a name that happened to collide with
something else - e.g. a UNC share's own name - could not be changed
without moving what was mounted. Give it an override parameter that
wins over all three; nothing passes one yet.
* mount: add -volumeName to name the disk explicitly
Windows has no equivalent of the "weed fuse" -o passthrough that lets
a Linux or macOS mount override its derived volname, so a name picked
up from -dir - e.g. a UNC share's own name - could not be changed
short of moving what was mounted. -volumeName overrides it on every
platform.
* mount: document -volumeName
* mount: scope -volumeName's help text to macOS and Windows
Linux has no volume-label mount option for -volumeName to feed, so
the flag's own description says where it applies instead of leaving
that unstated.
* mount: forward -volumeName through the weed fuse option parser
weed fuse (the /etc/fstab helper) turns -o key=value into the same
MountOptions weed mount takes, but volumeName had no case, so it fell
through to being forwarded as a literal, unrecognized FUSE option
instead of ever reaching mountOptions.volumeName.
* mount: apply -volumeName to FsName on Linux and FreeBSD
FsName only ever took the filer address and -filer.path, so
-volumeName had nothing to override there and silently did nothing;
the skipAutofs case still forces "fuse", since that name is what
util-linux/mount requires to recognize the pseudo filesystem.
* filer: guard the FoundationDB value size limit, not the transaction limit
An entry's whole chunk list is one FoundationDB value, and FDB caps a value at
100,000 bytes while a transaction may reach 10MB. UpdateEntry checked the
transaction limit, so every entry between the two limits passed the guard and
was rejected by FDB itself with error 2103 (Value length exceeds limit). The
failure surfaced inside the store rather than at the guard, so the S3 layer
dropped the connection and clients saw a network fault instead of an error.
Check the value limit in UpdateEntry and KvPut instead, after gzip and before
the transaction, with an error that names the limit it hit. The removed
transaction-size constant guarded nothing else: DeleteFolderChildren batches by
entry count.
Refs #11158
* filer: fold at 500 chunks in the foundationdb build
Manifest packing is what keeps a large file's entry small, but it only ran once
a flat chunk list reached 10000 chunks. A FoundationDB value stops at 100,000
bytes and an entry's whole chunk list is one value, which at ~100 bytes per
chunk record is about 1000 chunks -- so on FDB the write always failed before
packing could help: a 3.3 GiB PutObject at the default -maxMB=4 was already
past the limit.
FoundationDB support is its own build (`go build -tags foundationdb`, shipped
as its own image), so the batch is a build-time choice and needs no negotiation
at run time. The tagged build folds at 500, every other build keeps 10000 and
is untouched.
500 is not arbitrary: a single fold level leaves (chunks/batch) manifest
pointers plus up to (batch-1) unfolded chunks in the entry, so the reachable
chunk count is highest when the two terms are near equal. For a 100,000-byte
budget that optimum is 500, which holds an entry inside the limit up to
~250,000 chunks -- ~1 TB at -maxMB=4, against ~4 GB before. Larger files need
nested packing, which no batch size substitutes for.
One binary serves every role in that image, so the filer and each client that
folds -- S3, mount, WebDAV, weed shell, filer.copy -- agree on the batch by
construction. A binary built with the tag but pointed at another store folds
earlier than that store requires, costing one manifest blob per 500 chunks and
one read to resolve it.
Fixes#11158
* filer: fold with rollback inside MaybeManifestize, not beside it
A fold that fails midway has already uploaded manifest blobs for its earlier
batches, and returns only the data chunks -- dropping the manifests it had
separated out of the caller's list. Both were wrong in ways that mattered:
- AppendToEntry assigned that shortened list straight to entry.Chunks and
created the entry, so an append to an already-folded file whose fold
failed lost every previously folded chunk. weed mount had the same shape.
- cleanupChunks logged the error as "not good, but should be ok" and then
returned it through a named result, failing the whole CreateEntry or
UpdateEntry, while the blobs it had written stayed behind referenced by
nothing.
The S3 path was alone in handling this, through a private helper beside
MaybeManifestize. A second entry point next to the one everything else calls
just means the wrong one gets used, so the behaviour moves inside
MaybeManifestize: on failure it returns inputChunks as it received them, and
hands the blobs it saved to a deleteChunks callback. The filer, S3 and
filer.copy pass their existing deleters -- filer.copy already cleans up this
way after a failed upload -- and mount, WebDAV and weed shell pass nil, which
reports the blobs rather than collecting them, as before. Each caller keeps its
own error policy: the filer HTTP PUT path and filer.copy still fail the request,
the rest still continue with the flat list, which is a correct entry.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3: HEAD with partNumber reports the part's size and range
HeadObject set its headers from the total object size and then only
validated the partNumber, so a client probing part 1 with HEAD got the
whole object's Content-Length and a 200 while the same GET returned the
part's size, a Content-Range and a 206.
Resolve the part's byte range before the headers are written, through the
range logic GetObject already used, and answer a partNumber HEAD as the
ranged HEAD that AWS documents.
* s3: answer an unsatisfiable partNumber with 416 InvalidPartNumber
GET and HEAD rejected a partNumber past the number of parts with 400
InvalidPart, the code for a missing part in CompleteMultipartUpload. AWS
answers a read of a part that does not exist with 416 InvalidPartNumber,
which lets a client probing for the part count tell the two apart.
The ceph suite pins RGW's 400 InvalidPart here, so the s3tests jobs patch
that expectation the way they already patch prefix ordering.
* s3: keep the whole-object checksum off a partNumber response
The stored checksum covers the whole object, so it is already withheld
from a ranged read. A partNumber HEAD now describes one part while the
request carries no Range header, so exclude it there too rather than
handing a client a checksum that does not match the bytes described.
* s3: resolve a partNumber against the parts the object records
Completion accepts ascending, not consecutive, part numbers, so the part
count is not the highest part number. Comparing the two rejected an
uploaded part 3 of a two-part object, and let a request for the absent
part 2 fall through to the positional chunk lookup and serve part 3's
bytes. Ask the recorded boundaries for the part instead, and keep the
count comparison for objects written before boundaries were stored.
* s3: apply a client Range within the part on HEAD too
GET narrowed the part by a Range sent alongside partNumber; HEAD reported
the whole part, so the two disagreed again for a request that carries
both. Move the narrowing into the shared range lookup so either verb
describes the same bytes.
* fix(chart): serve S3 internal gRPC with mTLS when security enabled
The security.toml generated by the chart has no [grpc.s3] section, so
security.LoadServerTLS(viper, "grpc.s3") returns nil in weed/command/s3.go
and the S3 server listens plaintext on its gRPC port (httpPort+10000 = 18333
by default). Workers dial that port with mTLS credentials (grpc.worker),
producing:
walker dispatch ...: rpc error: code = Unavailable desc = connection
error: desc = "transport: authentication handshake failed: tls: first
record does not look like a TLS handshake"
This breaks the s3_lifecycle worker's LifecycleDelete RPC path (recovery
walk, daily replay) and any S3->S3 IAM cache propagation would fail the
same way if clients enforced TLS.
Add [grpc.s3] reusing the client cert already mounted on s3 pods (or
s3.tlsSecret when set, mirroring the seaweedfs.s3.tlsArgs helper for the
HTTPS listener).
Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf)
* fix(chart): always use internal client cert for grpc.s3 identity
s3.tlsSecret is the public HTTPS listener certificate (possibly issued by
a public CA); internal gRPC peers only trust grpc.ca, so presenting it on
the internal gRPC port would break lifecycle/IAM RPC verification. Keep
the two trust domains separate.
Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.8-Flash-Next-ROCmFP4)
golang.org/x/image v0.44.0 is affected by CVE-2026-46603 (GO-2026-6222):
a denial of service via excessive memory allocation when decoding
malformed VP8L (lossless WebP) data. It is fixed in v0.45.0, released
2026-08-11.
The decoder is reachable from SeaweedFS: weed/images/resizing.go
blank-imports golang.org/x/image/webp, which registers the VP8L decoder
with image.Decode, so the filer image resizing path decodes attacker
supplied WebP data with the affected version.
This is a go.mod/go.sum only change produced by
`go get golang.org/x/image@v0.45.0 && go mod tidy`; no other dependency
moved. `go build ./weed/`, `go vet ./weed/images/...`,
`go test ./weed/images/...` and `go mod verify` all pass.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* filer: require a read token for the root listing
maybeCheckJwtAuthorization waved through every GET/HEAD on "/", so a filer
with jwt.filer_signing.read.key set still served its root directory listing --
entry names, sizes and chunks[].file_id -- to a caller holding no token at
all, and served the same listing to a token restricted by allowed_prefixes.
The exemption was added for health checks before the filer had /healthz and
/readyz. Both are registered on the default and read-only muxes ahead of the
"/" handler and answer without a token, so drop it.
Point the mTLS harness at /healthz, which is what it was probing for.
* filer: keep the jwt query parameter out of a proxied chunk request
The proxy stripped "jwt" from the forwarded query on reads only, on the
grounds that a writer's own credential travels there. It does not: an
uploader carries its AssignVolume token in the Authorization header, and the
query parameter on this path holds a filer credential.
Strip it for every method. A volume server has no business seeing a filer
token, and because security.GetJwt reads the query before the header,
relaying one would hide the writer's own token behind it.
* filer: dispatch the chunk proxy after the JWT gate
The ?proxyChunkId= branch returned before maybeCheckJwtAuthorization ran, so
GET, PUT, POST and DELETE against any needle in the cluster were reachable on
the filer's HTTP port with no filer credential, on a filer where every other
request answered 401. An anonymous caller read a stored object, replaced its
bytes, or deleted the needle, which the master's next vacuum makes permanent.
#10434 stopped the filer from minting a volume write token for that caller,
which closes the write half only where the volume server has a jwt.signing.key
of its own -- not the shipped default, and not what scaffold/security.toml
recommends for a filer deployment. The read half stayed open in every
configuration, because the filer mints the read token itself.
Move the dispatch below the gate. A file id carries no path, so a token
restricted by allowed_prefixes cannot be scoped against one and is refused
here; every consumer of this endpoint holds an unrestricted token.
* filer: mint the volume credential for a proxied write too
The proxy minted a volume token on reads and forwarded whatever the caller
sent on writes. #10434 made it that way because the branch ran ahead of the
JWT gate, so a token minted here would have been signed for an unauthenticated
caller; the branch now runs behind the gate, and the credential the caller
presents there is a filer one, which a volume server cannot validate and has
no business seeing.
Mint at the access level the request needs, and drop the caller's
Authorization when there is no key to mint from. A proxied uploader then needs
only the filer credential, instead of holding one for each hop with a single
header to put them in.
* mount, mq, filer.sync: send the filer credential for a proxied chunk
Every in-tree consumer of ?proxyChunkId= reached the filer anonymously: mount
and the broker put the AssignVolume token in the Authorization header, which
is a volume credential, and filer.sync sent nothing at all. That was enough
only while the branch ran ahead of the filer's JWT gate.
Build the URL through one helper, and pick the credential from the URL it
returns: a chunk proxied through a filer is a request to the filer, which
authorizes it and attaches the volume credential itself, so the token there is
a filer one at the access level the request needs.
* filer: honor -exposeDirectoryData
The flag was declared on all three commands that start a filer and read by
none of them: FilerOption.ExposeDirectoryData was only ever assigned from
filer.expose_directory_metadata in security.toml, so -exposeDirectoryData=false
silently left the listing exposed. Only the TOML key had any effect.
Plumb the flag through and let either switch turn the listing off.
* filer: count a proxied chunk request once
Moving the dispatch below the gate put it after the deferred request
observation, so every proxied chunk now landed in FilerRequestHistogram twice,
once under its HTTP method and once under chunkProxy. Name the deferred one
after the proxy instead, the way the unsupported-method branch already does,
which also gives the endpoint the status codes FilerRequestCounter records.
* filer: do not 404 a TUS session on a transient chunk-load failure
readTusSessionInfo already proved the session exists before
loadTusSessionChunks is called, so a failure there is a read failure,
not evidence the session is gone: a volume-server timeout or a
canceled request context surfaces through ListDirectoryEntries the
same way a missing session would.
Every such error was mapped to writeTusSessionNotFound, answering 404
to HEAD/PATCH and 204 to DELETE. A spec-compliant TUS client trusts
that and discards the session, orphaning every chunk it had committed
until the 24h expiry sweep, or forever if it never issues a DELETE.
Only an error matching filer_pb.ErrNotFound is now reported as not
found; anything else answers 500 so the client retries against the
same session instead of abandoning it.
* test: cover a TUS session's transient chunk-load failure
Adds a listErr hook to the in-memory test store, alongside the
existing commitErr/deleteErr, to simulate a store or RPC failure from
ListDirectoryEntries.
HEAD, PATCH and DELETE against a live session all answer with a
server error instead of a not-found status when the chunk listing
fails transiently, and the session is left on disk untouched. A
listing failure that genuinely means not found, filer_pb.ErrNotFound,
still answers 404 (204 for DELETE).
* filer.remote.sync: skip an upload whose source entry was deleted or rewritten
A replay from an earlier offset (-timeAgo) re-emits create and update
events for entries the filer has since deleted or rewritten. Their chunks
are gone from the volume servers, so the upload can never succeed, and
failing the event holds the sync offset before it: every restart of the
subscription replays it into the same dead chunks, and progress on
everything after it in the log is never persisted. One such entry stops
replication for the whole mount.
When the upload fails, look the entry up on the filer. Gone, or holding
other content than the event described, the event is superseded and is
skipped with an error log; the event that superseded it follows in the
log and brings the remote to the current state. Otherwise the failure
stands and the event is retried as before.
Fixes#11148
* filer.remote.sync: compare chunks by file id when deciding an event is superseded
filer.IsSameData compares chunk ETags, so a delete-and-recreate of
identical bytes, which stores the same content under new file ids and
drops the old ones, looked still as described and kept failing the event
on its dead chunks. Compare by file id with DoMinusChunks, the way the
filer itself decides which chunks an update leaves for deletion: the
event is superseded when the current entry no longer references every
chunk it named, and still as described when it does, including when more
chunks were appended after it.
* filer.remote.sync: ask the filer on the first failed upload attempt, not after the backoff
The superseded check ran after util.Retry had given up, so every dead
entry still cost the full retry cycle, about 13s, before it was skipped:
the SDK reports a missing chunk as "RequestError", which
IsTransientError takes as worth retrying. Move the check into the retry
loop with util.RetryOnError. Any failed attempt asks the filer, and the
loop stops at once when the entry is gone, surfacing errSuperseded for
the caller to skip. An entry the filer still holds keeps the retry policy
it had.
filer.remote.gateway shares retriedWriteFile and the same offset-pinning
processor, so its three call sites skip a superseded event the same way.
* filer: serve "//" paths at the cleaned path instead of redirecting
http.ServeMux redirects a non-canonical path ("//", "..") to its cleaned
form, but since Go 1.22 it builds the Location from the already-escaped
path, so it is percent-encoded twice (golang/go#79897). A client that
follows the redirect re-posts "/负极全景" as "/%25E8%25B4%259F...", and
the filer stores a directory literally named "%E8%B4%9F...".
Wrap the filer muxes in CleanPathHandler, which rewrites the request to
the same cleaned path ServeMux would have redirected to and dispatches
directly. The decoded name reaches the handler, the round trip goes
away, and clients that do not follow redirects work too.
Fixes#11125
* filer: keep RequestURI in step with the cleaned path
PostHandler derives storage rules, the bucket and the read-only check from
r.RequestURI while writing the entry at r.URL.Path. After CleanPathHandler
rewrote only the URL, a "//" or ".." request would be placed by the raw
path and written to the cleaned one. Rewrite RequestURI too, as the
redirect-following client used to.
* filer: match storage rules on the decoded write path
PostHandler resolved the storage rule from r.RequestURI, the raw
request-target. Clients percent-encode non-ASCII segments on the wire, so
a read-only or TTL rule configured on "/data/只读/" never matched a POST
to "/data/%E5%8F%AA%E8%AF%BB/" and the write went through. Use r.URL.Path,
the decoded path the entry is actually written to, as the header-based
destination check already does. The query string no longer reaches the
rule lookup, so the "?" trimming in the read-only error is gone.
2026-09-04 00:02:33 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: keep new volumes and writes off servers in maintenance mode
The master recorded a volume server's maintenance flag from the heartbeat
but never consulted it. A server in maintenance (#7977) is being drained,
yet the master kept creating volumes on it whenever it had free slots and
kept handing out its volumes for writes. Nothing on the volume server
blocks plain HTTP uploads either, so "read-only mode" was only a name.
Volume growth: a data node in maintenance mode reports zero free slots
through AvailableSpaceFor, which takes it out of every candidate list,
feasibility count and capacity reservation. Its slots still roll up into
its rack and data center, so the random offset drawn from those totals for
an other-rack or other-DC replica could land in space the walk then skips
and fail with "No free volume slot found!" while siblings had room; the
walk now folds the offset into the space that is actually eligible. This
also covers the pre-existing case of an over-committed sibling.
Assignment: a replica on a server in maintenance mode is treated like a
read-only replica in isAllWritable, so its volume leaves the writable
list and returns when the flag clears. Topology.SetDataNodeMaintenanceMode
re-evaluates the node's volumes on every change, since heartbeats are
digest-based and a full volume list may not follow for a long time. Reads
and lookups are untouched. The flag moves to an atomic so the assign and
growth paths can read it without the node lock.
Heartbeat: the Go volume server sent its state only when it changed, so a
master elected while a server sat in maintenance never learned about it.
The state now rides along on every heartbeat, as the Rust server already
does; the master's compare is an atomic swap, and only a change does work.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: hold maintenance mode through vacuum commit and mark-writable
SetVolumeAvailable and SetVolumeWritable put a volume back on the writable
list on the replica count alone. A vacuum that started before the server
entered maintenance, or a vacuum worker's mark-writable arriving after it,
handed the volume back to assignment with a replica on the draining server.
Heartbeats carry only changed volumes, so nothing re-evaluated it until the
volume itself changed.
Apply isAllWritable on both paths, the same test EnsureCorrectWritables
uses. Also pin that re-evaluating a volume a concurrent disconnect already
removed from its layout is a no-op.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: record a server's read-only notification on its node before judging the volume
A volume server notifies the master the moment it flips a volume between
read-only and writable, ahead of the heartbeat that repeats the flag. The
layout only set its per-location flag, so isAllWritable, which reads the
node's heartbeat copy, still saw the old value: a mark-writable was
withheld until the next heartbeat, and a re-evaluation landing between a
mark-readonly and its heartbeat put the volume back on the writable list.
Record the flag on the node's volume first. AddOrUpdateVolume keeps the
digest and the active volume count in step, so the heartbeat that follows
finds nothing to change.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: a read-only mark does not confirm a provisional volume
DataNode.SetVolumeReadOnly went through Disk.AddOrUpdateVolume, which
treats its input as a server report and so ended the grace period that
keeps a just-grown volume safe from a full report collected before the
grow. A volume marked read-only before its first report could then be
removed by that stale report.
Give Disk a SetVolumeReadOnly that flips the flag and keeps the digest and
active volume count in step without touching volumeAddedAt.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-03 23:50:12 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.remote.sync: confirm a missing RemoteEntry against the filer before re-uploading
The event is the entry as it was when the update was logged. A chmod or
utimes right after a write is logged while the sync is still uploading the
write, so it carries no RemoteEntry even though the object is on the remote
by the time it is processed. Gating on the event alone turned every such
update into a delete and a second upload of the same bytes; cp -p, rsync
and Django's FileSystemStorage all write that way.
Look up the filer's current entry when the event has no RemoteEntry: the
upload stamps it as soon as it completes, so the stamp is there for the
race and absent for a file that was never replicated. Skip the update when
the entry has since been deleted rather than upload from chunks that may be
gone; the delete event that follows removes the remote object.
Tests build entries from chunks, which is what IsSameData compares in
production, and cover both no-RemoteEntry cases through a stub filer.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.remote.sync: do not delete the remote object before overwriting it in place
The update write path deleted the old object and then wrote the new one,
even when both are the same key. S3, GCS and Azure all overwrite on write,
so the delete bought nothing and left the remote with no object between
the two calls, or at all if the write then failed and pinned the offset.
On a versioned remote bucket it also left a delete marker per rewrite.
Delete only when the key changes, which is what the delete was for.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.remote.sync: trim comments
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-03 19:14:34 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Maintenance mode exists to fence a volume server so it can be evacuated
without taking new writes (#7977), but the gate added in #8115 also
rejected the RPCs evacuation issues against the source: VolumeMarkReadonly
(the first step of every move, and the failure reported in #11066),
VolumeDelete (the last step), and VolumeEcShardsDelete (the last step for
EC shards). volumeServer.evacuate, volume.move and ec.balance therefore
all failed on exactly the server they were meant to drain.
Those three RPCs only remove data or restrict the server further, the same
class as DeleteCollection and the unmount RPCs that were never gated, so
they are exempted from the maintenance check in both the Go and Rust
volume servers. Everything that adds data or reopens the server for
writes (AllocateVolume, WriteNeedleBlob, BatchDelete, VolumeCopy,
ReceiveFile, EC generate/copy/rebuild, vacuum, tiering, VolumeMarkWritable)
stays blocked. A side effect is that scrub can now fence broken volumes
readonly on a server already in maintenance.
Fixes#11066
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
An entry whose content is rewritten unchanged before it first reached the
remote took the metadata-only branch, and UpdateFileMetadata returns early
when the extended attributes match without checking that the object is
there. shouldSendToRemote had already reported the entry as needing to be
sent, so the effect was that it stayed local for as long as its content did
not change, with the sync reporting healthy progress over it.
Require RemoteEntry to be set before treating an update as metadata-only.
Gating at the caller covers the S3, GCS and Azure clients, which share the
same early return.
Fixes#11139
2026-09-03 17:42:40 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Every image-signing job has failed since signing was added (#11129):
must provide --new-bundle-format or --bundle where applicable with
--signing-config or --use-signing-config
Cosign 3 turned on two defaults, not one. The action only disabled
--new-bundle-format to keep the .sig tag layout, but --use-signing-config
is still on, and cosign refuses that pairing because the signing-config
path has nowhere to write its verification material without a bundle.
Disabling it too falls back to the default Fulcio and Rekor URLs, the
same services the .sig layout always used. The verify step needs no
change: cosign verify looks for a referrer bundle first and falls back
to the .sig tag when there is none.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* storage: make DeleteVolume errors inspectable with errors.Is
An absent volume wraps ErrVolumeNotFound and an only-empty refusal now
wraps ErrVolumeNotEmpty with %w instead of %v, so callers no longer have
to match on the message.
Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm
* volume server: return NotFound and FailedPrecondition from VolumeDelete
An absent volume maps to codes.NotFound and a non-empty volume under
only_empty to codes.FailedPrecondition, so a caller retiring a volume can
treat NotFound as already done. The store message is kept in the status
description because the EC empty-replica sweep still matches on it.
Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm
* wdclient: add LookupVolumeIdsAuthoritative
Bypasses the vid map and asks the provider directly, for callers where a
stale positive location is unsafe.
Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm
* filer: add LookupDirectoryEntries batch lookup RPC
Up to 4096 exact-path lookups in one call, resolved concurrently with
results in request order, plus one deduplicated location lookup for every
volume the returned entries reference and per-fid read tokens when the
filer signs reads. unavailable_volume_is_miss lets cache-style callers
take an entry whose volume has no live location as a miss, resolved
against the master rather than the filer's location cache.
Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm
* filer: test that an expired file entry is deleted on read
Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm
* filer: test that AssignVolume and CreateEntry resolve the same TTL rule
Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm
* master: refuse partial lookups while warming up
LookupVolume returned Unavailable during warm-up only when every requested
volume was missing. A batch mixing a reported volume with one whose server
has not reconnected yet came back as a partial answer with a per-volume
not-found, which a caller treating the master as authoritative reads as
gone. Any not-found during warm-up is now Unavailable, which callers
already retry.
Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm
* filer: build batch test requests instead of copying a proto message
Copying a generated message copies its internal mutex, which go vet's
copylocks check rejects.
Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm
* filer: match ErrNotFound with errors.Is and state the miss rule's contract
A wrapped not-found from the store would otherwise be reported as an
error rather than a miss. The comments now say why a nil location map is
the only sign of an unanswered lookup: the provider returns nil when it
got no answer and a populated map, with unserved volumes reported as
errors, when the master did answer.
Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm
* volume server: map absent and non-empty VolumeDelete errors in the Rust server
Matches the Go server: an absent volume is NotFound and an only_empty
refusal is FailedPrecondition instead of Internal, with the messages the
EC empty-replica sweep matches on.
Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm
* filer: test that a malformed entry keeps its error outside cache mode
Same test file as the enterprise tree, so the next sync sees one version.
Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm
As an untyped constant it became int when passed to Infof, which
overflows on linux/386 and failed the 32-bit vet job. Every field it is
compared against is already uint64.
Claude-Session: https://claude.ai/code/session_015rYAmF8hV9yypb9yvy4A1z
* telemetry: tidy the server module after the protobuf bump
Claude-Session: https://claude.ai/code/session_01VGiphDxpsKMwu9XpUFhybQ
* telemetry: keep only clusters that store at least 10 GiB
Fresh weed server runs, CI jobs and throwaway containers each mint their
own cluster id. They came in at tens of thousands a day, were most of
the counted clusters and held almost none of the bytes, and the state
file and the metrics page grew with every one of them. Reports under
the floor are counted and dropped, and a state file written before the
floor sheds them on the first restart.
Claude-Session: https://claude.ai/code/session_01VGiphDxpsKMwu9XpUFhybQ
* master: report telemetry only once the cluster stores 10 GiB
A throwaway cluster no longer registers itself with its first report a
minute after start; a real one begins reporting at the first daily tick
after it crosses the floor.
Claude-Session: https://claude.ai/code/session_01VGiphDxpsKMwu9XpUFhybQ
* ci: composite action that signs and verifies an image with cosign
Keyless, by digest, with a verification pass against the calling workflow's
own identity right after signing. Signatures use the .sig tag layout rather
than the OCI-referrer bundle cosign 3 writes by default, since that is what
the verifiers people run today read. Dependabot is pointed at the action so
the cosign-installer pin keeps moving.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker release: sign every variant on both registries
The merge job signs each variant's multi-arch index on GHCR and Docker Hub
once the tag exists, recursively so the platform images are covered too.
latest re-tags the same manifest and inherits the signature.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker dev: sign the dev image
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker latest: sign a latest rebuilt by hand
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker release: sign the foundationdb image
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker: sign the per-version foundationdb and rocksdb builds
They push to the same repository as the releases, so an admission policy
that verifies chrislusf/seaweedfs would otherwise reject them.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker: document image signature verification
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* ci: pin the actions the signing jobs newly run by commit
These run with registry credentials and the OIDC token that signs under
the repository's identity, so a retargeted tag upstream must not be able
to reach them.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker latest: pass the dispatch tag through env, not the script
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker: complete Kyverno policy, digest note, identity scope
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker latest: keep the dispatch tag out of the manifest script too
The step predates signing, but the job now holds the OIDC identity.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* ci: pin every action in the jobs that sign
The jobs that hold the OIDC identity run these with registry credentials,
so a retargeted tag upstream must not reach them.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker release: copy and sign the digest the run created, pin the rest
crane copy and the signature both resolved the tag, which another
publisher could move between the two steps. The index digest is read once,
right after it is created, and the Docker Hub copy and both signatures use
it. The manual latest rebuild gets the same treatment. The actions in these
jobs are pinned to commits, crane to v0.22.0 by checksum, and the sparse
checkout no longer keeps the token.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker latest: the signing job checks out the workflow's own commit
The job only assembles and signs manifests, so nothing there needs the
source_ref checkout; the local signing action now comes from the same
revision as the workflow file that calls it.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker release: take the index digest from the create result
imagetools create writes the descriptor it pushed with --metadata-file
(buildx 0.32+, the runners ship 0.36), so the digest no longer comes from
re-resolving the tag even within the same step.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* http: try a volume server that failed to answer last
A cached location list is shuffled on every read, so once a replica dies
half the reads keep dialing it first and pay a connect failure or timeout
before the healthy replica answers. Remember, per host, when a request got
no answer at all and order such hosts last for the next half minute. Once
that passes, one read probes the host in its usual place while the others
keep it last until the probe settles, so a black-holed server costs one
stalled read per interval instead of one per read.
Nothing is ever skipped: a host that failed is still tried when the others
fail too. Any response, including an error status, counts as reachable.
Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs
* filer: refresh a chunk's locations after one of them fails
A mount's location cache is only relearned when every cached location
fails. When one replica dies and the other still answers, every read
succeeds and the dead replica stays in the cache, and in the shuffled
order it keeps being dialed first long after the master has dropped it.
When a read fails on one location and a later one answers, call the
refresh hook so the cached entry is dropped and looked up again. The read
that already paid for the failure returns its data; the reads after it
start from the locations the master knows now.
Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs
* http: claim the probe for every expired host, and try it first
The claim was only checked for the first url, so with two replicas whose
marks expired together the second was probed by every read at once. Claim
each expired host on its own and put the reads that won a claim ahead of
the reachable hosts, so a probe is always a real attempt and a lost claim
always means the host is tried last.
Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs
* filer: refresh a chunk's locations in the streaming read path too
The streaming loop had no refresh hook, so a manifest or streamed chunk
that failed on one cached location and was served by another kept the
stale entry until every location failed. Give it the same hook as the
buffered loop, built by one refreshUrls function shared by the reader
cache and the stream callers.
Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs
* http: probe at most one expired host per read
Claiming every expired host in one ordering left all but the first claim
without an attempt, since a read stops at its first answer, and a host that
had come back waited another interval for nothing. Claim only the first
expired host a read sees and leave the rest last and unclaimed, so each
following read probes one of them.
Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs
* test: start the live server before releasing the dead server's port
Closing the dead server first let the live server come up on the same
port, in which case the dead location answers and the partial failure
under test never happens.
Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs
The cached-location test needs two files on one volume, but it wrote six
files into the six volumes a 001 layout starts with, and every so often
each file landed on its own volume and the test had nothing to probe.
Seven files leave no way to spread them out.
Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs
* helm: values-driven labels on every ingress
Each ingress already takes annotations from values, but its labels were
a fixed block, so tools that select ingresses by label (ExternalDNS
label filters, for one) had nothing to key on. Every ingress block now
has a labels map rendered after the standard app.kubernetes.io labels,
including the Traefik IngressRouteTCP that shares the filer gRPC values.
Claude-Session: https://claude.ai/code/session_01L6eJGXtYkwe1W9QjGeUgr1
* helm ci: render check for ingress labels
Claude-Session: https://claude.ai/code/session_01L6eJGXtYkwe1W9QjGeUgr1
weedfs_stream_mutate_error_test.go names its *streamMutateError local
sme, which codespell reads as a misspelling of same/some. It is an
identifier, so exempt it beside the other variable-name entries.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: rebuild a missing .idx from the .dat
Pointing -dir.idx at a directory that holds no index aborted the whole
volume server: checkIdxFile found no .idx and load() called glog.Fatalf.
Every row of the index is derivable from the .dat, so walk it in append
order and write the index back, which reproduces byte for byte what the
server's own writes had left in the old directory.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: keep the index co-located with the data in the Rust server
Go's load() drops back to the data directory when an .idx already sits
beside the .dat, so naming a --dir.idx does not strand a pre-existing
index. Rust had no such adjustment: it opened the new directory with
create, and the volume came up on an empty index with every needle
invisible.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: rebuild a missing .idx from the .dat in the Rust server
Mirrors the Go side. Rust did not abort on a missing index the way
checkIdxFile did; it opened the new directory with create and mounted the
volume on an empty index, so every needle read as missing while the .dat
still held the data. Walk the .dat in append order and write the index
back, byte for byte what the server's own writes had left behind.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: stop the idx rebuild at a zero-padded .dat tail
An all-zero needle header is unwritten space, not a record. Go's .dat walk
keeps reading past it and would index a truncated data file's tail as
millions of needle 0 rows; the Rust walk already stops there. Stop the Go
rebuild at the same place.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: create the -dir.idx directory when it does not exist
Rust's DiskLocation creates the index directory as it takes it; Go only
resolved the path, so naming a directory that does not exist yet left every
volume unable to open or rebuild its index and took the server down.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: stop the idx rebuild at a torn .dat record
A crash between writing a needle's header and its body leaves a record
whose declared size runs past the end of .dat. Indexing it puts a row in
the .idx that points at bytes that do not exist, which fails every read of
that needle and trips the past-EOF check on the next load. Stop at the
first record that does not fit, in both servers.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: stop the idx rebuild at a negative-size header
A corrupt header whose size field is negative makes the .dat walk advance
backwards: NeedleBodyLength adds the negative size, so the next offset is
lower than the current one. The Go walk then reads at a negative offset and
the rebuild fails, which puts the volume server right back to exiting at
startup; the Rust walk seeks past EOF and truncates the index instead.
A negative size is never a record, so stop there.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: skip a volume whose index cannot be rebuilt, do not exit
glog.Fatalf calls os.Exit(255), so a rebuild that could not write -- a full
or read-only index directory -- put the server right back to dying at
startup for one bad volume. Return the error instead: loadExistingVolume
logs it and skips that volume, which is what the remote-volume branch just
above already does and what the Rust loader has always done.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: create the index directory from the rebuild too
The rebuild is the first thing to write into a fresh -dir.idx, and it runs
before the loaders that create the directory on their way to opening .idx.
Create it in both rebuilds so the ordering does not matter.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* ci: let codespell past the sme variable in the mount tests
weedfs_stream_mutate_error_test.go names its *streamMutateError local
sme, which codespell reads as a misspelling of same/some. It is an
identifier, so exempt it beside the other variable-name entries.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
-dir=/data fails on macOS, where the root filesystem is read-only, and
on any Linux box without root; -dir=./data is created on the spot.
go install of the weed package is refused because go.mod carries
replace directives, so the install script is the shortcut instead.
Claude-Session: https://claude.ai/code/session_014apMEkkquAtAYp89paTkAT
The Helm values put filer metadata on a claim through filer.data, which
is what the chart reads; enablePVC was rendering a hostPath. Claims use
the cluster default storage class instead of local-path. The AWS CLI
test carries its own credentials, the compose download includes the
Prometheus config the compose file mounts, and the disk-read claim is
per blob, since large files are chunked.
Claude-Session: https://claude.ai/code/session_014apMEkkquAtAYp89paTkAT
* filer: give the SQL stores' key-value reads their own connections
A listing holds the connection its rows are on for the whole iteration, and
FilerStoreWrapper calls maybeReadHardLink -> KvGet from inside that iteration,
so a hard-linked entry needs a second connection while the first is still busy.
Out of one bounded pool that is a deadlock: the listings fill the pool and then
wait for a connection none of them will release, and the wrapper's
context.WithoutCancel leaves the waiters without a deadline, so the filer stays
wedged rather than erroring.
The sqlite store shows it at its sharpest -- it allows a single connection, so
one listing over one hard-linked entry never returns. On postgres with
connection_max_open = 50, 60 concurrent listings over hard-linked entries made
no progress at all.
Key-value reads now run on their own pool, carved out of connection_max_open
rather than added to it, so the operator's cap still bounds what the store opens
against the database. An unbounded pool keeps a single pool: nothing can wait
there. sqlite's single connection becomes two, one per pool, and its writes get
a busy timeout so a write that meets the reader waits instead of failing.
Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t
* sqlite: keep both pools on one database, whatever the dbFile spells
A dbFile that already carries URI options got a second "?" appended, which the
driver reads as part of the preceding option value, and a bare :memory: is
private to each connection, so the key-value pool would open its own empty
database and every key-value operation would fail on a missing filemeta.
Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t
* sqlite: assert the busy timeout on the in-memory DSN too
Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t
* rdma: drop the sidecar prototype
The Rust engine under it never touched a wire: rdma.rs fabricates pattern
bytes and the crate's default feature is mock-ucx, with real-ucx unimplemented
since the directory landed. Nothing builds it, no CI runs it, and its only
consumer is weed mount's RDMA client, removed next. Two 22MB binaries were
committed along with it.
Claude-Session: https://claude.ai/code/session_01X3zhqLYwQwEQCrRbuzQKvy
* mount: remove the RDMA client that spoke to the deleted sidecar
Its only server was the sidecar's HTTP API, and the path could never have
worked in production anyway: it served a single chunk per call, ignored the
buffer's chunk boundaries, and had no test. Removing it also removes the
per-handle cumulative-offset cache, which nothing else used.
The -rdma.* mount flags go with it. They defaulted to off and pointed at an
address no released build ever listened on.
Claude-Session: https://claude.ai/code/session_01X3zhqLYwQwEQCrRbuzQKvy
chrislusfandChris Lu
franchbandClaude Fable 5.1
Chris LuandGitHub
Alex KandGitHub
