The queue holds sixteen sealed windows, which is a memory bound only
while a window is BufferSize. An entry larger than that grows its window
to fit, and the depth then multiplies straight through: sixteen queued
copies of a 100 MB window is 1.6 GB of flush data alone.
Account the queued bytes and make producers wait once they pass the
ceiling the depth was chosen for. What is charged is the pooled slab
rather than the window length, since mem.Allocate rounds up to a size
class and the queue holds the whole slab. A window larger than the whole
budget still goes through on its own, so an oversized entry is never
stuck.
Windows are admitted in the order they were sealed. A producer can now
park here for seconds, and letting a later window overtake an earlier one
would persist them out of order and walk lastFlushedOffset and
lastFlushTsNs backwards.
A window is copied into its slab under the write lock, before the
reservation is taken, so a burst of concurrent oversized writers would
each hold a full copy in hand while queueing up -- memory the budget
never sees. Large writers wait for queue headroom before they take the
lock, which throttles the burst; it does not bound it, since a writer
that passes the check still seals unconditionally.
Take any room in the queue before the shutdown escape, too: the window is
already sealed by then, so dropping it loses records the caller was told
were accepted. A shutdown that races a full queue can still drop one --
that predates this change and needs the flush loop's lifetime reworked.
Size a grown window to the entry rather than to twice it: the extra room
only bought space for a second oversized record in the same window, which
doubles the flush copy and the snapshot taken of it. The overflow guard
halved its bound for that doubled allocation, so raise it to match what
is now allocated and what maxBufferSize documents.
* filer: write the metadata log in pieces a volume server will accept
A single oversized metadata event grows the log buffer past the volume
server's fileSizeLimitMB, and the flush of that buffer is then rejected
forever: the retry loop has no exit, so the blob at the head of the queue
blocks every later flush and the metadata feed stalls until restart.
Split the flushed buffer into BufferSize pieces, on record boundaries
where possible so each piece still decodes on its own, and retry each
piece separately so a partial success is not replayed. Log files are
already read as a chunk stream, with a whole-file fallback when a chunk
does not decode standalone, so a record may cross a piece boundary.
* log_buffer: let go of a window array grown for an oversized entry
An entry larger than BufferSize grows the window array to 2*size+4, and
window arrays cycle through SealBuffer rather than being freed. One such
entry therefore leaves every later window carrying its size, and
currentSnapshotView allocates a snapshot as wide as the array on each
window, so a few KB of metadata keeps paying for it.
Drop the array when SealBuffer hands it back. Growth is on demand, so
the next oversized entry just reallocates.
* iceberg maintenance: store merged data files as chunks, not inline
saveFilerFile had no size threshold, so compaction wrote whole merged
parquet files -- hundreds of MB -- as Entry.Content. That puts the
parquet bytes verbatim in the filer store and sends them through the
metadata change log again as one event.
Keep manifests and metadata JSON inline, upload anything larger to
volume servers in chunks, assigning through the filer so the path's
storage rules apply.
* filer: follow the file size limit the volume servers report
The starting piece size is a constant, so a cluster whose
-fileSizeLimitMB is set below it would reject every piece and wedge just
the same. The rejection names the limit, so take it from there and
re-cut the rest of the flush to fit.
Piece the buffer one at a time rather than up front, since the size can
change partway through a flush.
* util: match transient error messages case-insensitively, expose the message form
The same condition reaches different layers capitalized differently -- a
volume server relays its idle timeout as "I/O timeout" inside a JSON string --
and the callers that grew their own substring lists all lower-case first.
Also split out IsTransientErrorMessage for the paths that carry only the text,
such as the per-file status strings in a batch delete response, and pick up
"no route to host" and "network is unreachable" from the gRPC classifier.
* filersink: classify transient network errors through util.IsTransientError
The local list caught i/o timeout, connection reset, and broken pipe but not
connection refused, no such host, unexpected EOF, the syscall errnos, or the
gRPC and S3 overload codes. Keep only the bare io.EOF case, which is transient
here -- a truncated chunk read -- but a clean stream end elsewhere.
* filer deletion: reuse util.IsTransientErrorMessage for the network patterns
Six of the sixteen patterns were already covered. Keep the ones specific to
this pipeline -- read-only volumes, lookup failures, backpressure -- and note
why context cancellation stays retryable here: it decides whether to requeue
the deletion, not whether to retry a call.
* wdclient: fold the shared classifier into the volume lookup retry check
The string tail duplicated the shared list and missed the syscall errnos and
net.Error timeouts. Keep "connection" and "timeout", which are broader than
the shared classifier on purpose: a volume lookup is a cheap read-only call.
* util: retry transient errors, not just the ones containing "transport"
util.Retry only retried when the error string contained "transport", so a
plain "read: connection reset by peer" from S3 got zero retries. Classify
the error instead: net timeouts, connection resets, and the throttling and
overload replies S3 and gRPC return are all worth another attempt, while a
cancelled or expired context is not.
* filer sync: hold the sync offset behind a failed event
A sync job that returned an error was logged and forgotten, and the
watermark advanced past it anyway. The offset is the durable resume point,
so the event was never replayed: for filer.remote.sync that left the file
present locally, absent on the remote, with no RemoteEntry and nothing to
retry it.
Pin the watermark at the oldest failed event. Later events keep flowing,
but the persisted offset stays behind the failure, so a restart replays it.
* Add GitHub Actions workflow for codespell on master
* Add rudimentary codespell config
* Tune codespell config: skip generated code, ignore camelCase, whitelist domain terms
Add camelCase/PascalCase regex to ignore common Go/Rust/JS identifiers
like allLocations, publishErr, ReadInside, FlushInterval. Also skip
templ-generated *_templ.go files, and whitelist a handful of
short/domain-specific words (visibles, fo, te, ser, bject, unparseable,
keep-alives, tread, anc, ue) that show up as false positives across the
tree.
Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Fix ambiguous typos and protect false positives
Fixes typos that codespell reports with multiple candidate suggestions
(so `codespell -w` cannot auto-apply them), plus one inline pragma and
one config entry to protect legitimate identifiers.
Manual fixes (single correct answer chosen from context):
- pattens -> patterns (5x) in filer/upload/shell flag help strings
- finded -> found (2x) in tarantool storage.lua comment
- spacify -> specify (2x) in helm chart values.yaml comment
- wether -> whether in skiplist.go docstring
- simpe -> simple in mq schema test case name
False-positive protection:
- Add `//codespell:ignore` next to `source GET's` (possessive of HTTP
verb) in s3api_object_handlers_copy_stream.go
- Whitelist `auther` in .codespellrc — it's a local variable meaning
"authenticator" in weed/security/tls.go, not a typo of "author".
Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Extend codespell ignore list: .git-meta path and thirdparty groupId
Also skip `.git-meta` (scratch dir for commit messages that may contain
typo words verbatim) and whitelist `thirdparty` — it appears as the
literal Maven groupId `org.apache.hadoop.thirdparty` in hdfs3 poms
and cannot be renamed.
Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* [DATALAD RUNCMD] Fix non-ambiguous typos with codespell -w
Auto-applied fixes to the 44 remaining single-suggestion typos across
docs, comments, log messages, tests, config, and one Java pom.
=== Do not change lines below ===
{
"chain": [],
"cmd": "uvx codespell -w",
"exit": 0,
"extra_inputs": [],
"inputs": [],
"outputs": [],
"pwd": "."
}
^^^ Do not change lines above ^^^
* Revert breaking codespell fixes; whitelist unknwon and atleast
Two of the auto-applied `codespell -w` fixes were false positives that
would break the build/tests:
- go.mod: `github.com/unknwon/goconfig` is a real Go module path — the
upstream author's GitHub handle is literally `unknwon`. Renaming to
`unknown` would fail dependency resolution.
- test/benchmark/fuse_db/bin/{sqlite_verify.py,run_mysql.sh,run_sqlite.sh}:
`atleast` is a literal CLI mode value (a string constant compared and
passed as a positional argument). Rewriting to `at least` splits it
into two arguments and breaks the mode check.
Reverted those files and whitelisted both words in .codespellrc so
future runs won't re-suggest the same broken fixes.
Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* generate vtproto marshalers for filer_pb and use them on the metadata log path
Reflection-based proto.Unmarshal allocates a fresh message tree through
reflect.New on every call. On the metadata subscription fan-out the same
event is decoded once per subscriber, so reflect.New tops the decode
churn under many mounts.
Generate MarshalVT/UnmarshalVT/SizeVT for filer.proto (a separate
filer_vtproto.pb.go, filer.pb.go untouched) and call them on the log
entry marshal and the subscribe/replay decode paths. UnmarshalVT
allocates message structs directly and copies byte and string fields, so
it stays wire-compatible with proto.Unmarshal and preserves the
non-aliasing the persisted-log cache depends on.
For SubscribeMetadataResponse this cuts decode allocations 69 -> 50 and
~4.5us -> ~2.1us per event; the win scales with subscriber overlap.
* marshal log entries directly into the buffer
SizeVT is allocation-free and MarshalToSizedBufferVT writes into a
pre-sized slice, so the log entry can be marshaled straight into
logBuffer.buf. This drops the per-entry MarshalVT allocation and the
follow-up copy on the write path.
* expand vtproto benchmarks: marshal, decode, and marshal-into-buffer by chunk count
Parametrize by nested-message count (chunks per event) and add encode +
zero-alloc marshal-into-buffer benchmarks alongside the decode one, so
the write-path win from MarshalToSizedBufferVT is measurable too.
* keep proto.Unmarshal for metadata events to preserve UTF-8 validation
UnmarshalVT skips proto3's UTF-8 validation of string fields, so a
SubscribeMetadataResponse with an invalid-UTF-8 string (e.g. Directory
"\xff") that proto.Unmarshal rejects would decode and reach path
filtering and subscribers. Decode events with proto.Unmarshal again;
UnmarshalVT stays on the log entry paths, whose only variable-length
fields are bytes and so carry no UTF-8 constraint.
Tests cover the codec difference and that a malformed event is skipped
before delivery.
* route log buffer flush copies through the shared slab pool
Each flush copied the sealed window into a bytes.Buffer drawn from a
package-local sync.Pool. GC drains that pool, so once collections run
often (e.g. under GOMEMLIMIT) most flushes miss and Write grows a fresh
window-sized array, the dominant bytes.growSlice source under write load.
Copy into a size-classed slab from weed/util/mem instead. Slabs are
reused process-wide and returned to their exact size class after the
flush, so variable-sized flushes across many partitions stop churning
mismatched buffers.
* guard nil slab in flush releaseMemory
mem.Free(nil) resolves to the smallest slot pool and stores a zero-cap
slice, so a later mem.Allocate could return nil and panic. The current
call sites never pass nil, but the guard keeps a double release harmless.
* s3: accept raw semicolons in query strings
Go's url.ParseQuery drops any key=value pair containing a raw ';'. A
presigned PUT that signs content-type carries
X-Amz-SignedHeaders=content-type%3Bhost; when a client or proxy decodes
the %3B, the parameter vanished and the upload failed with
MissingFields, while AWS accepts the raw ';' as query data. Re-encode
it before routing so the pair survives parsing and signature
verification.
* iam, iceberg: recover raw-semicolon query pairs on the other listeners
The standalone IAM API verifies SigV4 with a canonical query recomputed
from the parsed query, and Iceberg REST warehouse/parent values may
legally contain ';'. Move the normalization middleware to util/http and
attach it to both routers.
TestSharedSnapshotConcurrentIntegrity writes as fast as one core can
marshal for two seconds, so its transient heap scales with host speed.
On linux/386 a fast runner walks the heap into the 4GB address-space
ceiling and the run dies with out of memory. Set a 1GB memory limit for
the duration of the test so the GC paces the writer instead of the
address space; the seal-and-recycle race being tested is unaffected.
* log_buffer: share one window snapshot across all subscriber reads
Every in-memory read handed each subscriber a private pooled copy of the
window it wanted, so N subscribers reading the same data cost N copies of
up to 8MB each -- and slow consumers (grpc send backpressure) held those
copies live for their whole iteration. With hundreds of mount subscribers
that multiplied into gigabytes of live heap on the filer.
Share the bytes instead of copying per reader:
- Sealed windows get a lazily created GC-owned snapshot, made once by the
first reader and handed out zero-copy to the rest. The snapshot travels
with its window when SealBuffer shifts slots, so recycling the sealed
array never invalidates it.
- The current window keeps a shared snapshot of its append-only prefix
buf[:pos], extended on demand; each byte is copied once per window
(writer-rate-bound) instead of once per reader. At seal a fully
extended prefix becomes the sealed window's snapshot.
ReadFromBuffer now reports whether the returned buffer is a pooled copy
(flush path) or a shared view that must not be released; the read loops
only recycle pooled buffers.
With 200 subscribers consuming at grpc pace over sealed and current
windows, peak live heap drops from 5.2GB to 178MB.
* log_buffer: clear released read buffer so a panic cannot double-free it
The read loops release the previous iteration's pooled buffer and then
call ReadFromBuffer. If that call panicked before reassigning bytesBuf,
the deferred cleanup would put the same buffer into the pool a second
time, letting two future readers share one backing array. Nil the
pointer at the release site so the defer sees nothing to free.
* log_buffer: read only ts_ns during metadata-read binary search
readTs ran a full proto.Unmarshal of each LogEntry just to compare
timestamps during ReadFromBuffer's binary search, allocating fresh
slices for the data and key byte fields on every probe. Under
metadata-subscription fan-out this decode churn dominated allocations
and drove heavy GC, inflating filer RSS well past the live heap.
Scan the wire format for ts_ns (field 1) instead and skip the data/key
payloads without copying them. readTs is now zero-alloc; the binary
search no longer touches the event payloads at all.
* log_buffer: alias event payloads instead of copying them on delivery
The subscribe read loops decoded each LogEntry with a full proto.Unmarshal,
allocating fresh slices for the data and key byte fields on every entry
(protobuf consumeBytesNoZero). When many metadata subscribers each re-read
the in-memory log window under a reconnect storm, that per-entry copy was a
large share of allocation churn -- and for entries a subscriber filters out
by prefix, the copied payload was never even looked at.
Decode with a wire scan that points data/key at sub-slices of the source
buffer instead of copying. The aliased slices are valid only for the
duration of the eachLogEntryFn callback; every current subscriber either
re-decodes the event into its own struct or hands it to a synchronous grpc
Send, so none retain them. Per delivered entry the loop decode drops from
176 B / 2 allocs to zero.
With replication 001 and one node down, ~2/3 of volumes have their
replica on the dead node. The primary write lands locally but the
replica forward fails, so the volume returns 500 "failed to write to
replicas". #9744 made that upload fail fast so the client re-assigns,
but the client retry never fired: the reassign gate matched a fixed
list of error substrings that didn't include this one. The mount
surfaced I/O error and dropped the chunk, leaving missing lines and
null-byte gaps in an append workload while a node rebooted.
Decide reassignment by HTTP status instead of matching message text.
On the write path a volume server only 5xxs on a ReplicatedWrite
failure (local disk, replica peer down, under-replication) — all of
which a different volume dodges — so any 5xx reassigns; a no-response
transport failure (the assigned target itself is down) reassigns too;
a 4xx is a genuine client error and is surfaced. doUploadData tags its
errors with the response status via uploadStatusError, and the gate
moves from util.MultiRetry(errList) to util.RetryOnError(predicate).
This drops the fragile substring list (and the message-prefix
constants and guard test it needed); store_replicate.go is untouched.
A read or a replicated write to a volume server that is TCP-reachable but not
answering -- one still loading its volumes after a restart, or reached over a
stale keep-alive to a container that came back on a new IP -- blocked forever:
the shared HTTP transport had a dial timeout but no response timeout.
Add ResponseHeaderTimeout so a chunk read fails over to another replica and a
replicated write fails fast for the client to retry, and IdleConnTimeout so
pooled connections to a departed server are evicted instead of reused.
* util: don't let the activity timeout clobber externally-set conn deadlines
util.Conn extended the connection deadline at the start of every
Read/Write. net/http's server also sets deadlines directly on the same
conn - abortPendingRead sets one in the past to interrupt the pending
background read after each response. The activity extension raced with
and silently overwrote that interrupt, leaving the read blocked (and the
server's conn.serve goroutine stuck in abortPendingRead) until the full
-idleTimeout (default 30s) expired.
Wedged connections count as active, so the volume server's graceful HTTP
drain waited out its whole 30s StopTimeout on shutdown - observed as
weed mini taking ~30s to exit in the FUSE integration tests after any
filer->volume traffic.
Track externally-set deadlines, suspend the activity extension while one
is in force, and serialize deadline updates with a mutex. Activity still
extends both directions at once: a long write-only response must keep
the read deadline alive too, or the server's background read would time
out and cancel the in-flight request.
* util: extend read/write deadlines independently when one side is external
A server-configured WriteTimeout keeps an external write deadline in
force for the whole request, which previously suspended the activity
extension entirely - leaving the read deadline stale from before the
request and letting net/http's background read time out mid-response.
Extend each direction independently instead.
Introduce security.BearerPrefix ("Bearer ", RFC 6750) and use it
everywhere an "Authorization: Bearer <token>" header is constructed,
replacing the scattered "BEARER "/"Bearer " string literals. SeaweedFS
matches the scheme case-insensitively when parsing (security.GetJwt), so
behavior is unchanged; this removes the magic string and settles the
casing on the standard form. The parser's upper-case comparison stays as
is on purpose.
* fix(chunk_cache): close data/index files on initialization error
* chunk_cache: assign outer err on the .dat open path
The error-path defer keys off the function-level err, but the .dat
OpenFile used := and shadowed it, so that path relied on nothing being
open yet rather than the cleanup invariant. Assign the outer err so
every error return is uniform.
* chunk_cache: verify descriptor closure on POSIX, not just Windows
os.Remove succeeds on open files on Linux/macOS, so the removal check
only proved closure on Windows. Compare the open-fd count before and
after the failed load; gate the removal check to Windows.
---------
Co-authored-by: Contributor <contributor@example.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
util/http: lazily init the global HTTP client
GetGlobalHttpClient returned a nil client until InitGlobalHttpClient ran,
which only happens in weed.go's main. Anything that starts a command
in-process bypasses that: the admin server's metrics goroutine seeds a
dashboard sample on startup, reaching fetchPublicUrlMap -> GetGlobalHttpClient().Do,
and nil-derefs the receiver in GetHttpScheme.
Init the client on first Get via sync.Once so it is never nil regardless of
the startup path. InitGlobalHttpClient keeps its eager-init role through the
same Once.
* fix(mount): run entry invalidations off the meta-cache apply loop
The apply loop ran invalidateFunc inline, which acquires the open file
handle's lock in fhLockTable. Meanwhile flushMetadataToFiler holds that
same fh lock and then waits on the apply loop (applyLocalMetadataEvent).
When both target the same open file concurrently, the loop blocks on the
fh lock while the lock holder blocks on the loop: an ABBA deadlock that
backs up every later readdir/flush and hangs the mount.
Fix: dispatch entry invalidations to a dedicated FIFO worker goroutine so
the apply loop never blocks on locks held by goroutines waiting on it.
Adds a regression test reproducing the interleaving.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* perf(mount): update invalidate counter once per batch
Run the batch's invalidateFunc calls without re-taking invalidateMu per
item, then bump invalidateProcessed and broadcast once after the loop.
WaitForEntryInvalidations only needs the count to reach its target and a
batch always completes together, so the per-item lock + broadcast was
wasted work.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* mount: extract the invalidate worker into util.AsyncBatchWorker
The apply loop's off-thread entry-invalidation queue was a one-off mutex +
cond + slice + counters living inside MetaCache. Pull it out as a generic
unbounded FIFO worker so the deadlock-avoidance contract (never block the
producer, drain on shutdown, wait-for-quiesce) lives in one place.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix(ec): check decode .idx writes and fsync decoded .dat/.idx
WriteIdxFileFromEcIndex silently dropped io.Copy and Write errors, so a
short or failed write of the reconstructed .idx went unnoticed and the
caller proceeded to delete the source EC shards. Propagate those errors.
Also fsync the decoded .dat and .idx before returning, so the bytes are
durable before the shards that produced them are removed cluster-wide.
Mirror the .idx fsync into the Rust volume server (its .dat already
syncs and its writes already propagate errors).
* fix(ec): publish decoded .dat/.idx atomically via temp file and rename
WriteDatFile and WriteIdxFileFromEcIndex wrote in place at the final
name with O_TRUNC. A crash mid-write left a truncated .dat/.idx at the
final name beside the still-present EC shards; on restart that partial
file could be mounted as the live volume even though the shards held the
real data. Write to a .tmp file, fsync it, then rename into place and
fsync the directory, so the final name is only ever absent or complete.
A failed decode removes its own temp file rather than leaking it.
Add util.FsyncDir as the shared directory-fsync primitive and reuse the
Rust volume server's fsync_dir for the mirrored change.
* fix(ec): propagate .ecj read errors in the Rust decoder
Path::exists returned false for any error (permission denied, transient
IO), silently skipping the deletion journal and resurrecting deleted
needles as live. Read the journal directly and treat only NotFound as
absent, propagating other errors. The Go decoder already behaves this
way (FileExists returns false only for IsNotExist, then the open
surfaces other errors).
* fix(ec): remove rename destination on Windows in the Rust decoder publish
std::fs::rename does not replace an existing file on every Windows
version. Remove the destination first under a Windows guard before the
atomic publish rename, matching the compaction commit path.
* fix(util): ignore comment only sql input
Problem: sqlutil.SplitStatements strips SQL comments while scanning, but when no statements remain it falls back to returning the original query. Inputs that contain only comments are therefore reported as executable SQL statements.
Root cause: The no-statements fallback did not distinguish a real single statement from input that had been fully removed by comment filtering.
Fix: Remove the original-query fallback and return an explicit empty slice when scanning produces no statements.
Reproduction: env GOCACHE=/private/tmp/seaweedfs-go-cache go test ./weed/util/sqlutil -run TestSplitStatements -count=1 failed before the fix because comment-only inputs returned the comment text as a statement.
Validation: gofmt -w weed/util/sqlutil/splitter.go weed/util/sqlutil/splitter_test.go; env GOCACHE=/private/tmp/seaweedfs-go-cache go test ./weed/util/sqlutil -run TestSplitStatements -count=1; env GOCACHE=/private/tmp/seaweedfs-go-cache go test ./weed/util/sqlutil -count=1; git diff --check; git diff --cached --check.
Duplicate check: Searched /private/tmp/seaweedfs-codex0610-old-branch-index.tsv and existing tests for sqlutil, SplitStatements, comments, and comment-only. Old PostgreSQL query branches cover malformed wire frames and SQL engine numeric parsing, not comment-only statement splitting.
Co-authored-by: Codex <noreply@openai.com>
* Update weed/util/sqlutil/splitter.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
---------
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
The eachLogDataFn error path printed the full LogEntry proto. For an
entry carrying a large chunk manifest that is hundreds of KB of escaped
bytes in a single log line, burying the actual error -- often just a
subscriber disconnect -- at the very end. Log the key, timestamp,
offset and data size instead.
* Bound the metadata-log flush queue
A stalled flush, e.g. slow volume servers under a reconnect storm, let up
to 256 queued 8MB buffer copies pin two gigabytes per log buffer while
producers kept filling the queue. Cap the queue at 16 so a sustained
stall backpressures writers instead of growing the heap. The flush
goroutine never feeds back into the buffer (system-log paths skip event
notification), so blocked producers cannot deadlock the consumer.
* Don't drop a force-flushed buffer on a full queue
ForceFlush enqueued with a two-second timeout, but by then the live
buffer was already sealed and reset, so a timed-out send silently lost
the copy. Block until the flush is queued; the wait for completion stays
bounded since the data is durable once the flush loop drains it.
* Never close the flush channel
ShutdownLogBuffer closed flushChan while producers could still be
blocked sending into it, which panics. Terminate loopFlush with a nil
sentinel instead, so the channel is never closed, and give every
producer-side send a shutdown escape so none parks forever once the
flush loop exits. Everything queued before the sentinel still drains,
preserving IsAllFlushed semantics.
* Copy the shutdown flush under the buffer lock
Every other copyToFlush call site holds the lock; the shutdown path read
the live buffer unlocked while producers could still be appending.
* fix(http): accept no content delete responses
Problem: util/http.Delete reports an error for a successful HTTP 204 No Content response.
Root cause: Delete only treats 200 OK, 202 Accepted, and 404 Not Found as non-error responses, omitting the standard 204 status commonly returned by DELETE endpoints.
Fix: Include http.StatusNoContent in the Delete success status set.
Reproduction: go test ./weed/util/http -run TestDeleteTreatsNoContentAsSuccess -count=1 fails before the fix with an empty error for a 204 response.
Validation: go test ./weed/util/http -run TestDeleteTreatsNoContentAsSuccess -count=1; go test ./weed/util/http -count=1; git diff --check; git diff --cached --check
Co-authored-by: Codex <noreply@openai.com>
* Update weed/util/http/http_global_client_util_test.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
---------
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Problem: RandomUint64 generated eight random bytes but returned int32, truncating the value before mount file and directory handles converted it to uint64. This reduced handle entropy to 32 bits and produced sign-extended handle values.\n\nRoot cause: the helper cast BytesToUint64 to int32 and exposed int32 as its return type.\n\nFix: make RandomUint64 return uint64 and return the full BytesToUint64 result.\n\nReproduction: go test ./weed/util -run TestRandomUint64ReturnsUint64 -count=1 failed before the fix because RandomUint64() had kind int32.\n\nValidation: gofmt -w weed/util/bytes.go weed/util/bytes_test.go; git diff --check; go test ./weed/util -run TestRandomUint64ReturnsUint64 -count=1; go test ./weed/util -count=1; go test ./weed/mount -count=1; git diff --cached --check
* ip.bind: bind outbound connections to the configured address
-ip.bind only governed listeners; outbound gRPC and HTTP connections let
the OS pick the source IP, which may not even be able to reach the
target. Mirror the bind address into a process-global source address and
apply it to outbound TCP dials: the gRPC context dialer, the per-client
HTTP transports, and the default transport. Loopback targets and unix
sockets keep the OS-chosen source so same-host traffic still works.
* ip.bind: first-write-wins source IP, skip on address-family mismatch
Make SetOutboundLocalIP first-write-wins so a `weed server` component's own
bind setting (run in its goroutine) can't clobber the process-wide source
address the top-level -ip.bind already established for the other components.
Skip source binding when the target is a literal IP of a different family
than the bind address, since forcing a mismatched source fails the dial.
* security: reload JWT signing keys on SIGHUP
Signing keys were read once in the server constructors and never
refreshed. After a key rotation (Secret update, divergent reads) the
in-memory key stayed stale and every request kept failing "wrong jwt"
until the affected process was restarted.
Add Guard.UpdateSigningKeys and call it from the master, volume and
filer reload paths and the s3 reload hook, next to the existing
whitelist refresh. Make the global chunk-read JWT cache reloadable via
an atomic swap, and register the master's Reload with grace.OnReload --
it was never wired, so the master ignored SIGHUP entirely.
Mirror the same refresh in the Rust volume server's SIGHUP handler.
* security: swap signing keys behind an atomic pointer
Addresses review feedback on the in-place key swap: SigningKey is a
[]byte, so reassigning the Guard fields while a request handler reads
them is a data race that can tear the multi-word slice header and read
out of bounds.
Hold the four signing-key fields in an immutable signingConfig snapshot
behind atomic.Pointer; UpdateSigningKeys swaps the whole pointer, so a
reader sees either the old keys or the new ones. Reads go through new
SigningKey/ExpiresAfterSec/ReadSigningKey/ReadExpiresAfterSec accessors.
The Rust guard is already safe: every read and the SIGHUP write go
through the shared RwLock<Guard>.
* security: fold whitelist + auth state into the atomic snapshot
Review follow-up. UpdateSigningKeys still wrote isWriteActive while the
request path read it (and the whitelist maps) unsynchronized, so a SIGHUP
under load could expose an inconsistent mix of activation bits and
whitelist contents.
Move all hot-reloadable Guard state -- keys, expirations, whitelist, and
the activation flags -- into a single immutable guardState swapped behind
one atomic.Pointer. The Update* methods take a small mutex to serialize
the read-modify-write; readers stay lock-free. The concurrency test now
also rotates the whitelist and probes IsWhiteListed under -race.
Also read each signing key once per branch in the volume/filer JWT auth
checks, so a reload landing mid-check can't take the allow-fast-path
after auth was enabled or verify against a different key than the branch
saw.
ReadFromBuffer and HasData() take the read lock separately, so a write
that lands between them can make a subscriber which just read a
momentarily empty buffer return ResumeFromDiskError even though the data
is now servable from memory. Re-read under a fresh lock and only bail
when the position is genuinely behind the in-memory window (flushed to
disk); otherwise loop back and read it.
Explain:
- problem: Delete and DeleteProxied could panic on malformed URLs when a JWT was provided.
- root cause: maybeAddAuth was called before checking the error returned by http.NewRequest, so req could be nil.
- fix: return the request construction error before adding the Authorization header.
- validation: go test ./weed/util/http -run 'TestDelete(ReturnsInvalidRequestErrorBeforeAddingAuth|ProxiedReturnsInvalidRequestErrorBeforeAddingAuth)' -count=1; git diff --check
* fix(http): handle invalid gzip stream errors
Explain:
- problem: ReadUrlAsStream could panic when a response claimed gzip encoding but the body was not a valid gzip stream.
- root cause: the gzip reader error was ignored and a nil reader was deferred and read from.
- fix: return the gzip.NewReader error before registering Close or reading.
- validation: go test ./weed/util/http -run TestReadUrlAsStreamReturnsGzipReaderError -count=1; git diff --check.
* test: avoid closing shared global HTTP client in unit test
* fix(util): guard BytesToUint{16,32,64} against short input
length is computed as uint, so length-1 on an empty slice underflows
to MaxUint and the loop indexes b[0] on a zero-length slice. BytesToUint16
also indexed b[0]/b[1] with no length check. All call sites today gate
the slice length explicitly, so this hardens the API for new callers
rather than fixing a live crash.
Return 0 on short input, matching the existing variable-length contract.
* BytesToUint16: match variable-length contract of the 32/64 helpers
A 1-byte slice should return uint16(b[0]) rather than 0, matching how
BytesToUint32 and BytesToUint64 treat short input.
NameList, NameBatch and their serde were an earlier in-memory directory
batch implementation. The redis3 filer store uses its own ItemList
backed by Redis sorted sets, so these types had no production callers
(NameList only via its own test, LoadNameList none at all). Drop them
and the now-orphaned NameBatchData proto message, regenerating
skiplist.pb.go with the repo-standard protoc-gen-go v1.36.6.
* fix(redis3): prevent filer crash from inconsistent skiplist ends
DeleteByKey updated the two ends asymmetrically: the start side decided
whether to clear StartLevels[index] by comparing a cached reference key,
while the end side cleared EndLevels[index] structurally. The redis3
ItemList re-keys a node while keeping its id, so that cached key drifts.
When such a node was the only one at a level and got deleted, the stale
key comparison left StartLevels dangling while EndLevels was cleared to
nil. The next InsertByKey then dereferenced a nil EndLevels[0] and took
down the whole filer during a rename or delete.
Match the deleted node by its unique id so both ends stay consistent,
and guard each end in InsertByKey so an already-corrupted skiplist
persisted in Redis self-heals instead of crashing on load.
* fix(redis3): propagate errors from WriteName node split
The case 2.3 split path returned nil instead of the error in seven
branches. Because the split runs a multi-step sequence (DeleteByKey on
the skiplist, ItemAdd, redis range-delete, ItemAdd), a swallowed failure
let WriteName report success while the skiplist was half-updated, which
the caller then persisted - silently corrupting the directory listing
and setting up the very inconsistent-ends state that crashes the filer.
Chris LuandGitHub
github-actions[bot]
patrickandGitHub
Bruce ZouandGitHub
7y-9andGitHub