* redis2: route the orphan cleanup existence checks to the master
* scaffold: the redis_cluster2 read routing key is useReadOnly
* ci: run the gated redis store tests
* redis2: poll for the redis expiry instead of a fixed sleep
* redis2: assert the value key exists before testing its expiry
The four targets ran in a shell loop at ~3m13s each, so the job took
13m29s and gated the whole workflow by itself: everything else finished
within 8m13s. A matrix runs them concurrently, ~3m45s wall clock.
fail-fast is off so one broken target still reports the other three,
instead of one target per push.
* mount: add the WinFsp filesystem adapter
WinFsp speaks a path-based FUSE dialect; weed/mount implements the
inode-based raw protocol the Linux kernel uses. This translates between
them so Windows runs the same filesystem code as everywhere else rather
than a second implementation: paths resolve to inodes one Lookup at a
time, and the raw operations run unchanged underneath.
Errno translation is spelled out rather than passed through. Go numbers
Windows errnos as offsets from APPLICATION_ERROR, so the raw value would
mean something unrelated by the time WinFsp read it.
Hard links return ENOSYS since WinFsp has none, and byte-range locks stay
with its kernel driver rather than the mount's lock table.
Not reachable from the mount command yet.
* mount: build the winfsp errno table with explicit precedence
Platforms alias errnos differently: freebsd has no ENODATA and linux makes
ENOATTR the same value as it. A map literal with colliding constant keys
does not compile, so build the table and let the first entry win, keeping
the general codes their own meaning.
* mount: wire the winfsp adapter into the mount command
RunMount was one function doing filer setup, mount-point preparation and
serving. The setup is the same everywhere, so it moves to mount_common.go
and each platform keeps only what differs.
Windows differs mostly in the mount point: WinFsp wants a drive letter or
a path that does not exist yet, so none of the unix preparation applies,
and a bad one is worth rejecting up front because WinFsp reports failure
as a bare false. Adds -windows.caseInsensitive for software that expects
Windows naming rules.
* ci: mount on windows and exercise it
Builds weed.exe, installs WinFsp, starts a cluster, mounts S: and runs a
test suite against it: round trips at several sizes, offset writes,
rename, delete, nested directories, concurrent writers, and a directory
wide enough to stand in for the case that prompted this.
Nothing else here can run the Windows mount, so without this the adapter
is only known to compile.
* ci: build the windows mount without cgo
The runner has MinGW, so cgo is on by default and cgofuse compiles its
cgo variant, which needs WinFsp's headers. The nocgo variant loads the
DLL at run time and is what the released weed.exe uses.
* mount: make the winfsp path splitting portable and test it
resolve and resolveParent had the splitting inline in a windows-tagged
file, so the cases that matter most there — both separators, empty and
dot components, the root having no parent to create in — could not be
tested on any runner that builds this.
* test: check the windows mount persists across a remount
Reading a file back through the same live mount proves nothing about
durability; the answer can come from the mount's own caches. Write the
fixtures, confirm the filer serves them with the mount out of the path,
then re-read after a teardown and remount.
* test: cover the windows mount operations that had none
Truncate, append, chtimes and the hard-link refusal were implemented but
never exercised, and the errno table was only unit-tested for mapping,
never end to end. Adds names that have to survive the UTF-16 boundary,
rename over an existing target and across directories, and concurrent
handles on one file rather than one file each.
* ci: dial the filer over ipv4 and run the persistence phases
localhost resolves to ::1 first on windows and the cluster binds ipv4
only, so the mount's grpc dial was refused while the http readiness
probe passed by falling back to ipv4.
* ci: pin the cluster to loopback and probe ports by connecting
weed mini advertises the runner's LAN address and binds filer grpc there,
so the mount's dial to 127.0.0.1:18888 was refused while http answered.
The readiness probe also passed with nothing on 18888: Test-NetConnection
reported success for a port that then refused a connection, so it now
opens a socket instead.
* ci: report listening ports before mounting
The readiness probe connects to the filer grpc port and the mount is then
refused on it, which cannot both be true; print the actual state.
* ci: run the cluster, mount and tests in one step
The runner tears down a step's process tree when its shell exits, so the
cluster started in an earlier step was already gone: the readiness probe
passed against a live filer, the step ended, and the mount then found
nothing listening. A diagnostic step reported no weed.exe at all.
Everything that needs those processes alive now shares a step.
* mount: key windows file io on the handle, not the path
Read and Write walked the path on every call to fill in a NodeId the raw
filesystem never reads: both look the file up by handle. Under eight
writers creating files in one directory the walk transiently missed and
the write failed with ENOENT before reaching the filesystem at all.
Same for flush, fsync and the release calls. O_EXCL now fails on an
existing name instead of taking it over, and Symlink is refused: the
entry is easy to create but WinFsp only follows it once the reparse
point is wired up, so it read back as an empty file.
* mount: translate cgofuse open flags for windows
cgofuse reports MSVC's numbering and the raw filesystem tests Go's, so
only the access mode and O_TRUNC lined up: O_EXCL arrived as O_APPEND and
O_CREAT as nothing at all.
Also report which handle a failed write was using, to tell a handle that
was never issued from one released while still in use.
* mount: report which step of a windows create failed
A concurrent create fails with ENOENT and the path walk, the parent
lookup and the create itself are indistinguishable from the caller.
* ci: send weed logs to stderr on windows
glog writes to its own files by default, so the mount's own error output
never reached the redirected log. Its flags are global and have to come
before the subcommand.
* mount: resolve known paths from the inode table on windows
Every create walked the parent chain with a filer lookup per component.
With eight writers creating files in one directory that is hundreds of
concurrent lookups of the same parent, and lookupEntry reports an
authoritative ENOENT when the directory is cached, the entry is not in
the cache and the inode table has no record — a window a concurrent
refresh can open for a directory that plainly exists.
A path the mount already tracks now resolves straight out of that table.
* test: sync the windows persistence fixtures before closing
The mount is killed rather than unmounted, so anything still queued for
flush is legitimately lost and the test was measuring crash durability
while calling it persistence. A 9MB file lost four chunks that way.
* mount: keep the lookup refresh on the target path
Resolving a tracked path straight from the inode table skipped Lookup,
which is also what refreshes the entry: a truncate then read back the
pre-truncate size. Only the parent chain takes the shortcut now, which
is where the concurrent creates were racing anyway.
* mount: log every windows resolve failure
Open suppressed ENOENT and Getattr logged nothing, which hid the two
callbacks that can report a missing file during a create.
* mount: drop dot entries from windows directory listings
readdir reports "." and ".." for the kernel, but Windows enumerates a
directory without them and displays whatever it is handed, so a folder of
200 files listed 202. Go's ReadDir filters them, which is why only the
PowerShell walk caught it.
* mount: flush queued writes when windows mount is interrupted
The signal handler exits the process the moment its hooks return, so the
WaitForAsyncFlush after Serve never ran on ctrl-c and queued writes were
dropped.
* mount: let windows mount over an empty directory
WinFsp turns a directory mount point into a reparse point, which NTFS
allows on an empty directory and refuses on a populated one. The check
rejected every existing directory, so the ordinary habit of creating the
mount point first failed with a message saying it should not exist.
CI now mounts over a pre-created directory and writes through it.
* ci: run the windows mount check on any pull request
It is the only thing that exercises the Windows mount, so restricting it
to pull requests based on master skipped it for stacked ones. Replaces
the branch name that was pushed to trigger it.
* mount: do not log a missing windows entry as an error
Windows probes for entries that do not exist as a matter of course, so
ENOENT from getattr and open is an answer rather than a fault and would
have filled the log.
* mount: take the fast path for parent chains in every windows resolve
Narrowing it to resolveParent left Getattr and Open re-walking the parent
with a filer lookup per component, and those are what Windows calls
before a create: eight writers in one directory still raced a meta cache
refresh there. Only the final component needs the Lookup refresh.
The pass that suggested otherwise came from a run five times slower than
the failing ones, where the race had no room to appear.
* mount: drop the windows path resolution shortcut
Resolving from the inode table skipped the Lookup that refreshes an
entry, and a truncate then read back its old size. Applying it only to
the parent chain kept truncate correct but left concurrent creates
failing, and applying it to the final component too inverted that. The
two cannot both be satisfied this way, so this returns to looking up
every component and leaves the concurrent create failure open.
* mount: fall back to the open handle when a deferred entry is evicted
A create that defers the filer write leaves the entry only in the local
cache. Creating many files at once pushes the directory past the hot
threshold and evicts it, taking that placeholder with it, so a lookup
went to the filer, found nothing, and reported a file that plainly
exists as missing.
The handle still holding the unflushed entry is authoritative for it.
Caught by concurrent creates over a Windows mount, which resolves a path
on every call rather than relying on a kernel dentry cache.
* mount: let cgofuse resolve to the version the module graph requires
rclone already depends on cgofuse at a newer commit than the v1.6.0 pin,
so readonly builds refused the go.mod until it matched what MVS picks.
The interface and flag values the adapter uses are unchanged there.
* mount: wait for a pending async flush before looking up on the filer
Open, unlink and rename already wait, but a plain lookup went straight
to the filer and read pre-close metadata: truncate a file, close it, and
a path probe during the flush window reported the old size. The kernel
attr cache hides this on linux; a front end that resolves paths on every
operation hit it directly.
* mount: reject a umask wider than the file mode it becomes
ParseUint allowed 64 bits and the result is narrowed to os.FileMode,
which is 32, so an out-of-range umask truncated silently instead of
being reported as unparseable.
* mount: address review findings on the windows mount
WaitForAsyncFlush closed its channel unconditionally and shutdown reaches
it from both the interrupt hook and the path that resumes after serving,
so a ctrl-c could panic on a second close.
The deferred-entry fallback read an open handle's entry without its lock,
which is what the other two readers of that field take so FromPbEntry
does not walk the chunk slice mid-append. The async-flush wait also sat
ahead of the meta cache, making every stat of a recently closed file
queue behind uploads; it belongs just before the filer is consulted.
Windows entries were persisted as uid 0: the raw filesystem stores
InHeader's owner and the adapter left it zero. They now carry the
identity the mount was started with.
The errno table used Linux numbering while cgofuse decodes MSVC's, so
ENAMETOOLONG arrived as EDEADLK and five others were likewise wrong; a
windows test pins each value to cgofuse's own constant.
Also: break the filer handshake loop on success rather than always
running ten rounds, accept a drive letter written S:\\, report a missing
WinFsp instead of panicking, keep commas out of the volume label, and
drop -windows.caseInsensitive, which told WinFsp the mount folds case
while lookups stayed exact.
* mount: return windows lookup references so the inode table stays bounded
Every operation that hands back an EntryOut grants a reference the Linux
kernel returns with FORGET. WinFsp has no FORGET, so the adapter took one
per path component per call, plus one per child of every readdirplus, and
never gave any back: inodeToPath grew for the life of the mount. Walking
the 200k-file directory this exists for stranded 200k references.
The adapter now plays the part the kernel plays. Each resolution releases
what it took, and an open handle keeps the reference for its inode until
Release, counted because the raw filesystem reuses one handle for repeated
opens. Holding it is not optional: completeAsyncFlush skips the metadata
flush when the saved path no longer maps to the inode, so releasing early
would lose a close's metadata.
Also stops persisting the display owner. -o uid=-1 makes WinFsp report the
calling user whatever we say, but the value handed to the raw filesystem is
written to the filer, and 4294967295 is what every other client would read.
-windows.uid and -windows.gid set what is recorded.
* mount: fix windows behaviours the reference implementations guard against
WinFsp has no ro option — it discards the flag and leaves the volume
writable — so -readOnly accepted writes and deletes. The refusal now
happens in the operations themselves.
Windows sends times around its own 1601 epoch, which arrive as a large
negative second count; casting them through stored a year-1601 timestamp
that every other client then read. Those are now left alone. rclone
carries the same guard.
Chown returned ENOSYS, and WinFsp passes a chown failure straight out of
SetSecurity, so Explorer's Security tab and icacls failed for edits that
were not about ownership. It now accepts and discards.
Only create and mkdir presented a caller; the rest sent uid 0, which
hasAccess treats as root, so deletes and renames skipped the permission
check that creates got. Every operation presents the same identity now.
A drive letter written S:\ reached WinFsp unnormalised, which recognises
a drive only as exactly two characters and then failed as a directory
path. A test also pins the open flag translation, since swapping O_EXCL
and O_TRUNC would turn 'fail if it exists' into 'truncate it'.
* mount: answer windows getattr and truncate from the open handle
WinFsp keeps the path a handle was opened with and never updates it when
the file is renamed, so resolving the path again fails on a handle that is
still perfectly valid — the ordinary write-temp-then-rename save pattern.
The handle already knows its inode, which also removes a full path walk
from two operations WinFsp calls constantly.
Readlink on the root now refuses. WinFsp probes there to decide whether
the volume has symlinks and enables them unless it fails, and with them on
it resolves a path a component at a time, each one reaching us as its own
walk — all for a feature Symlink already refuses.
* mount: require the windows mount directory not to exist
WinFsp creates the directory itself with FILE_CREATE and removes it when
the filesystem goes away, so an existing one — empty or not — fails with
"mount point in use". Allowing an empty directory was wrong, and the CI
check that appeared to prove otherwise was the vacuous one: listing a
plain directory succeeds whether or not anything is mounted on it, so the
step passed while the mount had failed and the writes went to local disk.
That check now waits for the reparse point, which is what caught this.
* mount: apply review comments on the windows mount
-windows.uid and -windows.gid reached the adapter but not the filesystem
parameters, which is what carries the owner written to the filer, so the
flags changed nothing.
Readdir re-resolved the path while Getattr and Truncate answer from the
handle; a directory renamed during an enumeration then failed on the
stale path WinFsp still holds.
Utimens now honours UTIME_OMIT instead of writing whatever came with it.
* mount: tag the unix-only lock tests away from windows
The production lock files were tagged when the package was made to build
on windows, but the tests that exercise them were not, so anything that
compiles tests for windows still failed on syscall.F_WRLCK.
* ci: vet the mount tests for each target too
Only compiling the non-test build let an untagged test keep a per-OS
syscall constant without anything noticing.
* mount: keep the xattr flag constants off freebsd
x/sys/unix has no XATTR_CREATE or XATTR_REPLACE there, and weedfs_xattr.go
is already tagged away from freebsd for that reason. Putting them in a
!windows file dragged them back in, so master stopped building for
freebsd.
* ci: cross-compile freebsd and darwin too
The windows-only check missed a freebsd break in the very file it was
added to guard, because nothing else on a pull request compiles them.
* mount: drop the unused go-fuse fs package dependency
WFS embedded fs.Inode but never used any of its methods, and the only
other reference was RENAME_EXCHANGE, a constant sitting next to three
literals. Removing both drops fs and five internal packages from the
mount build graph.
* mount: build the package on windows
Windows has no fcntl lock types, no O_ACCMODE and no x/sys/unix, so a
handful of constants kept weed/mount pinned to unix even though the code
using them is portable in-memory logic. Route them through per-OS shims
and give setBlksize a windows no-op.
The POSIX lock table now compiles on windows but stays unreachable:
WinFsp resolves byte-range locks in its own kernel driver, so nothing
will feed it there.
go.mod points at a go-fuse branch commit and needs repinning to a release
tag once that lands.
* ci: cross-compile for windows
Nothing caught the unix-only constants creeping into weed/mount until a
release build failed.
* mount: let readdir feed a sink instead of the kernel buffer
doReadDirectory wrote directly into fuse.DirEntryList, which is the
kernel's wire format. A front end that is not the kernel would have to
pack entries only to parse them straight back out.
Route it through DirEntrySink instead. ReadDir and ReadDirPlus pass the
reply buffer, so nothing changes for the FUSE server.
* mount: pin go-fuse v2.9.4 for the windows build
Add path filters to workflows that fired on every PR/push regardless
of the diff: CodeQL, go build, the e2e/EC/vacuum/TLS/plugin-worker
integration suites, the Kafka and Postgres gateways, the S3 suites
(Ceph s3tests, s3-go, s3-tables, proxy-signature, https, example,
filer-group), TUS, and the dev binary/container builds. Each scopes
to its subsystem under weed/, its test dir, go.mod/go.sum, and the
workflow file, so docs-, helm-, terraform-, rust- or java-only
changes no longer trigger a full compile-and-test fleet.
386 test binaries execute natively on the amd64 runner, so the suite
catches what vet cannot: unaligned 64-bit atomics and arithmetic that
wraps at runtime. -short keeps the e2e suites on amd64 only.
* fix(tests): keep EC e2e fid cookie arithmetic in uint32
The cookie constants 0x9490CA00 and 0x9500CA00 were added to the int
loop variable before conversion, overflowing 32-bit int at compile
time on linux/386 and linux/arm. Convert the loop variable instead so
the addition stays in uint32.
* fix(tests): pass s3client max backoff in milliseconds
MaxBackoffDelay is documented as milliseconds and multiplied by 1e6
before use, but the example set it to 5s in nanoseconds, yielding an
absurd backoff on 64-bit and a compile-time int overflow on 32-bit.
* ci: type-check code and tests for linux/386
64-bit-only constant arithmetic keeps slipping into test files and
breaking 32-bit downstream builds. Vet the whole root module under
GOOS=linux GOARCH=386 so these fail in CI instead of after release.
* fix(tests): convert s3client backoff to Duration before scaling
The ms-to-ns multiplication ran in int, wrapping at runtime on 32-bit;
scale by time.Millisecond after the Duration conversion instead.
* check for nil needle map before compaction sync
When CommitCompact runs concurrently, it sets v.nm = nil under
dataFileAccessLock. CompactByIndex does not hold that lock, so
v.nm.Sync() can hit a nil pointer. Add an early nil check to
return an error instead of crashing.
Fixes#8591
* guard copyDataBasedOnIndexFile size check against nil needle map
The post-compaction size validation at line 538 accesses
v.nm.ContentSize() and v.nm.DeletedSize(). If CommitCompact has
concurrently set v.nm to nil, this causes a SIGSEGV. Skip the
validation when v.nm is nil since the actual data copy uses local
needle maps (oldNm/newNm) and is unaffected.
Fixes#8591
* use atomic.Bool for compaction flags to prevent concurrent vacuum races
The isCompacting and isCommitCompacting flags were plain bools
read and written from multiple goroutines without synchronization.
This allowed concurrent vacuums on the same volume to pass the
guard checks and run simultaneously, leading to the nil pointer
crash. Using atomic.Bool with CompareAndSwap ensures only one
compaction or commit can run per volume at a time.
Fixes#8591
* use go-version-file in CI workflows instead of hardcoded versions
Use go-version-file: 'go.mod' so CI automatically picks up the Go
version from go.mod, avoiding future version drift. Reordered
checkout before setup-go in go.yml and e2e.yml so go.mod is
available. Removed the now-unused GO_VERSION env vars.
* capture v.nm locally in CompactByIndex to close TOCTOU race
A bare nil check on v.nm followed by v.nm.Sync() has a race window
where CommitCompact can set v.nm = nil between the two. Snapshot
the pointer into a local variable so the nil check and Sync operate
on the same reference.
* add dynamic timeouts to plugin worker vacuum gRPC calls
All vacuum gRPC calls used context.Background() with no deadline,
so the plugin scheduler's execution timeout could kill a job while
a large volume compact was still in progress. Use volume-size-scaled
timeouts matching the topology vacuum approach: 3 min/GB for compact,
1 min/GB for check, commit, and cleanup.
Fixes#8591
* Revert "add dynamic timeouts to plugin worker vacuum gRPC calls"
This reverts commit 80951934c3.
* unify compaction lifecycle into single atomic flag
Replace separate isCompacting and isCommitCompacting flags with a
single isCompactionInProgress atomic.Bool. This ensures CompactBy*,
CommitCompact, Close, and Destroy are mutually exclusive — only one
can run at a time per volume.
Key changes:
- All entry points use CompareAndSwap(false, true) to claim exclusive
access. CompactByVolumeData and CompactByIndex now also guard v.nm
and v.DataBackend with local captures.
- Close() waits for the flag outside dataFileAccessLock to avoid
deadlocking with CommitCompact (which holds the flag while waiting
for the lock). It claims the flag before acquiring the lock so no
new compaction can start.
- Destroy() uses CAS instead of a racy Load check, preventing
concurrent compaction from racing with volume teardown.
- unmountVolumeByCollection no longer deletes from the map;
DeleteCollectionFromDiskLocation removes entries only after
successful Destroy, preventing orphaned volumes on failure.
Fixes#8591
* fix: use keyed fields in struct literals
- Replace unsafe reflect.StringHeader/SliceHeader with safe unsafe.String/Slice (weed/query/sqltypes/unsafe.go)
- Add field names to Type_ScalarType struct literals (weed/mq/schema/schema_builder.go)
- Add Duration field name to FlexibleDuration struct literals across test files
- Add field names to bson.D struct literals (weed/filer/mongodb/mongodb_store_kv.go)
Fixes go vet warnings about unkeyed struct literals.
* fix: remove unreachable code
- Remove unreachable return statements after infinite for loops
- Remove unreachable code after if/else blocks where all paths return
- Simplify recursive logic by removing unnecessary for loop (inode_to_path.go)
- Fix Type_ScalarType literal to use enum value directly (schema_builder.go)
- Call onCompletionFn on stream error (subscribe_session.go)
Files fixed:
- weed/query/sqltypes/unsafe.go
- weed/mq/schema/schema_builder.go
- weed/mq/client/sub_client/connect_to_sub_coordinator.go
- weed/filer/redis3/ItemList.go
- weed/mq/client/agent_client/subscribe_session.go
- weed/mq/broker/broker_grpc_pub_balancer.go
- weed/mount/inode_to_path.go
- weed/util/skiplist/name_list.go
* fix: avoid copying lock values in protobuf messages
- Use proto.Merge() instead of direct assignment to avoid copying sync.Mutex in S3ApiConfiguration (iamapi_server.go)
- Add explicit comments noting that channel-received values are already copies before taking addresses (volume_grpc_client_to_master.go)
The protobuf messages contain sync.Mutex fields from the message state, which should not be copied.
Using proto.Merge() properly merges messages without copying the embedded mutex.
* fix: correct byte array size for uint32 bit shift operations
The generateAccountId() function only needs 4 bytes to create a uint32 value.
Changed from allocating 8 bytes to 4 bytes to match the actual usage.
This fixes go vet warning about shifting 8-bit values (bytes) by more than 8 bits.
* fix: ensure context cancellation on all error paths
In broker_client_subscribe.go, ensure subscriberCancel() is called on all error return paths:
- When stream creation fails
- When partition assignment fails
- When sending initialization message fails
This prevents context leaks when an error occurs during subscriber creation.
* fix: ensure subscriberCancel called for CreateFreshSubscriber stream.Send error
Ensure subscriberCancel() is called when stream.Send fails in CreateFreshSubscriber.
* ci: add go vet step to prevent future lint regressions
- Add go vet step to GitHub Actions workflow
- Filter known protobuf lock warnings (MessageState sync.Mutex)
These are expected in generated protobuf code and are safe
- Prevents accumulation of go vet errors in future PRs
- Step runs before build to catch issues early
* fix: resolve remaining syntax and logic errors in vet fixes
- Fixed syntax errors in filer_sync.go caused by missing closing braces
- Added missing closing brace for if block and function
- Synchronized fixes to match previous commits on branch
* fix: add missing return statements to daemon functions
- Add 'return false' after infinite loops in filer_backup.go and filer_meta_backup.go
- Satisfies declared bool return type signatures
- Maintains consistency with other daemon functions (runMaster, runFilerSynchronize, runWorker)
- While unreachable, explicitly declares the return satisfies function signature contract
* fix: add nil check for onCompletionFn in SubscribeMessageRecord
- Check if onCompletionFn is not nil before calling it
- Prevents potential panic if nil function is passed
- Matches pattern used in other callback functions
* docs: clarify unreachable return statements in daemon functions
- Add comments documenting that return statements satisfy function signature
- Explains that these returns follow infinite loops and are unreachable
- Improves code clarity for future maintainers
Chris LuandGitHub
Chris Lu
yulai.li
Konstantin Lebedev
naveensrinivasan