* s3tables: add ClickHouse iceberg catalog integration test
* ci: run the ClickHouse iceberg catalog test
* s3tables: bound setup HTTP calls in the ClickHouse test
* s3tables: pin the ClickHouse writer image dependencies
* helm: mount an enterprise license Secret into every component
Running the enterprise image under this chart meant hand-rolling
extraVolumes and extraVolumeMounts on every component. Missing one is
easy and quiet: a component without the license silently drops to
community mode, and on the admin that surfaces only as Data Recovery and
Point-in-Time Recovery refusing to enable, with the master looking fine.
Add global.seaweedfs.license.existingSecret. The Secret is mounted
read-only into master, volume, filer, s3, sftp, admin, worker and
all-in-one, and SEAWEED_LICENSE points every process at the file rather
than relying on the binary's search paths, which depend on the working
directory.
The mount is a directory, never a subPath: kubelet refreshes Secret
contents in place, but a subPath is resolved once at container start and
never updates, which would break license renewal. There is deliberately
no checksum annotation on the pod template either — that would roll every
pod on renewal, the opposite of what is wanted. Verified on kind: the
renewed file reached a running master ~70s after the Secret was patched,
same pod UID, restartCount 0.
Also documents that only the master re-reads the license on a timer
today; the other components pick a renewal up on their next restart.
* helm: keep master data on a claim by default
The master's -mdir holds its Raft log and snapshots, and with them the
cluster's topology UUID — the identity an enterprise license is issued
against. It defaulted to a hostPath under /ssd, which does not follow a
rescheduled pod: the master came back with an empty data directory, a
freshly generated cluster UUID, and a license that no longer matched.
With the chart's default of a single master replica there is no peer to
recover the identity from either.
Default master.data.type to persistentVolumeClaim, sized 1Gi (Raft state
is small). hostPath stays available for anyone who wants it.
This is breaking for existing releases: volumeClaimTemplates is immutable,
so helm upgrade on a release installed with the old default fails with
"updates to statefulset spec for fields other than ... are forbidden".
Verified on kind, along with both ways out — pinning
master.data.type=hostPath upgrades cleanly, and the documented migration
(stop the master, pre-seed a claim named after the StatefulSet, upgrade)
preserves the cluster UUID. Seeding has to happen while the master is
stopped; copying into a live pod loses the state, since the running
master rewrites its Raft files before the restart.
* helm: mount the license on masters only
The master is what reads the license file: it validates it, enforces the
capacity limit and binds it to the cluster UUID. Mounting the Secret on
volume, filer, s3, sftp, admin and worker put it in six more containers
that never look at it, so drop it there and keep master plus all-in-one,
which runs `weed server -master`.
Two fixes from review while here:
- project only the configured key out of the Secret, so an unrelated
key in the same Secret is not exposed to the container. Verified the
key-scoped projection still updates in place: patching the Secret
reached the running master in ~50s, same pod UID, restartCount 0.
- drop SEAWEED_LICENSE from merged extraEnvironmentVars while a
license Secret is configured. It used to be possible to render the
key twice in one container, with the user's value winning over the
path the chart actually mounts.
CI now pins the scope (master only, all-in-one separately), the
key-scoped projection, readOnly, and that SEAWEED_LICENSE renders once.
* helm: fix the documented master-data migration
The seed pod in the migration never mounted the claim it was supposed to
seed, so following the steps verbatim copied the Raft state onto the
pod's ephemeral filesystem and threw it away with the pod — landing the
reader in exactly the empty-claim, new-cluster-UUID state the section
exists to avoid. Give the pod the volume.
The names were assembled as <release>-seaweedfs-*, which is wrong
whenever the release name already contains the chart name or an override
is set; read the StatefulSet name from the cluster instead and derive the
claim from it. Also scope the procedure to the chart's single-master
default, and create the Secret in the release namespace.
Trims the enterprise prose this section had accumulated: this is the OSS
chart, and the master-data default is a durability fix that stands on its
own.
* helm: quote the projected license key, reserve it on the secret env path
A secretKey that YAML reads as a non-string (123, yes, no) rendered
unquoted into the volume's items, so the projection would not match the
Secret's key. Quote both key and path.
all-in-one renders secretExtraEnvironmentVars itself, outside the merge
helper that already drops SEAWEED_LICENSE, so an entry there could still
render the variable twice. Skip it there too while a license Secret is
configured. The master template has no such block, so this is the only
remaining path.
* helm: correct the license helper comments after scoping to masters
* helm: keep hostPath as the master data default
Defaulting master.data.type to a claim broke every existing release:
volumeClaimTemplates is immutable on a StatefulSet, so helm upgrade
failed with "updates to statefulset spec for fields other than ... are
forbidden" before it changed anything.
Keep hostPath as the default and document the claim as the option to
choose — for a new install, or for an existing one via the migration
already in the README. The chart supported both types all along; only
the default moves back.
Every immutable field of every rendered StatefulSet is now identical to
upstream under default values, so an in-place upgrade cannot trip the
API. Verified on kind: install with the unmodified upstream chart,
upgrade to this branch (ok), upgrade again turning the license Secret on
(ok, volume added in place). A fresh install with
master.data.type=persistentVolumeClaim binds its claim as before.
The whole PR is additive now: nothing renders differently until a value
is set.
* helm: scope the migration's StatefulSet lookup to the release
* helm: trim the comments added by this change
* s3: give a versioned metadata-only copy its own chunks
A self-copy that only rewrites metadata clones the source entry, chunk fids
and all, and writes the clone back. With no versioning that is exactly right:
the clone replaces the entry it came from, so one entry owns the needles the
whole time. Under versioning the clone lands in a new .versions/ file and the
source stays live, and nothing refcounts a plain shared chunk list -- deleting
either version (a NoncurrentVersionExpiration rule, say) frees needles the
other still points at, and the next vacuum makes that permanent. rclone hits
this on every upload, since it stamps mtime with exactly this copy.
Take the metadata-only path only where the write replaces the entry it read:
the bare key of a bucket without versioning. Versioned, suspended, and
versionId-pinned copies fall through to the regular copy path, which gives the
destination its own chunks.
* s3: reencrypt a versioned SSE-KMS key rotation instead of reusing the chunks
A same-object copy that changes the KMS key id hands the source chunks straight
back, on the assumption that the copy overwrites the entry they came from. A
versioned bucket writes a new version beside the source instead, so the two end
up sharing needles that nothing refcounts, and deleting either one frees the
other's data. Reuse the chunks only when the destination really is the source
entry; otherwise fall through to the reencrypt path, which also gives the new
version the key it asked for rather than leaving it on the old one.
* s3: make one predicate decide whether a copy replaces its source
The metadata-only branch and the key-rotation strategy both answer the same
question -- does this copy write back to the entry it read -- so let them share
one predicate instead of pairing a same-destination check with it separately at
each site.
* test(s3): fail the copy regression tests when the vacuum does not run
The helper swallowed a failed or non-200 request to the master, so a vacuum
that never ran turned both chunk-ownership assertions into no-ops: the
tombstoned needles were still readable and the surviving version looked fine
either way. Require the endpoint, the request, and a 200.
* ci(s3): run every versioning test in the regression gate
The gate named the tests it wanted, so a new regression test sat there
uncovered until someone remembered this file -- it fooled me into thinking two
tests added in this PR never ran anywhere, when the comprehensive job had them
all along. Invert it: run everything, and name a test only to keep it out. The
delete job beside this one already works that way, and the suite costs about
two minutes.
Only the pagination stress tests are excluded; they build 1500+ versions, skip
themselves without ENABLE_STRESS_TESTS, and have their own make target.
Go's regexp has no negation, so the pattern is still assembled from a listing,
the way the volume-server integration workflow does it. Note the trailing $$:
make eats a lone trailing $ and takes the anchor with it.
Every test bucket is its own collection and each grows 7 volumes, so the
suite asks for 140 while the job caps the volume server at 100. Slots come
back only when the volume server's next full heartbeat tells the master the
deleted collections are gone, and the suite finishes inside one 5s pulse:
the run survives on whichever buckets happened to be dropped before that
single tick. The last run cleared by two slots, this one wedged the final
PutObject with 'No writable volumes and no free volumes left'.
Both workflows split the suite with ^Test[A-H] / ^Test[I-S] / ^Test[T-Z].
Test names cluster, so shard 2 drew 50 of the 114 grpc tests and 30 of
the 64 http ones, and spent 13m51s against 7m48s and 9m09s for its peers.
Listing the tests and dealing them out one at a time splits them 38/38/38
and 21/22/21, and keeps splitting evenly as tests are added. The pattern
is computed once into the environment rather than repeated in the summary
step, where the two copies had to be kept in agreement by hand.
The four targets ran in a shell loop at ~3m13s each, so the job took
13m29s and gated the whole workflow by itself: everything else finished
within 8m13s. A matrix runs them concurrently, ~3m45s wall clock.
fail-fast is off so one broken target still reports the other three,
instead of one target per push.
* volume: skip directory fsync on Windows
* ci: run the windows jobs for the whole vacuum path
Both windows jobs start the same weed mini cluster, so both exercise the
volume server's vacuum path, but only one of them watched a single file
in it. Cover the compact, reconcile and load files in both.
* volume: report a failed makeupDiff instead of discarding it
The cleanup removes assigned to the same err the makeupDiff failure was
held in, so an aborted compaction returned nil once both removes
succeeded. The master then recorded the vacuum as committed and the
volume reloaded against the discarded generation.
* volume: correct the fsyncDir comments after the windows skip
Both comments described the old shape, where windows fell through to a
sync whose error was swallowed.
* volume: keep the makeupDiff failure ahead of its cleanup errors
A failed remove of .cpd/.cpx outranked the failure that abandoned the
compaction, so the caller saw the cleanup error instead of the cause.
Log it and return the original, matching the Rust do_commit_compact. A
leftover temp file is rolled back by reconcile on the next start.
Key the Rust cargo cache on the toolchain that built it
The cache key was rust-<Cargo.lock hash> with a bare rust- restore prefix, so
one seaweed-volume/target survived across runner images. cargo tracks its own
inputs but not the runner's C toolchain, so build-script output for C
dependencies is reused even when the system libc underneath it changed.
That is how the Rust jobs got wedged: the cached aws-lc-sys objects reference
__isoc23_sscanf and __isoc23_strtol, symbols glibc only grew in 2.38, while the
jobs link on ubuntu-22.04 with glibc 2.35. Every job died at
rust-lld: error: undefined symbol: __isoc23_sscanf
>>> referenced by bcm.c in archive libaws_lc_sys-*.rlib
with nothing in the tree to explain it, and no amount of re-running helped
because the poisoned entry was hit every time.
Fold the glibc and rustc versions into the key so a toolchain change misses the
cache and rebuilds instead of producing an unlinkable target/.
* mount: report . and .. from windows directories
WinFsp strips the dot entries for the root itself and expects every other
directory to report them, the way a real NTFS enumeration does: its
dirctl test asserts a subdirectory's first two entries are "." and ".."
and that a hundred files enumerate as 102 entries. Dropping them
unconditionally is what fails querydir_test.
The Go test that guarded the old behaviour went with it: os.File.Readdir
filters dot entries itself, so it could never have observed either way.
* mount: give the windows dot entries their directory type
The readdir fills an attribute block only for real children, so "." and
".." arrived with a zeroed one and were reported with mode 0. Windows
refuses to enumerate a directory whose first entry is not marked as a
directory, which is the assertion querydir_test fails on with
STATUS_OBJECT_NAME_NOT_FOUND.
They now carry the type the readdir already knew. The explorer walk also
names any unexpected entry rather than only counting, so a dot entry
leaking through reads differently from a missing file.
* ci: run WinFsp's conformance suite against the windows mount
The FUSE mount is held to pjdfstest with an empty known-failures list;
the Windows mount had 24 hand-written tests. winfsp-tests is what WinFsp
uses to check a filesystem behaves like NTFS, and --fuse-external points
it at ours instead of the bundled memfs, so it is the same bar in the
same shape: anything failing that is not listed is a regression.
It reaches oplocks, security descriptors, POSIX unlink-and-rename and
directory-buffer resumption — the places a Windows filesystem actually
breaks, and none of which the current suite touches.
known_failures.txt starts with the four groups that cannot pass by
construction. The first run will show what else needs listing.
* ci: make the conformance runner fail loudly instead of running empty
The first run reported "0 excluded entries" and then died with
STATUS_DLL_NOT_FOUND, so it never tested anything while looking like a
normal failing run.
winfsp-tests links against winfsp-x64.dll, which the installer puts
somewhere the loader does not search, so the WinFsp bin directory goes on
PATH. A missing or empty known-failures list is now an error rather than
a silent run with nothing excluded, which would read as a clean sweep
with no known failures. ${env:ProgramFiles(x86)} needs the braces, and a
mount point without a trailing separator makes Join-Path build a path
relative to the drive's current directory rather than its root.
* ci: read winfsp-tests failures from its report, and list the real ones
The first run exited zero with 30 of 50 tests reporting KO, and the job
went green: --no-abort keeps the suite going past a failure and the exit
code stops reflecting them, so trusting it meant the check could not fail.
The report is now parsed for KO lines and each one named in the error.
known_failures.txt is populated from that run rather than guessed. The
groups are real gaps, not suite quirks: cached and overlapped IO fails as
a block, delete-while-open has no pending state, Windows file attributes
and creation time are not round-tripped, and directory enumeration does
not resume from a marker.
* ci: stop excluding the extended attribute tests
Forwarding landed, so the group runs instead of being taken on trust —
which is the only coverage it has had.
* mount: add the WinFsp filesystem adapter
WinFsp speaks a path-based FUSE dialect; weed/mount implements the
inode-based raw protocol the Linux kernel uses. This translates between
them so Windows runs the same filesystem code as everywhere else rather
than a second implementation: paths resolve to inodes one Lookup at a
time, and the raw operations run unchanged underneath.
Errno translation is spelled out rather than passed through. Go numbers
Windows errnos as offsets from APPLICATION_ERROR, so the raw value would
mean something unrelated by the time WinFsp read it.
Hard links return ENOSYS since WinFsp has none, and byte-range locks stay
with its kernel driver rather than the mount's lock table.
Not reachable from the mount command yet.
* mount: build the winfsp errno table with explicit precedence
Platforms alias errnos differently: freebsd has no ENODATA and linux makes
ENOATTR the same value as it. A map literal with colliding constant keys
does not compile, so build the table and let the first entry win, keeping
the general codes their own meaning.
* mount: wire the winfsp adapter into the mount command
RunMount was one function doing filer setup, mount-point preparation and
serving. The setup is the same everywhere, so it moves to mount_common.go
and each platform keeps only what differs.
Windows differs mostly in the mount point: WinFsp wants a drive letter or
a path that does not exist yet, so none of the unix preparation applies,
and a bad one is worth rejecting up front because WinFsp reports failure
as a bare false. Adds -windows.caseInsensitive for software that expects
Windows naming rules.
* ci: mount on windows and exercise it
Builds weed.exe, installs WinFsp, starts a cluster, mounts S: and runs a
test suite against it: round trips at several sizes, offset writes,
rename, delete, nested directories, concurrent writers, and a directory
wide enough to stand in for the case that prompted this.
Nothing else here can run the Windows mount, so without this the adapter
is only known to compile.
* ci: build the windows mount without cgo
The runner has MinGW, so cgo is on by default and cgofuse compiles its
cgo variant, which needs WinFsp's headers. The nocgo variant loads the
DLL at run time and is what the released weed.exe uses.
* mount: make the winfsp path splitting portable and test it
resolve and resolveParent had the splitting inline in a windows-tagged
file, so the cases that matter most there — both separators, empty and
dot components, the root having no parent to create in — could not be
tested on any runner that builds this.
* test: check the windows mount persists across a remount
Reading a file back through the same live mount proves nothing about
durability; the answer can come from the mount's own caches. Write the
fixtures, confirm the filer serves them with the mount out of the path,
then re-read after a teardown and remount.
* test: cover the windows mount operations that had none
Truncate, append, chtimes and the hard-link refusal were implemented but
never exercised, and the errno table was only unit-tested for mapping,
never end to end. Adds names that have to survive the UTF-16 boundary,
rename over an existing target and across directories, and concurrent
handles on one file rather than one file each.
* ci: dial the filer over ipv4 and run the persistence phases
localhost resolves to ::1 first on windows and the cluster binds ipv4
only, so the mount's grpc dial was refused while the http readiness
probe passed by falling back to ipv4.
* ci: pin the cluster to loopback and probe ports by connecting
weed mini advertises the runner's LAN address and binds filer grpc there,
so the mount's dial to 127.0.0.1:18888 was refused while http answered.
The readiness probe also passed with nothing on 18888: Test-NetConnection
reported success for a port that then refused a connection, so it now
opens a socket instead.
* ci: report listening ports before mounting
The readiness probe connects to the filer grpc port and the mount is then
refused on it, which cannot both be true; print the actual state.
* ci: run the cluster, mount and tests in one step
The runner tears down a step's process tree when its shell exits, so the
cluster started in an earlier step was already gone: the readiness probe
passed against a live filer, the step ended, and the mount then found
nothing listening. A diagnostic step reported no weed.exe at all.
Everything that needs those processes alive now shares a step.
* mount: key windows file io on the handle, not the path
Read and Write walked the path on every call to fill in a NodeId the raw
filesystem never reads: both look the file up by handle. Under eight
writers creating files in one directory the walk transiently missed and
the write failed with ENOENT before reaching the filesystem at all.
Same for flush, fsync and the release calls. O_EXCL now fails on an
existing name instead of taking it over, and Symlink is refused: the
entry is easy to create but WinFsp only follows it once the reparse
point is wired up, so it read back as an empty file.
* mount: translate cgofuse open flags for windows
cgofuse reports MSVC's numbering and the raw filesystem tests Go's, so
only the access mode and O_TRUNC lined up: O_EXCL arrived as O_APPEND and
O_CREAT as nothing at all.
Also report which handle a failed write was using, to tell a handle that
was never issued from one released while still in use.
* mount: report which step of a windows create failed
A concurrent create fails with ENOENT and the path walk, the parent
lookup and the create itself are indistinguishable from the caller.
* ci: send weed logs to stderr on windows
glog writes to its own files by default, so the mount's own error output
never reached the redirected log. Its flags are global and have to come
before the subcommand.
* mount: resolve known paths from the inode table on windows
Every create walked the parent chain with a filer lookup per component.
With eight writers creating files in one directory that is hundreds of
concurrent lookups of the same parent, and lookupEntry reports an
authoritative ENOENT when the directory is cached, the entry is not in
the cache and the inode table has no record — a window a concurrent
refresh can open for a directory that plainly exists.
A path the mount already tracks now resolves straight out of that table.
* test: sync the windows persistence fixtures before closing
The mount is killed rather than unmounted, so anything still queued for
flush is legitimately lost and the test was measuring crash durability
while calling it persistence. A 9MB file lost four chunks that way.
* mount: keep the lookup refresh on the target path
Resolving a tracked path straight from the inode table skipped Lookup,
which is also what refreshes the entry: a truncate then read back the
pre-truncate size. Only the parent chain takes the shortcut now, which
is where the concurrent creates were racing anyway.
* mount: log every windows resolve failure
Open suppressed ENOENT and Getattr logged nothing, which hid the two
callbacks that can report a missing file during a create.
* mount: drop dot entries from windows directory listings
readdir reports "." and ".." for the kernel, but Windows enumerates a
directory without them and displays whatever it is handed, so a folder of
200 files listed 202. Go's ReadDir filters them, which is why only the
PowerShell walk caught it.
* mount: flush queued writes when windows mount is interrupted
The signal handler exits the process the moment its hooks return, so the
WaitForAsyncFlush after Serve never ran on ctrl-c and queued writes were
dropped.
* mount: let windows mount over an empty directory
WinFsp turns a directory mount point into a reparse point, which NTFS
allows on an empty directory and refuses on a populated one. The check
rejected every existing directory, so the ordinary habit of creating the
mount point first failed with a message saying it should not exist.
CI now mounts over a pre-created directory and writes through it.
* ci: run the windows mount check on any pull request
It is the only thing that exercises the Windows mount, so restricting it
to pull requests based on master skipped it for stacked ones. Replaces
the branch name that was pushed to trigger it.
* mount: do not log a missing windows entry as an error
Windows probes for entries that do not exist as a matter of course, so
ENOENT from getattr and open is an answer rather than a fault and would
have filled the log.
* mount: take the fast path for parent chains in every windows resolve
Narrowing it to resolveParent left Getattr and Open re-walking the parent
with a filer lookup per component, and those are what Windows calls
before a create: eight writers in one directory still raced a meta cache
refresh there. Only the final component needs the Lookup refresh.
The pass that suggested otherwise came from a run five times slower than
the failing ones, where the race had no room to appear.
* mount: drop the windows path resolution shortcut
Resolving from the inode table skipped the Lookup that refreshes an
entry, and a truncate then read back its old size. Applying it only to
the parent chain kept truncate correct but left concurrent creates
failing, and applying it to the final component too inverted that. The
two cannot both be satisfied this way, so this returns to looking up
every component and leaves the concurrent create failure open.
* mount: fall back to the open handle when a deferred entry is evicted
A create that defers the filer write leaves the entry only in the local
cache. Creating many files at once pushes the directory past the hot
threshold and evicts it, taking that placeholder with it, so a lookup
went to the filer, found nothing, and reported a file that plainly
exists as missing.
The handle still holding the unflushed entry is authoritative for it.
Caught by concurrent creates over a Windows mount, which resolves a path
on every call rather than relying on a kernel dentry cache.
* mount: let cgofuse resolve to the version the module graph requires
rclone already depends on cgofuse at a newer commit than the v1.6.0 pin,
so readonly builds refused the go.mod until it matched what MVS picks.
The interface and flag values the adapter uses are unchanged there.
* mount: wait for a pending async flush before looking up on the filer
Open, unlink and rename already wait, but a plain lookup went straight
to the filer and read pre-close metadata: truncate a file, close it, and
a path probe during the flush window reported the old size. The kernel
attr cache hides this on linux; a front end that resolves paths on every
operation hit it directly.
* mount: reject a umask wider than the file mode it becomes
ParseUint allowed 64 bits and the result is narrowed to os.FileMode,
which is 32, so an out-of-range umask truncated silently instead of
being reported as unparseable.
* mount: address review findings on the windows mount
WaitForAsyncFlush closed its channel unconditionally and shutdown reaches
it from both the interrupt hook and the path that resumes after serving,
so a ctrl-c could panic on a second close.
The deferred-entry fallback read an open handle's entry without its lock,
which is what the other two readers of that field take so FromPbEntry
does not walk the chunk slice mid-append. The async-flush wait also sat
ahead of the meta cache, making every stat of a recently closed file
queue behind uploads; it belongs just before the filer is consulted.
Windows entries were persisted as uid 0: the raw filesystem stores
InHeader's owner and the adapter left it zero. They now carry the
identity the mount was started with.
The errno table used Linux numbering while cgofuse decodes MSVC's, so
ENAMETOOLONG arrived as EDEADLK and five others were likewise wrong; a
windows test pins each value to cgofuse's own constant.
Also: break the filer handshake loop on success rather than always
running ten rounds, accept a drive letter written S:\\, report a missing
WinFsp instead of panicking, keep commas out of the volume label, and
drop -windows.caseInsensitive, which told WinFsp the mount folds case
while lookups stayed exact.
* mount: return windows lookup references so the inode table stays bounded
Every operation that hands back an EntryOut grants a reference the Linux
kernel returns with FORGET. WinFsp has no FORGET, so the adapter took one
per path component per call, plus one per child of every readdirplus, and
never gave any back: inodeToPath grew for the life of the mount. Walking
the 200k-file directory this exists for stranded 200k references.
The adapter now plays the part the kernel plays. Each resolution releases
what it took, and an open handle keeps the reference for its inode until
Release, counted because the raw filesystem reuses one handle for repeated
opens. Holding it is not optional: completeAsyncFlush skips the metadata
flush when the saved path no longer maps to the inode, so releasing early
would lose a close's metadata.
Also stops persisting the display owner. -o uid=-1 makes WinFsp report the
calling user whatever we say, but the value handed to the raw filesystem is
written to the filer, and 4294967295 is what every other client would read.
-windows.uid and -windows.gid set what is recorded.
* mount: fix windows behaviours the reference implementations guard against
WinFsp has no ro option — it discards the flag and leaves the volume
writable — so -readOnly accepted writes and deletes. The refusal now
happens in the operations themselves.
Windows sends times around its own 1601 epoch, which arrive as a large
negative second count; casting them through stored a year-1601 timestamp
that every other client then read. Those are now left alone. rclone
carries the same guard.
Chown returned ENOSYS, and WinFsp passes a chown failure straight out of
SetSecurity, so Explorer's Security tab and icacls failed for edits that
were not about ownership. It now accepts and discards.
Only create and mkdir presented a caller; the rest sent uid 0, which
hasAccess treats as root, so deletes and renames skipped the permission
check that creates got. Every operation presents the same identity now.
A drive letter written S:\ reached WinFsp unnormalised, which recognises
a drive only as exactly two characters and then failed as a directory
path. A test also pins the open flag translation, since swapping O_EXCL
and O_TRUNC would turn 'fail if it exists' into 'truncate it'.
* mount: answer windows getattr and truncate from the open handle
WinFsp keeps the path a handle was opened with and never updates it when
the file is renamed, so resolving the path again fails on a handle that is
still perfectly valid — the ordinary write-temp-then-rename save pattern.
The handle already knows its inode, which also removes a full path walk
from two operations WinFsp calls constantly.
Readlink on the root now refuses. WinFsp probes there to decide whether
the volume has symlinks and enables them unless it fails, and with them on
it resolves a path a component at a time, each one reaching us as its own
walk — all for a feature Symlink already refuses.
* mount: require the windows mount directory not to exist
WinFsp creates the directory itself with FILE_CREATE and removes it when
the filesystem goes away, so an existing one — empty or not — fails with
"mount point in use". Allowing an empty directory was wrong, and the CI
check that appeared to prove otherwise was the vacuous one: listing a
plain directory succeeds whether or not anything is mounted on it, so the
step passed while the mount had failed and the writes went to local disk.
That check now waits for the reparse point, which is what caught this.
* mount: apply review comments on the windows mount
-windows.uid and -windows.gid reached the adapter but not the filesystem
parameters, which is what carries the owner written to the filer, so the
flags changed nothing.
Readdir re-resolved the path while Getattr and Truncate answer from the
handle; a directory renamed during an enumeration then failed on the
stale path WinFsp still holds.
Utimens now honours UTIME_OMIT instead of writing whatever came with it.
* mount: tag the unix-only lock tests away from windows
The production lock files were tagged when the package was made to build
on windows, but the tests that exercise them were not, so anything that
compiles tests for windows still failed on syscall.F_WRLCK.
* ci: vet the mount tests for each target too
Only compiling the non-test build let an untagged test keep a per-OS
syscall constant without anything noticing.
* mount: keep the xattr flag constants off freebsd
x/sys/unix has no XATTR_CREATE or XATTR_REPLACE there, and weedfs_xattr.go
is already tagged away from freebsd for that reason. Putting them in a
!windows file dragged them back in, so master stopped building for
freebsd.
* ci: cross-compile freebsd and darwin too
The windows-only check missed a freebsd break in the very file it was
added to guard, because nothing else on a pull request compiles them.
* mount: drop the unused go-fuse fs package dependency
WFS embedded fs.Inode but never used any of its methods, and the only
other reference was RENAME_EXCHANGE, a constant sitting next to three
literals. Removing both drops fs and five internal packages from the
mount build graph.
* mount: build the package on windows
Windows has no fcntl lock types, no O_ACCMODE and no x/sys/unix, so a
handful of constants kept weed/mount pinned to unix even though the code
using them is portable in-memory logic. Route them through per-OS shims
and give setBlksize a windows no-op.
The POSIX lock table now compiles on windows but stays unreachable:
WinFsp resolves byte-range locks in its own kernel driver, so nothing
will feed it there.
go.mod points at a go-fuse branch commit and needs repinning to a release
tag once that lands.
* ci: cross-compile for windows
Nothing caught the unix-only constants creeping into weed/mount until a
release build failed.
* mount: let readdir feed a sink instead of the kernel buffer
doReadDirectory wrote directly into fuse.DirEntryList, which is the
kernel's wire format. A front end that is not the kernel would have to
pack entries only to parse them straight back out.
Route it through DirEntrySink instead. ReadDir and ReadDirPlus pass the
reply buffer, so nothing changes for the FUSE server.
* mount: pin go-fuse v2.9.4 for the windows build
* helm: give the bucket hook the credentials weed shell needs
The post-install bucket hook pipes s3.configure into `weed shell`. Since
#9442 the filer only serves its IAM gRPC service to callers presenting an
admin-signed JWT, and since #9536 `weed shell` mints that token itself -
but only if it can find the filer signing key. The hook job sees neither
a security.toml nor the WEED_* environment overrides: its env list is
hardcoded, and it is the only workload in the chart without a
security.toml mount.
So with jwtSigning.filerWrite=true the hook logs
error: failed to get user anonymous: rpc error: code = Unauthenticated
desc = missing authorization metadata
and `weed shell` exits 0 regardless, which leaves the Job green while
anonymousRead is never applied.
Render the merged extraEnvironmentVars into the job's env - keeping
non-string values as valueFrom, so keys held in a Secret stay a
reference - and mount security.toml the way every other workload does.
The hardcoded WEED_CLUSTER_* entries go away because those values are
part of the global extraEnvironmentVars defaults and would otherwise
render twice.
Signed-off-by: Sebastian Preisner <preisner@puzzle-itc.de>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm ci: report a missing bucket hook Job instead of crashing
Signed-off-by: Sebastian Preisner <preisner@puzzle-itc.de>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm ci: assert where the bucket hook mounts security.toml
A mount under the wrong path leaves weed shell without the key just as
surely as no mount at all, so check the target, not only the name.
Signed-off-by: Sebastian Preisner <preisner@puzzle-itc.de>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm: keep the bucket hook's cluster endpoints chart-computed
Dropping the hardcoded WEED_CLUSTER_* left the readiness waits
dereferencing $WEED_CLUSTER_SW_*, which exist only while the values keep
the default alias. Rename the cluster, or clear extraEnvironmentVars, and
the hook polls "http://" for five minutes and then fails the release.
Derive the env names from the alias and render the addresses from the
chart, as all-in-one already does.
* helm ci: assert the bucket hook follows a renamed cluster alias
Renames the cluster and drops the default addresses, then checks that
every env name the readiness waits dereference is set, and that the
address is the chart's rather than the one left in the values.
* helm ci: check the hook's endpoints and security.toml source
The bucket hook tests took two shortcuts a wrong render slips through.
The alias test only rejected the stale master address the values kept, so
any other wrong address passed and the filer address was never compared at
all. And the security.toml test matched the volume by name, which a
same-named emptyDir or a foreign ConfigMap satisfies while `weed shell`
still has no signing key.
Compare both cluster addresses against the Services the chart renders, in
the default and the renamed-alias case, and assert the volume is backed by
the chart's security-config ConfigMap. Checked by pointing the volume at
another ConfigMap and the filer env at a wrong but non-empty address:
both now fail, both passed before.
Signed-off-by: Sebastian Preisner <preisner@puzzle-itc.de>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Sebastian Preisner <preisner@puzzle-itc.de>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* helm: optional NetworkPolicy per component
In a namespace with a default-deny policy the chart cannot be installed:
the components never reach each other, and the post-install bucket hook
waits on the master and filer until it gives up.
networkPolicy.enabled renders one policy per component, selecting its
pods by the standard app.kubernetes.io labels and admitting the other
pods of the release on the ports that component listens on. The port
lists come from the same values as the containerPorts, and CI asserts
the two agree. Restricting egress is a second opt-in with extraEgress
for the filer store and notification sinks, which the chart cannot
know about.
Closes#10421
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm: say "changed" instead of "retuned" in the policy comments
codespell reads "retuned" as a misspelling of "returned" and fails the
spelling job.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm: refuse empty port and DNS peer lists instead of widening
In a NetworkPolicy an empty ports list means every port and a missing peer
selector means every pod, so `kubeApiServer.ports: []` silently opened the
API server CIDRs on all ports, and nulling a DNS selector rendered
`podSelector: null`, which is every pod in kube-system. Both now fail the
render, and the DNS rule emits only the selectors that are set.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm: gate the bucket hook Job and its policy on one helper
Both were deriving the same condition from the same values, kept in step by
a comment. seaweedfs.bucketHookEnabled makes it one definition, so adding an
S3 mode cannot leave the Job running without its policy - which under
default-deny means the hook hangs. CI pins the pairing across the eleven
modes that decide it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm: say that an egress default-deny needs both toggles
networkPolicy.enabled on its own only covers a default-deny that restricts
ingress. Where Egress is in its policyTypes as well, which is the usual
baseline, the components still cannot resolve DNS and egress.enabled is
required too. Both values and the README said the first half of that.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm: quote the label values the policies emit
The same int-coercion the hook templates were fixed for, in the file this PR
adds: unquoted, a release named 123 renders app.kubernetes.io/instance as a
YAML integer in the metadata, the podSelector and both peer selectors, and
the API server rejects the object - a policy that silently never applies.
CI now renders the chart as release "123" and fails if any policy label
comes out as a non-string.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm: gate the resize hook policy on the same lookup as its Job
The policy rendered whenever the hook was enabled, which is the default, so
every release carrying networkPolicy got one - as a pre-install hook that
Helm never collects, left in the namespace after uninstall. It also made
egress.kubeApiServer.cidrs mandatory for every release, since the policy
claims the API server, for a Job that only runs on an upgrade that grows a
PVC.
Move the command computation the Job is gated on into a helper and read it
from both.
* helm: drop the API server rule from the admin policy
No seaweedfs binary talks to the Kubernetes API - there is no client-go in
go.mod - so the rule granted admin an egress path it never uses, and forced
anyone running admin with egress on to name an API server address for it.
The pod-RW ClusterRole the comment was reasoning from is a leftover from a
migration and is not read by any component.
* helm: reject a networkPolicy.components key that names no component
The component names are not guessable - objectstorage-provisioner,
seaweedfs-all-in-one, volume-<name> - and a typo silently dropped the rules
it was carrying. Check against every component the chart can produce, not
the enabled ones, so a values file shared across releases can still hold
overrides for a component this one leaves off.
* helm: name the all-in-one policy after the workload
componentName prefixes the release fullname onto the suffix it is given, and
the component label already starts with seaweedfs-, so the policy came out as
<release>-seaweedfs-seaweedfs-all-in-one.
* helm ci: assert the denied probe failed rather than that it did not succeed
kubectl run's "pod/x created" was captured alongside the pod log, so an
empty log would still not match exit=0 and the denial would pass without
anything having been tested.
* helm: document what turning the network policies on costs
Three things the values did not say: a Prometheus outside the release stops
scraping and nothing reports it, the resize hook's policy is a hook resource
that uninstall leaves behind, and the DNS selectors are wrong on OpenShift.
* helm: spell out the managed distributions codespell reads as a typo
codespell has AKS in its dictionary as a misspelling of ASK, so the DNS
selector note failed the spelling job.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm ci: read the probe verdict from the marker line, not the whole log
The default-deny step decides both probes from the pod log, but kubectl logs
returns stderr as well, and busybox wget reports "download timed out" there
even under -q. The denied probe therefore produced two lines beginning with
wget:, which matches neither exit=0 nor exit=*, so a correct denial landed in
the arm meant for a probe that produced no result at all and failed the job.
The earlier form hid this behind a catch-all that treated anything without
exit=0 as a denial; tightening that assertion made the stray line fatal
without narrowing the input it reads.
Pick the marker line out instead. The trailing || true is required: the step
runs under bash -e, so a grep that matches nothing would abort it rather than
reach the arm that reports an empty result, which is the case that assertion
exists to catch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Sebastian Preisner <preisner@puzzle-itc.de>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* helm: label the hook Job pods so selectors can reach them
The bucket hook pod carries only managed-by/instance and the volume
resize hook pod carries no labels at all, so nothing keyed on the
standard app.kubernetes.io set can address either of them. Give both
the same name/chart/managed-by/instance/component labels the other
workloads use, with the component naming each hook.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm ci: assert the whole label set on the Job and its pod
The block checked three keys and only looked at the pod's component, so a
missing chart/managed-by label, a wrong component on the Job, or a Job and
pod that disagree would all have passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm ci: compare the hook's release labels against a real workload
Presence alone let a wrong value through. The name/instance/chart values now
have to match a workload that already renders them, which also keeps the
chart version out of the test. managed-by stays a presence check: the chart
puts it on workload metadata but not on pod templates.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* helm: quote the resize hook's managed-by and instance labels
Matches the bucket hook, and keeps a numeric release name a string instead
of an int the API server rejects.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Sebastian Preisner <preisner@puzzle-itc.de>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* ci: move the fusermount3 repair into a composite action
Three copies of the same block were already drifting apart, and the
target comes from PATH: only ever add setuid root to a root-owned,
non-symlink binary under the system bin paths, and say why otherwise.
* test: say that the process exited in the wait errors
"process exit status 1 before ... accepted connections" is missing its
verb. Also mark the SIGTERM return discarded - it fails with
os.ErrProcessDone exactly when the select below already handles it.
* ci: prefer the distro fusermount3 over escalating a shadow copy
The shadowing /usr/local/bin/fusermount3 is not root-owned either, so
setting its setuid bit would have handed root to a binary the runner
user owns - the repair now symlinks the distro one earlier in PATH and
touches nothing, keeping the in-place chmod for a root-owned binary with
no distro alternative. A setuid bit only grants root when root owns the
file, so accept an existing one only then.
* ci: run the FUSE workflows when the shared action changes
Their paths filters listed each workflow file but not the composite
action all three now call.
* ci: restore the setuid bit on a shadowed fusermount3
Newer ubuntu-22.04 runner images carry a source-built fusermount3 in
/usr/local/bin that shadows the distro one in PATH and is not setuid
root. go-fuse looks the helper up through PATH, so every unprivileged
mount fails with "mount failed: Operation not permitted".
* test: fail a fuse test as soon as its mount process dies
A mount that cannot mount at all exits within a second, but the harness
still waited out the 30s readiness timeout and then reported "mount
point not ready within timeout", leaving the real cause buried in the
log tail. Watch the child processes and report their exit instead.
* mount: report a failed mount without a goroutine dump
A mount failure is an environment problem - no /dev/fuse, fusermount not
setuid, stale mount point - and the all-goroutine stack dump Fatalf adds
buries the one line that says so.
The copying and tagging tests force-drop each bucket's collection at the
master so volume slots are freed deterministically between tests. But the
copy-tests CI job runs its master on 9336 and the tagging Makefile on 9338,
while the tests default to 9333 — the cleanup dialed a dead port and quietly
no-oped. Each test bucket then grows 7 volumes against -volume.max=100, and
whenever async deletion lagged the data node ran out of slots and PutObject
500ed with "No writable volumes and no free volumes left".
Set MASTER_ENDPOINT where the master port is non-default: the copy-tests
workflow step, and the copying/tagging Makefiles (derived from MASTER_PORT).
ci: build telemetry server from its own module in deploy workflow
telemetry/server has had its own go.mod since #9924, so building
./telemetry/server/main.go from the repo root fails with 'no required
module provides package'. Build from within the module instead.
* Add GitHub Actions workflow for codespell on master
* Add rudimentary codespell config
* Tune codespell config: skip generated code, ignore camelCase, whitelist domain terms
Add camelCase/PascalCase regex to ignore common Go/Rust/JS identifiers
like allLocations, publishErr, ReadInside, FlushInterval. Also skip
templ-generated *_templ.go files, and whitelist a handful of
short/domain-specific words (visibles, fo, te, ser, bject, unparseable,
keep-alives, tread, anc, ue) that show up as false positives across the
tree.
Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Fix ambiguous typos and protect false positives
Fixes typos that codespell reports with multiple candidate suggestions
(so `codespell -w` cannot auto-apply them), plus one inline pragma and
one config entry to protect legitimate identifiers.
Manual fixes (single correct answer chosen from context):
- pattens -> patterns (5x) in filer/upload/shell flag help strings
- finded -> found (2x) in tarantool storage.lua comment
- spacify -> specify (2x) in helm chart values.yaml comment
- wether -> whether in skiplist.go docstring
- simpe -> simple in mq schema test case name
False-positive protection:
- Add `//codespell:ignore` next to `source GET's` (possessive of HTTP
verb) in s3api_object_handlers_copy_stream.go
- Whitelist `auther` in .codespellrc — it's a local variable meaning
"authenticator" in weed/security/tls.go, not a typo of "author".
Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Extend codespell ignore list: .git-meta path and thirdparty groupId
Also skip `.git-meta` (scratch dir for commit messages that may contain
typo words verbatim) and whitelist `thirdparty` — it appears as the
literal Maven groupId `org.apache.hadoop.thirdparty` in hdfs3 poms
and cannot be renamed.
Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* [DATALAD RUNCMD] Fix non-ambiguous typos with codespell -w
Auto-applied fixes to the 44 remaining single-suggestion typos across
docs, comments, log messages, tests, config, and one Java pom.
=== Do not change lines below ===
{
"chain": [],
"cmd": "uvx codespell -w",
"exit": 0,
"extra_inputs": [],
"inputs": [],
"outputs": [],
"pwd": "."
}
^^^ Do not change lines above ^^^
* Revert breaking codespell fixes; whitelist unknwon and atleast
Two of the auto-applied `codespell -w` fixes were false positives that
would break the build/tests:
- go.mod: `github.com/unknwon/goconfig` is a real Go module path — the
upstream author's GitHub handle is literally `unknwon`. Renaming to
`unknown` would fail dependency resolution.
- test/benchmark/fuse_db/bin/{sqlite_verify.py,run_mysql.sh,run_sqlite.sh}:
`atleast` is a literal CLI mode value (a string constant compared and
passed as a positional argument). Rewriting to `at least` splits it
into two arguments and breaks the mode check.
Reverted those files and whitelisted both words in .codespellrc so
future runs won't re-suggest the same broken fixes.
Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* helm: generate the SFTP host key per install
The SFTP secret template shipped one fixed ed25519 host key, so every
install that did not override it presented the same host identity.
Generate the key at install time instead, following the
getOrGeneratePassword pattern: an existing secret keeps its key across
upgrades, except the previously bundled one, which is replaced with a
freshly generated key on the next upgrade.
* helm: create the SFTP host-keys secret the deployments mount
Both the sftp and all-in-one deployments mount /etc/sw/ssh from
<fullname>-sftp-ssh-secret, but no template created it, so a default
install could not start its pod and host keys only reached the server
when enableAuth happened to mount them elsewhere. Create the secret
with a generated ed25519 key, keeping whatever keys an existing secret
already holds. The sshPrivateKey default becomes empty: the file it
pointed at only exists when enableAuth mounts /etc/sw, and a configured
but missing key file is fatal to the server, while hostKeysFolder now
always has a key.
* helm: test SFTP host key generation and secret lifecycle
Template checks: keys render into the secret the deployments mount,
parse as PKCS#8 ed25519, differ between installs, and the render
carries no key material from the chart itself; existingSshConfigSecret
and all-in-one wiring covered. On the kind cluster, exercise the
secret lifecycle: a generated key survives upgrades, the key earlier
chart versions bundled is replaced, and operator-managed keys are kept
untouched. chart-testing now also installs with sftp enabled, where
the pod only becomes ready if the server loads the generated host
key.
* helm: treat a whitespace-only stored SFTP host key as missing
A whitespace-only secret value skipped regeneration and then rendered
an empty key file.
* helm: mount the SFTP host keys secret at the configured hostKeysFolder
The secret was mounted at a fixed /etc/sw/ssh, so a custom
sftp.hostKeysFolder pointed the server at an empty directory. Mount at
the configured path in both the sftp and all-in-one deployments, and
pin flag/mount agreement in the rendering tests.
The dev container workflow prebuilds the Rust volume server from
seaweed-volume/ but did not watch that directory, so Rust build fixes
landed without this workflow validating them.
* fix(master): let the growth initiator wait for the growth it triggered
The growth-in-flight shed also fired on the request that initiated the
growth: it sets the pending flag right before the shed check, so a
cold-start assign enqueued growth and immediately failed itself with
"volume growth in progress". With no concurrent assigns around to pick
up the freshly grown volume, a single writer against an empty cluster
never completes a write despite ample free space.
Claim the pending flag with a compare-and-swap so exactly one request
becomes the initiator, triggering growth at most once, and let it wait
for that growth to land. Everyone else still sheds retryably instead of
pinning a goroutine: followers behind an in-flight growth, an initiator
whose growth concluded without yielding a writable volume, and an
initiator whose growth outlives the 10s wait budget, which previously
surfaced a non-retryable error (gRPC Unknown, HTTP 406) even though a
retry would have succeeded moments later.
* fix(master): stop assign waits when the request is cancelled
The assign retry loops slept through client cancellation, keeping a
goroutine spinning for the rest of the 10s budget after the caller had
gone; StreamAssign also ran assigns on a background context detached
from the stream. Wait on the request context and pass the stream
context through.
* topology: drop the unconditional grow-request setter
Growth is only claimed through AddGrowRequestIfAbsent's compare-and-swap
now; keeping the raw Store(true) around invites the check-then-set race
back.
* test: cover cold-start first write with a real cluster
Boot a fresh master plus three empty volume servers and require the very
first assign - HTTP and gRPC, each on a cold volume layout, no client
retries - to complete a write. The assign that triggers volume growth
must wait for it rather than answering "volume growth in progress";
unit tests stub the topology, so only a real cluster exercises the
assign-grow-wait path end to end.
* ci: add per-process memory sampler for perf jobs
Samples VmRSS once a second into a CSV and records peak VmHWM per process
on stop. Linux only; reads /proc/<pid>/status.
* ci: run perf benchmarks on the Rust volume server and report memory
Matrix the throughput and S3 jobs over go/rust volume servers, using a
standalone master (plus filer for S3) and swapping only the volume binary
so the two are directly comparable. Sample peak RSS in every job and surface
it per impl in the run summary.
* ci: harden mem sampler arg handling and peak fallback
Guard against missing args under set -u, and fall back to the max RSS
sampled when a process exits before VmHWM can be read.
Drop max-parallel so the 13 per-platform builds run together instead of two
waves of 8 (rocksdb was queuing behind the cap and starting ~8 min late).
Keep cache-to mode=max for rocksdb: its RocksDB static_lib compile is
sha-independent, so it caches across releases and stops being the ~16-min
long-pole that gates the merge fan-in. go-build variants stay mode=min.
docker release: build per-platform on native runners, drop mode=max cache
The build job built every platform of a variant on one runner, so 2-4 Go
cross-compiles fought over a single 2-vCPU box and arm64 ran in an emulated
context. Split the matrix to one platform per job on a native runner
(amd64/386 on ubuntu-latest, arm64/arm-v7 on ubuntu-24.04-arm); only arm/v7
still needs QEMU, and only for its final apk stage. Each job pushes by
digest, and a new merge job assembles the multi-arch tag with imagetools
and mirrors it to Docker Hub.
cache-to mode=max -> mode=min: BRANCH=sha cache-busts the heavy go-build
layer every release, so writing all intermediate layers to the gha backend
spent 3-11 min per variant on a cache the next release's sha can never hit.
* test: add self-contained S3 read/write load tool
Concurrent PUT/GET against the S3 gateway, reporting requests/sec,
transfer rate, and latency percentiles. Built on the aws-sdk-go-v2
client the S3 tests already use, so no extra benchmark binary is needed.
* ci: add performance workflow
Three parallel jobs: cpu/heap pprof of the server under write load,
native throughput via weed benchmark plus the Go micro-benchmarks, and
an S3 read/write benchmark against the gateway. Runs on push to master
and manual dispatch with tunable duration, object count, size, and
concurrency.
Add path filters to workflows that fired on every PR/push regardless
of the diff: CodeQL, go build, the e2e/EC/vacuum/TLS/plugin-worker
integration suites, the Kafka and Postgres gateways, the S3 suites
(Ceph s3tests, s3-go, s3-tables, proxy-signature, https, example,
filer-group), TUS, and the dev binary/container builds. Each scopes
to its subsystem under weed/, its test dir, go.mod/go.sum, and the
workflow file, so docs-, helm-, terraform-, rust- or java-only
changes no longer trigger a full compile-and-test fleet.
386 test binaries execute natively on the amd64 runner, so the suite
catches what vet cannot: unaligned 64-bit atomics and arithmetic that
wraps at runtime. -short keeps the e2e suites on amd64 only.
* fix(tests): keep EC e2e fid cookie arithmetic in uint32
The cookie constants 0x9490CA00 and 0x9500CA00 were added to the int
loop variable before conversion, overflowing 32-bit int at compile
time on linux/386 and linux/arm. Convert the loop variable instead so
the addition stays in uint32.
* fix(tests): pass s3client max backoff in milliseconds
MaxBackoffDelay is documented as milliseconds and multiplied by 1e6
before use, but the example set it to 5s in nanoseconds, yielding an
absurd backoff on 64-bit and a compile-time int overflow on 32-bit.
* ci: type-check code and tests for linux/386
64-bit-only constant arithmetic keeps slipping into test files and
breaking 32-bit downstream builds. Vet the whole root module under
GOOS=linux GOARCH=386 so these fail in CI instead of after release.
* fix(tests): convert s3client backoff to Duration before scaling
The ms-to-ns multiplication ran in int, wrapping at runtime on 32-bit;
scale by time.Millisecond after the Duration conversion instead.
* ci(s3tables): drop docker pre-pull from Lakekeeper job
The lakekeeper repro is pure Go against the local weed binary; the job
kept failing on Docker Hub timeouts pulling python:3 and localstack
images the test never runs. Also drop the stale python-in-docker
comments left from the old harness.
* ci(s3tables): serve python:3 from GHA cache in the STS job
Retried pulls still die when both mirror.gcr.io and registry-1.docker.io
are unreachable from the runner. Cache the saved image tarball under a
weekly key: an exact hit skips the registry entirely, a miss pulls fresh
and refreshes the cache, and a stale tarball from a previous week is the
fallback when Docker Hub is down.
* ci(spark): pre-pull the spark tag the test actually runs
The workflow warmed apache/spark:3.5.8 with retries while the
testcontainers setup runs apache/spark:3.5.1, so the real image was
pulled at test time with no retry at all.
* ci(s3tables): route Docker Hub pulls through mirror, drop unused buildx
The integration jobs set up docker/setup-buildx-action only to docker
pull/run images; the buildx bootstrap pulls moby/buildkit from
registry-1.docker.io, which times out and fails the whole job before any
test runs. These jobs never docker build with buildx, so the setup is
pure overhead and an extra registry dependency.
Replace it with a daemon registry-mirror pointing at mirror.gcr.io (a
pull-through cache for Docker Hub) and retry the pre-pulls a few times.
That removes the buildkit pull entirely and routes the rest through the
cache, with graceful fallback to Docker Hub on a miss.
* ci: route Docker Hub through mirror in remaining docker test workflows
Same registry-1.docker.io timeout fix for the other integration jobs.
s3-spark only docker pulls/runs an image, so drop the vestigial buildx
setup and pull through the mirror with retries, matching s3-tables.
kafka-quicktest, s3-proxy-signature, e2e and postgres build/compose and
genuinely need buildx (e2e/postgres export a local layer cache, which the
default driver can't), so keep it and just configure the mirror first —
that way even the moby/buildkit bootstrap pull is served from the cache.
Left samba/pjdfstest alone: they build-push to a local registry and pull
from localhost, so buildx is required and there's no Docker Hub runtime
pull to mirror.
Chris LuandGitHub
Kobi HikriandGitHub
