* rust: a seaweed-common crate for the address and TLS helpers both crates carry
seaweed-volume and seaweed-worker are separate cargo trees with separate
lockfiles and no root manifest, so anything both of them need has had to be
written twice. Two of those copies are a correctness risk rather than a typing
cost, and this crate is where they stop being copies.
address.rs is the HTTP<->gRPC port rule: `host:port` means gRPC on port+10000,
`host:port.grpcPort` names it outright. The two copies had already drifted —
the worker's bracketed IPv6 literals, the volume server's did not — so the rule
lives here once, returning a typed AddressError whose Display text is the volume
server's original wording, with join_host_port public beside it. A test asserts
two of those messages in full rather than by substring, because the wording is
the contract its callers hand to a Status or an io::Error; the other three end
in a std ParseIntError message, which is std's to reword. The enum is
#[non_exhaustive] so a future variant is not a breaking change for either
consumer. The tests are both crates' cases together, plus the IPv6,
already-bracketed and normalisation cases neither copy covered on its own.
tls.rs is install_default_crypto_provider. Both binaries link aws-lc-rs and ring
transitively, so rustls cannot auto-select and tonic's client TLS panics on
first use; each binary has to pin one and it has to be the same one, which is
exactly the kind of choice that should not exist twice. It is safe to share
because `cargo tree -i rustls` resolves a single rustls in each tree (0.23.37 in
seaweed-volume, 0.23.43 in seaweed-worker) and cargo unifies all
semver-compatible `rustls = "0.23"` requirements into one crate per binary, so
this crate writes the same process-wide static its consumer reads. rustls is
already in both graphs — directly in the volume server, through tonic's
tls-aws-lc in seaweed-worker-core — so the dependency adds no crate to either.
rust-version is 1.91.1, the lower of the two consumers' floors, so depending on
this crate cannot raise either tree's MSRV; verified with
`cargo +1.91.1 check --all-targets`. The lockfile is committed even though this
is a library: CI builds it directly, so a committed lock is what makes those
runs reproducible and their caches stable.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* rust: take the address and TLS helpers from seaweed-common
Both public signatures are kept, so no caller outside the two wrapper files
changes. parse_grpc_address stays `Result<String, String>` and maps the typed
error through Display; server_to_grpc_address stays `Option<String>` and drops
it with .ok(). Their doc comments and the volume server's 13 call sites are
otherwise untouched.
Three behaviours change, each in the direction of the copy that was already
right:
- The volume server now brackets IPv6 literals. `::1:19333` used to come back as
`::1:29333`, which build_grpc_endpoint rejects with "invalid gRPC endpoint
http://::1:19333: invalid authority" — an IPv6 master or EC peer could not be
dialled at all. Two tests in grpc_client.rs pin it, one on the string and one
on the endpoint the string builds.
- The volume server now emits the *parsed* gRPC port of the dotted form instead
of the original text it had just validated, so `host:8080.018080` and
`host:8080.+18080` come back as `host:18080` rather than as authorities the
URI parser rejects. Same port either way; only malformed spellings change.
- The worker's dotted form now validates the HTTP port it discards.
`server_to_grpc_address("host:abc.18080")` used to answer Some("host:18080");
it now answers None, which is what the volume server's copy has always done.
install_default_crypto_provider becomes a re-export in both trees, so
`crate::security::tls::install_default_crypto_provider` and
`weed_lance_worker::tls::install_default_crypto_provider` still resolve. The
lance crate's `rustls = "0.23"` was its only direct use of rustls and goes away
with the body; seaweed-common states the same requirement, so neither the
resolved version nor the enabled features move in either lockfile.
The PEM test fixtures stay where they are. The two tests that use them are not
duplicates: the volume server's exercises build_grpc_endpoint, and the lance one
exists precisely because aws-lc-rs and ring are both linked in that crate's
graph. Only the literals are shared, and exporting test fixtures from a library
to dedupe two constants costs more than it saves.
A path dependency outside both trees means every build context that copies one
crate directory has to copy the other. The repo has one: the Rust source-build
stage of docker/Dockerfile.go_build, which now copies seaweed-common beside
seaweed-volume. Every workflow whose `paths:` filter keys on a crate directory
gains `seaweed-common/**` — the two Rust test workflows, rust_binaries_dev,
container_dev and performance. The tag- and dispatch-triggered ones
(rust_binaries_release, container_release_unified, container_latest) have no
`paths:` filter and need nothing.
The two Rust test workflows also run `cargo test` in seaweed-common, from their
unit-test job, because a path dependency is not a workspace member and neither
tree's own `cargo test` reaches it. Each step builds into its job's cached
target directory, and both cache keys now hash seaweed-common/Cargo.lock as well
so a change there invalidates the cache it would otherwise silently reuse.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* docker: keep go_build working for BRANCH revisions without seaweed-common
The rust_builder stage copies seaweed-common unconditionally now that seaweed-volume path-depends on it, but BRANCH can name any revision — including ones that predate the crate. Create the directory in the builder stage so the COPY always has a source; an empty dir beside an old seaweed-volume is harmless.
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* ci: composite action that signs and verifies an image with cosign
Keyless, by digest, with a verification pass against the calling workflow's
own identity right after signing. Signatures use the .sig tag layout rather
than the OCI-referrer bundle cosign 3 writes by default, since that is what
the verifiers people run today read. Dependabot is pointed at the action so
the cosign-installer pin keeps moving.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker release: sign every variant on both registries
The merge job signs each variant's multi-arch index on GHCR and Docker Hub
once the tag exists, recursively so the platform images are covered too.
latest re-tags the same manifest and inherits the signature.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker dev: sign the dev image
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker latest: sign a latest rebuilt by hand
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker release: sign the foundationdb image
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker: sign the per-version foundationdb and rocksdb builds
They push to the same repository as the releases, so an admission policy
that verifies chrislusf/seaweedfs would otherwise reject them.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker: document image signature verification
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* ci: pin the actions the signing jobs newly run by commit
These run with registry credentials and the OIDC token that signs under
the repository's identity, so a retargeted tag upstream must not be able
to reach them.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker latest: pass the dispatch tag through env, not the script
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker: complete Kyverno policy, digest note, identity scope
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker latest: keep the dispatch tag out of the manifest script too
The step predates signing, but the job now holds the OIDC identity.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* ci: pin every action in the jobs that sign
The jobs that hold the OIDC identity run these with registry credentials,
so a retargeted tag upstream must not reach them.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker release: copy and sign the digest the run created, pin the rest
crane copy and the signature both resolved the tag, which another
publisher could move between the two steps. The index digest is read once,
right after it is created, and the Docker Hub copy and both signatures use
it. The manual latest rebuild gets the same treatment. The actions in these
jobs are pinned to commits, crane to v0.22.0 by checksum, and the sparse
checkout no longer keeps the token.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker latest: the signing job checks out the workflow's own commit
The job only assembles and signs manifests, so nothing there needs the
source_ref checkout; the local signing action now comes from the same
revision as the workflow file that calls it.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker release: take the index digest from the create result
imagetools create writes the descriptor it pushed with --metadata-file
(buildx 0.32+, the runners ship 0.36), so the digest no longer comes from
re-resolving the tag even within the same step.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* worker: compile plugin.proto with the protoc that ships with the build
seaweed-volume already does this: protoc-bin-vendored carries the binary, so
the build needs no package manager and every build sees the same version. An
explicit PROTOC still wins, which is what lets the lance crates - whose own
build scripts read the same variable - share it.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* ci: point the worker builds at the vendored protoc
The jobs installed protobuf-compiler for lance's build scripts. They read
PROTOC, so pointing it at the binary protoc-bin-vendored already puts in the
registry serves them without a system package - one less apt call on the way
to a release, and the same protoc a developer's build uses.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* docs: say what the worker build needs from protoc
The lance crates' build scripts are the ones that need it, not ours, and they
take the same vendored binary.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* worker: name the binary weed-worker
It is the Rust side of `weed worker`, the way weed-volume is the Rust side of
`weed volume`, and lance is the first family of jobs it carries rather than
the only one it ever will. The crate keeps its own name: when a second family
arrives the bin target moves to a crate of its own, under this name.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* docker: ship the Rust maintenance worker in the image
Lance table buckets need a worker that can read the format, and until now the
only way to get one was a Rust toolchain and a cargo build. It now sits at
/usr/bin/weed-worker beside the Rust volume server, reached as
`docker run chrislusf/seaweedfs worker-rust --admin host:23646` — the verb
mirrors volume-rust, so plain `worker` still runs the Go one.
Taken pre-built or not at all: the lance jobs pull in arrow and datafusion, far
too large a tree to compile inside the image build, so an architecture CI did
not build for gets the empty placeholder the entrypoint refuses to exec, the
way the Rust volume server already does.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* ci: build the Rust worker for the container images
The same native cross-compile the volume server uses, so the release, latest
and dev images all carry it on amd64 and arm64. The artifact holds both
binaries now, so it is named for that rather than for the volume server.
Only the release directory each job builds is cached: with a debug profile
beside it the worker's target/ reaches 24GB, against a 10GB cache budget.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* ci: publish Rust worker binaries with the release
Linux amd64 and arm64 only: the worker runs beside the cluster it maintains,
and its dependency tree makes every extra target an expensive build.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* ci: build and test the Rust workers on change
Nothing built seaweed-worker in CI, so the release and the container images
would have been the first place a break showed up. Tests run in release too,
rather than compiling lance, arrow and datafusion again in another profile.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* docs: say how to get a released worker
Neither the image nor the release tarballs were mentioned; a toolchain and a
cargo build read as the only way in.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* ci: install protoc for the Rust worker builds
lance's crates compile their own protos, and unlike seaweed-volume they do not
vendor a protoc to do it with, so every job that builds the worker failed at
lance-encoding's build script.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* ci: do not persist credentials in the worker release checkout
The upload step is handed a token explicitly; a cargo build script should not
find another one sitting in the checkout's git config.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* docker: keep the worker's argument boundaries
Unquoted $@ splits on whitespace and expands globs, so an argument carrying
either arrived as something else.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
Every workflow that builds the Rust volume server first installed protoc
from a package manager - twelve steps across apt, brew and choco. That is
37s per job on a good day, and this week archive.ubuntu.com stalled long
enough for four jobs to burn their whole timeout without reaching a build.
protoc-bin-vendored ships the compiler as a build-dependency, so it now
arrives through the cargo registry the workflows already cache and there
is nothing left to install. cargo build works on a machine with no protoc
at all, which is worth as much locally as it is in CI.
It also pins the version. The apt protoc on ubuntu-22.04 is 3.12, old
enough to reject proto3 optional, which is why build.rs passes
--experimental_allow_proto3_optional; the vendored one is 31.1. The flag
stays, since it costs nothing and keeps a build against an older PROTOC
working, and an explicit PROTOC still overrides the vendored binary for
packagers who supply their own.
The dev container workflow prebuilds the Rust volume server from
seaweed-volume/ but did not watch that directory, so Rust build fixes
landed without this workflow validating them.
Add path filters to workflows that fired on every PR/push regardless
of the diff: CodeQL, go build, the e2e/EC/vacuum/TLS/plugin-worker
integration suites, the Kafka and Postgres gateways, the S3 suites
(Ceph s3tests, s3-go, s3-tables, proxy-signature, https, example,
filer-group), TUS, and the dev binary/container builds. Each scopes
to its subsystem under weed/, its test dir, go.mod/go.sum, and the
workflow file, so docs-, helm-, terraform-, rust- or java-only
changes no longer trigger a full compile-and-test fleet.
When cross-compiling aws-lc-sys for aarch64-unknown-linux-musl using
aarch64-linux-gnu-gcc, glibc's _FORTIFY_SOURCE generates calls to
__memcpy_chk, __fprintf_chk etc. which don't exist in musl, causing
linker errors. Disable it via CFLAGS_aarch64_unknown_linux_musl.
Both container_latest.yml and container_dev.yml use Dockerfile.go_build
which expects weed-volume-prebuilt/ with pre-compiled Rust binaries, but
neither workflow produced them, causing COPY failures during docker build.
Add build-rust-binaries jobs that natively cross-compile for amd64 and
arm64, then download and place the artifacts in the Docker build context.
Also fix the trivy-scan local build path in container_latest.yml.
Chris LuandGitHub