mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-30 03:35:37 +00:00
7643f4f541
* rust: a seaweed-common crate for the address and TLS helpers both crates carry seaweed-volume and seaweed-worker are separate cargo trees with separate lockfiles and no root manifest, so anything both of them need has had to be written twice. Two of those copies are a correctness risk rather than a typing cost, and this crate is where they stop being copies. address.rs is the HTTP<->gRPC port rule: `host:port` means gRPC on port+10000, `host:port.grpcPort` names it outright. The two copies had already drifted — the worker's bracketed IPv6 literals, the volume server's did not — so the rule lives here once, returning a typed AddressError whose Display text is the volume server's original wording, with join_host_port public beside it. A test asserts two of those messages in full rather than by substring, because the wording is the contract its callers hand to a Status or an io::Error; the other three end in a std ParseIntError message, which is std's to reword. The enum is #[non_exhaustive] so a future variant is not a breaking change for either consumer. The tests are both crates' cases together, plus the IPv6, already-bracketed and normalisation cases neither copy covered on its own. tls.rs is install_default_crypto_provider. Both binaries link aws-lc-rs and ring transitively, so rustls cannot auto-select and tonic's client TLS panics on first use; each binary has to pin one and it has to be the same one, which is exactly the kind of choice that should not exist twice. It is safe to share because `cargo tree -i rustls` resolves a single rustls in each tree (0.23.37 in seaweed-volume, 0.23.43 in seaweed-worker) and cargo unifies all semver-compatible `rustls = "0.23"` requirements into one crate per binary, so this crate writes the same process-wide static its consumer reads. rustls is already in both graphs — directly in the volume server, through tonic's tls-aws-lc in seaweed-worker-core — so the dependency adds no crate to either. rust-version is 1.91.1, the lower of the two consumers' floors, so depending on this crate cannot raise either tree's MSRV; verified with `cargo +1.91.1 check --all-targets`. The lockfile is committed even though this is a library: CI builds it directly, so a committed lock is what makes those runs reproducible and their caches stable. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * rust: take the address and TLS helpers from seaweed-common Both public signatures are kept, so no caller outside the two wrapper files changes. parse_grpc_address stays `Result<String, String>` and maps the typed error through Display; server_to_grpc_address stays `Option<String>` and drops it with .ok(). Their doc comments and the volume server's 13 call sites are otherwise untouched. Three behaviours change, each in the direction of the copy that was already right: - The volume server now brackets IPv6 literals. `::1:19333` used to come back as `::1:29333`, which build_grpc_endpoint rejects with "invalid gRPC endpoint http://::1:19333: invalid authority" — an IPv6 master or EC peer could not be dialled at all. Two tests in grpc_client.rs pin it, one on the string and one on the endpoint the string builds. - The volume server now emits the *parsed* gRPC port of the dotted form instead of the original text it had just validated, so `host:8080.018080` and `host:8080.+18080` come back as `host:18080` rather than as authorities the URI parser rejects. Same port either way; only malformed spellings change. - The worker's dotted form now validates the HTTP port it discards. `server_to_grpc_address("host:abc.18080")` used to answer Some("host:18080"); it now answers None, which is what the volume server's copy has always done. install_default_crypto_provider becomes a re-export in both trees, so `crate::security::tls::install_default_crypto_provider` and `weed_lance_worker::tls::install_default_crypto_provider` still resolve. The lance crate's `rustls = "0.23"` was its only direct use of rustls and goes away with the body; seaweed-common states the same requirement, so neither the resolved version nor the enabled features move in either lockfile. The PEM test fixtures stay where they are. The two tests that use them are not duplicates: the volume server's exercises build_grpc_endpoint, and the lance one exists precisely because aws-lc-rs and ring are both linked in that crate's graph. Only the literals are shared, and exporting test fixtures from a library to dedupe two constants costs more than it saves. A path dependency outside both trees means every build context that copies one crate directory has to copy the other. The repo has one: the Rust source-build stage of docker/Dockerfile.go_build, which now copies seaweed-common beside seaweed-volume. Every workflow whose `paths:` filter keys on a crate directory gains `seaweed-common/**` — the two Rust test workflows, rust_binaries_dev, container_dev and performance. The tag- and dispatch-triggered ones (rust_binaries_release, container_release_unified, container_latest) have no `paths:` filter and need nothing. The two Rust test workflows also run `cargo test` in seaweed-common, from their unit-test job, because a path dependency is not a workspace member and neither tree's own `cargo test` reaches it. Each step builds into its job's cached target directory, and both cache keys now hash seaweed-common/Cargo.lock as well so a change there invalidates the cache it would otherwise silently reuse. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docker: keep go_build working for BRANCH revisions without seaweed-common The rust_builder stage copies seaweed-common unconditionally now that seaweed-volume path-depends on it, but BRANCH can name any revision — including ones that predate the crate. Create the directory in the builder stage so the COPY always has a source; an empty dir beside an old seaweed-volume is harmless. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>
115 lines
5.3 KiB
Markdown
115 lines
5.3 KiB
Markdown
# SeaweedFS Rust workers
|
|
|
|
`weed/pb/plugin.proto` is a language-agnostic contract: a maintenance worker
|
|
connects out to admin, announces the job types it can detect and execute, and
|
|
answers requests on that one stream. `weed worker -admin=host:23646` is the Go
|
|
implementation of it from outside the admin process. This workspace is the Rust
|
|
one.
|
|
|
|
crates/core the contract: stream, handshake, heartbeat, registry, config forms
|
|
crates/lance maintenance jobs for Lance tables, and a binary
|
|
|
|
It also depends on `../seaweed-common`, a small crate outside this workspace
|
|
holding the few helpers the Rust volume server needs identically — the
|
|
HTTP<->gRPC address rule and the rustls provider choice.
|
|
|
|
`core` knows nothing about any job. A second worker is a new crate beside
|
|
`lance` that depends on it, not a fork of the protocol.
|
|
|
|
## Building
|
|
|
|
Requires Rust 1.94.1+ (2024 edition), matching `rust-version` in `Cargo.toml`.
|
|
The patch release matters: 1.94.0 does not build. The edition itself only needs
|
|
1.85; the higher floor comes from the dependency tree — lance's `aws` feature
|
|
pulls in the AWS SDK — so it moves with those crates. CI builds on the latest
|
|
stable.
|
|
|
|
`core` compiles `plugin.proto` with the protoc that protoc-bin-vendored ships,
|
|
the way seaweed-volume does, so it needs no system install.
|
|
|
|
The lance crates compile protos of their own, in their own build-script
|
|
processes, which nothing our build script sets can reach. They need a protoc of
|
|
their own: either one on PATH — `brew install protobuf`, `apt install
|
|
protobuf-compiler` — or `PROTOC` naming one. CI points it at the vendored
|
|
binary for the runner's platform, resolved from the version in `Cargo.lock`.
|
|
|
|
## Running
|
|
|
|
cargo run -p weed-lance-worker -- --admin 127.0.0.1:23646
|
|
|
|
The admin's *HTTP* address is what an operator has; the gRPC port is derived
|
|
from it the way the Go side does. Dialling the HTTP port fails as "frame with
|
|
invalid size", which reads like a protocol bug rather than a wrong port.
|
|
|
|
The binary is `weed-worker`, not `weed-lance-worker`: it is the Rust side of
|
|
`weed worker`, and lance is the first family of jobs it carries rather than the
|
|
only one it ever will.
|
|
|
|
Released builds do not need a toolchain. The worker ships inside the SeaweedFS
|
|
image, beside the Rust volume server, under the verb that mirrors
|
|
`volume-rust`:
|
|
|
|
docker run chrislusf/seaweedfs worker-rust --admin admin:23646
|
|
|
|
and as `weed-worker_linux_{amd64,arm64}.tar.gz` on each GitHub release. Both are
|
|
linux amd64/arm64 only — lance, arrow and datafusion make every extra target an
|
|
expensive build, and the worker runs beside the cluster it maintains. On an
|
|
architecture without a build the image carries an empty placeholder and the
|
|
entrypoint says so rather than failing as "not found".
|
|
|
|
## Metrics
|
|
|
|
cargo run -p weed-lance-worker -- --admin 127.0.0.1:23646 --metrics-port 9328
|
|
|
|
Serves `/health`, `/ready` and `/metrics` on that port, the same three the Go
|
|
worker serves under `weed worker -metricsPort`, so one scrape config covers
|
|
workers in either language. Off by default, and bound to loopback unless
|
|
`--metrics-ip` says otherwise, because the endpoint is unauthenticated. 9328
|
|
continues the series the other components use (master 9324, volume 9325, filer
|
|
9326, s3 9327); an IPv6 address works with or without brackets.
|
|
|
|
Grafana: the "Plugin Workers" row of `other/metrics/grafana_seaweedfs.json`
|
|
graphs these. Its panels filter on `$cluster`, which comes from the scrape job's
|
|
labels, so scrape the worker the way the rest of the cluster is scraped or the
|
|
row stays empty.
|
|
|
|
Names are `SeaweedFS_worker_*`, matching the Go side's convention. The pair
|
|
worth alerting on is `objects_seen_total` and `objects_skipped_total`: a sweep
|
|
that proposes nothing and a sweep that could read nothing look identical from
|
|
`proposals_total` alone.
|
|
|
|
SeaweedFS_worker_connected 1
|
|
SeaweedFS_worker_objects_seen_total{job_type="lance_compact"} 7
|
|
SeaweedFS_worker_proposals_total{job_type="lance_compact"} 2
|
|
SeaweedFS_worker_jobs_total{job_type="lance_compact",result="ok"} 2
|
|
SeaweedFS_worker_lance_fragments_removed_total 25
|
|
|
|
`/ready` follows the control stream: a worker whose admin has gone away is
|
|
running but is not going to do anything.
|
|
|
|
## Credentials
|
|
|
|
The worker holds none. It asks the namespace to describe a table with
|
|
`vend_credentials` and hands the `storage_options` that come back to lance. A
|
|
gateway without STS configured vends no credentials at all, so `--access-key`
|
|
and `--secret-key` supply a fallback; anything the namespace does vend wins over
|
|
them.
|
|
|
|
## State
|
|
|
|
All three jobs are implemented and tested end to end against a live gateway:
|
|
|
|
compaction result: 12 fragments became 1
|
|
reindex result: 512 uncovered rows became 0
|
|
cleanup result: removed 14 versions and 24272 bytes
|
|
|
|
`cargo test -p weed-lance-worker` runs them when `WEED_LANCE_NAMESPACE` names a
|
|
live namespace and skips otherwise, the way the Go integration tests skip
|
|
without Docker. Each test seeds the table it needs, including building a vector
|
|
index and then appending rows outside it, so a run does not depend on what the
|
|
previous one left behind — the first version of these did, and quietly stopped
|
|
testing anything once it had done its job.
|
|
|
|
The handshake, descriptor exchange and heartbeat work against a live admin,
|
|
which logs the worker connecting and prefetches all three descriptors.
|