1df8c05bc3
Three production `unsafe` blocks carried no `// SAFETY:` comment at all (`libc::fallocate`, `libc::sysinfo`, `libc::statvfs`), and nothing made that an error: `clippy::undocumented_unsafe_blocks` is a `restriction` lint, allow-by-default, and appeared nowhere in either crate. Turn it on in `seaweed-volume`'s `[lints.clippy]` and in the worker workspace's `[workspace.lints.clippy]`, then document what each block relies on. `memory_status.rs` and `disk_location.rs` get their blocks narrowed to the `zeroed()` and the libc call, so each comment sits next to the operation it justifies and the arithmetic is outside the block. Both turn the success test into an early return on failure; the casts, the multiplication order and the values returned on either path are unchanged. The bigger problem was in `config.rs`'s tests. `with_temp_env_var` and `with_cleared_security_env` called `std::env::set_var`/`remove_var`, claiming soundness because every caller holds `process_state_lock()`. That mutex only serialises the fourteen annotated tests in this module. The same lib test binary runs the `grpc_server.rs` tests, which bind a `TcpListener`, dial loopback and drive a multi-thread tokio runtime, and tonic/hyper/rustls/aws-sdk all read the environment lazily on those threads — which is exactly the race Rust 2024 made these calls unsafe for. `restore_env_var` had no SAFETY comment at all. `#[serial]` would not have helped: it serialises annotated tests, which the mutex already did. So the config layer no longer reads the environment implicitly. An `EnvLookup<'a> = &'a dyn Fn(&str) -> Option<OsString>` is threaded from the public entry points down to every reader — `HOME`, `USERPROFILE`, the twenty-four `WEED_*` keys and `SEAWEED_WRITE_QUEUE`. `parse_cli` and `parse_security_config` keep their signatures and pass `process_env`, a thin wrapper over `std::env::var_os`; `resolve_config` becomes `resolve_config_with_env` (private, one caller). Tests build one with `fake_env` instead, so no test touches the real environment and every `unsafe` in the module is gone. `process_state_lock()` stays, with a smaller job: `set_current_dir` is safe but still process-global, so the tests that move the working directory are still serialised against the ones that read it. Tests naming an explicit config file never reach that search and no longer take the lock. No production behaviour changes: the same keys are read in the same order with the same precedence, and `env_string` reproduces `std::env::var(key).ok()` — absent and non-UTF-8 both read as unset. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
SeaweedFS Rust workers
weed/pb/plugin.proto is a language-agnostic contract: a maintenance worker
connects out to admin, announces the job types it can detect and execute, and
answers requests on that one stream. weed worker -admin=host:23646 is the Go
implementation of it from outside the admin process. This workspace is the Rust
one.
crates/core the contract: stream, handshake, heartbeat, registry, config forms
crates/lance maintenance jobs for Lance tables, and a binary
core knows nothing about any job. A second worker is a new crate beside
lance that depends on it, not a fork of the protocol.
Building
Requires Rust 1.94.1+ (2024 edition), matching rust-version in Cargo.toml.
The patch release matters: 1.94.0 does not build. The edition itself only needs
1.85; the higher floor comes from the dependency tree — lance's aws feature
pulls in the AWS SDK — so it moves with those crates. CI builds on the latest
stable.
core compiles plugin.proto with the protoc that protoc-bin-vendored ships,
the way seaweed-volume does, so it needs no system install.
The lance crates compile protos of their own, in their own build-script
processes, which nothing our build script sets can reach. They need a protoc of
their own: either one on PATH — brew install protobuf, apt install protobuf-compiler — or PROTOC naming one. CI points it at the vendored
binary for the runner's platform, resolved from the version in Cargo.lock.
Running
cargo run -p weed-lance-worker -- --admin 127.0.0.1:23646
The admin's HTTP address is what an operator has; the gRPC port is derived from it the way the Go side does. Dialling the HTTP port fails as "frame with invalid size", which reads like a protocol bug rather than a wrong port.
The binary is weed-worker, not weed-lance-worker: it is the Rust side of
weed worker, and lance is the first family of jobs it carries rather than the
only one it ever will.
Released builds do not need a toolchain. The worker ships inside the SeaweedFS
image, beside the Rust volume server, under the verb that mirrors
volume-rust:
docker run chrislusf/seaweedfs worker-rust --admin admin:23646
and as weed-worker_linux_{amd64,arm64}.tar.gz on each GitHub release. Both are
linux amd64/arm64 only — lance, arrow and datafusion make every extra target an
expensive build, and the worker runs beside the cluster it maintains. On an
architecture without a build the image carries an empty placeholder and the
entrypoint says so rather than failing as "not found".
Metrics
cargo run -p weed-lance-worker -- --admin 127.0.0.1:23646 --metrics-port 9328
Serves /health, /ready and /metrics on that port, the same three the Go
worker serves under weed worker -metricsPort, so one scrape config covers
workers in either language. Off by default, and bound to loopback unless
--metrics-ip says otherwise, because the endpoint is unauthenticated. 9328
continues the series the other components use (master 9324, volume 9325, filer
9326, s3 9327); an IPv6 address works with or without brackets.
Grafana: the "Plugin Workers" row of other/metrics/grafana_seaweedfs.json
graphs these. Its panels filter on $cluster, which comes from the scrape job's
labels, so scrape the worker the way the rest of the cluster is scraped or the
row stays empty.
Names are SeaweedFS_worker_*, matching the Go side's convention. The pair
worth alerting on is objects_seen_total and objects_skipped_total: a sweep
that proposes nothing and a sweep that could read nothing look identical from
proposals_total alone.
SeaweedFS_worker_connected 1
SeaweedFS_worker_objects_seen_total{job_type="lance_compact"} 7
SeaweedFS_worker_proposals_total{job_type="lance_compact"} 2
SeaweedFS_worker_jobs_total{job_type="lance_compact",result="ok"} 2
SeaweedFS_worker_lance_fragments_removed_total 25
/ready follows the control stream: a worker whose admin has gone away is
running but is not going to do anything.
Credentials
The worker holds none. It asks the namespace to describe a table with
vend_credentials and hands the storage_options that come back to lance. A
gateway without STS configured vends no credentials at all, so --access-key
and --secret-key supply a fallback; anything the namespace does vend wins over
them.
State
All three jobs are implemented and tested end to end against a live gateway:
compaction result: 12 fragments became 1
reindex result: 512 uncovered rows became 0
cleanup result: removed 14 versions and 24272 bytes
cargo test -p weed-lance-worker runs them when WEED_LANCE_NAMESPACE names a
live namespace and skips otherwise, the way the Go integration tests skip
without Docker. Each test seeds the table it needs, including building a vector
index and then appending rows outside it, so a run does not depend on what the
previous one left behind — the first version of these did, and quietly stopped
testing anything once it had done its job.
The handshake, descriptor exchange and heartbeat work against a live admin, which logs the worker connecting and prefetches all three descriptors.