mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-22 07:54:17 +00:00
7643f4f541
* rust: a seaweed-common crate for the address and TLS helpers both crates carry seaweed-volume and seaweed-worker are separate cargo trees with separate lockfiles and no root manifest, so anything both of them need has had to be written twice. Two of those copies are a correctness risk rather than a typing cost, and this crate is where they stop being copies. address.rs is the HTTP<->gRPC port rule: `host:port` means gRPC on port+10000, `host:port.grpcPort` names it outright. The two copies had already drifted — the worker's bracketed IPv6 literals, the volume server's did not — so the rule lives here once, returning a typed AddressError whose Display text is the volume server's original wording, with join_host_port public beside it. A test asserts two of those messages in full rather than by substring, because the wording is the contract its callers hand to a Status or an io::Error; the other three end in a std ParseIntError message, which is std's to reword. The enum is #[non_exhaustive] so a future variant is not a breaking change for either consumer. The tests are both crates' cases together, plus the IPv6, already-bracketed and normalisation cases neither copy covered on its own. tls.rs is install_default_crypto_provider. Both binaries link aws-lc-rs and ring transitively, so rustls cannot auto-select and tonic's client TLS panics on first use; each binary has to pin one and it has to be the same one, which is exactly the kind of choice that should not exist twice. It is safe to share because `cargo tree -i rustls` resolves a single rustls in each tree (0.23.37 in seaweed-volume, 0.23.43 in seaweed-worker) and cargo unifies all semver-compatible `rustls = "0.23"` requirements into one crate per binary, so this crate writes the same process-wide static its consumer reads. rustls is already in both graphs — directly in the volume server, through tonic's tls-aws-lc in seaweed-worker-core — so the dependency adds no crate to either. rust-version is 1.91.1, the lower of the two consumers' floors, so depending on this crate cannot raise either tree's MSRV; verified with `cargo +1.91.1 check --all-targets`. The lockfile is committed even though this is a library: CI builds it directly, so a committed lock is what makes those runs reproducible and their caches stable. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * rust: take the address and TLS helpers from seaweed-common Both public signatures are kept, so no caller outside the two wrapper files changes. parse_grpc_address stays `Result<String, String>` and maps the typed error through Display; server_to_grpc_address stays `Option<String>` and drops it with .ok(). Their doc comments and the volume server's 13 call sites are otherwise untouched. Three behaviours change, each in the direction of the copy that was already right: - The volume server now brackets IPv6 literals. `::1:19333` used to come back as `::1:29333`, which build_grpc_endpoint rejects with "invalid gRPC endpoint http://::1:19333: invalid authority" — an IPv6 master or EC peer could not be dialled at all. Two tests in grpc_client.rs pin it, one on the string and one on the endpoint the string builds. - The volume server now emits the *parsed* gRPC port of the dotted form instead of the original text it had just validated, so `host:8080.018080` and `host:8080.+18080` come back as `host:18080` rather than as authorities the URI parser rejects. Same port either way; only malformed spellings change. - The worker's dotted form now validates the HTTP port it discards. `server_to_grpc_address("host:abc.18080")` used to answer Some("host:18080"); it now answers None, which is what the volume server's copy has always done. install_default_crypto_provider becomes a re-export in both trees, so `crate::security::tls::install_default_crypto_provider` and `weed_lance_worker::tls::install_default_crypto_provider` still resolve. The lance crate's `rustls = "0.23"` was its only direct use of rustls and goes away with the body; seaweed-common states the same requirement, so neither the resolved version nor the enabled features move in either lockfile. The PEM test fixtures stay where they are. The two tests that use them are not duplicates: the volume server's exercises build_grpc_endpoint, and the lance one exists precisely because aws-lc-rs and ring are both linked in that crate's graph. Only the literals are shared, and exporting test fixtures from a library to dedupe two constants costs more than it saves. A path dependency outside both trees means every build context that copies one crate directory has to copy the other. The repo has one: the Rust source-build stage of docker/Dockerfile.go_build, which now copies seaweed-common beside seaweed-volume. Every workflow whose `paths:` filter keys on a crate directory gains `seaweed-common/**` — the two Rust test workflows, rust_binaries_dev, container_dev and performance. The tag- and dispatch-triggered ones (rust_binaries_release, container_release_unified, container_latest) have no `paths:` filter and need nothing. The two Rust test workflows also run `cargo test` in seaweed-common, from their unit-test job, because a path dependency is not a workspace member and neither tree's own `cargo test` reaches it. Each step builds into its job's cached target directory, and both cache keys now hash seaweed-common/Cargo.lock as well so a change there invalidates the cache it would otherwise silently reuse. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docker: keep go_build working for BRANCH revisions without seaweed-common The rust_builder stage copies seaweed-common unconditionally now that seaweed-volume path-depends on it, but BRANCH can name any revision — including ones that predate the crate. Create the directory in the builder stage so the COPY always has a source; an empty dir beside an old seaweed-volume is harmless. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>
Docker
Compose V2
SeaweedFS now uses the v2 syntax docker compose
If you rely on using Docker Compose as docker-compose (with a hyphen), you can set up Compose V2 to act as a drop-in replacement of the previous docker-compose. Refer to the Installing Compose section for detailed instructions on upgrading.
Confirm your system has docker compose v2 with a version check
$ docker compose version
Docker Compose version v2.10.2
Try it out
wget https://raw.githubusercontent.com/seaweedfs/seaweedfs/master/docker/seaweedfs-compose.yml
docker compose -f seaweedfs-compose.yml -p seaweedfs up
Try latest tip
wget https://raw.githubusercontent.com/seaweedfs/seaweedfs/master/docker/seaweedfs-dev-compose.yml
docker compose -f seaweedfs-dev-compose.yml -p seaweedfs up
Verify an image signature
Every image CI pushes to chrislusf/seaweedfs and ghcr.io/chrislusf/seaweedfs is signed with cosign, keyless, by the GitHub Actions workflow that built it, so there is no key to fetch or pin. The signature is attached to the image digest and covers the multi-arch index and each platform image in it; latest is the release image under another tag and verifies the same way. Images published before September 2026 predate signing.
cosign verify \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
--certificate-identity-regexp '^https://github.com/seaweedfs/seaweedfs/\.github/workflows/container_release_unified\.yml@' \
chrislusf/seaweedfs:latest
cosign prints the digest it verified. Deploy by that digest, or let an admission controller resolve the tag, so what runs is what was checked.
The identity is https://github.com/seaweedfs/seaweedfs/.github/workflows/<workflow>@<ref>. The ref is refs/tags/<version> for a release and refs/heads/master when a variant was republished by hand. The workflow is container_release_unified.yml for the release images, container_dev.yml for dev, container_latest.yml for a latest rebuilt by hand, container_release_foundationdb.yml for the _large_disk_foundationdb release image, and container_foundationdb_version.yml or container_rocksdb_version.yml for the per-version builds. The regexp above accepts release images only; container_[a-z_]+\.yml@ accepts everything this repository publishes, dev included.
The same check as a Kyverno policy, release images only:
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: verify-seaweedfs-images
spec:
validationFailureAction: Enforce
webhookTimeoutSeconds: 30
rules:
- name: signed-by-the-release-workflow
match:
any:
- resources:
kinds:
- Pod
verifyImages:
- imageReferences:
- "docker.io/chrislusf/seaweedfs:*"
- "ghcr.io/chrislusf/seaweedfs:*"
attestors:
- entries:
- keyless:
issuer: https://token.actions.githubusercontent.com
subject: https://github.com/seaweedfs/seaweedfs/.github/workflows/container_release_unified.yml@refs/tags/*
Local Development
cd $GOPATH/src/github.com/seaweedfs/seaweedfs/docker
make
S3 cmd
list
s3cmd --no-ssl --host=127.0.0.1:8333 ls s3://
Build and push a multiarch build
Make sure that docker buildx is supported (might be an experimental docker feature)
BUILDER=$(docker buildx create --driver docker-container --use)
docker buildx build --pull --push --platform linux/386,linux/amd64,linux/arm64,linux/arm/v7,linux/arm/v6 . -t chrislusf/seaweedfs
docker buildx stop $BUILDER
Minio debugging
mc config host add local http://127.0.0.1:9000 some_access_key1 some_secret_key1
mc admin trace --all --verbose local