Files
seaweedfs/docker/Dockerfile.go_build
T
Chris LuandGitHub cc8364a03e Ship the Rust maintenance worker with the release (#10879)
* worker: name the binary weed-worker

It is the Rust side of `weed worker`, the way weed-volume is the Rust side of
`weed volume`, and lance is the first family of jobs it carries rather than
the only one it ever will. The crate keeps its own name: when a second family
arrives the bin target moves to a crate of its own, under this name.

Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm

* docker: ship the Rust maintenance worker in the image

Lance table buckets need a worker that can read the format, and until now the
only way to get one was a Rust toolchain and a cargo build. It now sits at
/usr/bin/weed-worker beside the Rust volume server, reached as
`docker run chrislusf/seaweedfs worker-rust --admin host:23646` — the verb
mirrors volume-rust, so plain `worker` still runs the Go one.

Taken pre-built or not at all: the lance jobs pull in arrow and datafusion, far
too large a tree to compile inside the image build, so an architecture CI did
not build for gets the empty placeholder the entrypoint refuses to exec, the
way the Rust volume server already does.

Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm

* ci: build the Rust worker for the container images

The same native cross-compile the volume server uses, so the release, latest
and dev images all carry it on amd64 and arm64. The artifact holds both
binaries now, so it is named for that rather than for the volume server.

Only the release directory each job builds is cached: with a debug profile
beside it the worker's target/ reaches 24GB, against a 10GB cache budget.

Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm

* ci: publish Rust worker binaries with the release

Linux amd64 and arm64 only: the worker runs beside the cluster it maintains,
and its dependency tree makes every extra target an expensive build.

Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm

* ci: build and test the Rust workers on change

Nothing built seaweed-worker in CI, so the release and the container images
would have been the first place a break showed up. Tests run in release too,
rather than compiling lance, arrow and datafusion again in another profile.

Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm

* docs: say how to get a released worker

Neither the image nor the release tarballs were mentioned; a toolchain and a
cargo build read as the only way in.

Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm

* ci: install protoc for the Rust worker builds

lance's crates compile their own protos, and unlike seaweed-volume they do not
vendor a protoc to do it with, so every job that builds the worker failed at
lance-encoding's build script.

Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm

* ci: do not persist credentials in the worker release checkout

The upload step is handed a token explicitly; a cargo build script should not
find another one sitting in the checkout's git config.

Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm

* docker: keep the worker's argument boundaries

Unquoted $@ splits on whitespace and expands globs, so an argument carrying
either arrived as something else.

Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
2026-08-22 10:22:33 -07:00

121 lines
5.2 KiB
Docker

# Pin the builder to the host arch and cross-compile the (CGO-free) Go binary,
# so arm64/arm/386 targets skip QEMU emulation of the whole compile.
FROM --platform=$BUILDPLATFORM golang:1.26-alpine AS builder
RUN apk add git g++ fuse
RUN mkdir -p /go/src/github.com/seaweedfs/
ARG BRANCH=${BRANCH:-master}
# Clone with full history and all tags to ensure all commits are available
RUN git clone --no-single-branch --tags https://github.com/seaweedfs/seaweedfs /go/src/github.com/seaweedfs/seaweedfs
ARG TAGS
RUN cd /go/src/github.com/seaweedfs/seaweedfs && \
(git checkout $BRANCH || \
(echo "Checkout failed, fetching all history..." && \
git fetch --all --tags --prune && \
git checkout $BRANCH) || \
(echo "ERROR: Branch/commit $BRANCH not found in repository" && \
echo "Available branches:" && git branch -a && exit 1))
ARG TARGETOS TARGETARCH TARGETVARIANT
RUN cd /go/src/github.com/seaweedfs/seaweedfs/weed \
&& export LDFLAGS="-X github.com/seaweedfs/seaweedfs/weed/util/version.COMMIT=$(git rev-parse --short HEAD)" \
&& export GOOS=$TARGETOS GOARCH=$TARGETARCH \
&& case "$TARGETARCH" in arm) export GOARM="${TARGETVARIANT#v}";; esac \
&& CGO_ENABLED=0 go build -tags "$TAGS" -ldflags "-extldflags -static ${LDFLAGS}" -o /go/bin/weed .
# Rust volume server: use pre-built binary from CI when available (placed in
# weed-volume-prebuilt/ by the build-rust-binaries job), otherwise compile
# from source. Pre-building avoids a multi-hour QEMU-emulated cargo build
# for non-native architectures.
FROM alpine:3.23 as rust_builder
ARG TARGETARCH
ARG TAGS
COPY weed-volume-prebuilt/ /prebuilt/
COPY weed-worker-prebuilt/ /prebuilt-worker/
COPY --from=builder /go/src/github.com/seaweedfs/seaweedfs/seaweed-volume /build/seaweed-volume
COPY --from=builder /go/src/github.com/seaweedfs/seaweedfs/weed /build/weed
WORKDIR /build/seaweed-volume
RUN if [ -f "/prebuilt/weed-volume-${TARGETARCH}" ]; then \
echo "Using pre-built Rust binary for ${TARGETARCH}" && \
cp "/prebuilt/weed-volume-${TARGETARCH}" /weed-volume; \
elif [ "$TARGETARCH" = "amd64" ] || [ "$TARGETARCH" = "arm64" ]; then \
apk add --no-cache musl-dev openssl-dev protobuf-dev git rust cargo; \
if [ "$TAGS" = "5BytesOffset" ]; then \
cargo build --release; \
else \
cargo build --release --no-default-features; \
fi && \
cp target/release/weed-volume /weed-volume; \
else \
echo "Skipping Rust build for $TARGETARCH (unsupported)" && \
touch /weed-volume; \
fi
# The Rust maintenance worker is taken pre-built or not at all: the lance jobs
# it carries pull in arrow and datafusion, a far larger dependency tree than
# the image build can carry, so an architecture CI did not build for gets the
# same empty placeholder the entrypoint refuses to exec.
RUN if [ -f "/prebuilt-worker/weed-worker-${TARGETARCH}" ]; then \
echo "Using pre-built Rust worker for ${TARGETARCH}" && \
cp "/prebuilt-worker/weed-worker-${TARGETARCH}" /weed-worker; \
else \
echo "No pre-built Rust worker for ${TARGETARCH}" && \
touch /weed-worker; \
fi
# Pre-built binaries arrive via GitHub Actions artifacts, which drop the
# executable bit, so the copied file is 0644 and exec fails with "Permission
# denied". Restore it (no-op for the empty placeholders, which stay size 0).
RUN chmod 0755 /weed-volume /weed-worker
FROM alpine AS final
LABEL author="Chris Lu"
COPY --from=builder /go/bin/weed /usr/bin/
# Copy Rust volume server binary (real binary on amd64/arm64, empty placeholder on other platforms)
COPY --from=rust_builder /weed-volume /usr/bin/weed-volume
# Same for the Rust maintenance worker, which serves Lance table buckets
COPY --from=rust_builder /weed-worker /usr/bin/weed-worker
RUN mkdir -p /etc/seaweedfs
COPY --from=builder /go/src/github.com/seaweedfs/seaweedfs/docker/filer.toml /etc/seaweedfs/filer.toml
COPY --from=builder /go/src/github.com/seaweedfs/seaweedfs/docker/entrypoint.sh /entrypoint.sh
# FIPS 140-3 mode is ON by default (Go 1.24+)
# To disable: docker run -e GODEBUG=fips140=off ...
# Install dependencies and create non-root user
RUN apk upgrade --no-cache && \
apk add --no-cache fuse curl su-exec libgcc libcrypto3 libssl3 && \
addgroup -g 1000 seaweed && \
adduser -D -u 1000 -G seaweed seaweed
# volume server gprc port
EXPOSE 18080
# volume server http port
EXPOSE 8080
# filer server gprc port
EXPOSE 18888
# filer server http port
EXPOSE 8888
# master server shared gprc port
EXPOSE 19333
# master server shared http port
EXPOSE 9333
# s3 server http port
EXPOSE 8333
# webdav server http port
EXPOSE 7333
# Create data directory and set proper ownership for seaweed user
RUN mkdir -p /data/filerldb2 && \
chown -R seaweed:seaweed /data && \
chown -R seaweed:seaweed /etc/seaweedfs && \
chmod 755 /entrypoint.sh
VOLUME /data
WORKDIR /data
# Entrypoint will handle permission fixes and user switching
ENTRYPOINT ["/entrypoint.sh"]
# Default to a complete single-process cluster (master+volume+filer+S3+admin)
# so the image is usable out of the box — including in environments like
# GitHub Actions service containers that cannot pass arguments to the entrypoint.
# Override with any other subcommand at `docker run` / compose time.
CMD ["mini", "-dir=/data"]