93baa334ea
create-relocatable-package.py currently (refs #4194) builds a compressed tar file, but does so using a painfully slow Python implementation of gzip, which is a problem considering the huge size (around 2 gigabytes) of Scylla's executable. On my machine, running it for a release build of Scylla takes a whopping 6 minutes. Just replacing the Python compression with a pipe to an external "gzip" process speeds up the run to just 2 minutes. But gzip is still not optimal, using only one thread even when on a many-core machine. If we switch to "pigz", a parallel implementation of "gzip", all cores are used and on my machine the compression speeds up to just 23 seconds - that's 15 times faster than before this patch. So this patch has create-relocatable-package.py use an external pigz process. "pigz" is now required on the build system (if you want to create packages), so is added to install-dependencies.sh. [avi: update toolchain] Signed-off-by: Nadav Har'El <nyh@scylladb.com> Message-Id: <20190212090333.3970-1-nyh@scylladb.com>
Official toolchain for ScyllaDB
While we aim to build out-of-the-box on recent distributions, this isn't always possible and not everyone runs a recent distribution. For this reason a version-controlled toolchain is provided as a docker image.
Quick start
If your workstation supports docker (without requiring sudo), you can build and run Scylla easily without setting up the build dependencies beforehand:
./tools/toolchain/dbuild ./configure.py
./tools/toolchain/dbuild ninja build/release/scylla
./tools/toolchain/dbuild ./build/release/scylla --developer-mode 1
The dbuild script
The script dbuild allows you to run any in that toolchain with
the working directory mounted:
./tools/toolchain/dbuild ./configure.py
./tools/toolchain/dbuild ninja
You can adjust the docker run command by adding more flags before the
command to be executed, separating the flags and the command with --.
This can be useful to attach more volumes (for data or ccache) and to
set environment variables:
./tools/toolchain/dbuild -v /var/cache/ccache:/var/cache/ccache -- ninja
The script also works from other directories, so if you have scylla-ccm checked
out alongside scylla, you can write
../scylla/tools/toolchain/dbuild ./ccm ...
You will have access to both scylla and scylla-ccm in the container.
Interactive mode is also supported, using the docker -i and -t flags:
./tools/toolchain/dbuild -it -- bash
this will drop you into a shell, with all of the toolchain accessible.
Obtaining the current toolchain
The toolchain is stored in a file called tools/toolchain/image. Normally,
dbuild will fetch the toolchain automatically. If you want to access
the toolchain explicitly, pull that image:
docker pull $(<tools/toolchain/image)
Building the toolchain
If you add dependencies (to install-dependencies.sh or
seastar/install-dependencies.sh) you should update the toolchain.
Run the command
docker build --no-cache -f tools/toolchain/Dockerfile .
and use the resulting image.
Publishing an image
If you're a maintainer, you can tag the image and push it
using docker push. Tags follow the format
scylladb/scylla-toolchain:fedora-29-[branch-3.0-]20181128. After the
image is pushed, update tools/toolchain/image so new
builds can use it automatically.
For master toolchains, the branch designation is omitted. In a branch, if there is a need to update a toolchain, the branch designation is added to the tag to avoid ambiguity.