ede1d248af
Currently, we use --sig-proxy to forward signals to the container. However, this
requires the container's co-operation, which usually doesn't exist. For example,
docker run --sig-proxy fedora:29 bash -c "sleep 5"
Does not respond to ctrl-C.
This is a problem for continuous integration. If a build is aborted, Jenkins will
first attempt to gracefully terminate the processes (SIGINT/SIGTERM) and then give
up and use SIGKILL. If the graceful termination doesn't work, we end up with an
orphan container running on the node, which can then consume enough memory and CPU
to harm the following jobs.
To fix this, trap signals and handle them by killing the container. Also trap
shell exit, and even kill the container unconditionally, since if Jenkins happens
to kill the "docker wait" process the regular paths will not be taken.
We lose a lot by running the container asynchronously with the dbuild shell
script, so we need to add it back:
- log display: via the "docker logs" command
- auto-removal of the container: add a "docker rm -f" command on signal
or normal exit
Message-Id: <20190424130112.794-1-avi@scylladb.com>
68 lines
1.3 KiB
Bash
Executable File
68 lines
1.3 KiB
Bash
Executable File
#!/bin/bash
|
|
|
|
here="$(realpath $(dirname "$0"))"
|
|
toplevel="$(realpath "$here/../..")"
|
|
group_args=()
|
|
docker_args=()
|
|
|
|
for gid in $(id -G); do
|
|
group_args+=(--group-add "$gid")
|
|
done
|
|
|
|
if [[ "$1" = -* ]]; then
|
|
while [[ "$1" != "--" && $# != 0 ]]; do
|
|
docker_args+=("$1")
|
|
shift
|
|
done
|
|
if [[ "$1" != "--" ]]; then
|
|
echo "Expected '--' to terminate docker flag list"
|
|
exit 1
|
|
fi
|
|
shift
|
|
fi
|
|
|
|
container=$(
|
|
docker run \
|
|
--detach=true \
|
|
--network host \
|
|
-u "$(id -u):$(id -g)" \
|
|
"${group_args[@]}" \
|
|
--cap-add SYS_PTRACE \
|
|
-v "$PWD:$PWD:z" \
|
|
-v "$toplevel:$toplevel:z" \
|
|
-v /tmp:/tmp:z \
|
|
-v /etc/passwd:/etc/passwd:ro \
|
|
-v /etc/group:/etc/group:ro \
|
|
-v /etc/localtime:/etc/localtime:ro \
|
|
-w "$PWD" \
|
|
"${docker_args[@]}" \
|
|
"$(<"$here/image")" \
|
|
"$@"
|
|
)
|
|
|
|
kill_it() {
|
|
if [[ -n "$container" ]]; then
|
|
docker rm -f "$container" > /dev/null
|
|
container=
|
|
fi
|
|
}
|
|
|
|
trap kill_it SIGTERM SIGINT SIGHUP EXIT
|
|
|
|
docker logs --follow "$container"
|
|
|
|
if [[ -n "$container" ]]; then
|
|
exitcode="$(docker wait "$container")"
|
|
else
|
|
exitcode=99
|
|
fi
|
|
|
|
kill_it
|
|
|
|
trap - SIGTERM SIGINT SIGHUP EXIT
|
|
|
|
# after "docker kill", docker wait will not print anything
|
|
[[ -z "$exitcode" ]] && exitcode=1
|
|
|
|
exit "$exitcode"
|