tools: toolchain: improve dbuild signal handing

Currently, we use --sig-proxy to forward signals to the container. However, this
requires the container's co-operation, which usually doesn't exist. For example,

    docker run --sig-proxy fedora:29 bash -c "sleep 5"

Does not respond to ctrl-C.

This is a problem for continuous integration. If a build is aborted, Jenkins will
first attempt to gracefully terminate the processes (SIGINT/SIGTERM) and then give
up and use SIGKILL. If the graceful termination doesn't work, we end up with an
orphan container running on the node, which can then consume enough memory and CPU
to harm the following jobs.

To fix this, trap signals and handle them by killing the container. Also trap
shell exit, and even kill the container unconditionally, since if Jenkins happens
to kill the "docker wait" process the regular paths will not be taken.

We lose a lot by running the container asynchronously with the dbuild shell
script, so we need to add it back:

 - log display: via the "docker logs" command
 - auto-removal of the container: add a "docker rm -f" command on signal
   or normal exit
Message-Id: <20190424130112.794-1-avi@scylladb.com>
This commit is contained in:
Avi Kivity
2019-04-24 16:01:12 +03:00
committed by Tomasz Grabiec
parent ee9f3388f6
commit ede1d248af

View File

@@ -21,9 +21,9 @@ if [[ "$1" = -* ]]; then
shift
fi
docker run \
--sig-proxy=true \
--rm \
container=$(
docker run \
--detach=true \
--network host \
-u "$(id -u):$(id -g)" \
"${group_args[@]}" \
@@ -38,3 +38,30 @@ docker run \
"${docker_args[@]}" \
"$(<"$here/image")" \
"$@"
)
kill_it() {
if [[ -n "$container" ]]; then
docker rm -f "$container" > /dev/null
container=
fi
}
trap kill_it SIGTERM SIGINT SIGHUP EXIT
docker logs --follow "$container"
if [[ -n "$container" ]]; then
exitcode="$(docker wait "$container")"
else
exitcode=99
fi
kill_it
trap - SIGTERM SIGINT SIGHUP EXIT
# after "docker kill", docker wait will not print anything
[[ -z "$exitcode" ]] && exitcode=1
exit "$exitcode"