tools: toolchain: improve dbuild signal handing
Currently, we use --sig-proxy to forward signals to the container. However, this
requires the container's co-operation, which usually doesn't exist. For example,
docker run --sig-proxy fedora:29 bash -c "sleep 5"
Does not respond to ctrl-C.
This is a problem for continuous integration. If a build is aborted, Jenkins will
first attempt to gracefully terminate the processes (SIGINT/SIGTERM) and then give
up and use SIGKILL. If the graceful termination doesn't work, we end up with an
orphan container running on the node, which can then consume enough memory and CPU
to harm the following jobs.
To fix this, trap signals and handle them by killing the container. Also trap
shell exit, and even kill the container unconditionally, since if Jenkins happens
to kill the "docker wait" process the regular paths will not be taken.
We lose a lot by running the container asynchronously with the dbuild shell
script, so we need to add it back:
- log display: via the "docker logs" command
- auto-removal of the container: add a "docker rm -f" command on signal
or normal exit
Message-Id: <20190424130112.794-1-avi@scylladb.com>
This commit is contained in:
committed by
Tomasz Grabiec
parent
ee9f3388f6
commit
ede1d248af
@@ -21,9 +21,9 @@ if [[ "$1" = -* ]]; then
|
||||
shift
|
||||
fi
|
||||
|
||||
docker run \
|
||||
--sig-proxy=true \
|
||||
--rm \
|
||||
container=$(
|
||||
docker run \
|
||||
--detach=true \
|
||||
--network host \
|
||||
-u "$(id -u):$(id -g)" \
|
||||
"${group_args[@]}" \
|
||||
@@ -38,3 +38,30 @@ docker run \
|
||||
"${docker_args[@]}" \
|
||||
"$(<"$here/image")" \
|
||||
"$@"
|
||||
)
|
||||
|
||||
kill_it() {
|
||||
if [[ -n "$container" ]]; then
|
||||
docker rm -f "$container" > /dev/null
|
||||
container=
|
||||
fi
|
||||
}
|
||||
|
||||
trap kill_it SIGTERM SIGINT SIGHUP EXIT
|
||||
|
||||
docker logs --follow "$container"
|
||||
|
||||
if [[ -n "$container" ]]; then
|
||||
exitcode="$(docker wait "$container")"
|
||||
else
|
||||
exitcode=99
|
||||
fi
|
||||
|
||||
kill_it
|
||||
|
||||
trap - SIGTERM SIGINT SIGHUP EXIT
|
||||
|
||||
# after "docker kill", docker wait will not print anything
|
||||
[[ -z "$exitcode" ]] && exitcode=1
|
||||
|
||||
exit "$exitcode"
|
||||
|
||||
Reference in New Issue
Block a user