7304a795e5
The "jobs" script is used to determine the amount of compilation parallelism on a machine. It attempts to ensure each GCC process has at least 4 GB of memory per core. However, in the worst case scenario, we could end up having the GCC processes take up all the system memory, forcin swapping or OOM killer to kick in. For example, on a 4 core machine with 16 GB of memory, this worst case scenario seems easy to trigger in practice. Fix up the problem by keeping a 1 GB of memory reserve for other processes and calculating parallelism based on that. Message-Id: <20200423082753.31162-1-penberg@scylladb.com>
43 lines
1.3 KiB
Python
Executable File
43 lines
1.3 KiB
Python
Executable File
#!/usr/bin/env python3
|
|
# -*- coding: utf-8 -*-
|
|
#
|
|
# Copyright (C) 2018 ScyllaDB
|
|
#
|
|
|
|
#
|
|
# This file is part of Scylla.
|
|
#
|
|
# Scylla is free software: you can redistribute it and/or modify
|
|
# it under the terms of the GNU Affero General Public License as published by
|
|
# the Free Software Foundation, either version 3 of the License, or
|
|
# (at your option) any later version.
|
|
#
|
|
# Scylla is distributed in the hope that it will be useful,
|
|
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
# GNU General Public License for more details.
|
|
#
|
|
# You should have received a copy of the GNU General Public License
|
|
# along with Scylla. If not, see <http://www.gnu.org/licenses/>.
|
|
#
|
|
|
|
# Guess and print out a good number of compiler jobs to
|
|
# run. Note that on an interactive desktop you will want
|
|
# to reduce this to allow more memory for your desktop,
|
|
# to avoid swapping. This is oriented at continuous
|
|
# integration machines that do not serve an interactive
|
|
# load as well.
|
|
|
|
import os
|
|
|
|
procs = os.sysconf('SC_NPROCESSORS_ONLN')
|
|
mem = os.sysconf('SC_PHYS_PAGES') * os.sysconf('SC_PAGESIZE')
|
|
|
|
mem_reserve = 1000000000
|
|
job_mem = 4000000000
|
|
|
|
jobs = min(procs, (mem-mem_reserve) // job_mem)
|
|
jobs = max(jobs, 1)
|
|
|
|
print(jobs)
|