This reverts commit 8366d2231d because it
causes the following "scylla_setup" failure on Ubuntu 16.04:
Command: 'sudo /usr/lib/scylla/scylla_setup --nic ens5 --disks /dev/nvme0n1 --swap-directory / '
Exit code: 1
Stdout:
Setting up libtomcrypt0:amd64 (1.17-7ubuntu0.1) ...
Setting up chrony (2.1.1-1ubuntu0.1) ...
Creating '_chrony' system user/group for the chronyd daemon…
Creating config file /etc/chrony/chrony.conf with new version
Processing triggers for libc-bin (2.23-0ubuntu11.2) ...
Processing triggers for ureadahead (0.100.0-19.1) ...
Processing triggers for systemd (229-4ubuntu21.29) ...
501 Not authorised
NTP setup failed.
Stderr:
chrony.service is not a native service, redirecting to systemd-sysv-install
Executing /lib/systemd/systemd-sysv-install enable chrony
Traceback (most recent call last):
File "/opt/scylladb/scripts/libexec/scylla_ntp_setup", line 63, in <module>
run('chronyc makestep')
File "/opt/scylladb/scripts/scylla_util.py", line 504, in run
return subprocess.run(cmd, stdout=stdout, stderr=stderr, shell=shell, check=exception, env=scylla_env).returncode
File "/opt/scylladb/python3/lib64/python3.8/subprocess.py", line 512, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['chronyc', 'makestep']' returned non-zero exit status 1.
Commit 8e1f7d4fc7 ("dist/common/scripts: drop makedirs(), use
os.makedirs()") dropped the "mkdirs()" function, but forgot to convert
the caller in scylla_kernel_check to os.mkdirs().
Message-Id: <20200923104510.230244-1-penberg@scylladb.com>
Seems like parse_scylla_dirs_with_default() and get_scylla_dirs() shares most of
the code, de-duplicate it.
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
scylla_util.py is a library for common functions across setup scripts,
it should not include private function of single file.
So move all those functions to caller file.
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
We need to skip internet access on offline installation.
To do this we need following changes:
- prevent running yum/apt for each script
- set default "NO" for scripts it requires package installation
- set default "NO" for scripts it requires internet access, such as NTP
See #7153Closes#7224
* syuu1228-offline_setup:
dist/common/scripts: skip internet access on offline installation
scylla_ntp_setup: use shutil.witch() to lookup command
We need to skip internet access on offline installation.
To do this we need following changes:
- prevent running yum/apt for each script
- set default "NO" for scripts it requires package installation
- set default "NO" for scripts it requires internet access, such as NTP
See #7153Fixes#7182
The command installed directory may different between distributions,
we can abstract the difference using shutil.witch().
Also the script become simpler than passing full path to os.path.exists().
hugepages and libhugetlbfs-bin packages is only required for DPDK mode,
and unconditionally installation causes error on offline mode, so drop it.
Fixes#7182
When configuration files for perftune contain invalid parameter, scylla_prepare
may cause traceback because error handling is not eough.
Throw all errors from create_perftune_conf(), catch them on scylla_prepare,
print user understandable error.
Fixes#6847
do_verify_package written only for .rpm/.deb, does not working correctly
for offline install(including nonroot).
We should check file existance for the environment, not by package existance
using rpm/dpkg.
Fixes#7075
"
not recommended setups will still run iotune
fixes#6631
"
* tarzanek-gcp-iosetup:
scylla_io_setup: Supported GCP VMs with NVMEs get out of box I/O configs
scylla_util.py: add support for gcp instances
scylla_util.py: support http headers in curl function
scylla_io_setup: refactor iotune run to a function
We should not fill entire disk space with swapfile, it's safer to limit
swap size 50% of diskspace.
Also, if 50% of diskspace <= 1GB, abort setup since it's out of disk space.
Converting memory & disk sizes to an int value of N gigabytes was too rough,
it become problematic in low memory size / low disk size environment, such as
some types of EC2 instances.
We should calculate in bytes.
To get better API of memory & disk statistics, switch to psutil.
With the library we don't need to parse /proc/meminfo.
[avi: regenerate tools/toolchain/image for new python3-psutils package]
Iotune isn't run for supported/recommended GCP instances anymore,
we set the I/O properties now based on GCP tables or our
measurements(where applicable).
Not recommended/supported setups will still run iotune.
Fixes#6631
GCP VMs with NVMEs that are recommended can be recognized now.
Detection is done using resolving of internal GCP metadata server.
We recommend 2 cpu instances at least. For more than 16 disks
we mandate at least 32 cpus. 50:1 disk to ram ratio also has to be
kept.
Instances that use NVMEs as root disks are also considered as unsupported.
Supported instances for NVMEs are n1, n2, n2d, c2 and m1-megamem-96.
All others are unsupported now.
var-lib-scylla.mount should wait for MDRAID initilization, so we need to add
'After=mdmonitor.service'.
However, currently mdmonitor.service fails to start due to no mail address
specified, we need to add the entry on mdadm.conf.
Fixes#6876
On some CLI tools, command options may different between latest version
vs older version.
To maximize compatibility of setup scripts, we should always use
relocatable CLI tools instead of distribution version of the tool.
Related #6954
On GCE, /dev/sda14 reported as unused disk but it's BIOS boot partition,
should not use for scylla data partition, also cannot use for it since it's
too small.
It's better to exclude such partiotion from unsed disk list.
Fixes#6636
On 2d63acdd6a we replaced 'ol' and 'amzn'
to 'oracle' and 'amazon', but distro.id() actually returns 'amzn' for
Amazon Linux 2, so we need to revert the change.
Fixes#6882
To make the "scylla_setup" interface similar to Docker image, let's add
a "--io-setup ENABLE" command line option. The old "--no-io-setup"
option is retained for compatibility.
Let's report each missing CPU feature individually, and improve the
error message a bit. For example, if the "clmul" instruction is missing,
the report looks as follows:
ERROR: You will not be able to run Scylla on this machine because its CPU lacks the following features: pclmulqdq
If this is a virtual machine, please update its CPU feature configuration or upgrade to a newer hypervisor.
Fixes#6528
