"
not recommended setups will still run iotune
fixes#6631
"
* tarzanek-gcp-iosetup:
scylla_io_setup: Supported GCP VMs with NVMEs get out of box I/O configs
scylla_util.py: add support for gcp instances
scylla_util.py: support http headers in curl function
scylla_io_setup: refactor iotune run to a function
We should not fill entire disk space with swapfile, it's safer to limit
swap size 50% of diskspace.
Also, if 50% of diskspace <= 1GB, abort setup since it's out of disk space.
Converting memory & disk sizes to an int value of N gigabytes was too rough,
it become problematic in low memory size / low disk size environment, such as
some types of EC2 instances.
We should calculate in bytes.
To get better API of memory & disk statistics, switch to psutil.
With the library we don't need to parse /proc/meminfo.
[avi: regenerate tools/toolchain/image for new python3-psutils package]
Iotune isn't run for supported/recommended GCP instances anymore,
we set the I/O properties now based on GCP tables or our
measurements(where applicable).
Not recommended/supported setups will still run iotune.
Fixes#6631
GCP VMs with NVMEs that are recommended can be recognized now.
Detection is done using resolving of internal GCP metadata server.
We recommend 2 cpu instances at least. For more than 16 disks
we mandate at least 32 cpus. 50:1 disk to ram ratio also has to be
kept.
Instances that use NVMEs as root disks are also considered as unsupported.
Supported instances for NVMEs are n1, n2, n2d, c2 and m1-megamem-96.
All others are unsupported now.
var-lib-scylla.mount should wait for MDRAID initilization, so we need to add
'After=mdmonitor.service'.
However, currently mdmonitor.service fails to start due to no mail address
specified, we need to add the entry on mdadm.conf.
Fixes#6876
On some CLI tools, command options may different between latest version
vs older version.
To maximize compatibility of setup scripts, we should always use
relocatable CLI tools instead of distribution version of the tool.
Related #6954
On GCE, /dev/sda14 reported as unused disk but it's BIOS boot partition,
should not use for scylla data partition, also cannot use for it since it's
too small.
It's better to exclude such partiotion from unsed disk list.
Fixes#6636
On 2d63acdd6a we replaced 'ol' and 'amzn'
to 'oracle' and 'amazon', but distro.id() actually returns 'amzn' for
Amazon Linux 2, so we need to revert the change.
Fixes#6882
To make the "scylla_setup" interface similar to Docker image, let's add
a "--io-setup ENABLE" command line option. The old "--no-io-setup"
option is retained for compatibility.
This adds a '--io-setup N' command line option, which users can pass to
specify whether they want to run the "scylla_io_setup" script or not.
This is useful if users want to specify I/O settings themselves in
environments such as Kubernetes, where running "iotune" is problematic.
Fixes#6587
On .deb package with new relocatable package format, all files moved to
under scylla/ directory.
So we need to call ./scylla/install.sh on debian/rules, but it does not work
correctly, since install.sh does not support calling from other directory.
To support this, we need to changedir to scylla top directory before
copying files.
There are a bunch of renames that are done if PRODUCT is not the
default, but the Python code for them is incorrect. Path.glob()
is not a static method, and Path does not support .endswith().
Fix by constructing a Path object, and later casting to str.
Let's report each missing CPU feature individually, and improve the
error message a bit. For example, if the "clmul" instruction is missing,
the report looks as follows:
ERROR: You will not be able to run Scylla on this machine because its CPU lacks the following features: pclmulqdq
If this is a virtual machine, please update its CPU feature configuration or upgrade to a newer hypervisor.
Fixes#6528
"coredumpctl info" behavior had been changed since systemd-v232, we need to
support both version.
Before systemd-v232, it was simple.
It print 'Coredump' field only when the coredump exists on filesystem.
Otherwise print nothing.
After the change made on systemd-v232, it become more complex.
It always print 'Storage' field even the coredump does not exists.
Not just available/unavailable, it describe more:
- Storage: none
- Storage: journal
- Storage: /path/to/file (inacessible)
- Storage: /path/to/file
To support both of them, we need to detect message version first, then
try to detect coredump path.
Fixes: #6789
reference: 47f5064207
Instead of running shutil.copy() for each *.{service,default},
create symlink for these files.
Python will copy original file when copying debian directory.
"
Follow scylla-server package changes, simplified .rpm/.deb build process which merge build scripts into single script.
"
* syuu1228-python3_simplified_pkg_scripts:
python3: simplified .deb build process
python3: simplified .rpm build process
We could hit "cannot serialize '_io.BufferedReader' object" when request get 404 error from the server
Now you will get legit error message in the case.
Fixes#6690
Print error message and exit with non-zero status by following condition:
- coredumpctl says the coredump file is inaccessible
- failed to detect coredump file path from 'coredumpctl info <pid>'
- deleting coredump file failed because the file is missing
Fixes#6654
Sync version number fixup from main package, contains #6546 and #6752 fixes.
Note that scylla-python3 likely does not affect this versioning issue,
since it uses python3 version, which normally does not contain 'rcX'.
When we started to porting bash script to python script, we are not able to use
subprocess.run() since EPEL only provides python 3.4, but now we have
relocatable python, so we can switch to it.
Currently we we mistakenly made two different way to detect distribution,
directly reading /etc/os-release and use distro package.
distro package provides well abstracted APIs and still have full access to
os-release informations, we should switch to it.
Fixes#6691
In the section explaining how to build a docker image for a self-built
Scylla executable, we have a warning that even if you already built
Scylla, build_reloc.sh will re-run configure.py and rebuild the executable
with slightly different options.
The re-run of configure.py and ninja still happens (see issue #6547) but
we no longer pass *different* options to configure.py, so the rebuild
usually doesn't do anything and finishes in seconds, and the paragraph
warning about the rebuild is no longer relevant.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20200621093049.975044-1-nyh@scylladb.com>
On Ubuntu 18.04 and ealier & Deiban 10 and ealier, /usr merge is not done, so
/usr/bin/systemd-escape and /bin/systemd-escape is different place, and we call
/usr/bin but Debian variants tries to install the command in /bin.
Drop full path, just call command name and resolve by default PATH.
Fixes: #6650
