Commit Graph

1591 Commits

Author SHA1 Message Date
Tomasz Grabiec
850d9cfb59 node-exporter: Disable hwmon collector
This collector reads nvme temperature sensor, which was observed to
cause bad performance on Azure cloud following the reading of the
sensor for ~6 seconds. During the event, we can see elevated system
time (up to 30%) and softirq time. CPU utilization is high, with
nvm_queue_rq taking several orders of magnitude more time than
normally. There are signs of contention, we can see
__pv_queued_spin_lock_slowpath in the perf profile, called. This
manifests as latency spikes and potentially also throughput drop due
to reduced CPU capacity.

By default, the monitoring stack queries it once every 60s.

Closes scylladb/scylladb#21165
2024-10-27 21:59:15 +02:00
Kefu Chai
8355056453 build: cmake: expose and use the path to iotune correctly
in 415c83fa, we introduced a regression which broke the build of
target of "package". because

- the IMPORT_LOCATION_<CONFIG> of the imported target of
  "Seastar::iotune" includes a literal `$<CONFIG>`
- we retrieve the property named "IMPORTED_LOCATION" from
  this target. but value of this property is empty.

so, when we copied this file, the "src" parameter passed to
`cmake -E copy` is actually an empty string.

in this change, we

- set the `IMPORTED_LOCATION_${CONFIG}` property with a
  correct path.
- retrieve the property with the right approach -- to use
  `TARGET_FILE` generator expression.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#21181
2024-10-21 10:32:51 +03:00
Kefu Chai
415c83fa67 build: cmake: consume Seastar using its .pc files
before this change, scylla's CMake-based system consumes Seastar
library by including it directly. but this failed to address the needs
of linking against Seastar shared libraries in Debug and Dev builds, while
linking against the static libraries in other builds. because Seastar
uses `BUILD_SHARED_LIBS` CMake variable to determine if it builds
shared libraries. and we cannot assign different values to this
CMake variable based on current configure type -- CMake does not
support. see https://gitlab.kitware.com/cmake/cmake/-/issues/19467

in order to address this problem, we have a couple possible
solutions:

- to enable Seastar to build both shared and static libraries in a
  pass. without sacrificing the performance, we have to build
  all object files twice: once with -fPIC, once without. in order
  to accompolish this goal, we need to develop a machinary to
  populate the same settings to these two builds. this would
  complicate the design of Seastar's building system further.
- to build Seastar libraries twice in scylla, we could use
  the ExternalProject module to implement this. but it'd be
  complicate to extract the compile options, and link options
  previously populated by Seastar's targets with CMake --
  we would have to replicate all of them in scylla. this is
  out of the question.
- to build Seastar libraries twice before building scylla,
  and let scylla to consume them using CMake config files or
  .pc files. this is a compromise. it enables scylla to
  drive the build of Seastar libraries and to consume
  the compile options and link options. the downside is:

  * the generated compilation database (compile_commands.json)
    does not include the commands building Seastar anymore.
  * the building system of scylla does not have finer graind
    control on the building process of seastar. for instance,
    we cannot specify the build dependency to a certain seastar
    library, and just build it instead of building the whole
    seastar project.

turns out the last approach is the best one we can have
at this moment. this is also the approach used by the existing
`configure.py`.

in this change, we

- add FindSeastar.cmake to

  * detect the preconfigured Seastar builds, and
  * extract the build options from .pc files
  * expose library targets to be consumed by parent project
- add Seastar as an external project, so we can build it from
  the parent project. BUILD_AWAYS is set to ensure that Seastar is
  rebuilt, as scylla developers are expected to modify Seastar
  occasionally. since the change in Seastar's SOURCE_DIR is not
  detectable via the ExternalProject, we have to rebuild it.

  this is atypical compared to standard ExternalProject usage:
  - Seastar's build system should already be configured at this point.
  - We maintain separate project variants for each configuration type.

  Benefits of this approach:
  - Allows the parent project to consume the compile options exposed by
    .pc file. as the compile options vary from one config to another.
  - Allows application of config-specific settings
  - Enables building Seastar within the parent project's build system
  - Facilitates linking of artifacts with the external project target,
    establishing proper dependencies between them
- preserve the existing machinery of including Seastar only when
  building without multi-config generator. this allows users who don't
  use mult-config generator to build Seastar in-the-tree. the typical
  use case is the CI workflows performing the static analysis.

we will update `configure.py` to merge the compilation database
of scylla and seastar.

Refs scylladb/scylladb#2717
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2024-10-18 08:36:52 +08:00
Kefu Chai
947d9d5a97 scylla_coredump_setup: fix typos in comment
these typos were identified by the codespell workflow.

and fixed a syntax error along the way.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#20877
2024-09-30 13:29:34 +03:00
Avi Kivity
5a470b2bfb Merge 'scylla_raid_setup: configure SELinux file context' from Takuya ASADA
On RHEL9, systemd-coredump fails to coredump on /var/lib/scylla/coredump because the service only have write acess with systemd_coredump_var_lib_t. To make it writable, we need to add file context rule for /var/lib/scylla/coredump, and run restorecon on /var/lib/scylla.

Fixes #19325

Closes scylladb/scylladb#20528

* github.com:scylladb/scylladb:
  scylla_raid_setup: configure SELinux file context
  scylla_coredump_setup: fix SELinux configuration for RHEL9
2024-09-29 12:53:00 +03:00
Kefu Chai
40f2d4c988 build: cmake: drop scylla-jmx from the build
in 3cd2a61736, we dropped scylla-jmx
from the build. but didn't update the CMake building system accordingly,
this broke the CMake build, as the dependencies pointing to jmx cannot
be found or fulfilled.

in this change, we remove all references to jmx in the CMake build.

Signed-off-by: Laszlo Ersek <laszlo.ersek@scylladb.com>

Closes scylladb/scylladb#20736
2024-09-23 14:20:42 +03:00
Takuya ASADA
3cd2a61736 dist: drop scylla-jmx
Since JMX server is deprecated, drop them from submodule, build system
and package definition.

Related scylladb/scylla-tools-java#370
Related #14856

Signed-off-by: Takuya ASADA <syuu@scylladb.com>

Closes scylladb/scylladb#17969
2024-09-13 07:59:45 +03:00
Takuya ASADA
0ac450de05 scylla_raid_setup: configure SELinux file context
On RHEL9, systemd-coredump fails to coredump on /var/lib/scylla/coredump
because the service only have write acess with systemd_coredump_var_lib_t.
To make it writable, we need to add file context rule for
/var/lib/scylla/coredump, and run restorecon on /var/lib/scylla.

Fixes #20573
2024-09-13 04:31:52 +09:00
Takuya ASADA
56c971373c scylla_coredump_setup: fix SELinux configuration for RHEL9
Seems like specific version of systemd pacakge on RHEL9 has a bug on
SELinux configuration, it introduced "systemd-container-coredump" module
to provide rule for systemd-coredump, but not enabled by default.
We have to manually load it, otherwise it causes permission error.

Fixes #19325
2024-09-13 04:31:16 +09:00
Takuya ASADA
e36c939505 dist: tune LimitNOFILES for large nodes
On very large node, LimitNOFILES=80000 may not enough size, it can cause
"Too many files" error.

To avoid that, let's increase LimitNOFILES on scylla_setup stage,
generate optimal value calurated from memory size and number of cpus.

Closes scylladb/scylla-enterprise#4304

Closes scylladb/scylladb#20443
2024-09-09 14:13:49 +03:00
Kefu Chai
6970c502c9 dist: drop %pretrans section
before this change, if user does not have `/bin/sh` around, when
installing scylla packages, the script in `%pretrans" is executed,
and fails due to missing `/bin/sh`. per
https://docs.fedoraproject.org/en-US/packaging-guidelines/Scriptlets/#pretrans

> Note that the %pretrans scriptlet will, in the particular case of
> system installation, run before anything at all has been installed.
> This implies that it cannot have any dependencies at all. For this
> reason, %pretrans is best avoided, but if used it MUST (by necessity)
> be written in Lua. See
> https://rpm-software-management.github.io/rpm/manual/lua.html for more
> information.

but we were trying to warn users upgrading from scylla < 1.7.3, which
was released 7 years ago at the time of writing.

in this change, we drop the `%pretrans` section. hopefuly they will
find their way out if they still exist.

Fixes scylladb/scylladb#20321

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#20365
2024-09-01 19:46:19 +03:00
Kefu Chai
a06e1c6545 scylla-housekeeping: use raw string to avoid using escape sequence
before this change, when running `scylla-housekeeping`:
```
/opt/scylladb/scripts/libexec/scylla-housekeeping:122: SyntaxWarning: invalid escape sequence '\s'
  match = re.search(".*http.?://repositories.*/scylladb/([^/\s]+)/.*/([^/\s]+)/scylladb-.*", line)
```

we could have the warning above. because `\s` is not a valid escape
sequence, but the Python interpreter accepts it as two separated
characters of `\s` after complaining. but it's still annoying.

so, let's use a raw string here.

Refs scylladb/scylladb#20317
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#20359
2024-09-01 18:59:23 +03:00
Kefu Chai
03ab80501f tools/scylla-nodetool: add restore integration
as we have an API for restore a keyspace / table, let's expose this feature
with nodetool. so we can exercise it without the help of scylla-manager
or 3rd-party tools with a user-friendly interface.

in this change:

* add a new subcommand named "restore" to nodetool
* add test to verify its interaction with the API server
* update the document accordingly.
* the bash completion script is updated accordingly.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2024-08-28 15:42:49 +03:00
Avi Kivity
72a85e3812 Merge 'Integrated backup' from Pavel Emelyanov
This adds minimal implementation of the start-backup API call.

The method starts a task that uploads all files from the given keyspace's snapshot to the requested endpoint/bucket. Arguments are:
- endpoint -- the ID in object_store.yaml config file
- bucket -- the target bucket to put objects into
- keyspace -- the keyspace to work on
- snapshot -- the method assumes that the snapshot had been already taken and only copies sstables from it

The task runs in the background, its task_id is returned from the method once it's spawned and it should be used via /task_manager API to track the task execution and completion (hint: it's good to have non-zero TTL value to make sure fast backups don't finish before the caller manages to call wait_task API).

Sstables components are scanned for all tables in the keyspace and are uploaded into the /bucket/${cf_name}/${snapshot_name}/ path.

refs: #18391

Closes scylladb/scylladb#19890

* github.com:scylladb/scylladb:
  tools/scylla-nodetool: add backup integration
  docs: Document the new backup method
  test/object_store: Test that backup task is abortable
  test/object_store: Add simple backup test
  test/object_store: Move format_tuples()
  test/pylib: Add more methods to rest client
  backup-task: Make it abortable (almost)
  code: Introduce backup API method
  database: Export parse_table_directory_name() helper
  database: Introduce format_table_directory_name() helper
  snapshot-ctl: Add config to snapshot_ctl
  snapshot-ctl: Add sstables::storage_manager dependency
  snapshot-ctl: Maintain task manager module
  snapshot-ctl: Add "snapshots" logger
  snapshot-ctl: Outline stop() method and constructor
  snapshot-ctl: Inline run_snapshot_list<>
  test/cql_test_env: Export task manager from cql test env
  task_manager: Print task ttl on start (for debugging)
  docs: Update object_storage.md with AWS_ environment
  docs: Restructure object_storage.md
2024-08-25 20:19:10 +03:00
Kefu Chai
f8931a4578 build: cmake: add "dist" target
since the rules generated by `configure.py` has this target, we need
to have an equivalent target as well in CMake-based buidling system.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#20265
2024-08-25 20:18:12 +03:00
Kefu Chai
969cbb75ce tools/scylla-nodetool: add backup integration
as we have an API for backup a keyspace, let's expose this feature
with nodetool. so we can exercise it without the help of scylla-manager
or 3rd-party tools with a user-friendly interface.

in this change:

* add a new subcommand named "backup" to nodetool
* add test to verify its interaction with the API server
* add two more route to the REST API mock server, as
  the test is using /task_manager/wait_task/{task_id} API.
  for the sake of completeness, the route for
  /task_manager/{part1} is added as well.
* update the document accordingly.
* the bash completion script is updated accordingly.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2024-08-22 19:48:06 +03:00
Yaron Kaikov
8221a178d8 Revert "dist: support nonroot and offline mode for scylla-housekeeping"
This reverts commit c3bea539b6.

Since it breaking offline-installer artifact-tests. Also, it seems that we should have merged it in the first place since we don't need scylla-housekeeping checks for offline-installer

Closes scylladb/scylladb#19976
2024-08-04 10:55:26 +03:00
Kefu Chai
67e07bee25 build: cmake: use per-mode build dir
The build_unified.sh script accepts a --build-dir option, which
specifies the directory used for storing temporary files extracted
from tarballs defined by the --pkgs option. When performing parallel
builds of multiple modes, it's crucial that each build uses a unique
build directory. Reusing the same build directory for different modes
can lead to conflicts, resulting in build failures or, more seriously,
the creation of tarballs containing corrupted files.

so, in this change, we specify a different directory for each mode,
so that they don't share the same one.

Refs scylladb/scylladb#2717
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#19905
2024-07-28 18:11:37 +03:00
Takuya ASADA
02b20089cb scylla_raid_setup: install update-initramfs when it's not available
scylla_raid_setup may fail on Ubuntu minimal image since it calls
update-initramfs without installing.

Closes scylladb/scylladb#19651
2024-07-24 11:55:16 +03:00
Takuya ASADA
c3bea539b6 dist: support nonroot and offline mode for scylla-housekeeping
Introduce support nonroot and offline mode for scylla-housekeeping.

Closes #13084

Closes scylladb/scylladb#13088
2024-07-23 07:57:32 +03:00
Kefu Chai
c52f49facb build: cmake: do not mark cqlsh noarch
in 3c7af287, cqlsh's reloc package was marked as "noarch", and its
filename was updated accordingly in `configure.py`, so let's update
the CMake building system accordingly.

this change should address the build failure of

```
08:48:14  [3325/4124] Generating ../Debug/dist/tar/scylla-cqlsh-6.1.0~dev-0.20240629.60955ead75ef.noarch.tar.gz
08:48:14  FAILED: Debug/dist/tar/scylla-cqlsh-6.1.0~dev-0.20240629.60955ead75ef.noarch.tar.gz /jenkins/workspace/scylla-master/scylla-ci/scylla/build/Debug/dist/tar/scylla-cqlsh-6.1.0~dev-0.20240629.60955ead75ef.noarch.tar.gz
08:48:14  cd /jenkins/workspace/scylla-master/scylla-ci/scylla/build/dist && /usr/bin/cmake -E copy /jenkins/workspace/scylla-master/scylla-ci/scylla/tools/cqlsh/build/scylla-cqlsh-6.1.0~dev-0.20240629.60955ead75ef.noarch.tar.gz /jenkins/workspace/scylla-master/scylla-ci/scylla/build/Debug/dist/tar/scylla-cqlsh-6.1.0~dev-0.20240629.60955ead75ef.noarch.tar.gz
08:48:14  Error copying file "/jenkins/workspace/scylla-master/scylla-ci/scylla/tools/cqlsh/build/scylla-cqlsh-6.1.0~dev-0.20240629.60955ead75ef.noarch.tar.gz" to "/jenkins/workspace/scylla-master/scylla-ci/scylla/build/Debug/dist/tar/scylla-cqlsh-6.1.0~dev-0.20240629.60955ead75ef.noarch.tar.gz".
```

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#19710
2024-07-19 08:00:17 +03:00
Yaron Kaikov
ddcc6ec1e4 dist/docker/debian/build_docker.sh: Build container based on Ubuntu24.04
Now that we added support for Ubuntu24.04 and also migrating our images
to be based on that
(https://github.com/scylladb/scylla-machine-image/pull/530), we should
also modify our docker image

Fixes: https://github.com/scylladb/scylladb/issues/19738

Closes scylladb/scylladb#19764
2024-07-17 18:45:48 +03:00
Takuya ASADA
373a7825b5 scylla-housekeeping: fix exception on parsing version string
Since Python 3.12, version parsing becomes strict, parse_version() does
not accept the version string like '6.1.0~dev'.
To fix this, we need to pass acceptable version string to parse_version() like
'6.1.0.dev0', which is allowed on Python version scheme.

Also, release canditate version like '6.0.0~rc3' has same issue, it
should be replaced to '6.0.0rc3' to compare in parse_version().

reference: https://packaging.python.org/en/latest/specifications/version-specifiers/

Fixes #19564

Closes scylladb/scylladb#19572
2024-07-12 03:23:34 +09:00
Takuya ASADA
db04f8b16e Revert "scylla-housekeeping: fix exception on parsing version string"
This reverts commit 65fbf72ed0, since
it breaks scylla-housekeeping and SCT because the patch modified
version string.

We shoudn't modify version string directly, need to pass
modified string just for parse_version() instead.
2024-07-12 03:23:34 +09:00
Tomas Nozicka
26466a3043 Allow configuring default loglevel with args for container images
Closes scylladb/scylladb#19671
2024-07-11 12:37:53 +03:00
Takuya ASADA
cbf33aba5c scylla_coredump_setup: install systemd-coredump before has_zstd()
On Ubuntu/Debian, we have to install systemd-coredump before
running has_ztd(), since it detect ZSTD support by running coredumpctl.

Move pkg_install('systemd-coredump') to the head of the script.

Fixes #19643

Closes scylladb/scylladb#19648
2024-07-08 15:04:34 +03:00
Takuya ASADA
09e22690dc scylla_coredump_setup: enable compress by default when zstd support detected
We disabled coredump compression by default because it was too slow,
but recent versions of systemd-coredump supports faster zstd based compression,
so let's enable compression by default when zstd support detected.

Related scylladb/scylla-machine-image#462

Closes scylladb/scylladb#18854
2024-07-04 10:38:22 +03:00
Takuya ASADA
65fbf72ed0 scylla-housekeeping: fix exception on parsing version string
Since Python 3.12, version parsing becomes strict, parse_version() does
not accept the version string like '6.1.0~dev'.
To fix this, we need to replace version string from '6.1.0~dev' to
'6.1.0.dev0', which is allowed on Python version scheme.

reference: https://packaging.python.org/en/latest/specifications/version-specifiers/

Fixes #19564

Closes scylladb/scylladb#19572
2024-07-04 10:27:51 +03:00
Israel Fruchter
3c7af28725 cqlsh: update cqlsh submodule
this change updates the cqlsh submodule:

* tools/cqlsh/ ba83aea3...73bdbeb0 (4):
  > install.sh: replace tab with spaces
  > define the the debug packge is empty
  > tests: switch from using cqlsh bash to the test the python file
  > package python driver as wheels

it also includes follow change to package cqlsh as a regular
rpm instead of as a "noarch" rpm:

so far cqlsh bundles the python-driver in, but only as source.
meaning the package wasn't architecture, and also didn't
have the libev eventloop compiled in.

Since from python 3.12 and up, that would mean we would
fallback into asyncio eventloop (which still exprimental)
or into error (once we'll sync with the driver upstream)

so to avoid those, we are change the packaging of cqlsh
to be architecture specific, and get cqlsh compiled, and bundle
all of it's requirements as per architecture installed bundle of wheels.
using `shiv`, i.e. one file virtualenv that we'll be packing
into our artifacts

Ref: https://github.com/scylladb/scylla-cqlsh/issues/90
Ref: https://github.com/scylladb/scylla-cqlsh/pull/91
Ref: https://github.com/linkedin/shiv

Closes scylladb/scylladb#19385

* tools/cqlsh ba83aea...242876c (1):
  > Merge 'package python driver as wheels' from Israel Fruchter

Update tools/cqlsh/ submodule

in which, the change of `define the the debug packge is empty`
should address the build failure like

```
Processing files: scylla-cqlsh-debugsource-6.1.0~dev-0.20240624.c7748f60c0bc.aarch64
error: Empty %files file /jenkins/workspace/scylla-master/next/scylla/tools/cqlsh/build/redhat/BUILD/scylla-cqlsh/debugsourcefiles.list
RPM build errors:
    Empty %files file /jenkins/workspace/scylla-master/next/scylla/tools/cqlsh/build/redhat/BUILD/scylla-cqlsh/debugsourcefiles.list
```

Closes scylladb/scylladb#19473
2024-06-26 12:07:21 +03:00
Yaniv Michael Kaul
9b0eb82175 dist/common/scripts/scylla_coredump_setup: fix typo
Does not able -> Unable

Signed-off-by: Yaniv Kaul <yaniv.kaul@scylladb.com>

Closes scylladb/scylladb#19328
2024-06-17 17:33:46 +03:00
Botond Dénes
c563acdbe9 Merge 'build: cmake: use path to be compatible with CI' from Kefu Chai
this change is created in the same spirit of 1186ddef16, which updated the rule for generating the stripped dist pkg, but it failed to update the one for generating the unstripped dist pkg. what's why we have build failure when the workflow is looking for the unstripped tar.gz:

```
08:02:47  ++ ls /jenkins/workspace/scylla-master/scylla-ci/scylla/build/RelWithDebInfo/dist/tar/scylla-unstripped-6.1.0~dev-0.20240613.d5bdddaeb40b.x86_64.tar.gz
08:02:47  ls: cannot access '/jenkins/workspace/scylla-master/scylla-ci/scylla/build/RelWithDebInfo/dist/tar/scylla-unstripped-6.1.0~dev-0.20240613.d5bdddaeb40b.x86_64.tar.gz': No such file or directory`
```

so, in this change, we fix the path.

Refs #2717

---

* cmake related change, hence no need to backport.

Closes scylladb/scylladb#19290

* github.com:scylladb/scylladb:
  build: cmake: use per-mode path for building unstripped_dist_pkg
  build: cmake: use path to be compatible with CI
2024-06-14 15:35:26 +03:00
Kefu Chai
4c1006a5bb dist: s/SafeConfigParser/ConfigParser/
`SafeConfigParser` was renamed to `ConfigParser` in Python 3.2,
and Python warns us:

> scylla-housekeeping:183: DeprecationWarning: The SafeConfigParser
> class has been renamed to ConfigParser in Python 3.2. This alias will
> be removed in Python 3.12. Use ConfigParser directly instead.

see https://docs.python.org/3.2/library/configparser.html#configparser.ConfigParser
and https://docs.python.org/3.1/library/configparser.html#configparser.SafeConfigParser

Fixes #13046
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#19285
2024-06-14 09:59:22 +03:00
Kefu Chai
28a4298005 build: cmake: use per-mode path for building unstripped_dist_pkg
`before this change, we use "scylla" as the dependecy of
unstripped_dist_pkg, but that's implies the scylla built with the
default mode. if the build rules is generated using the
multi-config generator, the default mode does not necessarily
identical to the current `$<CONFIG>`, so let's be more explicit.
otherwise, we could run into built failure like

```
FAILED: dist/RelWithDebInfo/scylla-unstripped-6.1.0~dev-0.20240614.5f36888e7fbd.x86_64.tar.gz /jenkins/workspace/scylla-master/scylla-ci/scylla/build/dist/RelWithDebInfo/scylla-unstripped-6.1.0~dev-0.20240614.5f36888e7fbd.x86_64.tar.gz
cd /jenkins/workspace/scylla-master/scylla-ci/scylla && scripts/create-relocatable-package.py --build-dir /jenkins/workspace/scylla-master/scylla-ci/scylla/build/RelWithDebInfo --node-exporter-dir /jenkins/workspace/scylla-master/scylla-ci/scylla/build/node_exporter --debian-dir /jenkins/workspace/scylla-master/scylla-ci/scylla/build/debian /jenkins/workspace/scylla-master/scylla-ci/scylla/build/dist/RelWithDebInfo/scylla-unstripped-6.1.0~dev-0.20240614.5f36888e7fbd.x86_64.tar.gz
ldd: /jenkins/workspace/scylla-master/scylla-ci/scylla/build/RelWithDebInfo/scylla: No such file or directory
Traceback (most recent call last):
  File "/jenkins/workspace/scylla-master/scylla-ci/scylla/scripts/create-relocatable-package.py", line 109, in <module>
    libs.update(ldd(exe))
                ^^^^^^^^
  File "/jenkins/workspace/scylla-master/scylla-ci/scylla/scripts/create-relocatable-package.py", line 37, in ldd
    for ldd_line in subprocess.check_output(
                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/subprocess.py", line 466, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ldd', '/jenkins/workspace/scylla-master/scylla-ci/scylla/build/RelWithDebInfo/scylla']' returned non-zero exit status 1.
```

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2024-06-14 13:27:26 +08:00
Kefu Chai
b94420a9dd build: cmake: use path to be compatible with CI
this change is created in the same spirit of 1186ddef16, which
updated the rule for generating the stripped dist pkg, but it
failed to update the one for generating the unstripped dist pkg.
what's why we have build failure when the workflow is looking for
the unstripped tar.gz:

```
08:02:47  ++ ls /jenkins/workspace/scylla-master/scylla-ci/scylla/build/RelWithDebInfo/dist/tar/scylla-unstripped-6.1.0~dev-0.20240613.d5bdddaeb40b.x86_64.tar.gz
08:02:47  ls: cannot access '/jenkins/workspace/scylla-master/scylla-ci/scylla/build/RelWithDebInfo/dist/tar/scylla-unstripped-6.1.0~dev-0.20240613.d5bdddaeb40b.x86_64.tar.gz': No such file or directory`
```

so, in this change, we fix the path.

Refs #2717

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2024-06-14 13:27:26 +08:00
Avi Kivity
dffd0901b3 dist: scylla_util: sysconfig_parser: replace deprecated ConfigParser.readfp
ConfigParser.readfp was deprecated in Python 3.2 and removed in Python 3.12.

Under Fedora 40, the container fails to launch because it cannot parse its
configuration.

Fix by using the newer read_file().

Closes scylladb/scylladb#19236
2024-06-12 10:07:10 +03:00
Takuya ASADA
33af97ca5a dist/docker: revert dropping systemd package
On 7ce6962141 we dropped openssh-server,
it also dropped systemd package and caused an error on Scylla Operator
(#17787).

This reverts dropping systemd package and fix the issue.

Fix #17787

Closes scylladb/scylladb#18643
2024-05-20 16:38:15 +03:00
Patryk Wrobel
ec820e214c scylla_io_setup: ensure correct RLIMIT_NOFILE for iotune
The default limit of open file descriptors
per process may be too small for iotune on
certain machines with large number of cores.

In such case iotune reports failure due to
unability to create files or to set up seastar
framework.

This change configures the limit of open file
descriptors before running iotune to ensure
that the failure does not occur.

The limit is set via 'resource.setrlimit()' in
the parent process. The limit is then inherited
by the child process.

Signed-off-by: Patryk Wrobel <patryk.wrobel@scylladb.com>

Closes scylladb/scylladb#18546
2024-05-13 08:35:52 +03:00
Yaron Kaikov
3eb81915c1 docker: drop jmx and tools-java from installation
Following the work done in dd0779675f,
removing the scylla-jmx and scylla-tools-java from our docker image

Closes scylladb/scylladb#18566
2024-05-13 07:24:23 +03:00
Takuya ASADA
9538af0d95 scylla_kernel_check: fix block device size error on latest mkfs.xfs
On latest mkfs.xfs, it does not allow to format a block device which is
smaller than 300MB.
There are options to ignore this validation but it is unsupported
feature, so it is better to increase the loopback image size to
"supported size" == 300MB.

reference: https://lore.kernel.org/all/164738662491.3191861.15611882856331908607.stgit@magnolia/

Fixes #18568

Closes scylladb/scylladb#18620
2024-05-13 07:23:29 +03:00
Kefu Chai
1186ddef16 build: cmake: use paths to be compatible with CI
our CI workflow for publishing the packages expects the tar balls
to be located under `build/$buildMode/dist/tar`, where `$buildMode`
is "release" or "debug".

before this change, the CMake building system puts the tar balls
under "build/dist" when the multi-config generator is used. and
`configure.py` uses multi-config generator.

in this change, we put the tar balls for redistribution under
`build/$<CONFIG>/dist/tar`, where `$<CONFIG>` is "RelWithDebInfo"
or "Debug", this works better with the CI workflow -- we just need
to map "release" and "debug" to "RelWithDebInfo" and "Debug" respectively.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2024-05-11 21:56:50 +08:00
Kefu Chai
0f85255c74 build: cmake build dist-unified by default
in the same spirit of d57a82c156, this change adds `dist-unified`
as one of the default targets. so that it is built by default.
the unified package is required to when redistributing the precompiled
packages -- we publish the rpm, deb and tar balls to S3.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2024-05-11 18:44:11 +08:00
Yaron Kaikov
2cf7cc1ea5 scylla_setup: Remove jmx and tools packages from being verified
Following
b8634fb244
machine image started to fail with the following error:
```
10:44:59  ␛[0;32m    googlecompute.gce: scylla-jmx package is not installed.␛[0m
10:44:59  ␛[1;31m==> googlecompute.gce: Traceback (most recent call last):␛[0m
10:44:59  ␛[1;31m==> googlecompute.gce:   File "/home/ubuntu/scylla_install_image", line 135, in <module>␛[0m
10:44:59  ␛[1;31m==> googlecompute.gce:     run('/opt/scylladb/scripts/scylla_setup --no-coredump-setup --no-sysconfig-setup --no-raid-setup --no-io-setup --no-ec2-check --no-swap-setup --no-cpuscaling-setup --no-ntp-setup', shell=True, check=True)␛[0m
10:44:59  ␛[1;31m==> googlecompute.gce:   File "/usr/lib/python3.10/subprocess.py", line 526, in run␛[0m
10:44:59  ␛[1;31m==> googlecompute.gce:     raise CalledProcessError(retcode, process.args,␛[0m
10:44:59  ␛[1;31m==> googlecompute.gce: subprocess.CalledProcessError: Command '/opt/scylladb/scripts/scylla_setup --no-coredump-setup --no-sysconfig-setup --no-raid-setup --no-io-setup --no-ec2-check --no-swap-setup --no-cpuscaling-setup --no-ntp-setup' returned non-zero exit status 1.␛[0m
```

It seems we no longer need to verify that jmx and tools-java packages are installed.

Closes scylladb/scylladb#18494
2024-05-02 13:30:50 +03:00
Takuya ASADA
b8634fb244 dist: stop installing scylla-tools, scylla-jmx by default
Since we added native nodetool, we no longer need to install scylla-tools
and scylla-jmx, drop them from scylla metapackage and make it optional
package.

Closes #18472

Closes scylladb/scylladb#18487
2024-05-01 22:15:40 +03:00
Kefu Chai
4433d2e10e build: cmake: let iotune depends on config specific file
before this change, in order to build `${iotune_path}`, we use
the rule to build `app_iotune` but this target is built using
the default build type, see
https://cmake.org/cmake/help/latest/variable/CMAKE_DEFAULT_BUILD_TYPE.html#variable:CMAKE_DEFAULT_BUILD_TYPE
so, if we want to build `${iotune_path}` for the configuration
which is not listed as the first item in `CMAKE_CONFIGURATION_TYPES`,
we would end up with copying an nonexistent file.

to address this issue, we override the this behavior using
the `$<OUTPUT_CONFIG:...>` generator-expression. so that we
can depend on non-unique path. and the file-level dependency
between ${iotune_path} and $<CONFIG>/iotune can be established.

see also
https://cmake.org/cmake/help/latest/generator/Ninja%20Multi-Config.html#custom-commands

Refs #2717

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#18395
2024-04-29 09:06:39 +03:00
Botond Dénes
7cbe5c78b4 install.sh: use the native nodetool directly
* tools/java b810e8b00e...4ee15fd9ea (1):
  > install.sh: don't install nodetool into /usr/bin

Add a bin/nodetool and install it to bin/ in install.sh. This script
simply forwards to scylla nodetool and it is the replacement for the
Java nodetool, which is dropped from the java-tools's install.sh, in the
submodule update also included in this patch.
With this change, we now hardwire the usage of the native nodetool, as
*the* nodetool, with the intermediary nodetool wrapper script removed
from the picture.
Bash completion was copied from the java tools repository and it is now
installed by the scylla package, together with nodetool.

The Java nodetool is still available as as a fall-back, in case the
native nodetool has problems, at the path of
/opt/scylladb/share/cassandra/bin/nodetool.

Testing

I tested upgrades on a DEB and RPM distro: Ubuntu and Fedora.
First I installed scylla-5.4, then I installed the packages for this PR.
On Ubuntu, I had to use dpkg -i --auto-deconfigure, otherwise, dpkg would
refuse to install the new packages because they break the old ones. No
extra flags were required on Fedora.
In both cases, /usr/bin/nodetool was changed from a thunk calling the
Java nodetool (from 5.4) to the native launcher script from this PR.
/opt/scylladb/share/cassandra/bin/nodetool remained in place and still
works after the upgrade.

I also verified that --nonroot installs also work. Nodetool works both
when called with an absolute path, or when ~/scylladb/bin is added to
$PATH.

Fixes: #18226
Fixes: #17412

Closes scylladb/scylladb#18255

[avi: reset submodule to actual hash we ended up with]
2024-04-25 22:52:00 +03:00
Yaniv Kaul
2ce2649ec1 Typo: you -> your
Signed-off-by: Yaniv Kaul <yaniv.kaul@scylladb.com>

Closes scylladb/scylladb#17806
2024-04-04 14:55:46 +03:00
Kefu Chai
d57a82c156 build: cmake: add dist-* targets to the default build target
also, add a target of `dist-server`, which mirrors the structure
of the targets created by `configure.py`, and it is consistent
with the ones defined by `build_submodule()`.

so that they are built when our CI runs `ninja -C $build`. CI
expects that all these rpm and deb packages to built when
`ninja -C $build` finishes. so that it can continue with
building the container image. let's make it happen. so that
the CMake-based rules can work better with CI.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2024-03-18 20:02:43 +08:00
Kefu Chai
384e9e9c7c build: cmake: put server deb packages under build/dist/$<CONFIG>/debian
this change is a follow up of ca7f7bf8e2, which changed the output path
to build/$<CONFIG>/debian. but what dist/docker/debian/build_docker.sh
expects is `build/dist/$config/debian/*.deb`, where `$config` is the
normalized mode, when the debian packages are built using CMake
generated rules, `$mode` is CMake configuration name, i.e., `$<CONFIG>`.
so, ca7f7bf8e2 made a mistake, as it does not match the expectation of
`build_docker.sh`.

in this change, this issue is addressed. so we use the same path
in both `dist/CMakeLists.txt` and `dist/docker/debian/build_docker.sh`.

apply the same change to `dist-server-rpm`.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2024-03-18 14:21:39 +08:00
Kefu Chai
80d2981473 dist/docker: collect deb packages from different dir for CMake builds
CMake generate debian packages under build/$<CONFIG>/debian instead of
build/$mode/debian. so let's translate $mode to $<CONFIG> if
build.ninja is found under build/ directory, as configure.py puts
build.ninja under $top_srcdir, while CMake puts it under build/ .

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#17592
2024-03-06 10:13:47 +02:00
Botond Dénes
d37ac1545b Merge 'build: cmake: fixes for debian packaging' from Kefu Chai
- changes to use build/$<CONFIG> for build directory
- add ${CMAKE_BINARY_DIR}/debian as a dep
- generate deb packages under build/$<CONFIG>/debian

Closes scylladb/scylladb#17560

* github.com:scylladb/scylladb:
  build: cmake: generate deb packages under build/$<CONFIG>/debian
  build: cmake: add ${CMAKE_BINARY_DIR}/debian as a dep
  build: cmake: use build/$<CONFIG>/ instead of build
  build: cmake: always pass absolute path for add_stripped()
2024-03-06 10:12:18 +02:00