Commit Graph

228 Commits

Author SHA1 Message Date
Avi Kivity
a55b434a2b treewide: extent copyright statements to present day 2021-06-06 19:18:49 +03:00
Pavel Solodovnikov
e0749d6264 treewide: some random header cleanups
Eliminate not used includes and replace some more includes
with forward declarations where appropriate.

Signed-off-by: Pavel Solodovnikov <pa.solodovnikov@scylladb.com>
2021-06-06 19:18:49 +03:00
Avi Kivity
f3e8e625c0 Update tools/java submodule (toppartitions single jmx call)
* tools/java fd92603b99...599b2368d6 (1):
  > toppartitions: Fix toppartitions to only jmx once

Ref #8459.
2021-05-27 16:57:57 +03:00
Avi Kivity
eed89a9b56 Update tools/jmx submodule (toppartitions multi-sampler query)
* tools/jmx 440313e...a7c4c39 (1):
  > storage_service: Fix getToppartitions to always return both reads and writes
2021-05-11 18:39:10 +03:00
Botond Dénes
824b49aeb4 tools/scylla-sstable-index: use defer() to close sstables manager
So it is closed when loading the sstable throws an exception too.
Failing to close the manager will mask the real error as the user will
only see the assert failure due to failing to close the manager.

Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Message-Id: <20210429092248.50968-1-bdenes@scylladb.com>
2021-04-29 12:50:25 +03:00
Avi Kivity
7a6b678044 Update tools/java submodule for EveryWhere compaction strategy
* tools/java 57eb143119...fd92603b99 (1):
  > Add EverywhereStrategy class
2021-04-27 12:23:23 +03:00
Avi Kivity
34b57688b9 tools: toolchain: dbuild: define die() earlier
die() is called before it is defined, so it doesn't work. Move it eariler.

Ref #8520.

Closes #8523
2021-04-22 15:38:10 +02:00
Nadav Har'El
91249e9683 Update tools/java submodule
* tools/java 5756445ec7...57eb143119 (1):
  > sstableloader: Handle non-prepared batches with ":" in identifier names

Fixes #8230.
2021-04-06 16:37:03 +03:00
Avi Kivity
bfed9c15d5 Update tools/java submodule
* tools/java fb21784b91...5756445ec7 (1):
  > sstableloader: fix handling of rewritten partition

Ref scylladb/scylla-tools-java#238.
2021-04-01 16:07:46 +03:00
Avi Kivity
bbec43f9a1 Update tools/java submodule
* tools/java ccc4201ded...fb21784b91 (2):
  > fix: Add dummy implementation of getToppartitions
  > nodetool: Make toppartitions call the generic endpoint

Fixes #4520.
2021-03-31 17:38:03 +03:00
Pekka Enberg
aec33c599b Update tools/python3 submodule
* tools/python3 6f3bcbe...ad04e8e (2):
  > dist/debian: fix renaming debian/scylla-* files rule
  > fix license of package build script to AGPL
2021-03-29 11:50:24 +03:00
Pekka Enberg
203b7394d7 Update tools/java submodule
* tools/java 7b66b7a0fc...ccc4201ded (1):
  > dist/debian: fix renaming debian/scylla-* files rule
2021-03-29 11:50:19 +03:00
Avi Kivity
ec4d91f9eb tools: toolchain: dbuild: improve cgroupv2 detection code
dbuild detects if the kernel is using cgroupv2 by checking if the
cgroup2 filesystem is mounted on /sys/fs/cgroup. However, on Ubuntu
20.10, the cgroup filesystem is mounted on /sys/fs/cgroup and the
cgroup2 filesystem is mounted on /sys/fs/cgroup/unified. This second
mount matches the search expression and gives a false positive.

Fix by adding a space at the end; this will fail to match
/sys/fs/cgroup/unified.

Closes #8355
2021-03-29 09:31:29 +03:00
Avi Kivity
46185d7d82 Update tools/jmx submodule
* tools/jmx 9c687b5...440313e (1):
  > storage_service: Add a generic toppartitions endpoint
2021-03-25 12:36:10 +02:00
Avi Kivity
957259fab7 tools: toolchain: prepare: adjust manifest manipulations
The manifest manipulation commands stopped working with podman 3;
the containers-storage: prefix now throws errors.

Switch to `buildah manifest`; since we're building with buildah,
we might as well maintain the manifest with buildah as well.

Closes #8231
2021-03-23 09:18:19 +02:00
Takuya ASADA
0424a41e30 tools/toolchain: stop ignoring error on install-dependencies.sh, run jmx/java script correctly
We should run install-dependencies.sh with -e option to prevent ignoring
error in the script.
Also, need to add tools/jmx/install-dependencies.sh and
tools/java/install-dependencies.sh, to fix 'No such file or directory'
error on them.

Fixes #8293

Closes #8294

[avi: did not regenerate toolchain image, since no new packages are
      installed]
2021-03-18 11:24:18 +01:00
Avi Kivity
b91d6776a0 Update tools/java submodule
* tools/java fdc8fcc22c...7b66b7a0fc (1):
  > dist/redhat: add support SLES
2021-03-18 11:24:18 +01:00
Avi Kivity
11308c05f4 Update tools/jmx submodule
* tools/jmx 15c1d4f...9c687b5 (1):
  > dist/redhat: add support SLES
2021-03-17 09:59:22 +02:00
Avi Kivity
f326a2253c Update tools/java submodule
* tools/java 2c6110500c...fdc8fcc22c (1):
  > sstableloader: Use compound "where" restrictions for clustering
2021-03-15 11:19:22 +02:00
Avi Kivity
a49c4ab754 Update tools/java submodule
* tools/java c5d9e8513e...2c6110500c (1):
  > cassandra.in.sh: Add path to rack/dc properties file to classpath

Fixes #7930.
2021-03-11 12:03:01 +02:00
Nadav Har'El
ad563c6279 Update tools/java submodule
Fixes an sstableloader bug where we quoted twice column names that
had to be quoted, and therefore failed on such tables - and in particular
Alternator tables which always have a column called ":attrs".

Fixes #8229

* tools/java 142f517a23...c5d9e8513e (1):
  > sstableloader: Only escape column names once
2021-03-07 10:33:49 +02:00
Pekka Enberg
6066db7c90 Update tools/jmx submodule
* tools/jmx bac7d0b...15c1d4f (2):
  > StorageService: Add a method to return the uptime
  > Bump Jackson version in scylla-apiclient
2021-03-04 10:56:37 +02:00
Pekka Enberg
1d8a94f941 Update tools/jmx submodule
* tools/jmx c2fc96b...bac7d0b (1):
  > Merge 'Fix locking in APIBuilder.remove()' from Pekka Enberg
2021-03-03 18:30:48 +02:00
Avi Kivity
facc7c370e Update tools/jmx submodule
* tools/jmx 8073af6...c2fc96b (1):
  > APIBuilder: Remove RW-lock in JMX server repository wrapper

Fixes #7991.
2021-03-03 15:41:09 +02:00
Avi Kivity
86d8977c96 Update tools/python3 submodule
* tools/python3 199ac90...6f3bcbe (2):
  > Add support pip modules
  > create-relocatable-package.py: add support python libraries in /usr/local
2021-03-01 10:10:13 +02:00
Avi Kivity
8ac0d6d15d Update tools/jmx submodule
* tools/jmx bf8bb16...8073af6 (1):
  > CompactionManager: add the compaction id when available

Fixes #7927.
2021-03-01 10:09:16 +02:00
Pekka Enberg
d483922671 Update tools/java submodule
* tools/java 0187829d5e...142f517a23 (2):
  > nodetool: Enable resetlocalschema
  > sstableloader: Make progress printout less eager.
2021-02-19 12:37:04 +02:00
Pekka Enberg
843bf57c3c Update tools/jmx submodule
* tools/jmx 949cefc...bf8bb16 (1):
  > Merge 'dist/debian: fix renaming debian/scylla-* files rule' from Takuya ASADA
2021-02-18 10:35:00 +02:00
Avi Kivity
713a159600 tools: toolchain: add simplified procedure for creating dbuild images
The current procedure for building images is complicated, as it
requires access to x86_64, aarch64, and s390x machines. Add an alternative
procedure that is fully automated, as it relies on emulation on a single
machine.

It is slow, but requires less attention.

Closes #8024
2021-02-04 15:37:36 +02:00
Avi Kivity
bd7fbcc0cf tools: toolchain: dbuild: keep original user's groups
The supplementary groups are removed by default, so add them back.
Supplementary groups are useful for group-shared directories like
ccache.

I added them to the podman-only branch since I don't know if this
works for docker. If a docker user verifies it works there too,
we can move it to the generic code.

Closes #8020
2021-02-04 15:36:55 +02:00
Avi Kivity
10606aadb5 Update tools/java submodule
* tools/java 78c8ef4f54...0187829d5e (1):
  > nodetool: alternate way to specify table name which includes a dot

Fixes #6521.
2021-02-03 11:27:33 +02:00
Avi Kivity
285303b131 Update tools/jmx submodule
* tools/jmx 2c95650...949cefc (2):
  > dist/redhat: stop using systemd macros, call systemctl directly
  > Remove obsolete FIXME

See scylladb/scylla-jmx#94.
2021-02-02 11:29:36 +02:00
Avi Kivity
b32ece6975 Update tools/java submodule
* tools/java 4a55b81941...78c8ef4f54 (1):
  > nodetool: do no treat table name with dot as a secondary index

Fixes #6521.
2021-01-28 16:16:47 +02:00
Pekka Enberg
9fc83ac627 Update tools/java submodule
* tools/java 8080009794...4a55b81941 (1):
  > cassandra.in.sh: remove debug message
2021-01-26 15:56:58 +02:00
Pekka Enberg
8258556832 Update tools/python3 submodule
* tools/python3 c579207...199ac90 (1):
  > dist: debian: adjust .orig tarball name for .rc releases
2021-01-24 21:30:59 +02:00
Pekka Enberg
7d98e05923 Update tools/python3 submodule
* tools/python3 1763a1a...c579207 (1):
  > dist/debian: handle rc version correctly
2021-01-21 10:41:33 +02:00
Avi Kivity
daa0e964fc dbuild: avoid --pids-limit with podman and cgroupsv1
Podman doesn't correctly support --pids-limit with cgroupsv1. Some
versions ignore it, and some versions reject the option.

To avoid the error, don't supply --pids-limit if cgroupsv2 is not
available (detected by its presence in /proc/filesystems). The user
is required to configure the pids limit in
/etc/containers/containers.conf.

Fixes #7938.

Closes #7939
2021-01-21 10:41:33 +02:00
Botond Dénes
7eb8c71342 tools/scylla-types: add link to cql3-type-mapping.md
Just like scylla-sstable-index, scylla-types accepts types in (short)
cassandra class name notation. The mapping from the clq3 type names to
the class names is not straight-forward in all cases, so provide a link
to a table which lists the cassandra class name of all supported types
(and more).

Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Message-Id: <20210120083816.37774-2-bdenes@scylladb.com>
2021-01-20 10:50:33 +02:00
Botond Dénes
882ade7c6a types/scylla-sstable-index: update URL to cql3-type-mapping.md
Said document was recently moved but the URL was not updated.

Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Message-Id: <20210120083816.37774-1-bdenes@scylladb.com>
2021-01-20 10:50:33 +02:00
Avi Kivity
1716359455 Update tools/jmx submodule
* tools/jmx 20469bf...2c95650 (1):
  > install.sh: set a valid WorkingDirectory for nonroot offline install
2020-12-28 21:19:04 +02:00
Botond Dénes
978ec7a4bb tools: introduce scylla-sstable-index
A tool which lists all partitions contained in an sstable index. As all
partitions in an sstable are indexed, this tool can be used to find out
what partitions are contained in a given sstable.

The printout has the following format:
$pos: $human_readable_value (pk{$raw_hex_value})

Where:
* $pos: the position of the partition in the (decompressed) data file
* $human_readable_value: the human readable partition key
* $raw_hex_value: the raw hexadecimal value of the binary representation
  of the partition key

For now the tool requires the types making up the partition key to be
specified on the command line, using the `--type|-t` command line
argument, using the Cassandra type class name notation for types.
As these are not assumed to be widely known, this patch includes a
document mapping all cql3 types to their Cassandra type class name
equivalent (but not just).

Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Message-Id: <20201208092323.101349-1-bdenes@scylladb.com>
2020-12-15 18:46:47 +02:00
Takuya ASADA
ebc4076fa5 tools: toolchain: add node_exporter
Download node_exporter in frozen image to prepare adding node_exporter
to relocatable pacakge.

Related #2190

Closes #7765

[avi: updated toolchain, x86_64/aarch64/s390x]
2020-12-14 20:34:17 +02:00
Avi Kivity
65a0244614 Update tools/jmx submodule
* tools/jmx 6174a47...20469bf (1):
  > column_family: Return proper cardinality for toppartitions requests
2020-12-13 13:51:38 +02:00
Botond Dénes
ba7cf2f5fd tools/scylla-types: update name in description to use - instead of _
The executable was rename from using _ to using - to at one point but
apparently the description wasn't updated.

Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Message-Id: <20201207161626.79013-1-bdenes@scylladb.com>
2020-12-07 18:34:52 +02:00
Pekka Enberg
c84754a634 Update tools/java submodule
* tools/java ad48b44a26...8080009794 (1):
  > sstableloader: Fix command line parsing of "ignore-missing-columns"
2020-11-27 15:19:48 +02:00
Avi Kivity
82b508250e tools: toolchain: dbuild: don't confine with seccomp
Some systems (at least, Centos 7, aarch64) block the membarrier()
syscall via seccomp. This causes Scylla or unit tests to burn cpu
instead of sleeping when there is nothing to do.

Fix by instructing podman/docker not to block any syscalls. I
tested this with podman, and it appears [1] to be supported on
docker.

[1] https://docs.docker.com/engine/security/seccomp/#run-without-the-default-seccomp-profile

Closes #7661
2020-11-20 09:11:52 +02:00
Pekka Enberg
ba39bfa1be dist-check: Fix script name to work on Windows filesystem
Asias He reports that git on Windows filesystem is unhappy about the
colon character (":") present in dist-check files:

$ git reset --hard origin/master
error: invalid path 'tools/testing/dist-check/docker.io/centos:7.sh'
fatal: Could not reset index file to revision 'origin/master'.

Rename the script to use a dash instead.

Closes #7648
2020-11-19 13:16:30 +02:00
Avi Kivity
017174670b Update frozen toolchain for python3-urwid-2.1.2
urwid 2.1.0 struggles with some locale settings. 2.1.2
fixes the problem.

Fixes #7487.
2020-11-11 11:54:05 +02:00
Michał Chojnowski
f6c33f5775 dbuild: export $HOME seen by dbuild, not by $tool
The default of DBUILD_TOOL=docker requires passwordless access to docker
by the user of dbuild. This is insecure, as any user with unconstrained
access to docker is root equivalent. Therefore, users might prefer to
run docker as root (e.g. by setting DBUILD_TOOL="sudo docker").

However, `$tool -e HOME` exports HOME as seen by $tool.
This breaks dbuild when `$tool` runs docker as a another user.
`$tool -e HOME="$HOME"` exports HOME as seen by dbuild, which is
the intended behaviour.

Closes #7555
2020-11-05 18:44:50 +01:00
Michał Chojnowski
8f74c7e162 dbuild: Replace stray use of docker with $tool
Instead of invoking `$tool`, as is done everywhere else in dbuild,
kill_it() invoked `docker` explicitly. This was slightly breaking the
script for DBUILD_TOOL other than `docker`.

Closes #7554
2020-11-05 18:44:49 +01:00