Commit Graph

195 Commits

Author SHA1 Message Date
Pekka Enberg
9fc83ac627 Update tools/java submodule
* tools/java 8080009794...4a55b81941 (1):
  > cassandra.in.sh: remove debug message
2021-01-26 15:56:58 +02:00
Pekka Enberg
8258556832 Update tools/python3 submodule
* tools/python3 c579207...199ac90 (1):
  > dist: debian: adjust .orig tarball name for .rc releases
2021-01-24 21:30:59 +02:00
Pekka Enberg
7d98e05923 Update tools/python3 submodule
* tools/python3 1763a1a...c579207 (1):
  > dist/debian: handle rc version correctly
2021-01-21 10:41:33 +02:00
Avi Kivity
daa0e964fc dbuild: avoid --pids-limit with podman and cgroupsv1
Podman doesn't correctly support --pids-limit with cgroupsv1. Some
versions ignore it, and some versions reject the option.

To avoid the error, don't supply --pids-limit if cgroupsv2 is not
available (detected by its presence in /proc/filesystems). The user
is required to configure the pids limit in
/etc/containers/containers.conf.

Fixes #7938.

Closes #7939
2021-01-21 10:41:33 +02:00
Botond Dénes
7eb8c71342 tools/scylla-types: add link to cql3-type-mapping.md
Just like scylla-sstable-index, scylla-types accepts types in (short)
cassandra class name notation. The mapping from the clq3 type names to
the class names is not straight-forward in all cases, so provide a link
to a table which lists the cassandra class name of all supported types
(and more).

Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Message-Id: <20210120083816.37774-2-bdenes@scylladb.com>
2021-01-20 10:50:33 +02:00
Botond Dénes
882ade7c6a types/scylla-sstable-index: update URL to cql3-type-mapping.md
Said document was recently moved but the URL was not updated.

Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Message-Id: <20210120083816.37774-1-bdenes@scylladb.com>
2021-01-20 10:50:33 +02:00
Avi Kivity
1716359455 Update tools/jmx submodule
* tools/jmx 20469bf...2c95650 (1):
  > install.sh: set a valid WorkingDirectory for nonroot offline install
2020-12-28 21:19:04 +02:00
Botond Dénes
978ec7a4bb tools: introduce scylla-sstable-index
A tool which lists all partitions contained in an sstable index. As all
partitions in an sstable are indexed, this tool can be used to find out
what partitions are contained in a given sstable.

The printout has the following format:
$pos: $human_readable_value (pk{$raw_hex_value})

Where:
* $pos: the position of the partition in the (decompressed) data file
* $human_readable_value: the human readable partition key
* $raw_hex_value: the raw hexadecimal value of the binary representation
  of the partition key

For now the tool requires the types making up the partition key to be
specified on the command line, using the `--type|-t` command line
argument, using the Cassandra type class name notation for types.
As these are not assumed to be widely known, this patch includes a
document mapping all cql3 types to their Cassandra type class name
equivalent (but not just).

Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Message-Id: <20201208092323.101349-1-bdenes@scylladb.com>
2020-12-15 18:46:47 +02:00
Takuya ASADA
ebc4076fa5 tools: toolchain: add node_exporter
Download node_exporter in frozen image to prepare adding node_exporter
to relocatable pacakge.

Related #2190

Closes #7765

[avi: updated toolchain, x86_64/aarch64/s390x]
2020-12-14 20:34:17 +02:00
Avi Kivity
65a0244614 Update tools/jmx submodule
* tools/jmx 6174a47...20469bf (1):
  > column_family: Return proper cardinality for toppartitions requests
2020-12-13 13:51:38 +02:00
Botond Dénes
ba7cf2f5fd tools/scylla-types: update name in description to use - instead of _
The executable was rename from using _ to using - to at one point but
apparently the description wasn't updated.

Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Message-Id: <20201207161626.79013-1-bdenes@scylladb.com>
2020-12-07 18:34:52 +02:00
Pekka Enberg
c84754a634 Update tools/java submodule
* tools/java ad48b44a26...8080009794 (1):
  > sstableloader: Fix command line parsing of "ignore-missing-columns"
2020-11-27 15:19:48 +02:00
Avi Kivity
82b508250e tools: toolchain: dbuild: don't confine with seccomp
Some systems (at least, Centos 7, aarch64) block the membarrier()
syscall via seccomp. This causes Scylla or unit tests to burn cpu
instead of sleeping when there is nothing to do.

Fix by instructing podman/docker not to block any syscalls. I
tested this with podman, and it appears [1] to be supported on
docker.

[1] https://docs.docker.com/engine/security/seccomp/#run-without-the-default-seccomp-profile

Closes #7661
2020-11-20 09:11:52 +02:00
Pekka Enberg
ba39bfa1be dist-check: Fix script name to work on Windows filesystem
Asias He reports that git on Windows filesystem is unhappy about the
colon character (":") present in dist-check files:

$ git reset --hard origin/master
error: invalid path 'tools/testing/dist-check/docker.io/centos:7.sh'
fatal: Could not reset index file to revision 'origin/master'.

Rename the script to use a dash instead.

Closes #7648
2020-11-19 13:16:30 +02:00
Avi Kivity
017174670b Update frozen toolchain for python3-urwid-2.1.2
urwid 2.1.0 struggles with some locale settings. 2.1.2
fixes the problem.

Fixes #7487.
2020-11-11 11:54:05 +02:00
Michał Chojnowski
f6c33f5775 dbuild: export $HOME seen by dbuild, not by $tool
The default of DBUILD_TOOL=docker requires passwordless access to docker
by the user of dbuild. This is insecure, as any user with unconstrained
access to docker is root equivalent. Therefore, users might prefer to
run docker as root (e.g. by setting DBUILD_TOOL="sudo docker").

However, `$tool -e HOME` exports HOME as seen by $tool.
This breaks dbuild when `$tool` runs docker as a another user.
`$tool -e HOME="$HOME"` exports HOME as seen by dbuild, which is
the intended behaviour.

Closes #7555
2020-11-05 18:44:50 +01:00
Michał Chojnowski
8f74c7e162 dbuild: Replace stray use of docker with $tool
Instead of invoking `$tool`, as is done everywhere else in dbuild,
kill_it() invoked `docker` explicitly. This was slightly breaking the
script for DBUILD_TOOL other than `docker`.

Closes #7554
2020-11-05 18:44:49 +01:00
Pekka Enberg
edf04cd348 Update tools/python3 submodule
* tools/python3 cfa27b3...1763a1a (1):
  > Relocatable Package: create product prefixed relocatable archive
2020-11-04 14:24:20 +02:00
Pekka Enberg
5519ce2f0e Update tools/jmx submodule
* tools/jmx c51906e...6174a47 (2):
  > Relocatable Package: create product prefixed relocatable archive
  > build(deps-dev): bump junit from 4.8.2 to 4.13.1
2020-11-04 14:24:15 +02:00
Avi Kivity
b6f8bb6b77 tools/toolchain: update maintainer instructions
The instructions are updated for multiarch images (images that
can be used on x86 and ARM machines).

Additionally,
 - docker is replaced with podman, since that is now used by
   developers. Docker is still supported for developers, but
   the image creation instructions are only tested with podman.
 - added instructions about updating submodules
 - `--format docker` is removed. It is not necessary with
   more recent versions of docker.

Closes #7521
2020-11-02 10:29:54 +02:00
Avi Kivity
b45c933036 tools: toolchain: update for gcc-10.2.1-6.fc33.x86_64 2020-11-01 19:18:00 +02:00
Avi Kivity
17226f2f6c tools: toolchain: update to Fedora 33 with clang 11
Update the toolchain to Fedora 33 with clang 11 (note the
build still uses gcc).

The image now creates a /root/.m2/repository directory; without
this the tools/jmx build fails on aarch64.

Add java-1.8.0-openjdk-devel since that is where javac lives now.
Add a JAVA8_HOME environment variable; wihtout this ant is not
able to find javac.

The toolchain is enabled for x86_64 and aarch64.
2020-10-28 20:21:44 +02:00
Avi Kivity
5ff5d43c7a Update tools/java submodule
* tools/java e97c106047...ad48b44a26 (1):
  > build: Add generated Thrift sources to multi-Java build
2020-10-28 16:52:25 +02:00
Avi Kivity
6eb3ba74e4 Update tools/java submodule
* tools/java f2e8666d7e...e97c106047 (1):
  > Relocatable Package: create product prefixed relocatable archive
2020-10-28 08:47:49 +02:00
Amnon Heiman
52db99f25f scyllatop/livedata.py: Safe iteration over metrics
This patch change the code that iterates over the metrics to use a copy
of the metrics names to make it safe to remove the metrics from the
metrics object.

Fixes #7488

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2020-10-27 22:44:02 +02:00
Pekka Enberg
266d2b6f71 Update tools/jmx submodule
* tools/jmx c55f3f2...c51906e (1):
  > StorageService.java: Use the endpoint for getRangeToEndpointMap
2020-10-08 12:09:24 +03:00
Pekka Enberg
db6bb1ba91 Update tools/java submodule
* tools/java 4313155ab6...f2e8666d7e (1):
  > dist/debian/debian_files_gen.py: don't ignore permission error on shutil.rmtree()
2020-10-08 11:49:01 +03:00
Pekka Enberg
02bf30e9f5 Update tools/jmx submodule
* tools/jmx e3a381d...c55f3f2 (1):
  > dist/debian/debian_files_gen.py: don't ignore permission error on shutil.rmtree()
2020-10-08 11:48:57 +03:00
Pekka Enberg
16ed6fee40 Update tools/jmx submodule
* tools/jmx 25bcd76...e3a381d (1):
  > install.sh: show warning nonroot mode when systemd does not support user mode
2020-10-07 11:39:03 +03:00
Avi Kivity
6fdc8f28a9 Update tools/jmx submodule
* tools/jmx 45e4f28...25bcd76 (1):
  > install.sh: stop using symlinks for systemd units on nonroot mode

Fixes #7288.
2020-09-29 13:32:45 +03:00
Pekka Enberg
068b1e3470 Update tools/python3 submodule
* tools/python3 b4e52ee...cfa27b3 (1):
  > build: support passing product-version-release as a parameter
2020-09-28 10:53:11 +03:00
Avi Kivity
d45df0c705 Update tools/java submodule
* tools/java 583fc658df...4313155ab6 (1):
  > enable gettraceprobability
2020-09-24 12:19:29 +03:00
Pekka Enberg
bc13e596fe Update tools/java submodule
* tools/java d0cfef38d2...583fc658df (1):
  > build: support passing product-version-release as a parameter
2020-09-23 12:58:34 +03:00
Pekka Enberg
b0447f3245 Update tools/jmx submodule
* tools/jmx 6795a22...45e4f28 (1):
  > build: support passing product-version-release as a parameter
2020-09-23 12:58:30 +03:00
Pekka Enberg
ea8e545e4e Update tools/java submodule
* tools/java 2e2c056c07...d0cfef38d2 (1):
  > sstableloader: Support range boundary tombstones
2020-09-22 10:18:01 +02:00
Pekka Enberg
db272ba799 Update tools/java submodule
* tools/java 6c1c484140...2e2c056c07 (1):
  > sstableloader: Add verbose message if sstable/file cannot be opened
2020-09-17 17:19:41 +03:00
Pekka Enberg
9650b8d4b5 Update tools/java submodule
* tools/java b0114f64bc...6c1c484140 (1):
  > sstableloader: fix generating CQL statements
2020-09-17 12:25:08 +03:00
Avi Kivity
81844fb476 Update tools/java submodule
* tools/java 2d49ded77b...b0114f64bc (1):
  > Merge "dist: do not install build dependencies on build script" from Takuya

Fixes #7219.
2020-09-16 12:52:07 +03:00
Avi Kivity
888fde59f8 Update tools/jmx submodule
* tools/jmx d3096f3...6795a22 (1):
  > Merge "dist: do not install build dependencies on build script" from Takuya

Ref #7219.
2020-09-16 10:30:33 +03:00
Avi Kivity
764866ed02 Update tools/jmx submodule
* tools/jmx 8d92e54...d3096f3 (1):
  > dist: debian: fix detection of debuild
2020-09-13 16:26:53 +03:00
Avi Kivity
d55a8148ed tools: toolchain: update for gnutls-3.6.15
GNUTLS-SA-2020-09-04 / GNUTLS-SA-2020-09-04.

Fixes #7212.
2020-09-10 12:52:00 +03:00
Takuya ASADA
59a6e08cb9 Add support passing python3 dependencies from main repo to scylla-python3 script
We don't want to update scylla-python3 submodule for every python3 dependency
update, bring python3 package list to python3-dependencies.txt, pass it on
package building time.

See #6702
See scylladb/scylla-python3#6

[avi: add

* tools/python3 19a9cd3...b4e52ee (1):
  > Allow specify package dependency list by --packages

 to maintain bisectability]
2020-09-08 23:39:34 +03:00
Pekka Enberg
1ed9a336a5 Update tools/jmx submodule
* tools/jmx 12ab6aa...8d92e54 (1):
  > Merge 'JMX footprint work' from Calle
Fixes scylladb/scylla-jmx#133
Fixes scylladb/scylla-jmx#134
2020-09-07 13:56:47 +03:00
Pekka Enberg
478e831d4f Update tools/jmx submodule
* tools/jmx d5d1efd...12ab6aa (1):
  > Merge "Fix JMX startup after offline installation" from Amos

Fixes: scylladb/scylla#7098
Fixes: scylladb/scylla-jmx#129
2020-09-07 09:42:26 +03:00
Avi Kivity
37352a73b8 Update tools/python3 submodule
* tools/python3 f89ade5...19a9cd3 (1):
  > dist: redhat: reduce log spam from unpacking sources when building rpm
2020-09-01 12:36:24 +03:00
Pekka Enberg
0f1b54fa6e Update tools/java submodule
* tools/java d6c0ad1e2e...2d49ded77b (1):
  > sstableloader: remove wrong check that breaks range tombstones
2020-08-27 09:05:34 +03:00
Pekka Enberg
f7c5c48df6 Update tools/jmx submodule
* tools/jmx be8f1ac...d5d1efd (1):
  > dist/debian: Remove conflict tag for Java 11
2020-08-25 15:46:51 +03:00
Avi Kivity
5ec2eae247 build: add python3-redis to dependencies
Needed for redis tests.
2020-08-24 20:03:50 +03:00
Avi Kivity
b0ae9d0c7d Update tools/python3 submodule
* tools/python3 196be5a...f89ade5 (1):
  > reloc: cleanup deb builddir
2020-08-18 13:10:01 +03:00
Takuya ASADA
352a136ae2 scylla-python3: move scylla-python3 to separated repository
Except scylla-python3, each scylla package has its own git repository, same package script filename, same build directory structure.
To put python3 thing on scylla repo, we created 'python3' directory on multiple locations, made '-python3' suffixed files, dig deeper build directory not to conflict scylla-server package build.
We should move all scylla-python3 related files to new repository, scylla-python3.

To keep compatibility with current Jenkins script, provide packages on
build/ directory for now.

Fixes #6751
2020-08-18 09:34:08 +03:00