Ensure that the indices recorded in select_statement are passed to
result_set_builder when one is created for processing the cell values.
Signed-off-by: Dejan Mircevski <dejan@scylladb.com>
Extend the grammar file with GROUP BY, collect the column identifiers,
and store them in raw::select_statement.
Signed-off-by: Dejan Mircevski <dejan@scylladb.com>
There were many calls to read_keyspace_mutation. One in each function
that prepares a mutation for some other schema change.
With this patch they are all moved to a single location.
Tests: unit (dev, debug)
Signed-off-by: Rafael Ávila de Espíndola <espindola@scylladb.com>
Message-Id: <20190328024440.26201-1-espindola@scylladb.com>
Fedora28 python magic used to return a x-sharedlib mime type for .so files.
Fedora29 changed that to x-pie-executable, so the libraries are no longer
relocated.
Let's be more permissive and relocate everything that starts with application/.
Fixes#4396
Signed-off-by: Glauber Costa <glauber@scylladb.com>
Message-Id: <20190404140929.7119-1-glauber@scylladb.com>
This series fixes row level repair shutdown related issues we saw with
dtests, e.g., use after free of the repair meta object, fail to stop a
table during shutdown.
Fixes: #4044Fixes: #4314Fixes: #4333Fixes: #4380
Tests: repair_additional_test.py:RepairAdditionalTest.repair_abort_test
repair_additional_test.py:RepairAdditionalTest.repair_kill_2_test
* sestar-dev.git asias/repair.fix.shutdown.v1:
repair: Wait for pending repair_meta operation before removing it
repair: Check shutdown in row level repair
repair: Remove repair meta when node is dead
repair: Remove all row level repair during shtudown
* seastar 63d8607...6f73675 (5):
> Merge "seastar-addr2line: improve the context of backtraces" from Botond
> log: fix std::system_error ostream operator to print full error message
> Revert "threads: yield on get if we had run for too long."
> core/queue: Document concurrency constraints
> core/memory: Make small pools use the full span size
Fixes#4407.
Fixes#4316.
"
Calculation of IO properties is slightly wrong for i3.metal, because we get
the number of disks wrong. The reason for that is our check for ephemeral nvme
disks, that pre-date the time in which root devices were exposed as nvme devices
(nitro and metal instances).
"
toolchain updated with python3-psutil
* 'ec2fixes' of github.com:glommer/scylla:
scylla_util.py: do not include root disks in ephemeral list
scylla-python3: include the psutil module
fix typo in scylla_ec2_check
They are of type db::system_distributed_keyspace and
db::view::view_update_generator.
n1 is in normal status
n2 boots up and _sys_dist_ks or _view_update_generator are not
initialized
n1 runs stream, n2 is the follower.
n2 uses the _sys_dist_ks or _view_update_generator
"Assertion `local_is_initialized()' failed" is observed
Fixes#4360
Message-Id: <4ae13e1640ac8707a9ba0503a2744f6faf89ecf4.1554330030.git.asias@scylladb.com>
"
With these changes we avoid a std::vector<data_value> copy, which is
nice in itself, but also makes it possible to call get_list from other
shards.
"
* 'espindola/result-set-v3' of https://github.com/espindola/scylla:
Avoid copying a std::vector in get_list
query-result-set: add and use a get_ptr method
They are of type db::system_distributed_keyspace and db::view::view_update_generator.
n1 is in normal status
n2 boots up and _sys_dist_ks or _view_update_generator are not initialized
n1 runs repair, n2 is the follower.
n2 uses the _sys_dist_ks or _view_update_generator
"Assertion `local_is_initialized()' failed" is observed
Fixes#4360
Message-Id: <6616c21078c47137a99ba71baf82594ba709597c.1553742487.git.asias@scylladb.com>
For now this is just an optimization. But it also avoids copying
data_type, which will allow this be used across shards.
Signed-off-by: Rafael Ávila de Espíndola <espindola@scylladb.com>
This moves a copy up the call stack and makes it possible to avoid it
completely by passing a reference type to get_nonnull.
Signed-off-by: Rafael Ávila de Espíndola <espindola@scylladb.com>
When --abort-on-lsa-bad-alloc is enabled we want to abort whenever
we think we can be out of memory.
We covered failures due to bad_alloc thrown from inside of the
allocation section, but did not cover failures from reservations done
at the beginning of with_reserve(). Fix by moving the trap into
reserve().
Message-Id: <1553258915-27929-1-git-send-email-tgrabiec@scylladb.com>
Nitro instances (and metal ones) put their root device in nvme (as a
protocol. it is still EBS). Our algorithm so far has relied on parsing
the nvme devices to figure out which ones are ephemeral but it will
break for those instances.
Out of our supported instances so far, the i3.metal is the only one
in which this breaks.
Signed-off-by: Glauber Costa <glauber@scylladb.com>
Using a new python3 module has never been that easy! So we'll
unapologetically use psutil and don't even worry about whether or not
CentOS supports it (it doesn't)
Signed-off-by: Glauber Costa <glauber@scylladb.com>
* seastar 5572de7...63d8607 (6):
> test: verify that negative sleep time doesn't cause infinite sleep
> httpd: Change address handling to use socket_address
> dns: Change "unspecififed" address search type to retrive first avail
> Allow when_all and when_all_succeed to take function arguments
> when_all: abort if memory allocation fails
> inet_address: Add missing constructor impl.
We saw dtest failed to stop a node like:
```
ERROR: repair_one_missing_row_test (repair_additional_test.RepairAdditionalTest)
----------------------------------------------------------------------
Traceback (most recent
[2019.1.3.node1.repair.zip](https://github.com/scylladb/scylla/files/2723244/2019.1.3.node1.repair.zip)
call last):
File "/home/asias/src/cloudius-systems/scylla-dtest/repair_additional_test.py", line 2521, in repair_one_missing_row_test
return RepairAdditionalBase._repair_one_missing_row_test(self)
File "/home/asias/src/cloudius-systems/scylla-dtest/repair_additional_test.py", line 1842, in _repair_one_missing_row_test
self.check_rows_on_node(node2, nr_rows)
File "/home/asias/src/cloudius-systems/scylla-dtest/repair_additional_test.py", line 34, in check_rows_on_node
node.stop(wait_other_notice=True)
File "/home/asias/src/cloudius-systems/scylla-ccm/ccmlib/scylla_node.py", line 496, in stop
raise NodeError("Problem stopping node %s" % self.name)
NodeError: Problem stopping node node1
```
The problem is:
1) repair_meat is created
repair_meta -> repair_writer::create_writer() -> t.stream_in_progress()
repari_meta -> repair_reader::repair_reader -> cf.read_in_progress()
2) repair_meta is stored in _repair_metas map.
3) Shtudown repair, repair_meta is not removed from the _repair_metas map
4) Shutdown database which wait for the utils::phased_barrier.
To fix, we should stop and remove all the repair_meata from the _repair_metas map.
Tests: 30 successful runs of the repair_kill_2_test
Fixes: #4044
Repair follower nodes will create repair meta object when repair master
node starts a repair. Normally, the repair meta object is removed when
repair master finishes the repair and sends the verb
REPAIR_ROW_LEVEL_STOP to all the followers to remove the repair meta
object. In case of repair master was killed suddenly, no one will remove
the repair meta object.
To prevent keeping this repair meta object forever, we should remove
such objects when gossip detects a node is dead with the gossip
listener.
Fixes: #4380
Reviewed-by: Botond Dénes <bdenes@scylladb.com>
We remove repair_meta object in remove_repair_meta up receiving of stop
row level repair rpc verb. It is possible there is an pending operation
of repair_meta. To avoid use after free, we should not remove the
repair_meta object until all the pending operations are done.
Use a gate to protect it.
Fixes: #4333Fixes: #4314
Tests: 50 succesful run of repair_additional_test.py:RepairAdditionalTest.repair_kill_2_test
ignore_ready_future in load_new_ss_tables broke
migration_test:TestMigration_with_*.migrate_sstable_with_counter_test_expect_fail dtests.
The java.io.NotSerializableException in nodetool was caused by exceptions that
were too long.
This fix prints the problematic file names onto the node system log
and includes the casue in the resulting exception so to provide the user
with information about the nature of the error.
Fixes#4375
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Message-Id: <20190331154006.12808-1-bhalevy@scylladb.com>
"
To make offline installer easier we need to minimize dependencies as
possible.
Python dependencies are already dropped by adding relocatable python3 by
Glauber, now it's time to drop rest of command line tools which used by
scylla setup tools.
(even scripts are converted to python3, it still executes some external
commands, so these commands should be distributed with offline installer)
Note that some of CLI tools haven't added such as NTP and RAID stuff,
since these tools have daemons, not just CLI.
To use such stuff in offline mode, users have to install them manually.
But both NTP setup and RAID setup are optional, users still can run Scylla w/o
them.
"
Toolchain updated to docker.io/scylladb/scylla-toolchain:fedora-29-20190401
for changes in install-dependencies.sh; also updates to gnutls 3.6.7 security
release.
* 'reloc_clitools_v5' of https://github.com/syuu1228/scylla:
reloc: add relocatable CLI tools for scylla setup scripts
dist/redhat: drop systemd-libs from dependency
dist/redhat: drop file from dependency since it seems unused
dist/redhat: drop pciutils from dependency since it only used in DPDK mode
Truncate would make disk usage stat go wild because it isn't updated
when sstables are removed in table::discard_sstables(). Let's update
the stat after sstables are removed from the sstable set.
Fixes#3624.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <20190328154918.25404-1-raphaelsc@scylladb.com>
Since we don't use DPDK mode by default, and the mode is not officially
supported, drop pciutils from package dependency.
Users who want to use DPDK mode they neeed to install the package
manually.
* seastar 05efbce...5572de7 (5):
> posix_file_impl::list_directory: do not ignore symbolic link file type
> prometheus: yield explicitly after each metric is processed
> thread: add maybe_yield function
> metrics: add vector overload of add_group()
> memory: tone down message for memory allocator
Currently scylla-python3 package name is hardcorded, need to support
package name renaming just like on other scylla packages.
This is required to release enterprise version.
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20190329003941.12289-1-syuu@scylladb.com>
"
File must be either owned by the process uid
or have both read and write access to it,
so it could be (hard) linked when sysctl
fs.protected_hardlinks is enabled.
Fixes#3117
"
* 'projects/valid_owner_and_mode/v3-rebased' of https://github.com/bhalevy/scylla:
storage_service: handle load_new_sstables exception
init: validate file ownership and mode.
treewide: use std::filesystem
Files and directories must be owned by the process uid.
Files must have read access and directories must have
read, write, and execute access.
Refs #3117
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Rather than {std::experimental,boost,seastar::compat}::filesystem
On Sat, 2019-03-23 at 01:44 +0200, Avi Kivity wrote:
> The intent for seastar::compat was to allow the application to choose
> the C++ dialect and have seastar follow, rather than have seastar choose
> the types and have the application follow (as in your patch).
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
When generating view updates for base mutations when no pre-existing
data exists, we were forgetting to apply the tracked tombstones.
Fixes#4321
Tests: unit(dev)
* https://github.com/duarten/scylla materialized-views/4321/v1.1:
db/view: Apply tracked tombstones for new updates
tests/view_schema_test: Add reproducer for #4321
When generating view updates for base mutations when no pre-existing
data exists, we were forgetting to apply the tracked tombstones.
Fixes#4321
Tests: unit(dev)
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
