To make product name templatization works correctly, we cannot use
"debian/scylla-server" as package contents directory path,
need to use template like "debian/{{product}}-server" instead.
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20190517121946.18248-1-syuu@scylladb.com>
The calculation is done in a non preemptable loop over all tables, so if
numbers of tables is very large it may take a while since we also build
a string for gossiper state. Make the loop preemtable and also make
the string calculation more efficient by preallocating memory for it.
Message-Id: <20190516132748.6469-3-gleb@scylladb.com>
invoke_on_all() copies provided function for each shard it is executed
on, so by moving stats map into the capture we copy it for each shard
too. Avoid it by putting it into the top level object which is already
captured by reference.
Message-Id: <20190516132748.6469-2-gleb@scylladb.com>
All `table` instances currently unconditionally allocate a cell locker
for counter cells, though not all need one. Since the lockers occupy
quite a bit of memory (as reported in #4441), it's wasteful to
allocate them when unneeded.
Fixes#4441.
Tests: unit (dev, debug)
Signed-off-by: Dejan Mircevski <dejan@scylladb.com>
Message-Id: <20190515190910.87931-1-dejan@scylladb.com>
"
This series contains loosely related generic cleanup patches that the
timestamp based data segregation series depends on. Most of the patches
have to do with making headers self-sustainable, that is compilable on
their own. This was needed to be able to ensure that the new headers
introduced or touched by that series are self-sustainable too.
This series also introduces `schema_fwd.hh` which contains a forward
declaration of `schema` and `schema_ptr` classes. No effort was made to
find and replace all existing ad-hoc schema forward declarations in the
source tree.
"
* 'pre-timestamp-based-data-segregation-cleanup/v1' of https://github.com/denesb/scylla:
encoding_stats.hh: add missing include
sstables/time_window_compaction_strategy.hh: make self-sufficient
sstables/size_tiered_compaction_strategy.hh: make self-sufficient
sstables/compaction_strategy_impl.hh: make header self-sufficient
compaction_strategy.hh: use schema_fwd.hh
db/extensions.hh: use schema_fwd.hh
Add schema_fwd.hh
Makes opening all sstable components go through same file open
routine, optionally applying extensions to each (except TOC which
is special).
Also ensures we read Scylla metadata before other non-TOC
components, as we might need this for extensions (hint hint).
Message-Id: <20190513201821.14417-1-calle@scylladb.com>
"
Although CQL allows SELECT statements with both simple and aggregate
selectors, Scylla disallows them. This patch removes that restriction
and ensures that mixed simple/aggregate selection works as specified
both with and without GROUP BY.
Tests: unit (dev)
"
* 'aggregate-and-simple-select-together' of https://github.com/dekimir/scylla:
cql: Fix mixed selection with GROUP BY
cql: Allow mixing of aggregate and simple selectors
GROUP BY is currently supported by simple_selection, the class used
when all selectors are simple. But when selectors are mixed, we use
selection_with_processing, which does not yet support GROUP BY. This
patch fixes that.
It also adapts one testcase in filtering_test to the new behavior of
simple_selector. The test currently expects the last value seen, but
simple_selector now outputs the first value seen.
(More details: the WHERE clause implicitly selects the columns it
references, and unit tests are forced to provide expected values for
these columns. The user-visible result is unchanged in the test;
users never see the WHERE column values due to filtering in
cql::transport, outside unit tests.)
Signed-off-by: Dejan Mircevski <dejan@scylladb.com>
Scylla currently rejects SELECT statements with both simple and
aggregate selectors, but Cassandra allows them. This patch brings
parity to Scylla.
Fixes#4447.
Tests: unit (dev)
Signed-off-by: Dejan Mircevski <dejan@scylladb.com>
"
Commit d0f9e00 changed the representation of the gc_clock::duration
from int32_t to int64_t.
Mutation hashing uses appending_hash<gc_clock::time_point>, which by
default feeds duration::count() into the hasher. duration::rep changed
from int32_t to int64_t, which changes the value of the hash.
This affects schema digest and query digests, resulting in mismatches
between nodes during a rolling upgrade.
Fixes#4460.
Refs #4485.
"
* tag 'fix-gc_clock-digest-v2.1' of github.com:tgrabiec/scylla:
tests: Add test which verifies that schema digest stays the same
tests: Add sstables for the schema digest test
schema_tables, storage_service: Make schema digest insensitive to expired tombstones in empty partition
db/schema_tables: Move feed_hash_for_schema_digest() to .cc file
hashing: Introduce type-erased interface for the hasher
hashing: Introduce C++ concept for the hasher
hashers: Rename hasher to cryptopp_hasher
gc_clock: Fix hashing to be backwards-compatible
The _make_config_values macro reduces duplication (both the item name
and the types need to be available as C++ identifiers and as runtime
strings), but is hard to work with. The macro is huge and editors
don't handle it well, errors aren't identified at the correct
location, and since the macro doesn't have types, it's hard to
refactor.
This series replaces the macro with ordinary C++ code. Some repetition is
introduced, but IMO the result is easier to maintain than the macro. As a
bonus the bulk of the code is moved away from the header file.
Tests: unit (dev), manual testing of the config REST API
* https://github.com/avikivity/scylla config-no-macro/v2
config: make the named_value type name available without requiring
_make_config_values
config: remove value_status from named_value template parameter list
config: add named_value::value_as_json()
api: config: stop using _make_config_values
config: auto-add named_values into config_file
config: add allowed_values parameter to named_value constructor
config: convert _make_config_values to individual named_value member
declarations and initializers
Schema digest is calculated by querying for mutations of all schema
tables, then compacting them so that all tombstones in them are
dropped. However, even if the mutation becomes empty after compaction,
we still feed its partition key. If the same mutations were compacted
prior to the query, because the tombstones expire, we won't get any
mutation at all and won't feed the partition key. So schema digest
will change once an empty partition of some schema table is compacted
away.
That's not a problem during normal cluster operation because the
tombstones will expire at all nodes at the same time, and schema
digest, although changes, will change to the same value on all nodes
at about the same time.
This fix changes digest calculation to not feed any digest for
partitions which are empty after compaction.
The digest returned by schema_mutations::digest() is left unchanged by
this patch. It affects the table schema version calculation. It's not
changed because the version is calculated on boot, where we don't yet
know all the cluster features. It's possible to fix this but it's more
complicated, so this patch defers that.
Refs #4485.
Asd
The motivation is to allow hiding the definition of functions
accepting a hasher. For one, this reduces (re)complication times,
because we can put the definition in .cc
"
scylla_setup is currently broken for OEL. This happens because the
OS detection code checks for RHEL and Fedora. CentOS returns itself
as RHEL, but OEL does not.
"
* 'unbreakable' of github.com:glommer/scylla:
scylla_setup: be nicer about unrecognized OS
scylla_util: recognize OEL as part of the RHEL family
Right now if the user tries to execute this in an unrecognized OS, the
following will be thrown:
Traceback (most recent call last):
File "/usr/lib/scylla/libexec/scylla_setup", line 214, in <module>
do_verify_package('scylla-enterprise-jmx')
File "/usr/lib/scylla/libexec/scylla_setup", line 73, in do_verify_package
if res != 0:
UnboundLocalError: local variable 'res' referenced before assignment
It would be a lot nicer to exit gracefully and print a messge saying what
is going on. This was caught when running on OEL, which the previous patch
fixed. Still, there are other unknown OS out there the users may try to run
on.
Signed-off-by: Glauber Costa <glauber@scylladb.com>
Oracle Linux is a RHEL-like distribution and we support it just fine, but our
new incarnation of scylla_setup is failing to recognize it.
os-release for OEL is a bit different. It doesn't have an ID_LIKE string, and
only shows an ID string, which is set to 'ol'. So let's recognize this.
Fixes: #4493
Branches: 3.1
Signed-off-by: Glauber Costa <glauber@scylladb.com>
* seastar f73690e...3f7a5e1 (7):
> Revert "Make sure all allocations/deallocations are properly byte aligned"
> http: fix request content for POST requests
> doc: discourage generic lambdas and unconstrained templates
> smp: add smp_service_group for smp::submit_to() resource control
> Revert "smp: add smp_service_group for smp::submit_to() resource control"
> smp: add smp_service_group for smp::submit_to() resource control
> Make sure all allocations/deallocations are properly byte aligned
Commit d0f9e00 changed the representation of the gc_clock::duration
from int32_t to int64_t.
Mutation hashing uses appending_hash<gc_clock::time_point>, which by
default feeds duration::count() into the hasher. duration::rep changed
from int32_t to int64_t, which changes the value of the hash.
This affects schema digest and query digests, resulting in mismatches
between nodes during a rolling upgrade.
Fixes#4460.
(cherry picked from commit 549d0eb2f3)
"
This series contains fixes for GCC9 build, mostly corrections needed
after changes in libstdc++. With this series and a workaround for
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90415 (not included)
Scylla builds and passes unit tests with GCC9 (tested on Fedora 30,
development mode only).
Tests: unit(dev with gcc8 and gcc9).
"
* tag 'gcc9-fixes/v1' of https://github.com/pdziepak/scylla:
tests/imr: add missing noexcept
counters: bytes_view::pointer is not const pointer
imr/fundamental: use bytes_view::const_pointer for const pointer
In libstdc++ for gcc9 std::basic_string_view::pointer isn't const any
more. As a result the compiler is complaining about reinterpret_cast
casting away const. The solution is to use std::conditional<> to choose
between const pointer for counter view and non-const pointer for mutable
counter view.
In libstdc++ shipped with gcc9 std::basic_string_view::pointer is no
longer constant, which is causing the compiler to complain about
dropping const in reinterpret_cast. The solution is to use
std::basic_string_view::const_pointer.
"
Fix the broken logic that is meant to prevent sending hints when node is
in a DOWN NORMAL state.
"
* 'hinted_handoff_stop_sending_to_down_node-v2' of https://github.com/vladzcloudius/scylla:
hints_manager: rename the state::ep_state_is_not_normal enum value
hinted handoff: fix the logic that detects that the destination node is in DN state
hinted_handoff: sender::can_send(): optimize gossiper::is_alive(ep) check
hinted handoff: end_point_hints_manager::sender: use _gossiper instead of _shard_manager.local_gossiper()
types.cc: fix the compilation with fmt v5.3.0
Before ede1d248af, running "tools/toolchain/dbuild -it -- bash" was
a nice way to play in the toolchain environment, for example to start
a debugger. But that commit caused containers to run in detached mode,
which is incompatible with interactive mode.
To restore the old behavior, detect that the user wants interactive mode,
and run the container in non-detached mode instead. Add the --rm flag
so the container is removed after execution (as it was before ede1d248af).
Message-Id: <20190506175942.27361-1-avi@scylladb.com>
In 0ea6df, duration_test was made to link against all tests/*.o files.
This isn't necessary, as it only needs tests/exception_utils.o. This
patch narrows down duration_test's dependences to only
exception_utils.
Signed-off-by: Dejan Mircevski <dejan@scylladb.com>
Message-Id: <20190508211630.108228-1-dejan@scylladb.com>
Add new test cases:
- disallow creating a non-FULL index on frozen collections
- disallow repeated creation of a FULL index on frozen collections
- disallow FULL indexes on non-frozen collections
- disallow referencing frozen-map entries in the WHERE clause
Also add error-message expectations to existing test cases.
Fixes#3654.
Tests: unit (dev)
Signed-off-by: Dejan Mircevski <dejan@scylladb.com>
Message-Id: <20190509025806.124499-1-dejan@scylladb.com>
Since other build_*.sh are for running inside extracted relocatable
package, they have SCYLLA-PRODUCT-FILE on top of the directory,
but build_ami.sh is not running in such condition, we need to run
SCYLLA-VERSION-GEN first, then refer to build/SCYLLA-PRODUCT-FILE.
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20190509110621.27468-1-syuu@scylladb.com>
Rename this state value to better reflect the reality:
state::ep_state_is_not_normal -> state::ep_state_left_the_ring
The manager gets to this state when the destination Node has left the ring.
Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
When node is in a DN state its gossiper state may be NORMAL, SHUTDOWN
or "" depending on the use case.
In addition to that if node has been removed from the ring its state is
also going to be removed from the gossiper_state map.
Let's consider the above when deciding if node is in the DN state.
Fixes#4461
Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
AWS just released their new instances, the i3en instances. The instance
is verified already to work well with scylla, the only adjustments that
we need is advertise that we support it, and pre-fill the disk
information according to the performance numbers obtained by running the
instance.
Fixes#4486
Branches: 3.1
Signed-off-by: Glauber Costa <glauber@scylladb.com>
Message-Id: <20190508170831.6003-1-glauber@scylladb.com>
"
Cassandra has supported GROUP BY in SELECT statements since 2016
(v3.10), while ScyllaDB currently treats it as a syntax error. To
achieve parity with Cassandra in this important bit of functionality,
this patch adds full support for GROUP BY, from parsing to validation
to implementation to testing.
"
* 'groupby-implPP' of https://github.com/dekimir/scylla:
Implement grouping in selection processing
Propagate GROUP BY indices to result_set_builder
Process GROUP BY columns into select_statement
Parse GROUP BY clause, store column identifiers
Make result_set_builder obey its _group_by_cell_indices by recognizing
group boundaries and resetting the selectors.
Also make simple_selectors work correctly when grouping.
Fixes#2206.
Signed-off-by: Dejan Mircevski <dejan@scylladb.com>
Ensure that the indices recorded in select_statement are passed to
result_set_builder when one is created for processing the cell values.
Signed-off-by: Dejan Mircevski <dejan@scylladb.com>
