"
Make sure all headers compile on their own, without requiring any
additional includes externally.
Even though this requirement is not documented in our coding guides it
is still quasi enforced and we semi-regularly get and merge patches
adding missing includes to headers.
This patch-set fixes all headers and adds a `{mode}-headers` target that
can be used to verify each header. This target should be built by
promotion to ensure no new non-conforming code sneaks in.
Individual headers can be verified using the
`build/dev/path/to/header.hh.o` target, that is generated for every
header.
The majority of the headers was just missing `seastarx.hh`. I think we
should just include this via a compiler flag to remove the noise from
our code (in a followup).
"
* 'compiling-headers/v2' of https://github.com/denesb/scylla:
configure.py: add {mode}-headers phony target
treewide: add missing headers and/or forward declarations
test/boost/sstable_test.hh: move generic stuff to test/lib/sstable_utils.hh
sstables: size_tiered_backlog_tracker: move methods out-of-line
sstables: date_tiered_compaction_strategy.hh: move methods out-of-line
* seastar 3c498abcab...92c488706c (14):
> dpdk: restore including reactor.hh
> tests: distributed_test: add missing #include <mutex>
> reactor: un-static-ify make_pollfn()
> merge: Reduce inclusions of reactor.hh
A few #includes added to compensate for this
> sharded: delete move constructor
> future: Avoid a move constructor call
> future: Erase types a bit more in then_wrapped
> memory: Drop a never nullopt optional
> semaphore: specify get_units and with_semaphore as noexcept
> spinlock.hh: Add include for <cassert> header
> dpdk: Avoid a variable sized array
> future: Add an explicit promise member to continuation
> net: remove smart pointer wrappers around pollable_fd
> Merge "cleanup reactor file functions" from Benny
Closes#3295
The error_injection class allows injecting custom handlers into normal control
flow at the pre-determined injection points.
This is especially useful in various testing scenarios:
* Throwing an exception at some rare and extreme corner-cases
* Injecting a delay to test for timeouts to be handled correctly
* More advanced uses with custom lambda as an injection handler
Injection points are defined by `inject` calls.
Enabling and disabling injections are done by the corresponding
`enable` and `disable` calls.
REST frontend APIs is provided for convenience.
Branch URL: https://github.com/alecco/scylla/tree/as_error_injection
Tests: unit {{dev}}, unit {{debug}}
* 'as_error_injection' of github.com:alecco/scylla:
api: add error injection to REST API
utils: add error injection
Error injection class is implemented in order to allow injecting
various errors (exceptions, stalls, etc.) in code for testing
purposes.
Error injection is enabled via compile flag
SCYLLA_ENABLE_ERROR_INJECTION
TODO: manage shard instances
Enable error injection in debug/dev/sanitize modes.
Unit tests for error injection class.
Closes#3295
Signed-off-by: Pavel Solodovnikov <pa.solodovnikov@scylladb.com>
Signed-off-by: Alejo Sanchez <alejo.sanchez@scylladb.com>
Reclaim consults the _regions vector, so we don't want it moving around while
allocating more capacity. For that we take the reclaim lock. However, that
can cause a false-positive OOM during startup:
1. all memory is allocated to LSA as part of priming (2baa16b371)
2. the _regions vector is resized from 64k to 128k, requiring a segment
to be freed (plenty are free)
3. but reclaiming_lock is taken, so we cannot reclaim anything.
To fix, resize the _regions vector outside the lock.
Fixes#6003.
Message-Id: <20200311091217.1112081-1-avi@scylladb.com>
Fixes#5902 by making the LIKE restriction keep a vector
of matchers and apply them all to the column value.
Tests: unit (dev)
* dekimir/multiple-likes:
cql3: Allow repeated LIKE on same column
cql3: Forbid calling LIKE::values()
cql3: Move LIKE::_last_pattern to matcher
Merged patch series from Avi Kivity:
boost/multiprecision is a heavyweight library, pulling in 20,000 lines of code into
each header that depends on it. It is used by converting_mutation_partition_applier
and types.hh. While the former is easy to put out-of-line, the latter is not.
All we really need is to forward-declare boost::multiprecision::cpp_int, but that
is not easy - it is a template taking several parameters, among which are non-type
template parameters also defined in that header. So it's quite difficult to
disentangle, and fragile wrt boost changes.
This patchset introduces a wrapper type utils::multiprecision_int which _can_
be forward declared, and together with a few other small fixes, manages to
uninclude boost/multiprecision from most of the source files. The total reduction
in number of lines compiled over a full build is 324 * 23,227 or around 7.5
million.
Tests: unit (dev)
Ref #1https://github.com/avikivity/scylla uninclude-boost-multiprecision/v1
Avi Kivity (5):
converting_mutation_partition_applier: move to .cc file
utils: introduce multiprecision_int
tests: cdc_test: explicitly convert from cdc::operation to uint8_t
treewide: use utils::multiprecision_int for varint implementation
types: forward-declare multiprecision_int
configure.py | 2 +
concrete_types.hh | 2 +-
converting_mutation_partition_applier.hh | 163 ++-------------
types.hh | 12 +-
utils/big_decimal.hh | 3 +-
utils/multiprecision_int.hh | 256 +++++++++++++++++++++++
converting_mutation_partition_applier.cc | 188 +++++++++++++++++
cql3/functions/aggregate_fcts.cc | 10 +-
cql3/functions/castas_fcts.cc | 28 +--
cql3/type_json.cc | 2 +-
lua.cc | 38 ++--
mutation_partition_view.cc | 2 +
test/boost/cdc_test.cc | 6 +-
test/boost/cql_query_test.cc | 16 +-
test/boost/json_cql_query_test.cc | 12 +-
test/boost/types_test.cc | 58 ++---
test/boost/user_function_test.cc | 2 +-
test/lib/random_schema.cc | 14 +-
types.cc | 20 +-
utils/big_decimal.cc | 4 +-
utils/multiprecision_int.cc | 37 ++++
21 files changed, 627 insertions(+), 248 deletions(-)
create mode 100644 utils/multiprecision_int.hh
create mode 100644 converting_mutation_partition_applier.cc
create mode 100644 utils/multiprecision_int.cc
The goal is to forward-declare utils::multiprecision_int, something
beyond my capabilities for boost::multiprecision::cpp_int, to reduce
compile time bloat.
The patch is mostly search-and-replace, with a few casts added to
disambiguate conversions the compiler had trouble with.
multiprecision_int is a wrapper around boost::multiprecision::cpp_int that adds
no functionality. The intent is to allow forward declration; cpp_int is so
complicated that just finding out what its true type is a difficult exercise, as
it depends on many internal declarations.
Because cpp_int uses expression templates, the implementation has to explicitly
cast to the desired type in many places, otherwise the C++ compile is presented
with too many choices, especially in conjunction with data_value (which can
convert from many different types too).
"
Timeouts defaulted to `db::no_timeout` are dangerous. They allow any
modifications to the code to drop timeouts and introduce a source of
unbounded request queue to the system.
This series removes the last such default timeouts from the code. No
problems were found, only test code had to be updated.
tests: unit(dev)
"
* 'no-default-timeouts/v1' of https://github.com/denesb/scylla:
database: database::query*(), database::apply*(): remove default timeouts
database: table::query(): remove default timeout
mutation_query: data_query(): remove default timeout
mutation_query: mutation_query(): remove default timeout
multishard_mutation_query: query_mutations_on_all_shards(): remove default timeout
reader_concurrency_semaphore: wait_admission(): remove default timeout
utils/logallog: run_when_memory_available(): remove default timeout
"
As part of avoiding static initialization order problems I want to
switch a few global sstring to constexpr std::string_view. The
advantage being that a constexpr variable doesn't need runtime
initialization and therefore cannot be part of a static initialization
order problem.
In order to do the conversion I needed to convert a few APIs to use
std::string_view instead of sstring and const sstring&.
These patches are the simple cases that are also an improvement in
their own right.
"
* 'espindola/string_view' of https://github.com/espindola/scylla: (22 commits)
test: Pass a string_view to create_table's callback
Pass string_view to the schema constructor
cql3: Pass string_view to the column_specification constructor
Pass string_view to keyspace_metadata::new_keyspace
Pass string_view to the keyspace_metadata constructor
utils: Use std::string as keys in nonstatic_class_registry
utils: Pass a string_view to class_registry::to_qualified_class_name
auth: Return a string_view from authorizer::qualified_java_name
auth: Return a string_view from authenticator::qualified_java_name
utils: Pass string_view to is_class_name_qualified
test: Pass a string_view to create_keyspace
Pass string_view to no_such_column_family's constructor
perf_simple_query: Pass a string_view to make_counter_schema
Pass string_view to the schema_builder constructor
types: Add more data_value constructors
transport: Pass a string_view to cql_server::connection::make_autheticate
transport: Pass a string_view to cql_server::response::write_string
cql3: Pass std::string_view to query_processor::compute_id
cql3: Remove unused variable
cql3: Pass a string_view to cf_statement::prepare_keyspace
...
The sstring::compare functions was never updated to work with
std::string_view. We could fix that, but it seems better to just
switch to std::string.
With a working compare function we can avoid copying the argument
passed to to_qualified_class_name when an entry is found in the map.
Signed-off-by: Rafael Ávila de Espíndola <espindola@scylladb.com>
No reason to disallow this. We still forbid mixing LIKE and non-LIKE
relations on the same column.
Fixes#5902.
Signed-off-by: Dejan Mircevski <dejan@scylladb.com>
Instead of keeping the LIKE pattern in a restriction object (as we
currently do), keep it in like_matcher. Also move the
pattern-idempotence check from the restriction to the matcher.
Signed-off-by: Dejan Mircevski <dejan@scylladb.com>
boost/regex has huge header dependencies amounting to tens of thousands of
lines. This are now replicated in 167 translation units.
This patch converts like_matcher to use the pointer-to-implementation
idiom, which reduces the number of translations including boost/regex
to 28.
Since regular expressions are relatively expensive, and like_matcher is
relatively rare, the extra memory usage and run time will be
negligible.
Message-Id: <20200211170152.809554-1-avi@scylladb.com>
The disk-error-handler is purely auxiliary thing that helps
propagating IO errors to the rest of the code. It well
deserves not sitting in the root namespace.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Message-Id: <20200207112443.18475-1-xemul@scylladb.com>
gcc 10 tightened its C++ includes to no longer provide ssize_t,
so we must get it from a C header instead.
Message-Id: <20200129205912.21139-1-avi@scylladb.com>
"
Throw an error in case we hit an invalid time UUID
rather than hitting an assert.
Fixes#5552
(Ref #5588 that was dequeued and fixed here)
Test: UUID_test, cql_query_test(debug)
"
* 'validate-time-uuid' of https://github.com/bhalevy/scylla:
cql3: abstract_function_selector: provide assignment_testable_source_context
test: cql_query_test: add time uuid validation tests
cql3: time_uuid_fcts: validate timestamp arg
cql3: make_max_timeuuid_fct: delete outdated FIXME comment
cql3: time_uuid_fcts: validate time UUID
test: UUID_test: add tests for time uuid
utils: UUID: create_time assert nanos_since validity
utils/UUID_gen: make_nanos_since
utils: UUID: assert UUID.is_timestamp
"
This PR makes named_value respect allowed_values and then use it to transition away from old deprecated RandomPartitioner and ByteOrderedPartitioner. Then it removes the code that's no longer used.
We want to remove deprecated partitioners because, on one hand, they lead to performance problems and hot nodes. Moreover, we're planning to unify the token representation which would allow per table partitioner support. That, in turn, is a feature helpful in multiple efforts like CDC, materialized views, secondary indexes and multi-tenancy.
tests: unit(dev)
"
* 'remove_deprecated_partitioners' of https://github.com/haaawk/scylla:
partitioners: remove random_partitioner
partitioners: Make it impossible to use RandomPartitioner
partitioners: remove byte_ordered_partitioner
partitioners: Make it impossible to use ByteOrderedPartitioner
partitioners: Remove leftovers of OrderPreservingPartitioner
i_partitioner.cc: stop including byte_ordered_partitioner.hh
i_partitioner.cc: stop including random_partitioner.hh
config: use allowed_values to verify named_value input
config: add operator<< for seed_provider_type
Safely convert millis to "nanos_since" (number of 100
nanseconds since START_EPOCH) while type casting to uint64_t
to avoid possible int overflow.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Even though we configure the set of accepted values for
some config flags, named_value ignore them.
This patch implements the checks that verify flag is
not set to the value that's not on the list.
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
This templates the code for listener_vector, renames it to
atomic_vector and moves it to the utils directory.
Signed-off-by: Rafael Ávila de Espíndola <espindola@scylladb.com>
"
Most of the code in `cell` and the `imr` infrastructure it is built on
is `noexcept`. This means that extra care must be taken to avoid rouge
exceptions as they will bring down the node. The changes introduced by
0a453e5d3a did just that - introduced rouge `std::bad_alloc` into this
code path by violating an undocumented and unvalidated assumption --
that fragment ranges passed to `cell::make_collection()` are nothrow
copyable and movable.
This series refactors `cell::make_collection()` such that it does not
have this assumption anymore and is safe to use with any range.
Note that the unit test included in this series, that was used to find
all the possible exception sources will not be currently run in any of
our build modes, due to `SEASTAR_ENABLE_ALLOC_FAILURE_INJECTION` not
being set. I plan to address this in a followup because setting this
flags fails other tests using the failure injection mechanism. This is
because these tests are normally run with the failure injection disabled
so failures managed to lurk in without anyone noticing.
Fixes: #5575
Refs: #5341
Tests: unit(dev, debug)
"
* 'data-cell-make-collection-exception-safety/v2' of https://github.com/denesb/scylla:
test: mutation_test: add exception safety test for large collection serialization
data/cell.hh: avoid accidental copies of non-nothrow copiable ranges
utils/fragment_range.hh: introduce fragment_range_view
A lightweight, trivially copyable and movable view for fragment ranges.
Allows for uniform treatment of all kinds of ranges, i.e. treating all
of them as a view. Currently `fragment_range.hh` provides lightweight,
view-like adaptors for empty and single-fragment ranges (`bytes_view`). To
allow code to treat owning multi-fragment ranges the shame way as the
former two, we need a view for the latter as well -- this is
`fragment_range_view`.
"
The original fix (10f6b125c8) didn't
take into account that if there was a failed memtable flush (Refs
flush) but is not a flushable memtable because it's not the latest in
the memtable list. If that happens, it means no other memtable is
flushable as well, cause otherwise it would be picked due to
evictable_occupancy(). Therefore the right action is to not flush
anything in this case.
Suspected to be observed in #4982. I didn't manage to reproduce after
triggering a failed memtable flush.
Fixes#3717
"
* tag 'avoid-ooming-with-flush-continuations-v2' of github.com:tgrabiec/scylla:
database: Avoid OOMing with flush continuations after failed memtable flush
lsa: Introduce operator bool() to occupancy_stats
lsa: Expose region_impl::evictable_occupancy in the region class
Different versions of boost have different rules for what conversions
from cpp_int to smaller intergers are allowed.
We already had a function that worked with all supported versions, but
it was not being use by lua.
Signed-off-by: Rafael Ávila de Espíndola <espindola@scylladb.com>
Message-Id: <20200104041028.215153-1-espindola@scylladb.com>
If any two directories of data/commitlog/hints/view_hints
are the same we still end up running verify_owner_and_mode
and disk_sanity(check_direct_io_support) in parallel
on the same directoriea and hit #5510.
This change uses std::set rather than std::vector to
collect a unique set of directories that need initialization.
Fixes#5510
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Message-Id: <20191225160645.2051184-1-bhalevy@scylladb.com>
The hints and view_hints directory has per-shard sub-dirs,
and the directories code tries to create, check and lock
all of them, including the base one.
The manipulations in question are excessive -- it's enough
to check and lock either the base dir, or all the per-shard
ones, but not everything. Let's take the latter approach for
its simplicity.
Fixes#5510
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Looks-good-to: Benny Halevy <bhalevy@scylladb.com>
Message-Id: <20191223142429.28448-1-xemul@scylladb.com>
The unordered_set is turned into vector since for fs::path
there's no hash() method that's needed for set.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Now the only future-able operation remained is the call to
parallel_for_each(), all the rest is non-blocking preparation,
so we can drop the seastar::async and just return the future
from parallel_for_each.
The indendation is now good, as in previous patch is was prepared
just for that.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
The goal is to drop the seastar::async() usage.
Currently we have two places that return futures -- calls to
parallel_for_each-s. We can either chain them together or,
since both are working on the same set of directories, chain
actions inside them.
For code simplicity I propose to chain actions.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
The list of paths that should be touch-and-locked is already
at hands, this shortens the code and makes it slightly faster
(in theory).
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
In order not to pollute the root dir place the code in
utils/ directory, "utils" namespace.
While doing this -- move the touch_and_lock from the
class declaration.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
The main.cc code that converts sstring to fs::path
will be patched soon, the file_desc::open belongs
to seastar and works on sstrings.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
`segment_manager' now uses a decorated version of `timed_out_error'
with hardcoded name. On the other hand `region_group' uses named
`on_request_expiry' within its `expiring_fifo'.
