Online discard asks the disk to erase flash memory cells as soon
as files are deleted. This gives the disk more freedom to choose
where to place new files, so it improves performance.
On older kernel versions, and on really bad disks, this can reduce
performance so we add an option to disable it.
Since fstrim is pointless when online discard is enabled, we
don't configure it if online discard is selected.
I tested it on an AWS i3.large instance, the flag showd up in
`mount` after configuration.
Closes#9608
On scylla_unit.py, we provide `systemd_unit.is_active()` to return `systemctl is-active` output.
When we introduced systemd_unit class, we just returned `systemctl is-active` output as string, but we changed the return value to bool after that (2545d7fd43).
This was because `if unit.is_active():` always becomes True even it returns "failed" or "inactive", to avoid such scripting bug.
However, probably this was mistake.
Because systemd unit state is not 2 state, like "start" / "stop", there are many state.
And we already using multiple unit state ("activating", "failed", "inactive", "active") in our Cloud image login prompt:
https://github.com/scylladb/scylla-machine-image/blob/next/common/scylla_login#L135
After we merged 2545d7fd43, the login prompt is broken, because it does not return string as script expected (https://github.com/scylladb/scylla-machine-image/issues/241).
I think we should revert 2545d7fd43, it should return exactly same value as `systemctl is-active` says.
Fixes#9627Fixesscylladb/scylla-machine-image#241Closes#9628
* github.com:scylladb/scylla:
scylla_ntp_setup: use string in systemd_unit.is_active()
Revert "scylla_util.py: return bool value on systemd_unit.is_active()"
Commit 3f56c49a9e put redis into protocol_servers list of storage
service. Since then there's no need in explicit stop_server call
on shutdown -- the protocol_servers thing will do it.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Message-Id: <20211109154259.1196-1-xemul@scylladb.com>
It is bad form to introduce branches just for statistics, since branches
can be expensive (even when perfectly predictable, they consume branch
history resources). Switch to simple addition instead; this should be
not cause any cache misses since we already touch other statistics
earlier.
The inputs are already boolean, but cast them to boolean just so it
is clear we're adding 0/1, not a count.
Closes#9626
Change b0a2a9771f broke
the generic api implementation of
is_native_transport_running that relied on
the addresses list being empty agter the server is stopped.
To fix that, this change introduces a pure virtual method:
protocol_server::is_server_running that can be implemented
by each derived class.
Test: unit(dev)
DTest: nodetool_additional_test.py:TestNodetool.binary_test
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Message-Id: <20211114135248.588798-1-bhalevy@scylladb.com>
dnf has replaced yum on Fedora and CentOS. On modern versions of Fedora,
you have to install an extra package to get the old name working, so
avoid that inconvenience and use dnf directly.
Closes#9622
Add "rows" field to system.large_partitions. Add partitions to the
table when they are too large or have too many rows.
Fixes#9506
Signed-off-by: Michael Livshin <michael.livshin@scylladb.com>
Closes#9577
* seastar a189cdc45...04c6787b3 (12):
> Convert std::result_of to std::invoke_result
> Merge "IO queue full-duplex mode" from Pavel E
> Merge "Report bytes/ops for R and W separately" from Pavel E
> websocket: override std::exception::what() correctly
> tests: websocket_test: remove unused lambda capture
> Merge "Improve IO classes preemption" from Pavel E
> Revert "Merge "Improve IO classes preemption" from Pavel E"
> Merge "Add skeleton implementation of a WebSocket server" from Piotr S
> Merge "Improve IO classes preemption" from Pavel E
> io_queue: Add starvation time metrics (per-class)
> Revert "Merge "Add skeleton implementation of a WebSocket server" from Piotr S"
> Merge "Add skeleton implementation of a WebSocket server" from Piotr S
No functional changes, but makes the code shorter and gets rid
of a few allocations.
Coroutinizing has_column_family_access is deliberately skipped and
commented, since some callers expect this function to throw instead
of returning an exceptional future.
Message-Id: <958848a1eeeef490b162d2d2b805c8a14fc9082b.1636704996.git.sarna@scylladb.com>
Fresh seastar has duplex IO queues (and some more goodies). The
former one needs respective changes in scylla-gdb.py
* xemul/br-gdb-duplex-ioqueues:
scylla-gdb: Support new fair_{queue|group}s layout
scylla-gdb: Add boost::container::small_vector wrapper
scylla-gdb: Fix indentation aft^w before next patch
In the recent seastar io_queues carry several fair_queues on board,
so do the io_groups. The queues are in boost small_vector, the groups
are in a vector of unique_ptrs. This patch adds this knowledge to
scylla-gdb script.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
The upcoming seastar update will have fair_groups and fair_queues to
become arrays. Thus scylla-gdb will need to iterate over them with
some sort of loop. This patch makes the queue/group prining indentation
to match this future loop body and prepares the loop variables while
at it.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
"
Scylla can be configured via a bunch of config files plus
a bunch of commandline options. Collecting these altogether
can be challenging.
The proposed table solves a big portion of this by dupming
the db::config contents as a table. For convenience (and,
maybe, to facilitate Benny's CLI) it's possible to update
the 'value' column of the table with CQL request.
There exists a PR with a table that exports loglevels in a
form of a table. The updating technique used in this set
is applicable to that table as well.
tests: compilation(dev, release, debug), unit(debug)
"
* 'br-db-config-virtual-table-3' of https://github.com/xemul/scylla:
tests: Unit test for system.config virtual table
system_keyspace: Table with config options
code: Push db::config down to virtual tables
storage_proxy: Propagate virtual table exceptions messages
table: Virtual writer hook (mutation applier)
table: Rewrap table::apply()
table: Mark virtual reader branch with unlikely
utils: Add config_src::source_name() method
utils: Ability to set_value(sstring) for an option
utils: Internal change of config option
utils: Mark some config_file methods noexcept
When building a docker we relay on `VERSION` value from
`SCYLLA-VERSION-GEN` . For `rc` releases only there is a different
between the configured version (X.X.rcX) and the actualy debian package
we generate (X.X~rcX)
Using a similar solution as i did in dcb10374a5Fixes: #9616Closes#9617
A config option value is reported as 'text' type and contains
a string as it would looks like in json config.
The table is UPDATE-able. Only the 'value' columnt can be set
and the value accepted must be string. It will be converted into
the option type automatically, however in current implementation
is't not 100% precise -- conversion is lexicographical cast which
only works for simple types. However, liveupdate-able values are
only of those types, so it works in supported cases.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
The db::config reference is available on the database, which
can be get from the virtual_table itself. The problem is that
it's a const refernece, while system.config will be updateable
and will need non-const reference.
Adding non-const get_config() on the database looks wrong. The
database shouldn't be used as config provider, even the const
one.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
The intention is to return some meaningful info to the CQL caller
if a virtual table update fails. Unfortunately the "generic" error
reporting in CQL is not extremely flexible, so the best option
seems to report regular write failre with custom message in it.
For now this only works for virtual table errors.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Symmetrically to virtual reader one, add the virtual writer
callback on a table that will be in charge of applying the
provided mutation.
If a virtual table doesn't override this apply method the
dedicated exception is thrown. Next patch will catch it and
propagate back to caller, so it's a new exception type, not
existing/std one.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
The main motivation is to have future returning apply (to be used
by next patches). As a side effect -- indentation fix and private
dirty_memory_region_group() method.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
There soon will appear an updateable system.config table that
will push sstrings into names_value-s. Prepare for this change
by adding the respective .set_value() call. Since the update
only works for LiveUpdate-able options, and inability to do it
can be propagated back to the caller make this method return
true/false whether the update took place or not.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
When a named_value is .set_value()-d the caller may specify the reason
for this change. If not specified it's set to None, but None means
"it was there by default and was't changed" so it's a bit of a lie.
Add an explicit Internal reason. It's actually used by the directories
thing that update all directories according to --workdir option.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
"
Draining the database is now scattered across the do_drain()
method of the storage_service. Also it tells shutdown drain
from API drain.
This set packs this logic into the database::drain() method.
tests: unit(dev), start-stop-drain(dev)
"
* 'br-database-drain' of https://github.com/xemul/scylla:
database, storage_service: Pack database::drain() method
storage_service: Shuffle drain sequence
storage_service, database: Move flush-on-drain code
storage_service: Remove bool from do_drain
The series fixes issues:
server may use the wrong configuration after applying a remote snapshot, causing a split-brain situation
assertion ins raft::server_impl::notify_waiters()
snapshot transfer to a server removed from the configuration should be aborted
cluster may become stuck when a follower takes a snapshot after an accepted entry that the leader didn't learn about
* scylla-dev/random-test-fixes-v2:
raft: rename rpc_configuration to configuration in fsm output
raft: test: test case for the issue #9552
raft: fix matching of a snapshotted log on a follower
raft: abort snapshot transfer to a server that was removed from the configuration
raft: fix race between snapshot application and committing of new entries
raft: test: add test for correct last configuration index calculation during snapshot application
raft: do not maintain _last_conf_idx and _prev_conf_idx past snapshot index
raft: correctly truncate the log in a persistence module during snapshot application
clang evaluates function arguments from left to right, while gcc does so
in reverse. Therefore, this code can be correct on clang and incorrect
on gcc:
```
f(x.sth(), std::move(x))
```
This patch fixes one such instance of this bug, in memtable.cc.
Fixes#9605.
Closes#9606
"
table_state is being introduced for compaction subsystem, to remove table dependency
from compaction interface, fix layer violations, and also make unit testing
easier as table_state is an abstraction that can be implemented even with no
actual table backing it.
In this series, compaction strategy interfaces are switching to table_state,
and eventually, we'll make compact_sstables() switch to it too. The idea is
that no compaction code will directly reference a table object, but only work
with the abstraction instead. So compaction subdirectory can stop
including database.hh altogether, which is a great step forward.
"
* 'table_state_v5' of https://github.com/raphaelsc/scylla:
sstable_compaction_test: switch to table_state
compaction: stop including database.hh for compaction_strategy
compaction: switch to table_state in estimated_pending_compactions()
compaction: switch to table_state in compaction_strategy::get_major_compaction_job()
compaction: switch to table_state in compaction_strategy::get_sstables_for_compaction()
DTCS: reduce table dependency for task estimation
LCS: reduce table dependency for task estimation
table: Implement table_state
compaction: make table param of get_fully_expired_sstables() const
compaction_manager: make table param of has_table_ongoing_compaction() const
Introduce table_state
Move run_with_compaction_disabled() into compaction manager
run_with_compaction_disabled() living in table is a layer violation as the
logic of disabling compaction for a table T clearly belongs to manager
and table shouldn't be aware of such implementation details.
This makes things less error prone too as there's no longer a need for
coordination between table and manager.
Manager now takes all the responsibility.
* 'move_disable_compaction_to_manager/v6' of https://github.com/raphaelsc/scylla:
compaction: move run_with_compaction_disabled() from table into compaction_manager
compaction_manager: switch to coroutine in compaction_manager::remove()
compaction_manager: add struct for per table compaction state
compaction_manager: wire stop_ongoing_compactions() into remove()
compaction_manager: introduce stop_ongoing_compactions() for a table
compaction_manager: prevent compaction from being postponed when stopping tasks
compaction_manager: extract "stop tasks" from stop_ongoing_compactions() into new function
The storage_service::do_drain() now ends up with shutting down
compaction manager, flushing CFs and shutting down commitlog.
All three belong to the database and deserve being packed into
a single database::drain() method.
A note -- these steps are cross-shard synchronized, but database
already has a barrier for that.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Right now the draining sequence is
- stop transport (protocol servers, gossiper, streaming)
- shutdown tracing
- shutdown compaction manager
- flush CFs
- drain batchlog manager
- stop migration manager
- shutdown commitlog
This violates the layering -- both batchlog and migration managers
are higher-level services than the database, so they should be
shutdown/drained before it, i.e. -- before shutting down compaction
manager and flushing all CFs.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Flushing all CFs on shutdown is now fully managed in storage service
and it looks weird. Some better place for it seems to be the database
itself.
Moving the flushing code also imples moving the drain_progress thing
and patching the relevant API call.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
The do_drain() today tells shutdown drain from API drain. The reason
is that compaction manager subscribes on the main's abort signal and
drains itself early. Thus, on regular drain it needs this extra kick
that would crash if called from shutdown drain.
This differentiation should sit in the compaction manager itself.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Last method in compaction_strategy using table. From now on,
compaction strategy no longer works directly with table.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
From now on, get_major_compaction_job() will use table_state instead of
a plain reference to table.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
From now on, get_sstables_for_compaction() will use table_state.
With table_state, we avoid layer violations like strategy using
manager and also makes testing easier.
Compaction unit tests were temporarily disabled to avoid a giant
commit which is hard to parse.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
