The stream_manager will bookkeep the streaming bandwidth option, to
subscribe on its changes it needs the config reference. It would be
better if it was stream_manager::config, but currently subscription on
db::config::<stuff> updates is not very shard-friendly, so we need to
carry the config reference itself around.
Similar trouble is there for compaction_manager. The option is passed
through its own config, but the config is created on each shard by
database code. Stream manager config would be created once by main code
on shard 0.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
It's going to control the bandwidth for the streaming prio class.
For now it's jsut added but does't work for real
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Current code accepts priotity class as an argument to various functions
that need it and all its callers use streaming class. Next patches will
needs to sometimes use default class, but it will require heavy patching
of the distributed loader. Things get simpler if the priority class is
kept on sstable_directory on start.
This change also simplifies the ongoing effort on unification of sched
and IO classes.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
We want to print the backwards fiber in reverse, starting with the
furthest-away task in the chain. For this, the task list returned by
`_walk()` has to be reversed.
Closes#11062
This work gets us a step closer to compaction groups.
Everything in compaction layer but compaction_manager was converted to table_state.
After this work, we can start implementing compaction groups, as each group will be represented by its own table_state. User-triggered operations that span the entire table, not only a group, can be done by calling the manager operation on behalf of each group and then merging the results, if any.
Closes#11028
* github.com:scylladb/scylla:
compaction: remove forward declaration of replica::table
compaction_manager: make add() and remove() switch to table_state
compaction_manager: make run_custom_job() switch to table_state
compaction_manager: major: switch to table_state
compaction_manager: scrub: switch to table_state
compaction_manager: upgrade: switch to table_state
compaction: table_state: add get_sstables_manager()
compaction_manager: cleanup: switch to table_state
compaction_manager: offstrategy: switch to table_state()
compaction_manager: rewrite_sstables(): switch to table_state
compaction_manager: make run_with_compaction_disabled() switch to table_state
compaction_manager: compaction_reenabler: switch to table_state
compaction_manager: make submit(T) switch to table_state
compaction_manager: task: switch to table_state
compaction: table_state: Add is_auto_compaction_disabled_by_user()
compaction: table_state: Add on_compaction_completion()
compaction: table_state: Add make_sstable()
compaction_manager: make can_proceed switch to table_state
compaction_manager: make stop compaction procedures switch to table_state
compaction_manager: make get_compactions() switch to table_state
compaction_manager: change task::update_history() to use table_state instead
compaction_manager: make can_register_compaction() switch to table_state
compaction_manager: make get_candidates() switch to table_state
compaction_manager: make propagate_replacement() switch to table_state
compaction: Move table::in_strategy_sstables() and switch to table_state
compaction: table_state: Add maintenance sstable set
compaction_manager: make has_table_ongoing_compaction() switch to table_state
compaction_manager: make compaction_disabled() switch to table_state
compaction_manager: switch to table_state for mapping of compaction_state
compaction_manager: move task ctor into source
Fix https://github.com/scylladb/scylla-docs/issues/4040
Fix https://github.com/scylladb/scylla-docs/issues/4128
This PR adds the upgrade guides from ScyllaDB Enterprise 2021.1 to 2022.1. They are based on the previous guides.
Closes#11036
* github.com:scylladb/scylla:
doc: add the description of the new metrics in 2022.1
doc: remove the upgrade guide for Ubuntu 16.04 (no longer supported in version 2022.1)
doc: remove the outdated warning
Update docs/upgrade/_common/upgrade-guide-from-2021.1-to-2022.1-ubuntu-and-debian.rst
Update docs/upgrade/_common/upgrade_to_2022_warning.rst
doc: add a space on line 60 to fix the warning
doc: document metric update for 2022.1
doc: add the upgrade guide from 2021.1 to 2022.1
This series adds removal of dropped table directory when it has no remaining snapshots.
There are 2 code paths that take of that:
1. when the table is dropped and there are no active snapshots for it (typically when auto_snapshot disabled).
2. or when the last snapshot is cleared, leaving no other snapshot for a dropped table.
Unit tests were extended to covert these scenarios.
Fixes#10896Closes#11001
* github.com:scylladb/scylla:
legacy_schema_migrator: simplify drop_legacy_tables
database: clear_snapshot: remove dropped table directory when it has no remaining snapshots
database: clear_snapshot: make it a coroutine and use thread
database_test: add clear_multiple_snapshots test
database: make drop_column_family private
schema_tables: merge_tables_and_views: use drop_table_on_all_shards
database_test: drop_table_with_snapshots: test auto_snapshot
database_test: populate_from_quarantine_works: pass optional db:config to do_with_some_data
database: drop_table_on_all_shards: remove table directory having no snapshots
sstables: define table_subdirectories
sstables: officially define pending_delete_dir
database: add drop_table_on_all_shards
There is no need for utils::make_joinpoint now
that the function calls replica::database::drop_table_on_all_shards.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
and use an async thread around `directory_lister`
rather than `lister::scan_dir` to simplify the implementation.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Based on the `clear_snapshot` test.
Test with multiple snapshots and different
combinations of parameters to database::clear_snapshot.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
So that the dropped table's directory can be
removed after it has been dropped on all shards
if it has no snapshots.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Refactor test_drop_table_with_auto_snapshot out of
drop_table_with_snapshots, adding a auto_snapshot param,
controlling how to configure the cql_test_env db:.config::auto_snapshot,
so we can test both cases - auto_snapshot enabled and disabled.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Instead of just `tmpdir_for_data`, so we can easily set auto_snapshot
for `drop_table_with_snapshots` in the next patch.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
If the table to remove has no snapshots then
completely remove its directory on storage
as the left-over directory slows down operations on the keyspace
and makes searching for live tables harder.
Fixes#10896
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Rather than using the "pending_delete" string
in `pending_delete_dir_basename()`, so it can
be orderly removed in the next patch.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Runs drop_column_family on all database shards.
Will be extended later to consider removing the table directory.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
compaction_manager.cc still cannot stop including replica/database.hh
because upgrade and scrub still take replica::database as param,
but I'll remove it soon in another series.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
rewrite_sstables() is used by maintenance compactions that perform
an operation on a single file at a time.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Now that submit() switched to table_state, compaction_reenabler
and friends can switch to table_state too.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
auto_compaction_disabled_by_user is a configuration that can be enabled
or disabled on a particular table. We're adding this interface to
avoid having to push the configuration for every compaction_state,
which would result in redundant information as the configuration
value is the same for all table states.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
The idea is that we'll have a single on-completion interface for both
"in-strategy" and off-strategy compactions, so not to pollute table_state
with one interface for each.
replica::table::on_compaction_completion is being moved into private namespace.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
compaction_manager needs this interface when setting the sstable
creation lambda in compaction_descriptor, which is then forwarded
into the actual compaction procedure.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
they're used to stop all ongoing compaction on behalf of a given table
T. Today, each table has a single table_state representing it, but after
we implement compaction groups, we'll need to call the procedure for
each group in a table. But the discussion doesn't belong here, as
compaction group work will only come later. By the time being, we're
only making compaction manager fully switch to table_state.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
The only external user of get_compactions() doesn't use any filtering,
so after table_state switch, one will be allowed to get all jobs
running associated with a table_state.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
propagate_replacement is used by incremental compaction to notify
ongoing compaction about sstable list updates, such that the
ongoing job won't hold reference to exhausted sstables.
So it needs to switch to table_state, too.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
in_strategy_sstables() doesn't have to be implemented in table, as it's
simply about main set with maintenance and staging files filtered out.
Also, let's make it switch to table_state as part of ongoing work.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
