Restrict readers based on their memory consumption, instead of the count
of the top-level readers. To do this an interposer is installed at the
input_stream level which tracks buffers emmited by the stream. This way
we can have an accurate picture of the readers' actual memory
consumption.
New readers will consume 16k units from the semaphore up-front. This is
to account their own memory-consumption, apart from the buffers they
will allocate. Creating the reader will be deferred to when there are
enough resources to create it. As before only new readers will be
blocked on an exhausted semaphore, existing readers can continue to
work.
Restrict readers based on their memory consumption, instead of the count
of the top-level readers. To do this an interposer is installed at the
input_stream level which tracks buffers emmited by the stream. This way
we can have an accurate picture of the readers' actual memory
consumption.
New readers will consume 16k units from the semaphore up-front. This is
to account their own memory-consumption, apart from the buffers they
will allocate. Creating the reader will be deferred to when there are
enough resources to create it. As before only new readers will be
blocked on an exhausted semaphore, existing readers can continue to
work.
Forward-declare untyped_result_set and untyped_result_set_row, and remove
the include from query_processor.hh.
Message-Id: <20170916170859.27612-3-avi@scylladb.com>
We apply deletion of scylla_tables.version to the incoming schema
mutations so that table schema version is recalculated after merge.
The mutations which we read from local schema tables may not have it
deleted in which case all tables would be considered as differing on
the presence of the version field. Avoid this by deleting the field
from old mutations as well.
* seastar 0083ee8...85ca12d (1):
> Merge "Run-time logging configuration" from Jesse
Includes patch from Jesse:
"Switch to Seastar for logging option handling
In addition to updating the abstraction layer for Seastar logging in `log.hh`,
the configuration system (`db/config.{hh,cc}`) has been updated in two ways:
- The string-map type for Boost.program_options is now defined in Seastar.
- A configuration value can be marked as `UsedFromSeastar`. This is like `Used`,
except the option is expected to be defined in the Boost.Program_options
description for Seastar. If the option is not defined in Seastar, or it is
defined with a different type, then a run-time exception is thrown early in
Scylla's initialization. This is necessary because logging options which are
now defined in Seastar were previously defined in Scylla and support for these
options in the YAML file cannot be dropped. In order to be able to verify that
options marked `UsedFromSeastar` are actually defined in Seastar, the
interface for adding options to `db::config` has changed from taking a
`boost::program_options::options_description_easy_init` (which is handle into
a `boost::program_options::options_description` which only allows adding
options) to taking a `boost::program_options::options_description`
directly (which also allows querying existing options).
Scylla also fully defers to Seastar's support for run-time logging
configuration."
Signed-off-by: Jesse Haber-Kucharsky <jhaberku@scylladb.com>
Message-Id: <ef26cffb91bef1ae95d508187a6dd861a6c4fc84.1503344007.git.jhaberku@scylladb.com>
The number of keysapce and column family metrics reported is
proportional to the number of shards times the number of keysapce/column
families.
This can cause a performance issue both on the reporting system and on
the collecting system.
This patch adds a configuration flag (set to false by default) to enable
or disable those metrics.
Fixes#2701
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
Message-Id: <20170821113843.1036-1-amnon@scylladb.com>
Some (most?) users don't read logs or release notes, so they won't notice
that the ByteOrdered and Random partitioners were deprecated in 2.0. Make
them notice by refusing to start with a deprecated partitioner, unless a
switch is explicitly enabled.
Message-Id: <20170820073424.8331-1-avi@scylladb.com>
Large allocations can require cache evictions to be satisfied, and can
therefore induce long latencies. Enable the seastar large allocation
warning so we can hunt them down and fix them.
Message-Id: <20170819135212.25230-1-avi@scylladb.com>
* Configuration for cluster_name is commented-out in config file.
* Default value set to empty string and if not rewritten by user then
warning is printed and value is reset to "ScyllaDB Cluster".
Fixes#2648.
Message-Id: <20170808113322.9313-1-daniel@scylladb.com>
* seastar f14d2a3...7a49ae5 (8):
> sharded: improve support for cooperating sharded<> services
> sharded: support for peer services
> semaphore: add a version of with_semaphore that takes a duration timeout
> scripts: perftune.py: fix the CPU mask generation for more than 64 CPUs
> Revert "future-utils: make when_all() (vector variant) exception safe"
> Revert "future-utils: fix gross compilation errors in when_all()"
> future-utils: fix gross compilation errors in when_all()
> future-utils: make when_all() (vector variant) exception safe
Includes change to batchlog_manager constructor to adapt it to
seastar::sharded::start() change.
"This series adds an option to use paging in internal query and use that for the
get compaction history function.
Internal paging will be done explicitly, to use paging, you first create a
state object (that contains the query as well) and use that state to get the
first page, the result will contain both the query result and a new state that
can be used to get the next page.
Fixes#2366"
* 'amnon/paged_compaction_history_v5' of github.com:cloudius-systems/seastar-dev:
system_keyspace: Use paging for get compaction history
Add paging for internal queries
query_options: Allows creating query_options from query_options
It specifies the maximum gossip shadow round time. It can be used to
reduce the gossip feature check time during node boot up.
For instance, when the first node in the cluster, which listed both
itself and other node as seed in the yaml config, boots up, it will try
to talk to other seed nodes which are not started yet. The gossip shadow
round will be used to fetch the feature info of the cluster. Since there
is no other seed node in the cluster, the shadow round will fail. User
can reduce the default shadow_round_ms option to reduce the boot time.
Fixes#2615
Message-Id: <10916ce9059f3c7f1a1fb465919ae57de3b67d59.1500540297.git.asias@scylladb.com>
If a node is notified of a schema change where the schema's dropped
columns have changes, that node will miss the changes to the dropped
columns. A scenario where this can happen is where a column c is
dropped, then added as a different typed, and then dropped again, with
a node n having seen the first drop and being notified of the
subsequent add and drop.
Fixes#2616
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <20170725170622.4380-1-duarte@scylladb.com>
"Fixes issues uncovered in longevity test (#2608).
Main problem is that due to time drift scylla_tables.version column
may not get deleted on all nodes doing the schema merge, which will
make some nodes come up with different table schema version than others.
The inconsistency will not heal because scylla_tables doesn't
take part in the schema sync. This is fixed by the last patch.
This will cause nodes to constantly try to sync the schema, which under
some conditions triggers #2617."
* tag 'tgrabiec/fix-table-schema-version-inconsistency-v1' of github.com:scylladb/seastar-dev:
schema_tables: Add scylla_tables to ALL
schema: Make schema_mutations equality consistent with digest
schema_tables: Extract compact_for_schema_digest()
schema_tables: Always drop scylla_tables::version
there could be a lot of compactions when querying for compaction
history.
This patch changes the query to use paging. It would collect all results
when returning to the caller.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
So that scylla_tables takes part in the digest and in mutations sent
as part of schema sync. Otherwise inconsistencies in scylla_tables
will not heal.
Refs #2608.
It can happen that due to time drift between nodes, the incoming
"version" cell will have higher timestamp than api::new_timestamp().
In such case the column would not be dropped and would cause version
mismatch between nodes.
Ensure it's always covered by using max of current time and cell's
timestamp.
Refs #2608.
"This patchset restricts background writers - such as compactions,
streaming flushes and memtable flushes to a maximum amount of CPU usage
through a seastar::thread_scheduling_group.
The said maximum is recommended to be set 50 % - it is default
disabled, but can be adjusted through a configuration option until we
are able to auto-tune this.
The second patch in this series provides a preview on how such auto-tune
would look like. By implementing a simple controller we automatically
adjust the quota for the memtable writer processes, so that the rate at
which bytes come in is equal to the rates at which bytes are flushed.
Tail latencies are greatly reduced by this series, and heavy spikes that
previously appeared on CPU-bound workloads are no more."
* 'memtable-controller-v5' of https://github.com/glommer/scylla:
simple controller for memtable/streaming writer shares.
restrict background writers to 50 % of CPU.
Some places remained where code looked directly at
system_keyspace::NAME to determine iff a ks is
considered special/system/protected. Including
schema digest calculation.
Export "is_system_keyspace" and use accordingly.
Message-Id: <1500469809-23546-1-git-send-email-calle@scylladb.com>
"Fixes schema layout incompatibility in a mixed 1.7 and 2.0 cluster (#2555)
by reverting back to using the old layout in memory and thus also
in across-node requests. We still use the new v3 layout in schema
tables (needed by drivers and external tools). Translations happen
when converting to/from schema mutations."
* tag 'tgrabiec/use-v2-schema-layout-in-memory-v2' of github.com:scylladb/seastar-dev:
schema: Revert back to the 1.7 layout of static compact tables in memory
schema: Use v3 column layout when converting to/from schema mutations
schema: Encapsulate column layout translations in the v3_columns class
We are using C* 3.x compatible layout in schema tables but want to
keep using the 1.7 layout in memory for compatibility during rolling
upgrade. This patch switches the schema and schema_builder classes
back to the old layout. Translation of layout happens when converting
to/from schema mutations.
Notable changes:
1) Includes a revert of commit 6260f31e08
"thrift: Update CQL mapping of static CFs".
2) Brings back the "default_validation_class" schema attribute. In v3
it can be dervied from column definitions, but in v2 it can't, so
we have to store it.
3) legacy_schema_migrator and schema_builder don't have to do
conversions to v3, this is now handled by the v3_columns
class. schema_builder works with the same layout as schema, that
is v2.
4) Includes a revert of commit 66991a7ccb
"v3 schema test fixes"
Fixes#2555.
This patch introduces a simple controller that will adjust memtables CPU
shares, trying to keep it around the soft limit: if we start going below
it means we're too fast (unless we are idle) and shares are adjusted
downwards. If we start going above it means we're too fast and shares
are adjusted upwards.
I have tested this extensively in a single-CPU setup with various
CPU-bound workloads while tracking virtual dirty and the results are
good, with virtual dirty fluctuating only slightly, somewhere within the
desired range.
Exceptions to this are:
1) when the load is very light - the idle system goes faster, and that's
ok
2) when the load is very high - as foreground requests dominate we can't
flush fast enough and hit the hard limit. However, in such scenarios
the memtable shares do hit its maximum, and the results are no worse
than they are right now and this will only be fixed by CPU-limiting the
actual requests.
This feature can be disabled with a config option - that is scheduled to
go away as we acquire more confidence in this. When the feature is
disabled, all background writers (streaming, compaction, memtables) will
share the same scheduling group, with static quotas.
Signed-off-by: Glauber Costa <glauber@scylladb.com>
