Instead of retrying, just drop mutations that raced with a truncate.
* git@github.com:duarten/scylla.git truncate-reorder/v1:
database: Rename replay_position_reordered_exception
database: Drop mutations that raced with truncate
(cherry picked from commit 63caa58b70)
"This patchset restricts background writers - such as compactions,
streaming flushes and memtable flushes to a maximum amount of CPU usage
through a seastar::thread_scheduling_group.
The said maximum is recommended to be set 50 % - it is default
disabled, but can be adjusted through a configuration option until we
are able to auto-tune this.
The second patch in this series provides a preview on how such auto-tune
would look like. By implementing a simple controller we automatically
adjust the quota for the memtable writer processes, so that the rate at
which bytes come in is equal to the rates at which bytes are flushed.
Tail latencies are greatly reduced by this series, and heavy spikes that
previously appeared on CPU-bound workloads are no more."
* 'memtable-controller-v5' of https://github.com/glommer/scylla:
simple controller for memtable/streaming writer shares.
restrict background writers to 50 % of CPU.
(cherry picked from commit c5ee62a6a4)
"Fixes schema layout incompatibility in a mixed 1.7 and 2.0 cluster (#2555)
by reverting back to using the old layout in memory and thus also
in across-node requests. We still use the new v3 layout in schema
tables (needed by drivers and external tools). Translations happen
when converting to/from schema mutations."
* tag 'tgrabiec/use-v2-schema-layout-in-memory-v2' of github.com:scylladb/seastar-dev:
schema: Revert back to the 1.7 layout of static compact tables in memory
schema: Use v3 column layout when converting to/from schema mutations
schema: Encapsulate column layout translations in the v3_columns class
(cherry picked from commit 1daf1bc4bb)
Some places remained where code looked directly at
system_keyspace::NAME to determine iff a ks is
considered special/system/protected. Including
schema digest calculation.
Export "is_system_keyspace" and use accordingly.
Message-Id: <1500469809-23546-1-git-send-email-calle@scylladb.com>
(cherry picked from commit 247c36e048)
Instead of copying and moving the bound, pass it by reference so the
transformer can decide whether it wants to copy or not. The only
caller so far doesn't want a copy and takes the value by reference,
which would be capturing a temporary value. Caught by the
view_schema_test with gcc7.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <20170705210255.29669-1-duarte@scylladb.com>
(cherry picked from commit 3dd0397700)
It is useful for larger cluster with larger gossip message latency. By
default the fd_max_interval_ms is 2 seconds which means the
failure_detector will ignore any gossip message update interval larger
than 2 seconds. However, in larger cluster, the gossip message udpate
interval can be larger than 2 seconds.
Fixes#2603.
Message-Id: <49b387955fbf439e49f22e109723d3a19d11a1b9.1500278434.git.asias@scylladb.com>
(cherry picked from commit adc5f0bd21)
branch 'tgrabiec/schema-migration-fixes' of github.com:scylladb/seastar-dev:
schema: Use proper name comparator
legacy_schema_migrator: Properly migrate non-UTF8 named columns
schema_tables: Store column_name in text form
legacy_schema_migrator: Migrate columns like Cassandra
schema_builder: Add factory method for default_names
legacy_schema_migrator: Simplify logic
thrift: Don't set regular_column_name_type
schema: Use proper column name type for static columns
schema: Fix column_name_type() for static compact tables
schema: Introduce clustering_column_at()
thrift: Reuse cell_comparator::to_sstring() for obtaining comparator type
partition_slice_builder: Use proper column's type instead of regular_column_name_type()
(cherry picked from commit 13caccf1cf)
Reduces view_schema_test runtime to 5 seconds, from 53 seconds on an NVMe disk
with write-back cache, and forever on a spinning disk.
Message-Id: <20170716081653.10018-1-avi@scylladb.com>
(cherry picked from commit d9c64ef737)
We will be creating links to those sstable's files, and those don't work
if the data directory and the test sstable are on different devices.
Copying the files to the same directory fixes the problem.
Message-Id: <20170716090405.14307-1-avi@scylladb.com>
(cherry picked from commit 9116dd91cb)
We don't ensure mutations are applied in memory following the order of their
replay positions. A memtable can thus be flushed with replay position rp,
with the new one being at replay position rp', where rp' < rp. This breaks
an intrinsic assumption in the code, which this series addresses.
Fixes#2074
branch memtable-flush/v3 of git@github.com:duarten/scylla.git:
commitlog: Always flush latest memtable
column_family: More precise count of switched memtables
column_family: Fix typo in pending_tasks metric name
column_family: More precise count of pending flushes
dirty_memory_manager: Remove unnecessary check from flush_one()
column_family: Don't rely on flush_queue to guarantee flushes finished
column_family: Don't bother closing the flush_queue on stop()
column_family: Stop using flush_queue
column_family: Remove outdated comment about the flush_queue
memtable: Stop tracking the highest flushed rp
(cherry picked from commit caa62f7f05)
In storage_proxy we arrange the mutations sent by the replicas in a
vector of vectors, such that each row corresponds to a partition key
and each column contains the mutation, possibly empty, as sent by a
particular replica.
There is reconciliation-related code that assumes that all the
mutations sent by a particular replica can be found in a single
column, but that isn't guaranteed by the way we initially arrange the
mutations.
This patch fixes this and enforces the expected order.
Fixes#2531Fixes#2593
Signed-off-by: Gleb Natapov <gleb@scylladb.com>
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <20170713162014.15343-1-duarte@scylladb.com>
(cherry picked from commit b8235f2e88)
Current algorithm was marking tables with regular columns not named
"value" as not dense, which doesn't have to be the case. It can be
either way.
It should be enough to look at clustering components. If there is a
clustering key, then table is dense if and only if all comparator
components belong to the clustering key.
If there is no clustering key, then if there are any regular columns
we're sure it's not dense.
Fixes#2587.
Message-Id: <1499877777-7083-1-git-send-email-tgrabiec@scylladb.com>
(cherry picked from commit 30ec4af949)
The default of 2ms is somewhat arbitrary. Now that we have a lot more
mileage deploying Scylla applications in production it does sound not
only arbitrary, but high.
In particular, it is really hard to achieve 1ms latencies in the face of
CPU-heavy workloads with it.
Signed-off-by: Glauber Costa <glauber@scylladb.com>
Message-Id: <1499354495-27173-1-git-send-email-glauber@scylladb.com>
(cherry picked from commit 780a6e4d2e)
"Currently new nodes calculate digests based on v3 schema mutations,
which are very different from v2 mutations. As a result they will
use schemas with different table_schema_version that the old nodes.
The old nodes will not recognize the version and will try to request
its definition. That will fail, because old nodes don't understand
v3 schema mutations.
To fix this problem, let's preserve the digests during migration,
so that they're the same on new and old nodes. This will allow
requests to proceed as usual.
This does not solve the problem of schema being changed during
the rolling upgrade. This is not allowed, as it would bring the
same problem back.
Fixes #2549."
* tag 'tgrabiec/use-consistent-schema-table-digests-v2' of github.com:cloudius-systems/seastar-dev:
tests: Add test for concurrent column addition
legacy_schema_migrator: Set digest to one compatible with the old nodes
schema_tables: Persist table_schema_version
schema_tables: Introduce system_schema.scylla_tables
schema_tables: Simplify read_table_mutations()
schema_tables: Resurrect v2 read_table_mutations()
system_keyspace: Forward-declare legacy schemas
legacy_schema_migrator: Take storage_proxy as dependency
(cherry picked from commit a397889c81)
DowngradingConsistencyRetryPolicy uses live replicas count from
Unavailable exception to adjust CL for retry, but when there are pending
nodes CL is increased internally by a coordinator and that may prevent
retried query from succeeding. Adjust live replica count in case of
pending node presence so that retried query will be able to proceed.
Fixes#2535
Message-Id: <20170710085238.GY2324@scylladb.com>
(cherry picked from commit 739dd878e3)
Use name of the existing preceeding column with restriction
(last_column) instead of assuming that the column right after the
current column already has restrictions.
This will yield an error message that is different from that of
Cassandra, albeit still a correct one.
Fixes#2421
Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Message-Id: <40335768a2c8bd6c911b881c27e9ea55745c442e.1499781685.git.bdenes@scylladb.com>
(cherry picked from commit 33bc62a9cf)
CQL reply may contain metadata that describes columns present in the
response including the information about their type.
However, Scylla incorrectly reports counter types as bigint. The
serialised format of counters and bigint is exactly the same, which
could explain why the problem hasn't been noticed earlier but it is a
bug nevertheless.
Fixes#2569.
Message-Id: <20170711130520.27603-1-pdziepak@scylladb.com>
(cherry picked from commit 5aa523aaf9)
Otherwise we may deadlock, as explained in commit 5e8f0efc8:
Table drop starts with creating a snapshot on all shards. All shards
must use the same snapshot timestamp which, among other things, is
part of the snapshot name. The timestamp is generated using supplied
timestamp generating function (joinpoint object). The joinpoint object
will wait for all shards to arrive and then generate and return the
timestamp.
However, we drop tables in parallel, using the same joinpoint
instance. So joinpoint may be contacted by snapshotting shards of
tables A and B concurrently, generating timestamp t1 for some shards
of table A and some shards of table B. Later the remaining shards of
table A will get a different timestamp. As a result, different shards
may use different snapshot names for the same table. The snapshot
creation will never complete because the sealing fiber waits for all
shards to signal it, on the same name.
Message-Id: <1499762663-21967-1-git-send-email-tgrabiec@scylladb.com>
(cherry picked from commit 310d2a54d2)
`r` is moved-from, and later captured in a different lambda. The compiler may
choose to move and perform the other capture later, resulting in a use-after-free.
Fix by copying `r` instead of moving it.
Discovered by sstable_test in debug mode.
Message-Id: <20170702082546.20570-1-avi@scylladb.com>
(cherry picked from commit 07b8adce0e)
Configuring cpufreq service on VMs/IaaS causes an error because it doesn't supported cpufreq.
To prevent causing error, skip whole configuration when the driver not loaded.
Fixes#2051
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1498809504-27029-1-git-send-email-syuu@scylladb.com>
(cherry picked from commit 1c35549932)
Repair today has a semaphore limiting the number of ongoing checksum
comparisons running in parallel (on one shard) to 100. We needed this
number to be fairly high, because a "checksum comparison" can involve
high latency operations - namely, sending an RPC request to another node
in a remote DC and waiting for it to calculate a checksum there, and while
waiting for a response we need to proceed calculating checksums in parallel.
But as a consequence, in the current code, we can end up with as many as
100 fibers all at the same stage of reading partitions to checksum from
sstables. This requires tons of memory, to hold at least 128K of buffer
(even more with read-ahead) for each of these fibers, plus partition data
for each. But doing 100 reads in parallel is pointless - one (or very few)
should be enough.
So this patch adds another semaphore to limit the number of checksum
*calculations* (including the read and checksum calculation) on each shard
to just 2. There may still be 100 ongoing checksum *comparisons*, in
other stages of the comparisons (sending the checksum requests to other
and waiting for them to return), but only 2 will ever be in the stage of
reading from disk and checksumming them.
The limit of 2 checksum calculations (per shard) applies on the repair
slave, not just to the master: The slave may receive many checksum
requests in parallel, but will only actually work on 2 at a time.
Because the parallelism=100 now rate-limits operations which use very little
memory, in the future we can safely increase it even more, to support
situations where the disk is very fast but the link between nodes has
very high latency.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20170703151329.25716-1-nyh@scylladb.com>
(cherry picked from commit d177ec05cb)
"Old and new nodes will advertise different schema version because
of different format of schema tables. This will result in attempts
to sync the schema by each of the node.
Currently this will result in scary error messages in logs about
sync failing due to not being able to find schema of given version.
It's benign, but may scare users. It the future incompatibilities
could result in more subtle errors. Better to inhibit it completely."
* 'tgrabiec/fix-schema-pull-errors-during-upgrade' of github.com:cloudius-systems/seastar-dev:
migration_manager: Give empty response to schema pulls from incompatible nodes
migration_manager: Don't pull schema from incompatible nodes
service: Advertise schema tables format version through gossip
(cherry picked from commit 91221e020b)
So that they are not left on disk even though we did a clean shutdown.
First part of the fix is to ensure that closed segments are recognized
as not allocating (_closed flag). Not doing this prevents them from
being collected by discard_unused_segments(). Second part is to
actually call discard_unused_segments() on shutdown after all segments
were shut down, so that those whose position are cleared can be
removed.
Fixes#2550.
Message-Id: <1499358825-17855-1-git-send-email-tgrabiec@scylladb.com>
(cherry picked from commit 6555a2f50b)
* tag 'tgrabiec/row-cache-metrics-v2' of github.com:cloudius-systems/seastar-dev:
row_cache: Switch _stats.hits/misses to row granularity
row_cache: Rename num_entries() to partitions() for clarity
row_cache: Track mispopulations also at row level
row_cache: Track row insertions
row_cache: Track row hits and misses
row_cache: Make mispopulation counter also apply for continuity information
row_cache: Add partition_ prefix to current counters
misc_services: Switch to using reads_with[_no]_misses counters
row_cache: Add metrics for operations on underlying reader
row_cache: Add reader-related metrics
row_cache: Remove dead code
(cherry picked from commit b1a0e37fcb)
"This series introduces selective_token_range_sharder and uses it in repair to
generate dht::token_range belongs to a specific shard."
* tag 'asias/repair-selective_token_range_sharder-v3' of github.com:cloudius-systems/seastar-dev:
repair: Use selective_token_range_sharder
tests: Add test_selective_token_range_sharder
dht: Add selective_token_range_sharder
(cherry picked from commit 66e56511d6)
The test put a wrapping range into a non-wrapping range variable.
This was harmless at the time this test was written, but newer code
may not be as forgiving so better use a non-wrapping range as intended.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20170704103128.29689-1-nyh@scylladb.com>
(cherry picked from commit d95f908586)
when do_for_each is in its last iteration and with_semaphore defers
because there's an ongoing cleanup, sstable object will be used after
freed because it was taken by ref and the container it lives in was
destroyed prematurely.
Let's fix it with a do_with, also making code nicer.
Fixes#2537.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <20170630035324.19881-1-raphaelsc@scylladb.com>
(cherry picked from commit b9d0645199)
Boot should not continue until a future returned by
wait_for_gossip_to_settle() is resolved. Commit 991ec4a16 mistakenly
broke that, so restore it back. Also fix calls for supervisor::notify()
to be in the right places.
Message-Id: <20170702082355.GQ14563@scylladb.com>
(cherry picked from commit d23111312f)
