The get_sstables_by_partition_key method used by the API to return a set of
sstables names that holds a given partition key.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
This patch adds a test to the has_partition_key method, it creates an
sstable with a partition key and then used that key in the
has_partition_key method to verify that it is there.
It creates a different key and use that to verify that a non exist key
return false.
Based on:
8daaf9833a
This patch adds a from_nodetool_style_string factory method to partition_key.
The string format is follows the nodetool format, that column in the
partition keys are split by ':'.
For example, if a partition key has two column col1 and col2, to get the
partition key that has col1 = val1 and col2 = val2:
val1:val2
"
SSTable 3.0 format introduces serialization header which is used in reading SSTables in that format.
This patchset implements loading of this new component of Statistics.db.
Tests: units (release)
"
* 'haaawk/sstables3/load_serialization_header_v2' of ssh://github.com/scylladb/seastar-dev:
Load serialization_header from statistics
Add parse for disk_array_vint_size
Add helpers to read/parse vints
Add signed_vint::serialized_size_from_first_byte
Add sstable::get_serialization_header
Move random_access_reader to separate header
250ms is too high of a period for memtable controller. Since memtable
flushes are relatively efficient, specially in comparison to
compactions, if the shares are high we can flush a lot of data down with
the high shares - so in the next adjustment period our shares will be
minuscule and we won't flush much at all.
This leads to oscillating behavior that is mitigated by adjusting
faster.
Signed-off-by: Glauber Costa <glauber@scylladb.com>
Message-Id: <20180508182746.28310-3-glauber@scylladb.com>
"Fixes a bug in partition_snapshot::merge_partition_versions(), which would not
attempt merging if the snapshot is attached to the latest version (in which
case _version is nullptr and _entry is != nullptr). This would cause
partition_version objects to accumulate if there was an older snapshot and it
went away before the latest snapshot. Versions will be removed when the whole
entry goes away (flush or eviction).
May cause performance problems.
Fixes #3402."
* 'tgrabiec/fix-merge_partition_versions' of github.com:tgrabiec/scylla:
mvcc: Test version merging when snapshots go away
anchorless_list: Make ranges conform to SinglePassRange
anchorless_list: Drop deprecated use of std::iterator
mvcc: Fix partition_snapshot::merge_partition_versions() to not leave latest versions unmerged
The newer version of iotune, recently merged to Seastar, accepts
a new parameter that tells us where should we store the properties
about the disk.
We are already generating that properties file for the AMI case.
Let's also pass that parameter when calling iotune.
Signed-off-by: Glauber Costa <glauber@scylladb.com>
Message-Id: <20180507175757.9144-1-glauber@scylladb.com>
Starting from 2018.1 and 2.2 there was a change in the repository path.
It was made to support multiple product (like manager and place the
enterprise in a different path).
As a result, the regular expression that look for the repository fail.
This patch change the way the path is searched, both rpm and debian
varations are combined and both options of the repository path are
unified.
See scylladb/scylla-enterprise#527
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
Message-Id: <20180429151926.20431-1-amnon@scylladb.com>
"
In SSTables 3.0, the base and increment fields have been swapped in
Bloom filters to reduce collisions (see CASSANDRA-8413). This affects
the resulting values written to Filter.db.
This patchset adds support for reading/writing Filter.db in the format
corresponding to the version of SSTables.
Tests: unit {release}
Filter.db files have been generated using Cassandra 3.11 with same data
as in unit tests and are validated to match those generated by Scylla.
"
* 'projects/sstables-30/write-filter/v1-2' of https://github.com/argenet/scylla:
Fix mistakes and typos in comments (minor clean-up)
Check Filter.db in SSTables 3.x write tests.
Support Bloom filter format used in SSTables 3.0.
Remove unused overload of i_filter::get_filter().
The two hash values, base and increment, used to produce indices for
setting bits in the filter, have been swapped in SSTables 3.0.
See CASSANDRA-8413 for details.
Signed-off-by: Vladimir Krivopalov <vladimir@scylladb.com>
Fixes crash in cql_tests.StorageProxyCQLTester.table_test
"avoid race condition when deleting sstable on behalf..." changed
discard_sstables behaviour to only return rp:s for sstables owned
and submitted for deletion (not all matching time stamp),
which can in some cases cause zero rp returned.
Message-Id: <20180508070003.1110-1-calle@scylladb.com>
When node is decommissioned/removed it will drain all its hints and all
remote nodes that have hints to it will drain their hints to this node.
What "drain" means? - The node that "drains" hints to a specific
destination will ignore failures and will continue sending hints till the end
of the current segment, erase it and move to the next one till there are
no more segments left.
After all hints are drained the corresponding hints directory is removed.
Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
Returning a future with an exception from end_point_manager::stop()
is practically useless because the best the caller can do is to log
it and continue as if it didn't happen because it has other things
to shut down.
Therefore in order to simplify the caller we will log the exception
if it happens and will always return a non-exceptional future.
Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
This ensures we respect the write timeout set by the client when
applying base writes, in case a writes takes too long to acquire the
row lock for the read-before-write phase of a materialized view
update.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <20180507132755.8751-1-duarte@scylladb.com>
* seastar ac02df7...840002c (20):
> dpdk: protect against missing statistics
> alien: make visible in documentation
> Merge "rewrite iotune to conform to the new ioscheduler" from Glauber
> app_template: Correct outdated comment
> apps, tests: Catch polymorphic exceptions by reference
> configure.py: Enhance detection for gcc -fvisibility=hidden bug
> reactor: add rudimentary task histogram reporting
> Revert "Merge "rewrite iotune to conform to the new ioscheduler" from Glauber"
> Merge "rewrite iotune to conform to the new ioscheduler" from Glauber
> build: Use the same warning name for Clang and GCC
> core/rwlock: Add support for timeouts
> fs qualification: protect against EINTR
> Docker: Fix failing build due to missing GNU make
> reactor: move optional to experimental so we compile with c++14
> future: remove allocation from future::get() thread context switch
> Merge "rpc streaming" from Gleb
> reactor: put mountpoint_params in seastar namespace
> Tutorial: in PDF version of tutorial, better backtick typesetting
> tutorial: support, and start using, links to other sections
> tutorial: improve second half of semaphores section
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
This patch adds a simple and naive mechanism to ensure a base replica
doesn't overwhelm a potentially overloaded view replica by sending too
many concurrent view updates. We add a semaphore to limit to 100 the
number of outstanding view updates. We limit globally per shard, and
not per destination view replica. We also limit statically.
Refs #2538
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <20180426134457.21290-2-duarte@scylladb.com>
The functions in cql_assertions.hh are very convenient, but have one
frustrating drawback: When you have many of those assertions in one
test, it's very hard to know *which* of the similar assertions failed.
The problem is that an error often looks like this:
unknown location(0): fatal error: in "test_many_columns":
std::runtime_error: Expected 2 row(s) but got 0
tests/cql_assertions.cc(131): last checkpoint
Which of the many similar checks in "test_many_columns" failed? Note the
unhelpful "unknown location" and also the "last checkpoint" points to code
in cql_assertions.cc, not in the actual test, so it is useless.
The root cause of these problems is that the Boost macros use the C
preprocessor __FILE__ and __LINE__, which in actual C++ functions like
is_rows() remembers its location, instead of the caller. Fixing this will
not be simple. But this patch has a much simpler solution - fixing the
"last checkpoint". What ruins the last checkpoint is the use of BOOST_REQUIRE
inside the cql_assertions.cc is_rows() - when that succeeds, it records
the location inside cql_assertions.cc (!) as the last success.
If we just replace BOOST_REQUIRE by our own test (just like in the rest of
the cql_assertions.cc code), this code will not override the last checkpoint.
The user can see the last real successful BOOST_REQUIRE, or use
BOOST_TEST_PASSPOINT() to set his own checkpoints between different parts of
the same test.
After this patch, and with adding BOOST_TEST_PASSPOINT() calls between
different parts of my test, the failure above now looks like:
unknown location(0): fatal error: in "test_many_columns":
std::runtime_error: Expected 2 row(s) but got 0
tests/secondary_index_test.cc(299): last checkpoint
The "last checkpoint" now shows me exactly where my failing check was.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20180501152638.26238-1-nyh@scylladb.com>
"This patchset adds ostream operators to result_message and uses them
in cql_assertions."
* tag 'result_message-print/v1.1' of https://github.com/avikivity/scylla:
tests: cql_assersions: improve error message when a row is not found
transport: add ostream support to result_message
transport: const correctness for result_message::accept()
Use utf8_type where warranted.
Fixes view_schema_test failure where the rows did not match. I don't
understand exactly why the failure happened (using the wrong type
should not cause a failure here), but the change fixes the problem.
Tests: view_schema_test (release)
Message-Id: <20180506130015.7450-1-avi@scylladb.com>
The visitor does not alter the result_message it is visiting (and
its signature indicates that) so accept() should be const-qualified
to indicate that and to allow visiting const result_message:s.
"
This patchset adds support for writing Statistics.db in the SSTables
'mc' (3.x) format. This file is essential for reading data stored in
Data.db as it contains base values used for delta encoding and types of
columns.
This patchset also fixes several bugs found in writing data and index
files as well as bugs in a statistics-related structure definition.
Tests: unit {debug, release}
All SSTables files for write unit tests are validated to be processed by
sstabledump and output is verified to show the expected data.
"
* 'projects/sstables-30/write-statistics/v1' of https://github.com/argenet/scylla:
Add test covering the composite partition key case.
Add Statistics.db files to write tests for SSTables 3.0.
Do not check rows and cells for expiration when writing them to the data file.
Fix promoted index serialization.
Fix the order of items in stats_metadata.
Fix timestamp_epoch value which was truncated on exceeding int32_t type limit.
Write serialization header to Statistics.db for SSTables 3.x.
Do not pass schema to metadata_collector::update(column_stats)
Collect metadata statistics when writing SSTables 3.0.
Call get_metadata_collector() instead of referencing sstable::_collector directly.
Fix logic of writing TTLed cells in SSTable 3.0 format.
Separate statistics for count of cells, columns and rows in column_stats.
Deserialize collection in a way that doesn't incur shared_ptr counter increment and is generally shorter.
Track both min & max values for timestamp, TTL and local deletion time in metadata_collector.
Add class for tracking both extremum values (min and max) on updates.
Mainly to check that the composite type is properly serialized when
writing serialization header to Statistics.db.
Signed-off-by: Vladimir Krivopalov <vladimir@scylladb.com>
For these tests to work, all time-related values are now fixed as these
are stored in Statistics.db files.
Signed-off-by: Vladimir Krivopalov <vladimir@scylladb.com>
Although this logic may be seen as a useful optimization, it hinders
unit tests writing SSTables 3.0 as those need to have fixed time-related
values to produce Statistics.db files with the same content on each run.
Signed-off-by: Vladimir Krivopalov <vladimir@scylladb.com>
There is a new field introduced in the SSTables 3.0 index file format
named 'partition_header_length' that can be used to skip over to the
first clustering row in a wide partition. This one has not been
previously written and caused malformed indices.
Updated the corresponding test to include a static row and write
multiple wide partitions.
Signed-off-by: Vladimir Krivopalov <vladimir@scylladb.com>
