This commit makes database, sstables and tests aware
of which large_partition_handler they use.
Proper large_partition_handler is retrievable from config information
and is based on existing compaction_large_partition_warning_threshold_mb
entry. Right now CQL TABLE variant of large_partition_handler is used
in the database.
Tests use a NOP version of large_partition_handler, which does not
depend on CQL queries at all.
Old versions of JsonCpp declare the following typedefs for internally
used aliases:
typedef long long int Int64;
typedef unsigned long long int UInt64;
In newer versions (1.8.x), those are declared as:
typedef int64_t Int64;
typedef uint64_t UInt64;
Those base types are not identical so in cases when a type has
constructors overloaded only for specific integral types (such as
Json::Value in JsonCpp or data_value in Scylla), an attempt to
pack/unpack an integer from/to a JSON object causes ambiguous calls.
Fixes#3208
Tests: unit {release}.
Signed-off-by: Vladimir Krivopalov <vladimir@scylladb.com>
Message-Id: <e9fff9f41e0f34b15afc90b5439be03e4295623e.1524556258.git.vladimir@scylladb.com>
"
This patchset is a part of a bigger effort for bringing our
microbenchmarking tests from the source tree to be used for regression
testing purposes with CI.
Now, it is possible to export results of tests run into JSON format that
can be stored in ElasticSearch and compared among runs to detect
performance degradation should it happen.
Example of JSON output (formatted for readability):
{
"results" :
{
"parameters" :
{
"read" : "64",
"read,skip,test_run_count" : "64,256,1",
"skip" : "256",
"test_run_count" : 1
},
"stats" :
{
"(KiB)" : 126960,
"aio" : 993,
"blocked" : 208,
"c blk" : 1,
"c hit" : 0,
"c miss" : 1,
"cpu" : 99.779365539550781,
"dropped" : 0,
"frag/s" : 311939.61559016741,
"frags" : 200000,
"idx blk" : 0,
"idx hit" : 0,
"idx miss" : 0,
"time (s)" : 0.641149729
}
},
"test_group_properties" :
{
"message" : "Testing scanning large partition with skips.\nReads whole range interleaving reads with skips according to read-skip pattern",
"name" : "large-partition-skips",
"needs_cache" : false,
"partition_type" : "large"
},
"versions" :
{
"scylla-server" :
{
"commit_id" : "4acfa17f4",
"date" : "20180306",
"run_date_time" : "2018-16-06 12:16:41",
"version" : "666.development"
}
}
}
"
* 'issues/2947/v6' of https://github.com/argenet/scylla:
Add support for JSON output format for perf_fast_forward results.
Wrap output for customization. Move all output handling to a single managing class.
The JSON output is arranged in a way that makes it easier to upload
results to ElasticSearch.
All the tests results are placed under the perf_forward_data_output/ directory
For test groups, we create separate subdirectories where we save results
from runs of tests in those groups.
For each test run, we store results in a separate file named:
<dash-separated-param-list>.<run-number>.json
where
<dash-separated-param-list> is a dash-separated list of parameters of the current
test, e.g., 1-64 (for read-skip pattern).
<run-number> is the number of run of this test with the specified
parameters. This is needed as the same list of parameters can be
used more than once (for instance, when cache is enabled).
Those numbers start with 1, i.e., 1, 2, 3.
So, the path to a resulting JSON file may look like:
perf_fast_forward_output/large-partition-skips/64-4096.1.json
Signed-off-by: Vladimir Krivopalov <vladimir@scylladb.com>
Instead of passing the output parameters to std::cout straight away, use
helper wrappers. This will allow us to add more formats for gathered
tests results.
Introduce helper writer classes hierarchy that can be extended to
support different output formats (JSON, XML, etc).
Signed-off-by: Vladimir Krivopalov <vladimir@scylladb.com>
Container indices are size_t, and in other places we gratuituously
declare a limit as unsigned and the loop index as signed.
Tests: unit (release)
Message-Id: <20180212121642.10525-1-avi@scylladb.com>
Recently, memtable flush in test requires storage service for tests,
or it fails with "Assertion `local_is_initialized()' failed".
storage_service_for_tests needs to run in a thread, that's why
flush_memtable was flattened.
Last but not least, we need to revert flushed memory account because
same memtable is used for all sstables in the perf test so as not
to trigger `_mt._flushed_memory <= _mt.occupancy().used_space()'
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <20171205012853.21559-1-raphaelsc@scylladb.com>
This will make migration to flat_mutation_reader much
easier and sstables::mutation_reader is going away with
this migration anyway.
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
Before the patch we appended and queried at the front. Insert at the
front instead, so that writes and reads overlap. Stresses eviction and
population more.
Message-Id: <1506369562-14892-1-git-send-email-tgrabiec@scylladb.com>
query::full_slice doesn't select any regular or static columns, which
is at odds with the expectations of its users. This patch replaces it
with the schema::full_slice() version.
Refs #2885
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <1507732800-9448-2-git-send-email-duarte@scylladb.com>
