Commit Graph

23 Commits

Author SHA1 Message Date
Benny Halevy
3f23016cc0 perf-simple-query: add mean and standard deviation stats
Currently, perf-simple-query summarizes the statistics
only for the throughput, printing the median,
median absolute deviation, minimum, and maximum.
But the throughput put is typically highly variable
and its median is noisy.

This patch calculates also the mean and standard deviation
and does that also for instructions_per_op and cpu_cycles_per_op
to present a fuller picture of the performance metrics.

Output example:
```
random-seed=3383668492
enable-cache=1
Running test with config: {partitions=10000, concurrency=100, mode=read, frontend=cql, query_single_key=no, counters=no}
Disabling auto compaction
Creating 10000 partitions...
95613.97 tps ( 63.1 allocs/op,   0.0 logallocs/op,  14.1 tasks/op,   42456 insns/op,   22117 cycles/op,        0 errors)
97538.45 tps ( 63.1 allocs/op,   0.0 logallocs/op,  14.1 tasks/op,   42454 insns/op,   22094 cycles/op,        0 errors)
95883.37 tps ( 63.1 allocs/op,   0.0 logallocs/op,  14.1 tasks/op,   42438 insns/op,   22268 cycles/op,        0 errors)
96791.45 tps ( 63.1 allocs/op,   0.0 logallocs/op,  14.1 tasks/op,   42433 insns/op,   22256 cycles/op,        0 errors)
97894.71 tps ( 63.1 allocs/op,   0.0 logallocs/op,  14.1 tasks/op,   42420 insns/op,   22010 cycles/op,        0 errors)

         throughput: mean=96744.39 standard-deviation=996.89 median=96791.45 median-absolute-deviation=861.02 maximum=97894.71 minimum=95613.97
instructions_per_op: mean=42440.08 standard-deviation=14.99 median=42437.59 median-absolute-deviation=13.58 maximum=42456.15 minimum=42420.10
  cpu_cycles_per_op: mean=22148.98 standard-deviation=110.43 median=22117.04 median-absolute-deviation=106.89 maximum=22267.70 minimum=22010.42
```

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>

Closes scylladb/scylladb#19450
2024-06-25 12:25:59 +03:00
Benny Halevy
fb3db7d81f perf-simple-query: add cpu_cycles / op metric
Example output:
```
bhalevy@[] scylla$ build/release/scylla perf-simple-query --default-log-level=error -c 1 --duration 10
random-seed=4058714023
enable-cache=1
Running test with config: {partitions=10000, concurrency=100, mode=read, frontend=cql, query_single_key=no, counters=no}
Disabling auto compaction
Creating 10000 partitions...
86912.75 tps ( 63.1 allocs/op,   0.0 logallocs/op,  14.1 tasks/op,   42346 insns/op,   22811 cycles/op,        0 errors)
91348.29 tps ( 63.1 allocs/op,   0.0 logallocs/op,  14.1 tasks/op,   42306 insns/op,   22362 cycles/op,        0 errors)
87965.84 tps ( 63.1 allocs/op,   0.0 logallocs/op,  14.1 tasks/op,   42338 insns/op,   22966 cycles/op,        0 errors)
90793.67 tps ( 63.1 allocs/op,   0.0 logallocs/op,  14.1 tasks/op,   42351 insns/op,   22783 cycles/op,        0 errors)
90104.27 tps ( 63.1 allocs/op,   0.0 logallocs/op,  14.1 tasks/op,   42358 insns/op,   22875 cycles/op,        0 errors)
90397.13 tps ( 63.1 allocs/op,   0.0 logallocs/op,  14.1 tasks/op,   42355 insns/op,   22735 cycles/op,        0 errors)
89142.39 tps ( 63.1 allocs/op,   0.0 logallocs/op,  14.2 tasks/op,   42363 insns/op,   22996 cycles/op,        0 errors)
90410.40 tps ( 63.1 allocs/op,   0.0 logallocs/op,  14.1 tasks/op,   42363 insns/op,   22725 cycles/op,        0 errors)
88173.10 tps ( 63.1 allocs/op,   0.0 logallocs/op,  14.2 tasks/op,   42366 insns/op,   23160 cycles/op,        0 errors)
88416.51 tps ( 63.1 allocs/op,   0.0 logallocs/op,  14.2 tasks/op,   42379 insns/op,   23102 cycles/op,        0 errors)

median 90104.26849997675
median absolute deviation: 1244.02
maximum: 91348.29
minimum: 86912.75
```

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>

Closes scylladb/scylladb#18818
2024-06-14 07:42:09 +03:00
Marcin Maliszkiewicz
6152223890 test: perf: extract result aggregation logic to a separate struct
It will be reused later by a new tool.
2024-05-09 13:58:29 +02:00
Benny Halevy
e5ca65f78b test/perf: report also log_allocations/op
Currently perf-simple-query --write ignores
log allocations that happen on the memtable
apply path.

This change adds tracking and accounting
of the number of log allocation,
and reporting of thereof.

For reference, here's the output of
build/release/scylla perf-simple-query --write --default-log-level=error --random-seed=1 -c 1
```
random-seed=1
enable-cache=1
Running test with config: {partitions=10000, concurrency=100, mode=write, frontend=cql, query_single_key=no, counters=no}
Disabling auto compaction
78073.55 tps ( 59.4 allocs/op,  16.3 logallocs/op,  14.3 tasks/op,   52991 insns/op,        0 errors)
77263.59 tps ( 59.3 allocs/op,  16.0 logallocs/op,  14.3 tasks/op,   53282 insns/op,        0 errors)
79913.07 tps ( 59.3 allocs/op,  16.0 logallocs/op,  14.3 tasks/op,   53295 insns/op,        0 errors)
79554.32 tps ( 59.3 allocs/op,  16.0 logallocs/op,  14.3 tasks/op,   53284 insns/op,        0 errors)
79151.53 tps ( 59.3 allocs/op,  16.0 logallocs/op,  14.3 tasks/op,   53289 insns/op,        0 errors)

median 79151.53 tps ( 59.3 allocs/op,  16.0 logallocs/op,  14.3 tasks/op,   53289 insns/op,        0 errors)
median absolute deviation: 761.54
maximum: 79913.07
minimum: 77263.59
```

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2024-05-02 18:42:41 +03:00
Kefu Chai
fe28aac440 test/perf: add fmt::formatter for perf_result_with_aio_writes
before this change, we rely on the default-generated fmt::formatter
created from operator<<, but fmt v10 dropped the default-generated
formatter.

in this change, we define formatters for `perf_result_with_aio_writes`,
and drop its operator<<.

Refs #13245

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#17849
2024-03-18 12:53:39 +02:00
Kefu Chai
2ccd9e695d test/perf: add fmt::formatters for scheduling_latency_measurer and perf_result
before this change, we rely on the default-generated fmt::formatter
created from operator<<, but fmt v10 dropped the default-generated
formatter.

in this change, we define formatters for

* scheduling_latency_measurer
* perf_result

and drop their operator<<:s

Refs #13245

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2024-02-23 10:17:50 +08:00
Lakshmi Narayanan Sreethar
76f0d5e35b reader_permit: store schema_ptr instead of raw schema pointer
Store schema_ptr in reader permit instead of storing a const pointer to
schema to ensure that the schema doesn't get changed elsewhere when the
permit is holding on to it. Also update the constructors and all the
relevant callers to pass down schema_ptr instead of a raw pointer.

Fixes #16180

Signed-off-by: Lakshmi Narayanan Sreethar <lakshmi.sreethar@scylladb.com>

Closes scylladb/scylladb#16658
2024-01-11 08:37:56 +02:00
Botond Dénes
e1b30f50be reader_concurrency_semaphore: add register_metrics constructor parameter
To be used in the next patch to control whether the semaphore registers
and exports metrics or not. We want to move metric registration to the
semaphore but we don't want all semaphores to export metrics. The
decision on whether a semaphore should or shouldn't export metrics
should be made on a case-by-case basis so this new parameter has no
default value (except for the for_tests constructor).
2023-12-13 06:25:45 -05:00
Botond Dénes
156e5d346d reader_permit: keep trace_state pointer on permit
And propagate it down to where it is created. This will be used to add
trace points for semaphore related events, but this will come in the
next patches.
2023-03-22 04:58:01 -04:00
Piotr Dulikowski
b9250a43e3 test: perf: count errors and report the count in results
Now, exceptions encountered during the test are counted as errors, and
the error count is reported at the end of the test.
2022-06-27 22:14:29 +02:00
Calle Wilund
5b60a6cf7c perf: Add aio_writes mixin for perf_results
Can be used with time_parallel_ex. Adds measurements for aio writes/aio written bytes.
2022-04-05 13:42:36 +00:00
Avi Kivity
fcb8d040e8 treewide: use Software Package Data Exchange (SPDX) license identifiers
Instead of lengthy blurbs, switch to single-line, machine-readable
standardized (https://spdx.dev) license identifiers. The Linux kernel
switched long ago, so there is strong precedent.

Three cases are handled: AGPL-only, Apache-only, and dual licensed.
For the latter case, I chose (AGPL-3.0-or-later and Apache-2.0),
reasoning that our changes are extensive enough to apply our license.

The changes we applied mechanically with a script, except to
licenses/README.md.

Closes #9937
2022-01-18 12:15:18 +01:00
Avi Kivity
cba07a3145 test: perf: fix format string for scheduling_latency_measurer
Need a colon to introduce the format after the default argument
specifier.

Found by fmt 8.

Closes #9644
2021-11-17 09:00:56 +02:00
Avi Kivity
27a2c74b64 test: replace seastar::sprint() with fmt::format()
sprint() is obsolete.
2021-10-27 17:02:00 +03:00
Benny Halevy
fe479aca1d reader_permit: add timeout member
To replace the timeout parameter passed
to flat_mutation_reader methods.

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2021-08-24 14:29:44 +03:00
Botond Dénes
c07db00b70 test: move away from make_permit()
Use the most appropriate up-front admission variant.
2021-07-14 17:19:02 +03:00
Botond Dénes
2454811dd6 test/perf: perf.hh: add reader_concurrency_semaphore_wrapper
A convenience, self-closing wrapper for those perf tests that have no
way to stop the semaphore and wait for it too.
2021-07-08 16:53:38 +03:00
Avi Kivity
a55b434a2b treewide: extent copyright statements to present day 2021-06-06 19:18:49 +03:00
Avi Kivity
863b49af03 test: perf: report instructions retired per operations
Instructions retired per op is a much more stable than time per op
(inverse throughput) since it isn't much affected by changes in
CPU frequencey or other load on the test system (it's still somewhat
affected since a slower system will run more reactor polls per op).
It's also less indicative of real performance, since it's possible for
fewer inststructions to execute in more time than more instructions,
but that isn't an issue for comparative tests).

This allows incremental changes to the code base to be compared with
more confidence.
2021-04-28 18:46:55 +03:00
Avi Kivity
498e6b9a64 test: perf: make executor_stats_snapshot() a member function of executor
I'd like to add an instructions counter which isn't accessible via
a global, so make the snapshot function a member. Out of respect to #1,
define functions for getting the number of allocations and tasks processed,
as they need heavy header files.
2021-04-28 18:38:35 +03:00
Avi Kivity
a43e896396 test: perf: don't truncate allocation/req and tasks/req report
I used {:.0} to truncate to integer, but apparently that resulted
in only one significant digit in the report, so 93.1 was reported as
90. Use the {:5.1f} to avoid truncation, and even get an extra
digit (we can have fractional tasks/op due to batching).

Current result is 93.1 allocs/op, 20.1 tasks/op (which suggests
batch size of around 10).

Closes #8550
2021-04-28 12:50:13 +02:00
Avi Kivity
202c631dee test: perf: perf_simple_query: collect allocation and task statistics
Calculate and display the number of memory allocations and tasks
executed per operation. Sample results (--smp 1):

180022.46 tps (90 allocs/op, 20 tasks/op)
178963.44 tps (90 allocs/op, 20 tasks/op)
178702.41 tps (90 allocs/op, 20 tasks/op)
177679.74 tps (90 allocs/op, 20 tasks/op)
179539.36 tps (90 allocs/op, 20 tasks/op)

median 178963.44 tps (90 allocs/op, 20 tasks/op)
median absolute deviation: 575.92
maximum: 180022.46
minimum: 177679.74

This allows less noisy tracking of how some changes impact performance.
2021-04-07 17:54:48 +03:00
Avi Kivity
3a90df39c5 perf: deinline some functions in perf.hh
Those functions were defined in a header, but not marked inline.
This made including the header from two source files impossible,
as the linker would complain about duplicate symbols.

Rather than making them inline, put them in a new source file
perf.cc as they don't need to be inline.
2021-04-07 17:51:58 +03:00