Merge 'tracing: fast slow query tracing' from Ivan Prisyazhnyy
The set of patches introduces a new tracing mode - `fast slow query tracing`. In this mode, Scylla tracks only tracing sessions and omits all tracing events if the tracing context does not have a `full_tracing` state set.
Fixes #2572
Motivation
---
We want to run production systems with that option always enabled so we could always catch slow queries without an overhead. The next step is we are gonna optimize further the costs of having tracing enabled to minimize session context handling overhead to allow it to be as transparent for the end-user as possible.
Fast tracing mode
---
To read the status do
$ curl -v http://localhost:10000/storage_service/slow_query
To enable fast slow-query tracing
$ curl -v --request POST http://localhost:10000/storage_service/slow_query\?fast=true\&enable=true
Potential optimizations
---
- remove tracing::begin(lazy_eval)
- replace tracing::begin(string) for enum to remove copying and memory allocations
- merge parameters allocations
- group parameters check for trace context
- delay formatting
- reuse prepared statement shared_ptr instead of both copying it and copying its query
Performance
---
100% cache hits
---
1 Core:
```
$ SCYLLA_HOME=/home/sitano.public/Projects/scylla build/release/scylla --smp 1 --cpuset 7 --log-to-syslog 0 --log-to-stdout 1 --default-log-level info --network-stack posix --workdir /home/sitano.public/Projects/scylla --developer-mode 1 --listen-address 0.0.0.0 --api-address 0.0.0.0 --rpc-address 0.0.0.0 --broadcast-rpc-address 172.18.0.1 --broadcast-address 127.0.0.1
./cassandra-stress write n=100000 no-warmup -pop seq=1..100000 -node 127.0.0.1 -log level=verbose -rate threads=1 -mode native cql3
curl --request POST http://localhost:10000/storage_service/slow_query\?fast\=false\&enable\=false
for i in $(seq 5); do
taskset -c 2,3,4,5 ./cassandra-stress read duration=5m -pop seq=1..100000 -node 127.0.0.1 -log level=verbose -rate threads=4 throttle=30000/s -mode native cql3
done
curl --request POST http://localhost:10000/storage_service/slow_query\?fast\=true\&enable\=true
for i in $(seq 5); do
taskset -c 2,3,4,5 ./cassandra-stress read duration=5m -pop seq=1..100000 -node 127.0.0.1 -log level=verbose -rate threads=4 throttle=30000/s -mode native cql3
done
curl --request POST http://localhost:10000/storage_service/slow_query\?fast\=false\&enable\=true
for i in $(seq 5); do
taskset -c 2,3,4,5 ./cassandra-stress read duration=5m -pop seq=1..100000 -node 127.0.0.1 -log level=verbose -rate threads=4 throttle=30000/s -mode native cql3
done
```
| qps | | |
-- | -- | -- | -- | --
| baseline | fast, slow | nofast, slow | %[1-fastslow/baseline]
| 29,018 | 26,468 | 23,591 | 8.79%
| 28,909 | 26,274 | 23,584 | 9.11%
| 28,900 | 26,547 | 23,598 | 8.14%
| 28,921 | 26,669 | 23,596 | 7.79%
| 28,821 | 26,385 | 23,601 | 8.45%
stdev | 70.24030182 | 150.9678774 | 6.670832032 |
avg | 28,914 | 26,469 | 23,594 |
stderr | 0.24% | 0.57% | 0.03% |
%[avg/baseline] | | **8.46%** | 18.40% |
8.46% performance degradation in `fast slow query mode` for pure in-memory workload with minimum traces.
18.40% performance degradation in `original slow query mode` for pure in-memory workload with minimum traces.
0% cache hits
---
1GB memory, 1 Core:
$ SCYLLA_HOME=/home/sitano.public/Projects/scylla build/release/scylla --memory 1G --smp 1 --cpuset 7 --log-to-syslog 0 --log-to-stdout 1 --default-log-level info --network-stack posix --workdir /home/sitano.public/Projects/scylla --developer-mode 1 --listen-address 0.0.0.0 --api-address 0.0.0.0 --rpc-address 0.0.0.0 --broadcast-rpc-address 172.18.0.1 --broadcast-address 127.0.0.1
2.4GB, 10000000 keys data:
$ ./cassandra-stress write n=10000000 no-warmup -pop seq=1..10000000 -node 127.0.0.1 -log level=verbose -rate threads=4 -mode native cql3
$ curl --request POST http://localhost:10000/storage_service/slow_query\?fast\=true\&enable\=true
CASSANDRA_STRESS prepared statements with BYPASS CACHE
$ taskset -c 2,3,4,5 ./cassandra-stress read duration=5m -pop seq=1..10000000 -node 127.0.0.1 -log level=verbose -rate threads=4 throttle=30000/s -mode native cql3
20000 reads IOPS, 100MB/s from disk
| qps | | |
-- | -- | -- | -- | --
| baseline reads | fast, slow reads | %[1-fastslow/baseline] |
| 9,575 | 9,054 | 5.44% |
| 9,614 | 9,065 | 5.71% |
| 9,610 | 9,066 | 5.66% |
| 9,611 | 9,062 | 5.71% |
| 9,614 | 9,073 | 5.63% |
stdev | 16.75410397 | 6.892024376 |
avg | 9,605 | 9,064 |
stderr | 0.17% | 0.08% |
%[avg/baseline] | | **5.63%** |
5.63% performance degradation in `fast slow query mode` for pure on-disk workload with minimum traces.
Closes #8314
* github.com:scylladb/scylla:
tracing: fast mode unit test
tracing: rest api for lightweight slow query tracing
tracing: omit tracing session events and subsessions in fast mode
a78f43b071