mirror of
https://github.com/scylladb/scylladb.git
synced 2026-05-02 06:05:53 +00:00
a78f43b071
The set of patches introduces a new tracing mode - `fast slow query tracing`. In this mode, Scylla tracks only tracing sessions and omits all tracing events if the tracing context does not have a `full_tracing` state set. Fixes #2572 Motivation --- We want to run production systems with that option always enabled so we could always catch slow queries without an overhead. The next step is we are gonna optimize further the costs of having tracing enabled to minimize session context handling overhead to allow it to be as transparent for the end-user as possible. Fast tracing mode --- To read the status do $ curl -v http://localhost:10000/storage_service/slow_query To enable fast slow-query tracing $ curl -v --request POST http://localhost:10000/storage_service/slow_query\?fast=true\&enable=true Potential optimizations --- - remove tracing::begin(lazy_eval) - replace tracing::begin(string) for enum to remove copying and memory allocations - merge parameters allocations - group parameters check for trace context - delay formatting - reuse prepared statement shared_ptr instead of both copying it and copying its query Performance --- 100% cache hits --- 1 Core: ``` $ SCYLLA_HOME=/home/sitano.public/Projects/scylla build/release/scylla --smp 1 --cpuset 7 --log-to-syslog 0 --log-to-stdout 1 --default-log-level info --network-stack posix --workdir /home/sitano.public/Projects/scylla --developer-mode 1 --listen-address 0.0.0.0 --api-address 0.0.0.0 --rpc-address 0.0.0.0 --broadcast-rpc-address 172.18.0.1 --broadcast-address 127.0.0.1 ./cassandra-stress write n=100000 no-warmup -pop seq=1..100000 -node 127.0.0.1 -log level=verbose -rate threads=1 -mode native cql3 curl --request POST http://localhost:10000/storage_service/slow_query\?fast\=false\&enable\=false for i in $(seq 5); do taskset -c 2,3,4,5 ./cassandra-stress read duration=5m -pop seq=1..100000 -node 127.0.0.1 -log level=verbose -rate threads=4 throttle=30000/s -mode native cql3 done curl --request POST http://localhost:10000/storage_service/slow_query\?fast\=true\&enable\=true for i in $(seq 5); do taskset -c 2,3,4,5 ./cassandra-stress read duration=5m -pop seq=1..100000 -node 127.0.0.1 -log level=verbose -rate threads=4 throttle=30000/s -mode native cql3 done curl --request POST http://localhost:10000/storage_service/slow_query\?fast\=false\&enable\=true for i in $(seq 5); do taskset -c 2,3,4,5 ./cassandra-stress read duration=5m -pop seq=1..100000 -node 127.0.0.1 -log level=verbose -rate threads=4 throttle=30000/s -mode native cql3 done ``` | qps | | | -- | -- | -- | -- | -- | baseline | fast, slow | nofast, slow | %[1-fastslow/baseline] | 29,018 | 26,468 | 23,591 | 8.79% | 28,909 | 26,274 | 23,584 | 9.11% | 28,900 | 26,547 | 23,598 | 8.14% | 28,921 | 26,669 | 23,596 | 7.79% | 28,821 | 26,385 | 23,601 | 8.45% stdev | 70.24030182 | 150.9678774 | 6.670832032 | avg | 28,914 | 26,469 | 23,594 | stderr | 0.24% | 0.57% | 0.03% | %[avg/baseline] | | **8.46%** | 18.40% | 8.46% performance degradation in `fast slow query mode` for pure in-memory workload with minimum traces. 18.40% performance degradation in `original slow query mode` for pure in-memory workload with minimum traces. 0% cache hits --- 1GB memory, 1 Core: $ SCYLLA_HOME=/home/sitano.public/Projects/scylla build/release/scylla --memory 1G --smp 1 --cpuset 7 --log-to-syslog 0 --log-to-stdout 1 --default-log-level info --network-stack posix --workdir /home/sitano.public/Projects/scylla --developer-mode 1 --listen-address 0.0.0.0 --api-address 0.0.0.0 --rpc-address 0.0.0.0 --broadcast-rpc-address 172.18.0.1 --broadcast-address 127.0.0.1 2.4GB, 10000000 keys data: $ ./cassandra-stress write n=10000000 no-warmup -pop seq=1..10000000 -node 127.0.0.1 -log level=verbose -rate threads=4 -mode native cql3 $ curl --request POST http://localhost:10000/storage_service/slow_query\?fast\=true\&enable\=true CASSANDRA_STRESS prepared statements with BYPASS CACHE $ taskset -c 2,3,4,5 ./cassandra-stress read duration=5m -pop seq=1..10000000 -node 127.0.0.1 -log level=verbose -rate threads=4 throttle=30000/s -mode native cql3 20000 reads IOPS, 100MB/s from disk | qps | | | -- | -- | -- | -- | -- | baseline reads | fast, slow reads | %[1-fastslow/baseline] | | 9,575 | 9,054 | 5.44% | | 9,614 | 9,065 | 5.71% | | 9,610 | 9,066 | 5.66% | | 9,611 | 9,062 | 5.71% | | 9,614 | 9,073 | 5.63% | stdev | 16.75410397 | 6.892024376 | avg | 9,605 | 9,064 | stderr | 0.17% | 0.08% | %[avg/baseline] | | **5.63%** | 5.63% performance degradation in `fast slow query mode` for pure on-disk workload with minimum traces. Closes #8314 * github.com:scylladb/scylla: tracing: fast mode unit test tracing: rest api for lightweight slow query tracing tracing: omit tracing session events and subsessions in fast mode