Dump a diagnostics report on each shard when receiving a SIGQUIT. The
report is logged with a dedicated logger, called diagnostics.
The report has multiple parts:
* seastar memory diagnostics, similar to that printed by the scylla
memory command (from scylla-gdb.py).
* reader concurrency semaphore diagnostics for each semaphore.
Example report:
INFO 2024-11-27 01:31:55,882 [shard 0:main] diagnostics - Diagnostics dump requested via SIGQUIT:
Dumping seastar memory diagnostics
Used memory: 3988M
Free memory: 58M
Total memory: 4G
Hard failures: 0
LSA
allocated: 4M
used: 16
free: 4G
Cache:
total: 1M
used: 642K
free: 398K
Memtables:
total: 3M
Regular:
real dirty: 0B
virt dirty: 0B
System:
real dirty: 3M
virt dirty: 3M
Replica:
Read Concurrency Semaphores:
user: 0/100, 0B/81M, queued: 0
streaming: 0/10, 0B/81M, queued: 0
system: 0/10, 0B/81M, queued: 0
compaction: 0/unlimited, 0B/unlimited
view update: 0/50, 0B/40M, queued: 0
Execution Stages:
apply stage:
Total: 0
Tables - Ongoing Operations:
Pending writes (top 10):
0 Total (all)
Pending reads (top 10):
0 Total (all)
Pending streams (top 10):
0 Total (all)
Small pools:
objsz spansz usedobj memory unused wst%
8 4K 858 16K 9K 58
10 4K 5 8K 8K 99
12 4K 5 8K 8K 99
14 4K 0 0B 0B 0
16 4K 2k 44K 15K 35
32 4K 4k 136K 16K 11
32 4K 8k 280K 24K 8
32 4K 3k 92K 6K 6
32 4K 4k 140K 21K 14
48 4K 3k 180K 25K 14
48 4K 2k 120K 27K 22
64 4K 2k 156K 18K 11
64 4K 19k 1M 11K 0
80 4K 3k 236K 16K 6
96 4K 6k 572K 49K 8
112 4K 2k 276K 72K 25
128 4K 477 80K 20K 25
160 4K 194 60K 30K 49
192 4K 1k 232K 39K 16
224 4K 2k 468K 15K 3
256 4K 182 100K 55K 54
320 8K 349 152K 43K 28
384 8K 332 288K 164K 56
448 4K 243 180K 74K 40
512 4K 256 244K 116K 47
640 16K 185 192K 76K 39
768 16K 394 432K 137K 31
896 8K 54 192K 144K 75
1024 4K 288 432K 144K 33
1280 32K 92 256K 140K 54
1536 32K 11 128K 111K 86
1792 16K 10 144K 126K 87
2048 8K 487 1M 90K 8
2560 64K 113 384K 100K 26
3072 64K 9 256K 228K 89
3584 32K 3 288K 277K 96
4096 16K 129 912K 396K 43
5120 128K 21 384K 275K 71
6144 128K 4 512K 486K 94
7168 64K 3 576K 553K 96
8192 32K 373 3M 56K 1
10240 64K 6 832K 770K 92
12288 64K 17 960K 756K 78
14336 128K 2 1M 1M 97
16384 64K 14 1M 992K 81
Page spans:
index size free used spans
0 4K 4K 5M 1k
1 8K 8K 2M 213
2 16K 16K 2M 106
3 32K 64K 6M 200
4 64K 64K 4M 71
5 128K 384K 3934M 31k
6 256K 1M 256K 5
7 512K 512K 512K 2
8 1M 2M 0B 2
9 2M 2M 2M 2
10 4M 4M 0B 1
11 8M 16M 0B 2
12 16M 32M 0B 2
13 32M 0B 32M 1
14 64M 0B 0B 0
15 128M 0B 0B 0
16 256M 0B 0B 0
17 512M 0B 0B 0
18 1G 0B 0B 0
19 2G 0B 0B 0
20 4G 0B 0B 0
21 8G 0B 0B 0
22 16G 0B 0B 0
23 32G 0B 0B 0
24 64G 0B 0B 0
25 128G 0B 0B 0
26 256G 0B 0B 0
27 512G 0B 0B 0
28 1T 0B 0B 0
29 2T 0B 0B 0
30 4T 0B 0B 0
31 8T 0B 0B 0
INFO 2024-11-27 01:31:55,882 [shard 0:main] diagnostics - Diagnostics dump requested via SIGQUIT:
Semaphore user with 0/100 count and 0/84850769 memory resources: user request, dumping permit diagnostics:
permits count memory table/operation/state
0 0 0B total
Stats:
permit_based_evictions: 0
time_based_evictions: 0
inactive_reads: 0
total_successful_reads: 0
total_failed_reads: 0
total_reads_shed_due_to_overload: 0
total_reads_killed_due_to_kill_limit: 0
reads_admitted: 0
reads_enqueued_for_admission: 0
reads_enqueued_for_memory: 0
reads_admitted_immediately: 0
reads_queued_because_ready_list: 0
reads_queued_because_need_cpu_permits: 0
reads_queued_because_memory_resources: 0
reads_queued_because_count_resources: 0
reads_queued_with_eviction: 0
total_permits: 0
current_permits: 0
need_cpu_permits: 0
awaits_permits: 0
disk_reads: 0
sstables_read: 0
INFO 2024-11-27 01:31:55,882 [shard 0:main] diagnostics - Diagnostics dump requested via SIGQUIT:
Semaphore streaming with 0/10 count and 0/84850769 memory resources: user request, dumping permit diagnostics:
permits count memory table/operation/state
0 0 0B total
Stats:
permit_based_evictions: 0
time_based_evictions: 0
inactive_reads: 0
total_successful_reads: 6
total_failed_reads: 0
total_reads_shed_due_to_overload: 0
total_reads_killed_due_to_kill_limit: 0
reads_admitted: 6
reads_enqueued_for_admission: 0
reads_enqueued_for_memory: 0
reads_admitted_immediately: 6
reads_queued_because_ready_list: 0
reads_queued_because_need_cpu_permits: 0
reads_queued_because_memory_resources: 0
reads_queued_because_count_resources: 0
reads_queued_with_eviction: 0
total_permits: 6
current_permits: 0
need_cpu_permits: 0
awaits_permits: 0
disk_reads: 0
sstables_read: 0
INFO 2024-11-27 01:31:55,882 [shard 0:main] diagnostics - Diagnostics dump requested via SIGQUIT:
Semaphore compaction with 0/2147483647 count and 0/9223372036854775807 memory resources: user request, dumping permit diagnostics:
permits count memory table/operation/state
0 0 0B total
Stats:
permit_based_evictions: 0
time_based_evictions: 0
inactive_reads: 0
total_successful_reads: 0
total_failed_reads: 0
total_reads_shed_due_to_overload: 0
total_reads_killed_due_to_kill_limit: 0
reads_admitted: 0
reads_enqueued_for_admission: 0
reads_enqueued_for_memory: 0
reads_admitted_immediately: 0
reads_queued_because_ready_list: 0
reads_queued_because_need_cpu_permits: 0
reads_queued_because_memory_resources: 0
reads_queued_because_count_resources: 0
reads_queued_with_eviction: 0
total_permits: 27
current_permits: 0
need_cpu_permits: 0
awaits_permits: 0
disk_reads: 0
sstables_read: 0
INFO 2024-11-27 01:31:55,882 [shard 0:main] diagnostics - Diagnostics dump requested via SIGQUIT:
Semaphore system with 0/10 count and 0/84850769 memory resources: user request, dumping permit diagnostics:
permits count memory table/operation/state
1 0 0B *.*/view_builder/active
1 0 0B total
Stats:
permit_based_evictions: 0
time_based_evictions: 0
inactive_reads: 0
total_successful_reads: 234
total_failed_reads: 0
total_reads_shed_due_to_overload: 0
total_reads_killed_due_to_kill_limit: 0
reads_admitted: 234
reads_enqueued_for_admission: 154
reads_enqueued_for_memory: 0
reads_admitted_immediately: 80
reads_queued_because_ready_list: 154
reads_queued_because_need_cpu_permits: 0
reads_queued_because_memory_resources: 0
reads_queued_because_count_resources: 0
reads_queued_with_eviction: 0
total_permits: 235
current_permits: 1
need_cpu_permits: 0
awaits_permits: 0
disk_reads: 0
sstables_read: 0
INFO 2024-11-27 01:31:55,882 [shard 0:main] diagnostics - Diagnostics dump requested via SIGQUIT:
Semaphore view_update with 0/50 count and 0/42425384 memory resources: user request, dumping permit diagnostics:
permits count memory table/operation/state
0 0 0B total
Stats:
permit_based_evictions: 0
time_based_evictions: 0
inactive_reads: 0
total_successful_reads: 0
total_failed_reads: 0
total_reads_shed_due_to_overload: 0
total_reads_killed_due_to_kill_limit: 0
reads_admitted: 0
reads_enqueued_for_admission: 0
reads_enqueued_for_memory: 0
reads_admitted_immediately: 0
reads_queued_because_ready_list: 0
reads_queued_because_need_cpu_permits: 0
reads_queued_because_memory_resources: 0
reads_queued_because_count_resources: 0
reads_queued_with_eviction: 0
total_permits: 0
current_permits: 0
need_cpu_permits: 0
awaits_permits: 0
disk_reads: 0
sstables_read: 0
Fixes: scylladb/scylladb#7400
Closes scylladb/scylladb#21692
ScyllaDB Documentation
This repository contains the source files for ScyllaDB Open Source documentation.
- The
devfolder contains developer-oriented documentation related to the ScyllaDB code base. It is not published and is only available via GitHub. - All other folders and files contain user-oriented documentation related to ScyllaDB Open Source and are sources for opensource.docs.scylladb.com.
To report a documentation bug or suggest an improvement, open an issue in GitHub issues for this project.
To contribute to the documentation, open a GitHub pull request.
Key Guidelines for Contributors
- The user documentation is written in reStructuredText (RST) - a plaintext markup language similar to Markdown. If you're not familiar with RST, see ScyllaDB RST Examples.
- The developer documentation is written in Markdown. See Basic Markdown Syntax for reference.
- Follow the ScyllaDB Style Guide.
To prevent the build from failing:
-
If you add a new file, ensure it's added to an appropriate toctree, for example:
.. toctree:: :maxdepth: 2 :hidden: Page X </folder1/article1> Page Y </folder1/article2> Your New Page </folder1/your-new-article> -
Make sure the link syntax is correct. See the guidelines on creating links
-
Make sure the section headings are correct. See the guidelines on creating headings Note that the markup must be at least as long as the text in the heading. For example:
---------------------- Prerequisites ----------------------
Building User Documentation
Prerequisites
- Python
- poetry
- make
See the ScyllaDB Sphinx Theme prerequisites to check which versions of the above are currently required.
Mac OS X
You must have a working Homebrew in order to install the needed tools.
You also need the standard utility make.
Check if you have these two items with the following commands:
brew help
make -h
Linux Distributions
Building the user docs should work out of the box on most Linux distributions.
Windows
Use "Bash on Ubuntu on Windows" for the same tools and capabilities as on Linux distributions.
Building the Docs
- Run
make previewto build the documentation. - Preview the built documentation locally at http://127.0.0.1:5500/.
Cleanup
You can clean up all the build products and auto-installed Python stuff with:
make pristine
Information for Contributors
If you are interested in contributing to Scylla docs, please read the Scylla open source page at http://www.scylladb.com/opensource/ and complete a Scylla contributor agreement if needed. We can only accept documentation pull requests if we have a contributor agreement on file for you.
Third-party Documentation
-
Do any copying as a separate commit. Always commit an unmodified version first and then do any editing in a separate commit.
-
We already have a copy of the Apache license in our tree, so you do not need to commit a copy of the license.
-
Include the copyright header from the source file in the edited version. If you are copying an Apache Cassandra document with no copyright header, use:
This document includes material from Apache Cassandra.
Apache Cassandra is Copyright 2009-2014 The Apache Software Foundation.