f1cc6252fd
Currently, Scylla logs a warning when it writes a cell, row or partition which are larger than certain configured sizes. These warnings contain the partition key and in case of rows and cells also the cluster key which allow the large row or partition to be identified. However, these keys can contain user-private, sensitive information. The information which identifies the partition/row/cell is also inserted into tables system.large_partitions, system.large_rows and system.large_cells respectivelly. This change removes the partition and cluster keys from the log messages, but still inserts them into the system tables. The logged data will look like this: Large cells: WARN 2024-04-02 16:49:48,602 [shard 3: mt] large_data - Writing large cell ks_name/tbl_name: cell_name (SIZE bytes) to sstable.db Large rows: WARN 2024-04-02 16:49:48,602 [shard 3: mt] large_data - Writing large row ks_name/tbl_name: (SIZE bytes) to sstable.db Large partitions: WARN 2024-04-02 16:49:48,602 [shard 3: mt] large_data - Writing large partition ks_name/tbl_name: (SIZE bytes) to sstable.db Fixes #18041 Closes scylladb/scylladb#18166
60 lines
2.3 KiB
ReStructuredText
60 lines
2.3 KiB
ReStructuredText
Large Partitions Hunting
|
|
========================
|
|
|
|
|
|
|
|
This document describes how to catch large partitions.
|
|
|
|
What Should Make You Want To Start Looking For A Large Partition?
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
Any of the following:
|
|
|
|
* Latencies on a single shard become very long (look at the "Scylla Overview Metrics" dashboard of `ScyllaDB Monitoring Stack <https://monitoring.docs.scylladb.com/stable/>`_).
|
|
* Oversized allocation warning messages in the log:
|
|
|
|
.. code-block:: none
|
|
|
|
seastar_memory - oversized allocation: 2842624 bytes, please report
|
|
|
|
* A warning of "Writing large (partition|row|cell)" is issued when writing to a table (usually happens during a compaction):
|
|
|
|
.. code-block:: none
|
|
|
|
WARN 2022-09-22 17:33:11,075 [shard 1]large_data - Writing large partition Some_KS/Some_table: [COL] (SIZE bytes) to SSTABLE_NAME
|
|
|
|
In this case, refer to :ref:`Troubleshooting Large Partition Tables <large-partition-table-configure>` for more information.
|
|
|
|
What To Do When You Suspect You May Have A Large Partition?
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
For each table you suspect run:
|
|
|
|
.. code-block:: console
|
|
|
|
nodetool flush <keyspace name> <table name>
|
|
nodetool cfstats <keyspace name>.<table name> | grep "Compacted partition maximum bytes"
|
|
|
|
|
|
For example:
|
|
|
|
.. code-block:: console
|
|
|
|
nodetool cfstats demodb.tmcr | grep "Compacted partition maximum bytes"
|
|
|
|
Compacted partition maximum bytes: 1188716932
|
|
|
|
Using system tables to detect large partitions, rows, or cells
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
Large rows and large cells are listed in the ``system.large_rows`` and ``system.large_cells`` tables, respectively. See :doc:`Scylla Large Rows and Cells Tables </troubleshooting/large-rows-large-cells-tables/>` for more information.
|
|
|
|
|
|
When Compaction Creates an Error
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
When compaction or writing to a table results in a "Writing a partition with too many rows" warning:
|
|
|
|
This warning indicates that there is a huge multi-row partition (based on the number of rows) and it is orthogonal
|
|
to the size-based warnings. The warning is controlled by ``compaction_rows_count_warning_threshold``, which is set in the scylla.yaml.
|
|
See :ref:`Troubleshooting Large Partition Tables <large-partition-table-configure>` for more information. |