58 lines
2.5 KiB
ReStructuredText
58 lines
2.5 KiB
ReStructuredText
ScyllaDB Fails to Start - SSTable Corruption Problem
|
|
=====================================================
|
|
|
|
This troubleshooting guide describes what to do when ScyllaDB fails to start due to a corrupted SSTables.
|
|
Corruption can be a result of a bug, disk issue or human error, for example deleting one of the SSTable files
|
|
|
|
|
|
Problem
|
|
^^^^^^^
|
|
ScyllaDB node fails to start, node status shows that the node is down (DN)
|
|
|
|
How to Verify
|
|
^^^^^^^^^^^^^
|
|
When there is an unknown problem, checking the :doc:`logs</getting-started/logging/>` is mandatory, using ``journalctl -xe``
|
|
|
|
For example:
|
|
|
|
.. code-block:: shell
|
|
|
|
scylla[28659]: [shard 0] database - Exception while populating keyspace '<mykeyspace>' with 'test' table from file '/var/lib/scylla/data/mykeyspace/test-fa9994e02fd811e7a4ee000000000000': sstables::malformed_sstable_exception (At directory:/var/lib/scylla/data/mykeyspace/test-fa9994e02fd811e7a4ee000000000000: no TOC found for SSTable with generation 2!. Refusing to boot)
|
|
|
|
In this scenario, a missing ``TOC`` file will prevent the ScyllaDB node from starting.
|
|
|
|
The SSTable corruption problem can be different, for example, other missing or unreadable files. The following solution applies to all scenarios.
|
|
|
|
Solution
|
|
^^^^^^^^
|
|
|
|
1. | Locate all the SSTable files with the reported corrupted generation number as reported in the log, in our example the generation number is **2**.
|
|
| By default the SSTable can be located under ``/var/lib/scylla/data/keyspace_name/table_name-UUID/``
|
|
|
|
2. Delete all the SSTables files that belong to the generation number
|
|
|
|
For example:
|
|
|
|
.. code-block:: shell
|
|
|
|
sudo rm test-ka-2*
|
|
|
|
-rw-r--r-- 1 scylla scylla 66 May 8 14:17 test-ka-2-CompressionInfo.db
|
|
-rw-r--r-- 1 scylla scylla 357 May 8 14:17 test-ka-2-Data.db
|
|
-rw-r--r-- 1 scylla scylla 10 May 8 14:17 test-ka-2-Digest.sha1
|
|
-rw-r--r-- 1 scylla scylla 24 May 8 14:17 test-ka-2-Filter.db
|
|
-rw-r--r-- 1 scylla scylla 140 May 8 14:17 test-ka-2-Index.db
|
|
-rw-r--r-- 1 scylla scylla 38 May 8 14:17 test-ka-2-ScyllaDB.db
|
|
-rw-r--r-- 1 scylla scylla 4446 May 8 14:17 test-ka-2-Statistics.db
|
|
-rw-r--r-- 1 scylla scylla 92 May 8 14:17 test-ka-2-Summary.db
|
|
|
|
3. Start ScyllaDB node
|
|
|
|
``sudo systemctl start scylla-server``
|
|
|
|
4. Verify that the node is up again using ``nodetool status``. The node status should be Up Normal (UN)
|
|
|
|
5. Run the :doc:`nodetool repair </operating-scylla/nodetool-commands/repair/>` command on the node with the corrupted SSTable
|
|
|
|
.. include:: /troubleshooting/_common/ts-return.rst
|