diff --git a/docs/troubleshooting/error-messages/critical-disk-utilization.rst b/docs/troubleshooting/error-messages/critical-disk-utilization.rst new file mode 100644 index 0000000000..6fada1110c --- /dev/null +++ b/docs/troubleshooting/error-messages/critical-disk-utilization.rst @@ -0,0 +1,31 @@ +Critical disk utilization +========================= + +Issue +^^^^^ + +SELECT/INSERT/UPDATE/DELETE cqlsh operations fail with ``Critical disk utilization: rejected write mutation`` error message. + + +For example + +.. code-block:: shell + + WriteFailure: Error from server: code=1500 [Replica(s) failed to execute write] message="Critical disk utilization: rejected write mutation" info={'consistency': 'QUORUM', 'required_responses': 2, 'received_responses': 1, 'failures': 2} + +Problem +^^^^^^^ + +Scaling out a cluster failed or is delayed. As a consequence the disk space becomes critically low and mechanisms preventing nodes going out of space have been activated. User write activity was paused, but the node remains able to migrate data to other nodes. + +How to Verify +^^^^^^^^^^^^^ + +Run ``nodetool status`` to check nodes are up and running. + +Run ``df -h /var/lib/scylla/`` to verify that the current disk utilization (in %) is higher than the configured threshold :ref:`critical_disk_utilization_level `. + +Solution +^^^^^^^^ + +Scale out the cluster. Adding new nodes to the same rack as the node that reached critical condition, will bring the disk utilization down below the critical threshold and the prevention mechanisms will be deactivated. diff --git a/docs/troubleshooting/error-messages/index.rst b/docs/troubleshooting/error-messages/index.rst index 0ca8c26e10..8da6176e3b 100644 --- a/docs/troubleshooting/error-messages/index.rst +++ b/docs/troubleshooting/error-messages/index.rst @@ -8,6 +8,7 @@ Error Messages address-already-in-use schema-mismatch invalid-ssl-prot-error + critical-disk-utilization .. panel-box:: @@ -22,3 +23,5 @@ Error Messages * :doc:`Schema Mismatch ` * :doc:`Invalid SSL Protocol ` + + * :doc:`Critical disk utilization `