docs: write about debug kill and dump (#4516)

* docs: write about debug kill and dump

Closes #4325

* wrap file tree in code blocks

docs/tools/README.md

@@ -9,16 +9,21 @@ parent:
 
 Tendermint has some tools that are associated with it for:
 
+- [Debugging](./debugging.md)
 - [Benchmarking](#benchmarking)
-- [Validation of remote signers](./remote-signer-validation.md)
 - [Testnets](#testnets)
-
+- [Validation of remote signers](./remote-signer-validation.md)
 
 ## Benchmarking
 
-Benchmarking is done with tm-load-test, for information on how to use the tool please visit the docs: https://github.com/interchainio/tm-load-test
+- https://github.com/interchainio/tm-load-test
 
+`tm-load-test` is a distributed load testing tool (and framework) for load
+testing Tendermint networks.
 
 ## Testnets
 
-The testnets tool is aimed at testing Tendermint with different configurations. For more information please visit: https://github.com/interchainio/testnets.
+- https://github.com/interchainio/testnets
+
+This repository contains various different configurations of test networks for,
+and relating to, Tendermint.

docs/tools/debugging.md

@@ -0,0 +1,57 @@
+# Debugging
+
+## tendermint debug kill
+
+Tendermint comes with a `debug` sub-command that allows you to kill a live
+Tendermint process while collecting useful information in a compressed archive.
+The information includes the configuration used, consensus state, network
+state, the node' status, the WAL, and even the stack trace of the process
+before exit. These files can be useful to examine when debugging a faulty
+Tendermint process.
+
+```sh
+tendermint debug kill <pid> </path/to/out.zip> --home=</path/to/app.d>
+```
+
+will write debug info into a compressed archive. The archive will contain the
+following:
+
+```
+├── config.toml
+├── consensus_state.json
+├── net_info.json
+├── stacktrace.out
+├── status.json
+└── wal
+```
+
+Under the hood, `debug kill` fetches info from `/status`, `/net_info`, and
+`/dump_consensus_state` HTTP endpoints, and kills the process with `-6`, which
+catches the go-routine dump.
+
+## tendermint debug dump
+
+Also, the `debug dump` sub-command allows you to dump debugging data into
+compressed archives at a regular interval. These archives contain the goroutine
+and heap profiles in addition to the consensus state, network info, node
+status, and even the WAL.
+
+```sh
+tendermint debug dump </path/to/out> --home=</path/to/app.d>
+```
+
+will perform similarly to `kill` except it only polls the node and
+dumps debugging data every frequency seconds to a compressed archive under a
+given destination directory. Each archive will contain:
+
+```
+├── consensus_state.json
+├── goroutine.out
+├── heap.out
+├── net_info.json
+├── status.json
+└── wal
+```
+
+Note: goroutine.out and heap.out will only be written if a profile address is
+provided and is operational. This command is blocking and will log any error.