cli: debug sub-command (#4227)

## Issue

Implement a new subcommand: tendermint debug. This subcommand itself has two subcommands:

    $ tendermint debug kill <pid> </path/to/out.zip> --home=</path/to/app.d>

Writes debug info into a compressed archive. The archive will contain the following:

├── config.toml
├── consensus_state.json
├── net_info.json
├── stacktrace.out
├── status.json
└── wal

The Tendermint process will be killed.

    $ tendermint debug dump </path/to/out> --home=</path/to/app.d>

This command will perform similar to kill except it only polls the node and dumps debugging data every frequency seconds to a compressed archive under a given destination directory. Each archive will contain:

├── consensus_state.json
├── goroutine.out
├── heap.out
├── net_info.json
├── status.json
└── wal

Note:

    goroutine.out and heap.out will only be written if a profile address is provided and is operational.
    This command is blocking and will log any error.

replaces: #3327
closes: #3249

## Commits:

* Implement debug tool command stubs

* Implement net getters and zip logic

* Update zip dir API and add home flag

* Add simple godocs for kill aux functions

* Move IO util to new file and implement copy WAL func

* Implement copy config function

* Implement killProc

* Remove debug fmt

* Validate output file input

* Direct STDERR to file

* Godoc updates

* Sleep prior to killing tail proc

* Minor cleanup of godocs

* Move debug command and add make target

* Rename command handler function

* Add example to command long descr

* Move current implementation to cmd/tendermint/commands/debug

* Update kill cmd long description

* Implement dump command

* Add pending log entry

* Add gosec nolint

* Add error check for Mkdir

* Add os.IsNotExist(err)

* Add to debugging section in running-in-prod doc
This commit is contained in:
Alexander Bezobchuk
2019-12-13 03:46:23 -05:00
committed by Anton Kaliaev
parent c9708105f0
commit 4bfec797e8
8 changed files with 537 additions and 1 deletions

View File

@@ -141,6 +141,19 @@ returns just the votes seen at the current height.
- [StackOverflow
questions](https://stackoverflow.com/questions/tagged/tendermint)
### Debug Utility
Tendermint also ships with a `debug` sub-command that allows you to kill a live
Tendermint process while collecting useful information in a compressed archive
such as the configuration used, consensus state, network state, the node' status,
the WAL, and even the stacktrace of the process before exit. These files can be
useful to examine when debugging a faulty Tendermint process.
In addition, the `debug` sub-command also allows you to dump debugging data into
compressed archives at a regular interval. These archives contain the goroutine
and heap profiles in addition to the consensus state, network info, node status,
and even the WAL.
## Monitoring Tendermint
Each Tendermint instance has a standard `/health` RPC endpoint, which