test/e2e: add random testnet generator (#5479)

Closes #5291. Adds a randomized testnet generator. Nightly CI job will be submitted separately. A few of the testnets can be a bit flaky, even after disabling known-faulty behavior and making minor tweaks, and the larger networks may be too resource-intensive to run in CI - this will be optimized separately.
This commit is contained in:
Erik Grinaker
2020-10-22 13:36:08 +02:00
committed by Erik Grinaker
parent e7568f9e0c
commit f9bfb40d53
14 changed files with 570 additions and 30 deletions
+18 -2
View File
@@ -3,8 +3,7 @@
Spins up and tests Tendermint networks in Docker Compose based on a testnet manifest. To run the CI testnet:
```sh
make docker
make runner
make
./build/runner -f networks/ci.toml
```
@@ -14,6 +13,23 @@ This creates and runs a testnet named `ci` under `networks/ci/` (determined by t
Testnets are specified as TOML manifests. For an example see [`networks/ci.toml`](networks/ci.toml), and for documentation see [`pkg/manifest.go`](pkg/manifest.go).
## Random Testnet Generation
Random (but deterministic) combinations of testnets can be generated with `generator`:
```sh
./build/generator -d networks/generated/
# Split networks into 8 groups (by filename)
./build/generator -g 8 -d networks/generated/
```
Multiple testnets can be run with the `run-multiple.sh` script:
```sh
./run-multiple.sh networks/generated/gen-group3-*.toml
```
## Test Stages
The test runner has the following stages, which can also be executed explicitly by running `./build/runner -f <manifest> <stage>`: