* e2e: setup testing for multi-version (#9819)
This pull requests sets up the e2e tests to be able to support multiple versions within the same test network. This is achieved through a few simple changes:
* Each node takes a `version` parameter in the testnet manifest. This dictates which version of the testapp to use. If not set, the locally available version is used.
* Adds a `testapp-docker.yml` workflow that publishes the testapp to docker hub so that tagged versions may be available for use in a multi-version test network.
This change does not actually add a testnetwork that does multi-version testing. Since no previous versions of the testapp have been published to dockerhub, there are not old versions available to test against. We'll either need to configure this after the next minor release which will trigger a testapp to be pushed to dockerhub, or push an image from the previous version of Tendermint so that the multiversion test has an old version to pull.
#### PR checklist
- [ ] Tests written/updated, or no tests needed
- [ ] `CHANGELOG_PENDING.md` updated, or no changelog entry needed
- [ ] Updated relevant documentation (`docs/`) and code comments, or no
documentation updates needed
(cherry picked from commit 5ba0d131c4)
# Conflicts:
# test/e2e/pkg/manifest.go
* fix conflict
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
Co-authored-by: William Banfield <wbanfield@gmail.com>
Fixes the issue observed in an [e2e test run](https://github.com/tendermint/tendermint/actions/runs/3588927225). The issue arises when the e2e runner process cannot connect to the remote node. In the previous version of this code, the runner would simply skip the transaction if the client couldn't connect. This pull request resurrects that behavior.
---
#### PR checklist
- [ ] Tests written/updated, or no tests needed
- [ ] `CHANGELOG_PENDING.md` updated, or no changelog entry needed
- [ ] Updated relevant documentation (`docs/`) and code comments, or no
documentation updates needed
(cherry picked from commit d09f4f503d)
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
The "toml" annotation of "send no load" had a typo (`send_no_laod`).
This is suspected to have caused failures in e2e, however I couldn't reproduce the error locally, so not sure this will fix it. Still, the typo needs to be fixed in any case.
---
#### PR checklist
- [x] Tests written/updated, or no tests needed
- [x] `CHANGELOG_PENDING.md` updated, or no changelog entry needed
- [x] Updated relevant documentation (`docs/`) and code comments, or no
documentation updates needed
(cherry picked from commit 739b92bf01)
Co-authored-by: Sergio Mena <sergio@informal.systems>
* Added print
* Fix unmarshall
* Fix unmarshalling
* Simplified steps to unmarshall
* minor
* Use 'encoding/hex'
* Forget about C, this is Go!
* gosec warning
* Set maximum payload size
* nosec annotation
(cherry picked from commit b42c439776)
Co-authored-by: Sergio Mena <sergio@informal.systems>
* Add transaction hash to raw data
* Add hash in formatted output
* Cosmetic
(cherry picked from commit cdd3479f20)
Co-authored-by: Sergio Mena <sergio@informal.systems>
This pull request adds the block time as the unix time since the epoch to the `report` tool's csv output.
```csv
...
a7a8b903-1136-4da1-97aa-d25da7b4094f,1614226790,1663707084905417366,4,200,1024
a7a8b903-1136-4da1-97aa-d25da7b4094f,1614196724,1663707084905417366,4,200,1024
a7a8b903-1136-4da1-97aa-d25da7b4094f,1613097336,1663707084905417366,4,200,1024
a7a8b903-1136-4da1-97aa-d25da7b4094f,1609365168,1663707084905417366,4,200,1024
a7a8b903-1136-4da1-97aa-d25da7b4094f,1617199169,1663707084905417366,4,200,1024
a7a8b903-1136-4da1-97aa-d25da7b4094f,1615197134,1663707084905417366,4,200,1024
a7a8b903-1136-4da1-97aa-d25da7b4094f,1610399447,1663707084905417366,4,200,1024
...
```
#### PR checklist
- [ ] Tests written/updated, or no tests needed
- [ ] `CHANGELOG_PENDING.md` updated, or no changelog entry needed
- [ ] Updated relevant documentation (`docs/`) and code comments, or no
documentation updates needed
(cherry picked from commit 5fe1a72416)
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
the `NewClient` method is called by the load test framework for each connection. This means that if multiple connections are instantiated, each connection will erroneously have its own UUID. This PR changes the UUID generation to happen at the _beginning_ of the script instead of on client creation so that each experimental run shares a UUID.
Caught while preparing the script for production readiness.
#### PR checklist
- [ ] Tests written/updated, or no tests needed
- [ ] `CHANGELOG_PENDING.md` updated, or no changelog entry needed
- [ ] Updated relevant documentation (`docs/`) and code comments, or no
documentation updates needed
(cherry picked from commit 59a711eabe)
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
* add separated runs by UUID (#9367)
This _should_ be the last piece needed for this tool.
This allows the tool to generate reports on multiple experimental runs that may have been performed against the same chain.
The `load` tool has been updated to generate a `UUID` on startup to uniquely identify each experimental run. The `report` tool separates all of the results it reads by `UUID` and performs separate calculations for each discovered experiment.
Sample output is as follows
```
Experiment ID: 6bd7d1e8-d82c-4dbe-a1b3-40ab99e4fa30
Connections: 1
Rate: 1000
Size: 1024
Total Valid Tx: 9000
Total Negative Latencies: 0
Minimum Latency: 86.632837ms
Maximum Latency: 1.151089602s
Average Latency: 813.759361ms
Standard Deviation: 225.189977ms
Experiment ID: 453960af-6295-4282-aed6-367fc17c0de0
Connections: 1
Rate: 1000
Size: 1024
Total Valid Tx: 9000
Total Negative Latencies: 0
Minimum Latency: 79.312992ms
Maximum Latency: 1.162446243s
Average Latency: 422.755139ms
Standard Deviation: 241.832475ms
Total Invalid Tx: 0
```
closes: #9352
#### PR checklist
- [ ] Tests written/updated, or no tests needed
- [ ] `CHANGELOG_PENDING.md` updated, or no changelog entry needed
- [ ] Updated relevant documentation (`docs/`) and code comments, or no
documentation updates needed
(cherry picked from commit 1067ba1571)
# Conflicts:
# go.mod
* fix merge conflict
* fix lint
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
Co-authored-by: William Banfield <wbanfield@gmail.com>
This pull request adds the loadtime tool. This tool leverages the tm-load-test framework. Using the framework means that the only real logic that needs to be written is the logic for Tx generation. The framework does the rest.
The tool writes a set of metadata into the transaction, including the current transaction rate, number of connections, specified size of the transaction, and the current time.
This changes the ResponsePrepareProposal type, substituting the []TxRecord for just []bytes. This change is made in the .proto file and in all necessary additional places in the code.
This is largely a cherry pick of #6755 with some additional fixups added where detected.
This change moves the blockchain package to a package called blocksync. Additionally, it renames the relevant uses of the term `fastsync` to `blocksync`.
closes: #9227
#### PR checklist
- [ ] Tests written/updated, or no tests needed
- [x] `CHANGELOG_PENDING.md` updated, or no changelog entry needed
- [x] Updated relevant documentation (`docs/`) and code comments, or no
documentation updates needed
* -----start------
* [cherrypicked] state: panic on ResponsePrepareProposal validation error (#8145)
* state: panic on ResponsePrepareProposal validation error
* lint++
Co-authored-by: Sam Kleinman <garen@tychoish.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* [cherrypicked] abci++: remove CheckTx call from PrepareProposal flow (#8176)
* [cherrypicked] abci++: correct max-size check to only operate on added and unmodified (#8242)
* [cherrypicked] Remove `ModifiedTxStatus` from the spec and the code (#8210)
* Outstanding abci-gen changes to 'pb.go' files
* Removed modified_tx_status from spec and protobufs
* Fix sed for OSX
* Regenerated abci protobufs with 'abci-proto-gen'
* Code changes. UTs e2e tests passing
* Recovered UT: TestPrepareProposalModifiedTxStatusFalse
* Adapted UT
* Fixed UT
* Revert "Fix sed for OSX"
This reverts commit e576708c61.
* Update internal/state/execution_test.go
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
* Update abci/example/kvstore/kvstore.go
Co-authored-by: M. J. Fromberger <fromberger@interchain.io>
* Update internal/state/execution_test.go
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
* Update spec/abci++/abci++_tmint_expected_behavior_002_draft.md
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
* Addressed some comments
* Added one test that tests error at the ABCI client + Fixed some mock calls
* Addressed remaining comments
* Update abci/example/kvstore/kvstore.go
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
* Update abci/example/kvstore/kvstore.go
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
* Update abci/example/kvstore/kvstore.go
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
* Update spec/abci++/abci++_tmint_expected_behavior_002_draft.md
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
* Addressed William's latest comments
* Adressed Michael's comment
* Fixed UT
* Some md fixes
* More md fixes
* gofmt
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
Co-authored-by: M. J. Fromberger <fromberger@interchain.io>
* make proto-gen
* Fixed testcase on PrepareProposal error
* mockery
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
Co-authored-by: Sam Kleinman <garen@tychoish.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: M. J. Fromberger <fromberger@interchain.io>
* ----start----
* [PARTIAL cherry-pick] ABCI Vote Extension 2 (#6885)
* Cherry-picked #6567: state/types: refactor makeBlock, makeBlocks and makeTxs (#6567)
* [Cherrypicked] types: remove panic from block methods (#7501)
* [cherrypicked] abci++: synchronize PrepareProposal with the newest version of the spec (#8094)
This change implements the logic for the PrepareProposal ABCI++ method call. The main logic for creating and issuing the PrepareProposal request lives in execution.go and is tested in a set of new tests in execution_test.go. This change also updates the mempool mock to use a mockery generated version and removes much of the plumbing for the no longer used ABCIResponses.
* make proto-gen
* Backported EvidenceList's method ToABCI from #7961
* make build
* Fix mockery for Mempool
* mockery
* Backported abci Application mocks from #7961
* mockery2
* Fixed new PrepareProposal test cases in state/execution_test.go
* Fixed returned errors in consensus/state.go
* lint
* Addressed @cmwaters' comment
Co-authored-by: mconcat <monoidconcat@gmail.com>
Co-authored-by: JayT106 <JayT106@users.noreply.github.com>
Co-authored-by: Sam Kleinman <garen@tychoish.com>
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
* e2e: abci protocol should be consistent across networks (#7078)
It seems weird in retrospect that we allow networks to contain
applications that use different ABCI protocols.
(cherry picked from commit f2a8f5e054)
I realized after my last commit that my change made a following line of code a bit redundant.
(alternatively my last change was redunadnt to the existing code.)
I took this oppertunity to make some minor cleanups and logging changes to the node changes which I hope will make tests a bit more clear.
(cherry picked from commit a374f74f7c)
Co-authored-by: Sam Kleinman <garen@tychoish.com>
