449 Commits

Author SHA1 Message Date
Mark Rushakoff
d433ebe68d Improve handling of -short flag in tests (#9075)
As a small developer quality of life improvement, I found many individual unit tests that take longer than around a second to complete, and set them to skip when run under `go test -short`.

On my machine, the wall timings for tests (with `go test -count=1 ./...` and optionally `-short` and `-race`) are roughly:

- Long tests, no race detector: about 1m42s
- Short tests, no race detector: about 17s
- Long tests, race detector enabled: about 2m1s
- Short tests, race detector enabled: about 28s

This PR is split into many commits each touching a single package, with commit messages detailing the approximate timing change per package.
2022-07-29 13:41:54 +00:00
Sam Kleinman
48147e1fb9 logging: implement lazy sprinting (#8898)
shout out to @joeabbey for the inspiration. This makes the lazy
functions internal by default to prevent potential misuse by external
callers.

Should backport cleanly into 0.36 and I'll handle a messy merge into 0.35
2022-07-27 19:16:51 +00:00
M. J. Fromberger
fa32078ceb Fix unbounded heap growth in the priority mempool. (#9052)
This is a manual forward-port of #8944 and related fixes from v0.35.x.

One difference of note is that the CheckTx response messages no longer have a
field to record an error from the ABCI application. The code is set up so that
these could be reported directly to the CheckTx caller, but it would be an API
change, and right now a bunch of test plumbing depends on the existing semantics.

Also fix up tests relying on implementation-specific mempool behavior.

- Commit was setting the expected mempool size incorrectly.
- Fix sequence test not to depend on the incorrect size.
2022-07-20 14:37:13 -07:00
M. J. Fromberger
18b5a500da Extract a library from the confix command-line tool. (#9012)
Pull out the library functionality from scripts/confix and move it to
internal/libs/confix. Replace scripts/confix with a simple stub that has the
same command-line API, but uses the library instead.

Related:

- Move and update unit tests.
- Move scripts/confix/condiff to scripts/condiff.
- Update test data for v34, v35, and v36.
- Update reference diffs.
- Update testdata README.
2022-07-15 07:29:34 -07:00
Sam Kleinman
d5fb82e414 p2p: make p2p.Channel an interface (#8967)
This is (#8446) pulled from the `main/libp2p` branch but without any
of the libp2p content, and is perhaps the easiest first step to enable
pluggability at the peer layer, and makes it possible hoist shims
(including for, say 0.34) into tendermint without touching the reactors.
2022-07-11 20:22:40 +00:00
Sam Kleinman
61ce384d75 p2p: make peer gossiping coinflip safer (#8949)
Closes #8948
2022-07-08 14:06:57 +00:00
Sam Kleinman
636320f901 p2p: delete cruft (#8958)
I think the decision in #8806 is that we shouldn't do this yet, so I think it's best to just drop this.
2022-07-07 16:29:50 +00:00
Sam Kleinman
d1a16e8ff0 p2p: simpler priority queue (#8929) 2022-07-07 12:13:52 -04:00
Callum Waters
27c523dccb mempool: return error when mempool is full and inbound tx is rejected (#8942)
Addresses: https://github.com/tendermint/tendermint/issues/8928
2022-07-06 18:17:56 +00:00
yihuang
be6d74e657 Work around indexing problem for duplicate transactions (forward port: #8625) (#8945)
Port the bug fix terra-money#76 to upstream. This is critical for ethermint json-rpc to work.

fix: prevent duplicate tx index if it succeeded before
fix: use CodeTypeOk instead of 0
fix: handle duplicate txs within the same block
Co-authored-by: jess jesse@soob.co

ref: #5281

Co-authored-by: M. J. Fromberger <fromberger@interchain.io>
2022-07-06 14:05:48 -04:00
Sergio Mena
331860c2a8 Restore Commit to the ABCI++ spec, and other late modifications (#8796)
* Added VoteExtensionsEnableHeight

* Fix reference to `modified`

* Removed old pseudo-code, now included in spec

* Removed markdown warnings in abci++_basic_concepts_002_draft.md

* Restored `Commit` in the Methods section

* Addressed remaining markdown warnings

* Revisited intro and basic concepts section

* Extra pass at all spec sections to recover Commit, and other ABCI++ spec modifications

* Fixed links

* make proto-gen

* Remove _primes_ from spec notation

* Update proto/tendermint/abci/types.proto

Co-authored-by: Callum Waters <cmwaters19@gmail.com>

* Update spec/abci++/abci++_tmint_expected_behavior_002_draft.md

Co-authored-by: Callum Waters <cmwaters19@gmail.com>

* Addressed @cmwaters' comments

* Addressed @angbrav's and @mpoke's comments on spec

* make proto-gen

* Fix MD anchor reference

* Clarify throughout the spec when `ProcessProposal` and `VerifyVoteExtension` are called

* Update spec/abci++/abci++_app_requirements_002_draft.md

Co-authored-by: M. J. Fromberger <fromberger@interchain.io>

* Update spec/abci++/abci++_app_requirements_002_draft.md

Co-authored-by: M. J. Fromberger <fromberger@interchain.io>

* Update spec/abci++/abci++_app_requirements_002_draft.md

Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>

* Update spec/abci++/abci++_basic_concepts_002_draft.md

Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>

* Update spec/abci++/abci++_basic_concepts_002_draft.md

Co-authored-by: M. J. Fromberger <fromberger@interchain.io>

* Update spec/abci++/abci++_basic_concepts_002_draft.md

Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>

* Update spec/abci++/abci++_methods_002_draft.md

Co-authored-by: M. J. Fromberger <fromberger@interchain.io>

* Update spec/abci++/abci++_tmint_expected_behavior_002_draft.md

Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>

* Addresed comments

* Renamed 'draft' files

* Adatped links to new filenames

* Fixed links and minor cosmetic changes

* Renamed 'byzantine_validators' to 'misbehavior' in ABCI++ spec and protobufs

* make proto-gen

* Renamed 'byzantine_validators' to 'misbehavior' in the code

* Fixed link

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <daniel.cason@usi.ch>

* Addressed @cason's comments

* Clarified conditions for `ProcessProposal` call at proposer

Co-authored-by: Callum Waters <cmwaters19@gmail.com>
Co-authored-by: M. J. Fromberger <fromberger@interchain.io>
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
Co-authored-by: Daniel <daniel.cason@usi.ch>
2022-07-04 16:25:48 +02:00
Ian Jungyong Um
1d96faa35a p2p: fix typo (#8922)
Fix minor typos in peermanager
2022-07-01 12:44:39 +00:00
William Banfield
921530c352 p2p: use correct context error (#8916)
handshakeCtx is the internal context carrying the timeout. Its error should be used for the error return.
2022-06-30 22:02:59 +00:00
William Banfield
5274f80de4 p2p: fix flakey test due to disconnect cooldown (#8917)
This test was made flakey by #8839. The cooldown period means that the node in the test will not try to reconnect as quickly as the test expects. This change makes the cooldown shorter in the test so that the node quickly reconnects.
2022-06-30 21:48:10 +00:00
Sam Kleinman
60881f1d06 p2p: stop mconn channel sends without timeout (#8906) 2022-06-30 09:01:02 -04:00
Sam Kleinman
013b46a6c3 libs/strings: move to internal (#8890)
I think we were leaving this library public because the SDK dependend
upon it, but the function the SDK was using was one that we'd removed
because *we* weren't using it any more, and I made a PR agasint the
SDK to clean that up.

ref: https://github.com/cosmos/cosmos-sdk/pull/12368
2022-06-27 21:44:50 +00:00
Sam Kleinman
52b6dc19ba p2p: remove dial sleep and provide disconnect cooldown (#8839)
Alternative proposal for #8826
2022-06-24 17:57:49 +00:00
William Banfield
5f5e74798b p2p: set empty timeouts to small values. (#8847)
These timeouts default to 'do not time out' if they are not set. This times up resources, potentially indefinitely. If node on the other side of the the handshake is up but unresponsive, the[ handshake call](edec79448a/internal/p2p/router.go (L720)) will _never_ return.

These are proposed values that have not been validated. I intend to validate them in a production setting.
2022-06-23 22:33:21 +00:00
Sam Kleinman
436a38f876 p2p: track peers by address (#8841) 2022-06-23 10:03:10 -04:00
Ian Jungyong Um
2e11760fbe p2p: fix typo (#8836) 2022-06-22 09:30:11 +02:00
William Banfield
8860e027a8 p2p: more dial routines (#8827)
The dial routines perform network i/o, which is a blocking call into the kernel. These routines are completely unable to do anything else while the dial occurs, so for most of their lifecycle they are sitting idle waiting for the tcp stack to hand them data. We should increase this value by _a lot_ to enable more concurrent dials. This is unlikely to cause CPU starvation because these routines sit idle most of the time. The current value causes dials to occur _way_ too slowly. 

Below is a graph demonstrating the before and after of this change in a testnetwork with many dead peers. You can observe that the rate that we connect to new, valid peers, is _much_ higher than previously. Change was deployed around the 31 minute mark on the graph.

![image](https://user-images.githubusercontent.com/4561443/174919007-50e4453a-edd8-41d0-97ee-dea8853d57f7.png)
2022-06-22 00:51:09 +00:00
Sam Kleinman
cfd13825e2 p2p: add eviction metrics and cleanup dialing error handling (#8819) 2022-06-21 20:44:14 +00:00
Sam Kleinman
28d3239958 p2p: wake dialing thread after sleep (#8803) 2022-06-20 15:47:56 +00:00
Ian Jungyong Um
e3e162ff10 p2p: fix typo (#8793)
Fix the typo in router
2022-06-19 15:26:25 +00:00
Sam Kleinman
4d820ff4f5 p2p: peer score should not wrap around (#8790) 2022-06-17 22:27:38 +00:00
Sam Kleinman
0ac03468d8 p2p: track peers stored on startup (#8787) 2022-06-17 18:10:10 +00:00
Sam Kleinman
9e5b13725d p2p: peer store and dialing changes (#8737) 2022-06-17 08:02:10 -04:00
Sergio Mena
a4f29bfd44 Don't check PBTS-timeliness when in replay mode (#8774)
Closes #8781

Temporary fix to this issue, so that e2e tests don't fail and potentially leave other problems uncovered.
2022-06-16 19:31:17 +00:00
Sam Kleinman
51b3f111dc p2p: fix mconn transport accept test (#8762)
Fix minor test incongruency missed earlier.
2022-06-14 23:48:48 +00:00
Sam Kleinman
979a6a1b13 p2p: accept should not abort on first error (#8759) 2022-06-14 19:12:53 -04:00
Sam Kleinman
bf1cb89bb7 Revert "p2p: self-add node should not error (tendermint#8753)" (#8757) 2022-06-14 20:55:10 +00:00
Sam Kleinman
7971f4a2fc p2p: self-add node should not error (#8753) 2022-06-14 16:45:05 +00:00
Jeeyong Um
a4cf8939b8 mempool: fix error message check in test (#8750) 2022-06-14 06:54:35 -04:00
Jeeyong Um
21bbbe3e2a mempool: fix typos in test (#8746) 2022-06-14 10:50:55 +02:00
Marko
82907c84fa sink/psql: json marshal instead of proto (#8637)
Storing transaction records as JSON makes it simpler for clients of the index.
2022-06-13 10:20:54 -07:00
William Banfield
75a12ea0c6 consensus: switch timeout message to be debug and clarify meaning (#8694) 2022-06-03 13:46:10 -04:00
Sergio Mena
ce6485fa70 Remove the abci responses type - prune legacy responses (#8673)
Closes #8069 

* Type `ABCIResponses` was just wrapping type `ResponseFinalizeBlock`. This patch removes the former.
* Did some renaming to avoid confusion on the data structure we are working with.
* We also remove any stale ABCIResponses we may have in the state store at the time of pruning

**IMPORTANT**: There is an undesirable side-effect of the unwrapping. An empty `ResponseFinalizeBlock` yields a 0-length proto-buf serialized buffer. This was not the case with `ABCIResponses`. I have added an interim solution, but open for suggestions on more elegant ones.
2022-06-02 19:13:08 +00:00
Sam Kleinman
666d93338a p2p: shed peers from store from other networks (#8678) 2022-06-02 11:14:25 -04:00
Sergio Mena
56fc80d66d abci: Move app_hash parameter from Commit to FinalizeBlock (#8664)
* Removed from proto

* make proto-gen

* make build works

* make some tests pass

* Fix TestMempoolTxConcurrentWithCommit

* Minor change

* Update abci/types/types.go

* Update internal/state/execution.go

* Update test/e2e/app/state.go

Co-authored-by: Callum Waters <cmwaters19@gmail.com>

* Updated changelog and `UPGRADING.md`

* Fixed abci-cli tests, and doc

* Addressed @cmwaters' comments

* Addressed @cmwaters' comments, part 2

Co-authored-by: Callum Waters <cmwaters19@gmail.com>
2022-06-01 18:53:10 +02:00
Sam Kleinman
d2ca0b868d consensus: gossip catchup sleeping protocol (#8652) 2022-06-01 10:17:17 +00:00
Sam Kleinman
bc24ae4642 rpc: deprecate/updates to broadcast tx (#8624) 2022-05-26 08:28:28 -04:00
Jasmina Malicevic
cb9722c2b0 abci: strip mempoolerror from responsechectx (#8620)
* abci:mempoolError from ResponseCheckTx
* responseCheckTx returns an error if Tendermint decides not to accept an app after CheckTx
*updated spec, upgrading.md and changelog.md
2022-05-26 11:18:27 +02:00
Callum Waters
b0ec8a0ea7 mempool: migrate rechecktx to be a consensus parameter (#8514) 2022-05-25 23:57:23 +02:00
Jasmina Malicevic
4c857a7ed2 abci: remove unused fields from CheckTXResponse (part 1) (#8605)
abci: Removed Info, Log, Events and GasUsed from ResponseCheckTx. 
spec: Updated info on ResponseCheckTx to reflect field removal.
2022-05-25 23:06:16 +02:00
M. J. Fromberger
a988cefe5d Update generated mocks after #8607. (#8612) 2022-05-25 15:48:56 +00:00
Sam Kleinman
d59a53be01 p2p: reduce ability of SendError to disconnect peers (#8597) 2022-05-24 11:19:32 -04:00
William Banfield
43313e9b85 abci++: add proto fields for enabling vote extensions (#8587)
This pull requests adds the protocol buffer field for the `ABCI.VoteExtensionsEnableHeight` parameter. This proto field is threaded throughout all of the relevant places where consensus params are used and referenced.

This PR also adds validation of the consensus param updates. Previous consensus param changes didn't depend on _previous_ versions of the params, so this change adds a method for validating against the old params as well.

closes: #8453
2022-05-23 18:23:23 +00:00
William Banfield
0cceadf4d4 abci++: add consensus parameter logic to control vote extension require height (#8547)
This PR makes vote extensions optional within Tendermint. A new ConsensusParams field, called ABCIParams.VoteExtensionsEnableHeight, has been added to toggle whether or not extensions should be enabled or disabled depending on the current height of the consensus engine. Related to: #8453
2022-05-20 17:46:52 -04:00
William Banfield
ad73e6da2f consensus: update state from store before use in reactor (#8576)
Closes: #8575

This PR aims to fix the `LastCommitRound can only be negative for initial height 0` issue we see in the e2e tests by initializing the `state` object before starting the receive routines in the consensus reactor. This is somewhat inelegant, but it should fix the issue.
2022-05-19 19:35:30 +00:00
M. J. Fromberger
4a9bbe047f Fix lock sequencing in socket client request tracking. (#8581)
* Fix lock sequencing in socket client request tracking.

It is not safe to check base service state (IsRunning) while holding the lock
for the client state. If we do, then during shutdown we may deadlock with the
invocation of the OnStop handler, which the base service executes while holding
the service lock.

* Enqueue pending requests before sending them to the server.

If we don't do this, the server can reply before the request lands in the
queue.  That will cause the receiver to terminate early for an unsolicited
response. So enqueue first: This is safe because we're doing it in the same
routine as services the channel, so we won't take another message till we are
safely past that point.

* Document what we did.
* Fix socket paths in tests.
2022-05-19 12:11:57 -07:00