* [cherrypicked] abci++: only include meaningful header fields in data passed-through to application (#8216)
* make proto-gen
* Fix kvstore tests
* Small changes to abci protobufs taken from v0.36.x
* make proto-gen (again)
* [Partial cherrypick] Restore `Commit` to the ABCI++ spec, and other late modifications (backport #8796) (#8936)
* Restore `Commit` to the ABCI++ spec, and other late modifications (#8796)
* Added VoteExtensionsEnableHeight
* Fix reference to `modified`
* Removed old pseudo-code, now included in spec
* Removed markdown warnings in abci++_basic_concepts_002_draft.md
* Restored `Commit` in the Methods section
* Addressed remaining markdown warnings
* Revisited intro and basic concepts section
* Extra pass at all spec sections to recover Commit, and other ABCI++ spec modifications
* Fixed links
* make proto-gen
* Remove _primes_ from spec notation
* Update proto/tendermint/abci/types.proto
Co-authored-by: Callum Waters <cmwaters19@gmail.com>
* Update spec/abci++/abci++_tmint_expected_behavior_002_draft.md
Co-authored-by: Callum Waters <cmwaters19@gmail.com>
* Addressed @cmwaters' comments
* Addressed @angbrav's and @mpoke's comments on spec
* make proto-gen
* Fix MD anchor reference
* Clarify throughout the spec when `ProcessProposal` and `VerifyVoteExtension` are called
* Update spec/abci++/abci++_app_requirements_002_draft.md
Co-authored-by: M. J. Fromberger <fromberger@interchain.io>
* Update spec/abci++/abci++_app_requirements_002_draft.md
Co-authored-by: M. J. Fromberger <fromberger@interchain.io>
* Update spec/abci++/abci++_app_requirements_002_draft.md
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
* Update spec/abci++/abci++_basic_concepts_002_draft.md
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
* Update spec/abci++/abci++_basic_concepts_002_draft.md
Co-authored-by: M. J. Fromberger <fromberger@interchain.io>
* Update spec/abci++/abci++_basic_concepts_002_draft.md
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
* Update spec/abci++/abci++_methods_002_draft.md
Co-authored-by: M. J. Fromberger <fromberger@interchain.io>
* Update spec/abci++/abci++_tmint_expected_behavior_002_draft.md
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
* Addresed comments
* Renamed 'draft' files
* Adatped links to new filenames
* Fixed links and minor cosmetic changes
* Renamed 'byzantine_validators' to 'misbehavior' in ABCI++ spec and protobufs
* make proto-gen
* Renamed 'byzantine_validators' to 'misbehavior' in the code
* Fixed link
* Update spec/abci++/abci++_basic_concepts.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_basic_concepts.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_basic_concepts.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_basic_concepts.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_basic_concepts.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_basic_concepts.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_basic_concepts.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_basic_concepts.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_basic_concepts.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_basic_concepts.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_basic_concepts.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_basic_concepts.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_basic_concepts.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_basic_concepts.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_basic_concepts.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_basic_concepts.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_basic_concepts.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_basic_concepts.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_basic_concepts.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_basic_concepts.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_basic_concepts.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_basic_concepts.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_methods.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_methods.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_methods.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_methods.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_methods.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_methods.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_methods.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_methods.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_methods.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_methods.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_methods.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_methods.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_methods.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_methods.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_methods.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_methods.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_methods.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_methods.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_methods.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_methods.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_methods.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_methods.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_methods.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_methods.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_methods.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_methods.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_methods.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Update spec/abci++/abci++_methods.md
Co-authored-by: Daniel <daniel.cason@usi.ch>
* Addressed @cason's comments
* Clarified conditions for `ProcessProposal` call at proposer
Co-authored-by: Callum Waters <cmwaters19@gmail.com>
Co-authored-by: M. J. Fromberger <fromberger@interchain.io>
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
Co-authored-by: Daniel <daniel.cason@usi.ch>
(cherry picked from commit 331860c2a8)
* Fixed merge conflicts
Co-authored-by: Sergio Mena <sergio@informal.systems>
* make proto-gen (and again)
* make build
* fix UTs
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* internal/proxy: add initial set of abci metrics (#7115)
This PR adds an initial set of metrics for use ABCI. The initial metrics enable the calculation of timing histograms and call counts for each of the ABCI methods. The metrics are also labeled as either 'sync' or 'async' to determine if the method call was performed using ABCI's `*Async` methods.
An example of these metrics is included here for reference:
```
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="0.0001"} 0
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="0.0004"} 5
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="0.002"} 12
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="0.009"} 13
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="0.02"} 13
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="0.1"} 13
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="0.65"} 13
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="2"} 13
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="6"} 13
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="25"} 13
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="+Inf"} 13
tendermint_abci_connection_method_timing_sum{chain_id="ci",method="commit",type="sync"} 0.007802058000000001
tendermint_abci_connection_method_timing_count{chain_id="ci",method="commit",type="sync"} 13
```
These metrics can easily be graphed using prometheus's `histogram_quantile(...)` method to pick out a particular quantile to graph or examine. I chose buckets that were somewhat of an estimate of expected range of times for ABCI operations. They start at .0001 seconds and range to 25 seconds. The hope is that this range captures enough possible times to be useful for us and operators.
* fixup
* fixups
* docs: add abci timing metrics to the metrics docs (#7311)
* format table
* -----start------
* [cherrypicked] state: panic on ResponsePrepareProposal validation error (#8145)
* state: panic on ResponsePrepareProposal validation error
* lint++
Co-authored-by: Sam Kleinman <garen@tychoish.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* [cherrypicked] abci++: remove CheckTx call from PrepareProposal flow (#8176)
* [cherrypicked] abci++: correct max-size check to only operate on added and unmodified (#8242)
* [cherrypicked] Remove `ModifiedTxStatus` from the spec and the code (#8210)
* Outstanding abci-gen changes to 'pb.go' files
* Removed modified_tx_status from spec and protobufs
* Fix sed for OSX
* Regenerated abci protobufs with 'abci-proto-gen'
* Code changes. UTs e2e tests passing
* Recovered UT: TestPrepareProposalModifiedTxStatusFalse
* Adapted UT
* Fixed UT
* Revert "Fix sed for OSX"
This reverts commit e576708c61.
* Update internal/state/execution_test.go
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
* Update abci/example/kvstore/kvstore.go
Co-authored-by: M. J. Fromberger <fromberger@interchain.io>
* Update internal/state/execution_test.go
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
* Update spec/abci++/abci++_tmint_expected_behavior_002_draft.md
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
* Addressed some comments
* Added one test that tests error at the ABCI client + Fixed some mock calls
* Addressed remaining comments
* Update abci/example/kvstore/kvstore.go
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
* Update abci/example/kvstore/kvstore.go
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
* Update abci/example/kvstore/kvstore.go
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
* Update spec/abci++/abci++_tmint_expected_behavior_002_draft.md
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
* Addressed William's latest comments
* Adressed Michael's comment
* Fixed UT
* Some md fixes
* More md fixes
* gofmt
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
Co-authored-by: M. J. Fromberger <fromberger@interchain.io>
* make proto-gen
* Fixed testcase on PrepareProposal error
* mockery
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
Co-authored-by: Sam Kleinman <garen@tychoish.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: M. J. Fromberger <fromberger@interchain.io>
* ----start----
* [PARTIAL cherry-pick] ABCI Vote Extension 2 (#6885)
* Cherry-picked #6567: state/types: refactor makeBlock, makeBlocks and makeTxs (#6567)
* [Cherrypicked] types: remove panic from block methods (#7501)
* [cherrypicked] abci++: synchronize PrepareProposal with the newest version of the spec (#8094)
This change implements the logic for the PrepareProposal ABCI++ method call. The main logic for creating and issuing the PrepareProposal request lives in execution.go and is tested in a set of new tests in execution_test.go. This change also updates the mempool mock to use a mockery generated version and removes much of the plumbing for the no longer used ABCIResponses.
* make proto-gen
* Backported EvidenceList's method ToABCI from #7961
* make build
* Fix mockery for Mempool
* mockery
* Backported abci Application mocks from #7961
* mockery2
* Fixed new PrepareProposal test cases in state/execution_test.go
* Fixed returned errors in consensus/state.go
* lint
* Addressed @cmwaters' comment
Co-authored-by: mconcat <monoidconcat@gmail.com>
Co-authored-by: JayT106 <JayT106@users.noreply.github.com>
Co-authored-by: Sam Kleinman <garen@tychoish.com>
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
* Added BlockStore.DeleteBlock()
* Added initial block pruner prototype
* wip
* Added BlockStore.PruneBlocks()
* Added consensus setting for block pruning
* Added BlockStore base
* Error on replay if base does not have blocks
* Handle missing blocks when sending VoteSetMaj23Message
* Error message tweak
* Properly update blockstore state
* Error message fix again
* blockchain: ignore peer missing blocks
* Added FIXME
* Added test for block replay with truncated history
* Handle peer base in blockchain reactor
* Improved replay error handling
* Added tests for Store.PruneBlocks()
* Fix non-RPC handling of truncated block history
* Panic on missing block meta in needProofBlock()
* Updated changelog
* Handle truncated block history in RPC layer
* Added info about earliest block in /status RPC
* Reorder height and base in blockchain reactor messages
* Updated changelog
* Fix tests
* Appease linter
* Minor review fixes
* Non-empty BlockStores should always have base > 0
* Update code to assume base > 0 invariant
* Added blockstore tests for pruning to 0
* Make sure we don't prune below the current base
* Added BlockStore.Size()
* config: added retain_blocks recommendations
* Update v1 blockchain reactor to handle blockstore base
* Added state database pruning
* Propagate errors on missing validator sets
* Comment tweaks
* Improved error message
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* use ABCI field ResponseCommit.retain_height instead of retain-blocks config option
* remove State.RetainHeight, return value instead
* fix minor issues
* rename pruneHeights() to pruneBlocks()
* noop to fix GitHub borkage
Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com>
* test functions take time.Now and other minor changes
* updated remaining test files
* Update validation_test.go
* fix typo
* go fmt
* import time
Co-authored-by: Marko <marbar3778@yahoo.com>
* evidence: introduce time.Duration to evidence params
- add time.duration to evidence
- this pr is taking pr #2606 and updating it to use both time and height
- closes#2565
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* fix testing and genesis cfg in signer harness
* remove debugging fmt
* change maxageheight to maxagenumblocks, rename other things to block instead of height
* further check of duration
* check duration to not send peers outdated evidence
* change some lines, onward and upward
* refactor evidence package
* add a changelog pending entry
* make mockbadevidence have time and use it
* add what could possibly be called a test case
* remove mockbadevidence and mockgoodevidence in favor of mockevidence
* add a comment for err that is returned
* add a changelog for removal of good & bad evidence
* add a test for adding evidence
* fix test
* add ev to types in testcase
* Update evidence/pool_test.go
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* Update evidence/pool_test.go
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* fix tests
* fix linting
Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com>
* lint: golint issue fixes
- on my local machine golint is a lot stricter than the bot so slowly going through and fixing things.
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* more fixes from golint
* remove isPeerPersistentFn
* add changelog entry
* libs/common: refactor libs/common 2
- move random function to there own pkg
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* change imports and usage throughout repo
* fix goimports
* add changelog entry
* types: change `Commit` to consist of just signatures
These are final changes towards removing votes from commit and leaving
only signatures (see ADR-25)
Fixes#1648
* bring back TestCommitToVoteSetWithVotesForAnotherBlockOrNilBlock
+ add absent flag to Vote to indicate that it's for another block
* encode nil votes as CommitSig with BlockIDFlagAbsent
+ make Commit#Precommits array of non-pointers
because precommit will never be nil
* add NewCommitSigAbsent and Absent() funcs
* uncomment validation in CommitSig#ValidateBasic
* add comments to ValidatorSet funcs
* add a changelog entry
* break instead of continue
continue does not make sense in these cases
* types: rename Commit#Precommits to Signatures
* swagger: fix /commit response
* swagger: change block_id_flag type
* fix merge conflicts
* Fix long line errors in abci, crypto, and libs packages
* Fix long lines in p2p and rpc packages
* Fix long lines in abci, state, and tools packages
* Fix long lines in behaviour and blockchain packages
* Fix long lines in cmd and config packages
* Begin fixing long lines in consensus package
* Finish fixing long lines in consensus package
* Add lll exclusion for lines containing URLs
* Fix long lines in crypto package
* Fix long lines in evidence package
* Fix long lines in mempool and node packages
* Fix long lines in libs package
* Fix long lines in lite package
* Fix new long line in node package
* Fix long lines in p2p package
* Ignore gocritic warning
* Fix long lines in privval package
* Fix long lines in rpc package
* Fix long lines in scripts package
* Fix long lines in state package
* Fix long lines in tools package
* Fix long lines in types package
* Enable lll linter
* Expose priv validators for use in testing
* Generalize block header validation test past height 1
* Remove ineffectual assignment
* Remove redundant SaveState call
* Reorder comment for clarity
* Use the block executor ApplyBlock function instead of implementing a stripped-down version of it
* Remove commented-out code
* Remove unnecessary test
The required tests already appear to be implemented (implicitly) through
the TestValidateBlockHeader test.
* Allow for catching of specific error types during TestValidateBlockCommit
* Make return error testable
* Clean up and add TestValidateBlockCommit code
* Fix formatting
* Extract function to create a new mock test app
* Update comment for clarity
* Fix comment
* Add skeleton code for evidence-related test
* Allow for addressing priv val by address
* Generalize test beyond a single validator
* Generalize TestValidateBlockEvidence past first height
* Reorder code to clearly separate tests and utility code
* Use a common constant for stop height for testing in state/validation_test.go
* Refactor errors to resemble existing conventions
* Fix formatting
* Extract common helper functions
Having the tests littered with helper functions makes them less easily
readable imho, so I've pulled them out into a separate file. This also
makes it easier to see what helper functions are available during
testing, so we minimize the chance of duplication when writing new
tests.
* Remove unused parameter
* Remove unused parameters
* Add field keys
* Remove unused height constant
* Fix typo
* Fix incorrect return error
* Add field keys
* Use separate package for tests
This refactors all of the state package's tests into a state_test
package, so as to keep any usage of the state package's internal methods
explicit.
Any internal methods/constants used by tests are now explicitly exported
in state/export_test.go
* Refactor: extract helper function to make, validate, execute and commit a block
* Rename state function to makeState
* Remove redundant constant for number of validators
* Refactor mock evidence registration into TestMain
* Remove extraneous nVals variable
* Replace function-level TODOs with file-level TODO and explanation
* Remove extraneous comment
* Fix linting issues brought up by GolangCI (pulled in from latest merge from develop)
