* [cherry-picked] abci++: add consensus parameter logic to control vote extension require height (#8547)
This PR makes vote extensions optional within Tendermint. A new ConsensusParams field, called ABCIParams.VoteExtensionsEnableHeight, has been added to toggle whether or not extensions should be enabled or disabled depending on the current height of the consensus engine. Related to: #8453
* Fix UTs
* fix blocksync reactor import of state store
* fixes1
* fixed_more_UTs
* Fix TestHandshakeReplaySome
* Fix all unit tests
* Added hunk in original commit
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
Co-authored-by: Callum Waters <cmwaters19@gmail.com>
* [cherry-picked] abci: Vote Extension 1 (#6646)
* add proto, add boilerplates
* add canonical
* fix tests
* add vote signing test
* Update internal/consensus/msgs_test.go
* modify state execution in progress
* add extension signing
* VoteExtension -> ExtendVote
* apply review
* update data structures
* Add comments
* Apply suggestions from code review
Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>
* *Signed -> *ToSign
* add Vote to RequestExtendVote
* apply reviews
* Apply suggestions from code review
Co-authored-by: Dev Ojha <ValarDragon@users.noreply.github.com>
Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>
* fix typo, modify proto
Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>
Co-authored-by: Dev Ojha <ValarDragon@users.noreply.github.com>
* [cherry-picked] ABCI Vote Extension 2 (#6885)
* add proto, add boilerplates
* add canonical
* fix tests
* add vote signing test
* Update internal/consensus/msgs_test.go
* modify state execution in progress
* add extension signing
* add extension signing
* VoteExtension -> ExtendVote
* modify state execution in progress
* add extension signing
* verify in progress
* modify CommitSig
* fix test
* apply review
* update data structures
* Apply suggestions from code review
* Add comments
* fix test
* VoteExtensionSigned => VoteExtensionToSigned
* Apply suggestions from code review
Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>
* *Signed -> *ToSign
* add Vote to RequestExtendVote
* add example VoteExtension
* apply reviews
* fix vote
* Apply suggestions from code review
Co-authored-by: Dev Ojha <ValarDragon@users.noreply.github.com>
Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>
* fix typo, modify proto
* add abcipp_kvstore.go
* add extension test
* fix test
* fix test
* fix test
* fit lint
* uncomment test
* refactor test in progress
* gofmt
* apply review
* fix lint
Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>
Co-authored-by: Dev Ojha <ValarDragon@users.noreply.github.com>
* [cheryy-picked] abci: PrepareProposal-VoteExtension integration [2nd try] (#7821)
* PrepareProposal-VoteExtension integration (#6915)
* make proto-gen
* Fix protobuf crash in e2e nightly tests
* Update types/vote.go
Co-authored-by: M. J. Fromberger <fromberger@interchain.io>
* Addressed @creachadair's comments
Co-authored-by: mconcat <monoidconcat@gmail.com>
Co-authored-by: M. J. Fromberger <fromberger@interchain.io>
* [cherry-picked] Vote extensions: new design (#8031)
* Changed the spec text to agreed VoteExtension solution
* Revert "Removed protobufs related to vote extensions"
This reverts commit 4566f1e302.
* Changes to ABCI protocol buffers
* Update spec/core/data_structures.md
Co-authored-by: M. J. Fromberger <fromberger@interchain.io>
* Update spec/core/data_structures.md
Co-authored-by: M. J. Fromberger <fromberger@interchain.io>
* Fix dangling link in ABCI++ readme
* Addressed comments
Co-authored-by: M. J. Fromberger <fromberger@interchain.io>
* [cherry-picked] abci++: Sync implementation and spec for vote extensions (#8141)
* Refactor so building and linting works
This is the first step towards implementing vote extensions: generating
the relevant proto stubs and getting the build and linter to pass.
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Fix typo
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Better describe method given vote extensions
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Fix types tests
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Move CanonicalVoteExtension to canonical types proto defs
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Regenerate protos including latest PBTS synchrony params update
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Inject vote extensions into proposal
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Thread vote extensions through code and fix tests
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Remove extraneous empty value initialization
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Fix lint
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Fix missing VerifyVoteExtension request data
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Explicitly ensure length > 0 to sign vote extension
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Explicitly ensure length > 0 to sign vote extension
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Remove extraneous comment
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Update privval/file.go
Co-authored-by: M. J. Fromberger <fromberger@interchain.io>
* Update types/vote_test.go
Co-authored-by: M. J. Fromberger <fromberger@interchain.io>
* Format
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Fix ABCI proto generation scripts for Linux
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Sync intermediate and goal protos
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Update internal/consensus/common_test.go
Co-authored-by: Sergio Mena <sergio@informal.systems>
* Use dummy value with clearer meaning
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Rewrite loop for clarity
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Panic on ABCI++ method call failure
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Add strong correctness guarantees when constructing extended commit info for ABCI++
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Add strong guarantee in extendedCommitInfo that the number of votes corresponds
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Make extendedCommitInfo function more robust
At first extendedCommitInfo expected votes to be in the same order as
their corresponding validators in the supplied CommitInfo struct, but
this proved to be rather difficult since when a validator set's loaded
from state it's first sorted by voting power and then by address.
Instead of sorting the votes in the same way, this approach simply maps
votes to their corresponding validator's address prior to constructing
the extended commit info. This way it's easy to look up the
corresponding vote and we don't need to care about vote order.
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Remove extraneous validator address assignment
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Sign over canonical vote extension
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Validate vote extension signature against canonical vote extension
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Update privval tests for more meaningful dummy value
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Add vote extension capability to E2E test app
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Disable lint for weak RNG usage for test app
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Use parseVoteExtension instead of custom parsing in PrepareProposal
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Only include extension if we have received txs
It's unclear at this point why this is necessary to ensure that the
application's local app_hash matches that committed in the previous
block.
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Require app_hash from app to match that from last block
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Add contrived (possibly flaky) test to check that vote extensions code works
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Remove workaround for problem now solved by #8229
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* add tests for vote extension cases
* Fix spelling mistake to appease linter
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Collapse redundant if statement
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Formatting
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Always expect an extension signature, regardless of whether an extension is present
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Votes constructed from commits cannot include extensions or signatures
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Pass through vote extension in test helpers
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Temporarily disable vote extension signature requirement
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Expand on vote equality test errors for clarity
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Expand on vote matching error messages in testing
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Allow for selective subscription by vote type
This is an attempt to fix the intermittently failing
`TestPrepareProposalReceivesVoteExtensions` test in the internal
consensus package.
Occasionally we get prevote messages via the subscription channel, and
we're not interested in those. This change allows us to specify what
types of votes we're interested in (i.e. precommits) and discard the
rest.
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Read lock consensus state mutex in test helper to avoid data race
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Revert BlockIDFlag parameter in node test
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Perform additional check in ProcessProposal for special txs generated by vote extensions
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* e2e: check that our added tx does not cause all txs to exceed req.MaxTxBytes
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Only set vote extension signatures when signing is successful
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Remove channel capacity constraint in test helper to avoid missing messages
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Add TODO to always require extension signatures in vote validation
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* e2e: reject vote extensions if the request height does not match what we expect
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* types: remove extraneous call to voteWithoutExtension in test
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Remove unnecessary address parameter from CanonicalVoteExtension
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* privval: change test vote type to precommit since we use an extension
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* privval: update signing logic to cater for vote extensions
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* proto: update field descriptions for vote message
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* proto: update field description for vote extension sig in vote message
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* proto/types: use fixed-length 64-bit integers for rounds in CanonicalVoteExtension
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* consensus: fix flaky TestPrepareProposalReceivesVoteExtensions
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* consensus: remove previously added test helper functionality
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* e2e: add error logs when we get an unexpected height in ExtendVote or VerifyVoteExtension requests
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* node_test: get validator addresses from privvals
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* privval/file_test: optimize filepv creation in tests
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* privval: add test to check that vote extensions are always signed
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Add a script to check documentation for ToC entries. (#8356)
This script verifies that each document in the docs and architecture directory
has a corresponding table-of-contents entry in its README file. It can be run
manually from the command line.
- Hook up this script to run in CI (optional workflow).
- Update ADR ToC to include missing entries this script found.
* build(deps): Bump async from 2.6.3 to 2.6.4 in /docs (#8357)
Bumps [async](https://github.com/caolan/async) from 2.6.3 to 2.6.4.
- [Release notes](https://github.com/caolan/async/releases)
- [Changelog](https://github.com/caolan/async/blob/v2.6.4/CHANGELOG.md)
- [Commits](https://github.com/caolan/async/compare/v2.6.3...v2.6.4)
---
updated-dependencies:
- dependency-name: async
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* privval/file_test: reset vote ext sig before signing
Signed-off-by: Thane Thomson <connect@thanethomson.com>
Co-authored-by: M. J. Fromberger <fromberger@interchain.io>
Co-authored-by: Sergio Mena <sergio@informal.systems>
Co-authored-by: William Banfield <wbanfield@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Fix cherry-picks
* make proto-gen
* make mockery
* fix build
* All units tests passing
* linter error
* Update consensus/state_test.go
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
* Addressed @williambanfield's comments
* Go, not C!
Co-authored-by: mconcat <monoidconcat@gmail.com>
Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>
Co-authored-by: Dev Ojha <ValarDragon@users.noreply.github.com>
Co-authored-by: M. J. Fromberger <fromberger@interchain.io>
Co-authored-by: Thane Thomson <connect@thanethomson.com>
Co-authored-by: William Banfield <wbanfield@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
* test-vectors for backwards compatibility:
- copy & paste test-vectors from v0.33.5 to ensure
backwards compatibility for vote's SignBytes
* WIP: everything besides time seems to match :-/
* almost
* Found the culprit: field nums weren't consecutive ints ...
* fix order of partset header too
* this last votes-related test can easily be fixed
* some minor changes and fix last failing test
* move proto types back to stdtime, fix various linting
* use libs/protoio
* remvoe commented code
* add comments
* fix tests
* uncomment testscases
* dont ignore error panic
* fix signable test
* fix happy path testing
* fix comment
Co-authored-by: Marko Baricevic <marbar3778@yahoo.com>
LastCommitRound should always be >= 0 for heights > 1.
In State.updateToState, last precommit is computed only when round
greater than -1 and has votes. But "LastCommit" is always updated
regardless of the condition. If there's no last precommit, "LastCommit"
is set to (*types.VoteSet)(nil). That's why "LastCommit" can be -1 for
heights > 1.
To fix it, only update State.RoundState.LastCommit when there is last
precommit.
Fixes#2737
Co-authored-by: Cuong Manh Le <cuong.manhle.vn@gmail.com>
* format: add format cmd & goimport repo
- replaced format command
- added goimports to format command
- ran goimports
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* fix outliers & undo proto file changes
* lint: golint issue fixes
- on my local machine golint is a lot stricter than the bot so slowly going through and fixing things.
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* more fixes from golint
* remove isPeerPersistentFn
* add changelog entry
* libs/common: Refactor libs/common 5
- move mathematical functions and types out of `libs/common` to math pkg
- move net functions out of `libs/common` to net pkg
- move string functions out of `libs/common` to strings pkg
- move async functions out of `libs/common` to async pkg
- move bit functions out of `libs/common` to bits pkg
- move cmap functions out of `libs/common` to cmap pkg
- move os functions out of `libs/common` to os pkg
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* fix testing issues
* fix tests
closes#41417
woooooooooooooooooo kill the cmn pkg
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* add changelog entry
* fix goimport issues
* run gofmt
* libs/common: Refactor libs/common 4
- move byte function out of cmn to its own pkg
- move tempfile out of cmn to its own pkg
- move throttletimer to its own pkg
ref #4147
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* add changelog entry
* fix linting issues
* libs/common: refactor libs/common 2
- move random function to there own pkg
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* change imports and usage throughout repo
* fix goimports
* add changelog entry
* types: change `Commit` to consist of just signatures
These are final changes towards removing votes from commit and leaving
only signatures (see ADR-25)
Fixes#1648
* bring back TestCommitToVoteSetWithVotesForAnotherBlockOrNilBlock
+ add absent flag to Vote to indicate that it's for another block
* encode nil votes as CommitSig with BlockIDFlagAbsent
+ make Commit#Precommits array of non-pointers
because precommit will never be nil
* add NewCommitSigAbsent and Absent() funcs
* uncomment validation in CommitSig#ValidateBasic
* add comments to ValidatorSet funcs
* add a changelog entry
* break instead of continue
continue does not make sense in these cases
* types: rename Commit#Precommits to Signatures
* swagger: fix /commit response
* swagger: change block_id_flag type
* fix merge conflicts
* Fix long line errors in abci, crypto, and libs packages
* Fix long lines in p2p and rpc packages
* Fix long lines in abci, state, and tools packages
* Fix long lines in behaviour and blockchain packages
* Fix long lines in cmd and config packages
* Begin fixing long lines in consensus package
* Finish fixing long lines in consensus package
* Add lll exclusion for lines containing URLs
* Fix long lines in crypto package
* Fix long lines in evidence package
* Fix long lines in mempool and node packages
* Fix long lines in libs package
* Fix long lines in lite package
* Fix new long line in node package
* Fix long lines in p2p package
* Ignore gocritic warning
* Fix long lines in privval package
* Fix long lines in rpc package
* Fix long lines in scripts package
* Fix long lines in state package
* Fix long lines in tools package
* Fix long lines in types package
* Enable lll linter
* New lint version upgrade
- linter was upgraded
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* enable-a;; is deprecated
* minor change
* another try
* some more changes
* some more changes
* reenable prealloc
* add version till bot is fixed
* Renamed wire.go to codec.go
- Wire was the previous name of amino
- Codec describes the file better than `wire` & `amino`
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* ide error
* rename amino.go to codec.go
* Remove deprecated PanicXXX functions from codebase
As per discussion over
[here](https://github.com/tendermint/tendermint/pull/3456#discussion_r278423492),
we need to remove these `PanicXXX` functions and eliminate our
dependence on them. In this PR, each and every `PanicXXX` function call
is replaced with a simple `panic` call.
* add a changelog entry
Before we're using it to get a round state in tests. Now it can be done
by calling csX.GetRoundState. We will need to rewrite
TestStateSlashingPrevotes and TestStateSlashingPrecommits, which are
commented right now, to not rely on EventDataRoundState#RoundState
field.
Refs #1527
* improve ResetTestRootWithChainID() concurrency safety
Rely on ioutil.TempDir() to create test root directories and ensure
multiple same-chain id test cases can run in parallel.
* Update config/toml.go
Co-Authored-By: alessio <quadrispro@ubuntu.com>
* clean up test directories after completion
Closes: #1034
* Remove redundant EnsureDir call
* s/PanicSafety()/panic()/s
* Put create dir functionality back in ResetTestRootWithChainID
* Place test directories in OS's tempdir
In modern UNIX and UNIX-like systems /tmp is very often
mounted as tmpfs. This might speed test execution a bit.
* Set 0700 to a const
* rootsDirs -> configRootDirs
* Don't double remove directories
* Avoid global variables
* Fix consensus tests
* Reduce defer stack
* Address review comments
* Try to fix tests
* Update CHANGELOG_PENDING.md
Co-Authored-By: alessio <quadrispro@ubuntu.com>
* Update consensus/common_test.go
Co-Authored-By: alessio <quadrispro@ubuntu.com>
* Update consensus/common_test.go
Co-Authored-By: alessio <quadrispro@ubuntu.com>
* types.NewCommit
* use types.NewCommit everywhere
* fix log in unsafe_reset
* memoize height and round in constructor
* notes about deprecating toVote
* bring back memoizeHeightRound
* types: memoize height/round in commit instead of first vote
* types: commit.ValidateBasic in VerifyCommit
* types: new CommitSig alias for Vote
In preparation for reducing the redundancy in Commits, we introduce the
CommitSig as an alias for Vote. This is non-breaking on the protocol,
and minor breaking on the Go API, as Commit now contains a list of
CommitSig instead of Vote.
* remove dependence on ToVote
* update some comments
* fix tests
* fix tests
* fixes from review
* add proposer info to EventCompleteProposal
* create separate EventData structure for CompleteProposal
* cant us rs.Proposal to get BlockID because it is not guaranteed to be set yet
* copying RoundState isnt helping us here
* add Step back to make compatible with original RoundState event. update changelog
* add NewRound event
* fix test
* remove unneeded RoundState
* put height round step into a struct
* pull out ValidatorInfo struct. add ensureProposal assert
* remove height-round-state sub-struct refactor
* minor fixes from review
* validate reactor messages
Refs #2683
* validate blockchain messages
Refs #2683
* validate evidence messages
Refs #2683
* todo
* check ProposalPOL and signature sizes
* add a changelog entry
* check addr is valid when we add it to the addrbook
* validate incoming netAddr (not just nil check!)
* fixes after Bucky's review
* check timestamps
* beef up block#ValidateBasic
* move some checks into bcBlockResponseMessage
* update Gopkg.lock
Fix
```
grouped write of manifest, lock and vendor: failed to export github.com/tendermint/go-amino: fatal: failed to unpack tree object 6dcc6ddc143e116455c94b25c1004c99e0d0ca12
```
by running `dep ensure -update`
* bump year since now we check it
* generate test/p2p/data on the fly using tendermint testnet
* allow sync chains older than 1 year
* use full path when creating a testnet
* move testnet gen to test/docker/Dockerfile
* relax LastCommitRound check
Refs #2737
* fix conflicts after merge
* add small comment
* some ValidateBasic updates
* fixes
* AppHash length is not fixed
* WIP: switching to fixed offsets for SignBytes
* add version field to sign bytes and update order
* more comments on test-cases and add a tc with a chainID
* remove amino:"write_empty" tag
- it doesn't affect if default fixed size fields ((u)int64) are
written or not
- add comment about int->int64 casting
* update CHANGELOG_PENDING
* update documentation
* add back link to issue #1622 in documentation
* remove JSON tags and add (failing test-case)
* fix failing test
* update test-vectors due to added `Type` field
* change Type field from string to byte and add new type alias
- SignedMsgType replaces VoteTypePrevote, VoteTypePrecommit and adds new
ProposalType to separate votes from proposal when signed
- update test-vectors
* fix remains from rebasing
* use SignMessageType instead of byte everywhere
* fixes from review
