* ABCI++ RFC
This commit adds an RFC for ABCI++, which is a collection of three new phases of communication between the consensus engine and the application.
Co-authored-by: Sunny Aggarwal <sunnya97@protonmail.ch>
* Fix bugs pointed out by @liamsi
* Update rfc/004-abci++.md
Co-authored-by: Federico Kunze <31522760+fedekunze@users.noreply.github.com>
* Fix markdown lints
* Update rfc/004-abci++.md
Co-authored-by: Ismail Khoffi <Ismail.Khoffi@gmail.com>
* Update rfc/004-abci++.md
Co-authored-by: Tess Rinearson <tess.rinearson@gmail.com>
* Update rfc/004-abci++.md
Co-authored-by: Tess Rinearson <tess.rinearson@gmail.com>
* Add information about the rename in the context section
* Bold RFC
* Add example for self-authenticating vote data
* More exposition of the term IPC
* Update pros / negatives
* Fix sentence fragment
* Add desc for no-ops
Co-authored-by: Sunny Aggarwal <sunnya97@protonmail.ch>
Co-authored-by: Federico Kunze <31522760+fedekunze@users.noreply.github.com>
Co-authored-by: Ismail Khoffi <Ismail.Khoffi@gmail.com>
Co-authored-by: Tess Rinearson <tess.rinearson@gmail.com>
Closes#1581
This fixes the error in #1581, and also documents the purpose of this line. It ensures that if a peer tells us an address we know about, whose ID is the same as our current ID, we ignore it.
This removes the previous case where the ID's matched, but the IP's did not, which could yield a potential overwrite of the IP associated with the address later on. (This then would yield an eclipse attack)
This was not a vulnerability before though, thanks to a defensive check here https://github.com/tendermint/tendermint/blob/95fc7e58eedd3b0a563b93a0691c9cbe6e10f017/p2p/pex/addrbook.go#L522)
Closes#1798
This is done by making every mempool tx maintain a list of peers who its received the tx from. Instead of using the 20byte peer ID, it instead uses a local map from peerID to uint16 counter, so every peer adds 2 bytes. (Word aligned to probably make it 8 bytes)
This also required resetting the callback function on every CheckTx. This likely has performance ramifications for instruction caching. The actual setting operation isn't costly with the removal of defers in this PR.
* Make the mempool not gossip txs back to peers its received it from
* Fix adversarial memleak
* Don't break interface
* Update changelog
* Forgot to add a mtx
* forgot a mutex
* Update mempool/reactor.go
Co-Authored-By: ValarDragon <ValarDragon@users.noreply.github.com>
* Update mempool/mempool.go
Co-Authored-By: ValarDragon <ValarDragon@users.noreply.github.com>
* Use unknown peer ID
Co-Authored-By: ValarDragon <ValarDragon@users.noreply.github.com>
* fix compilation
* use next wait chan logic when skipping
* Minor fixes
* Add TxInfo
* Add reverse map
* Make activeID's auto-reserve 0
* 0 -> UnknownPeerID
Co-Authored-By: ValarDragon <ValarDragon@users.noreply.github.com>
* Switch to making the normal case set a callback on the reqres object
The recheck case is still done via the global callback, and stats
are also set via global callback
* fix merge conflict
* Addres comments
* Add cache tests
* add cache tests
* minor fixes
* update metrics in reqResCb and reformat code
* goimport -w mempool/reactor.go
* mempool: update memTx senders
I had to introduce txsMap for quick mempoolTx lookups.
* change senders type from []uint16 to sync.Map
Fixes DATA RACE:
```
Read at 0x00c0013fcd3a by goroutine 183:
github.com/tendermint/tendermint/mempool.(*MempoolReactor).broadcastTxRoutine()
/go/src/github.com/tendermint/tendermint/mempool/reactor.go:195 +0x3c7
Previous write at 0x00c0013fcd3a by D[2019-02-27|10:10:49.058] Read PacketMsg switch=3 peer=35bc1e3558c182927b31987eeff3feb3d58a0fc5@127.0.0.1
:46552 conn=MConn{pipe} packet="PacketMsg{30:2B06579D0A143EB78F3D3299DE8213A51D4E11FB05ACE4D6A14F T:1}"
goroutine 190:
github.com/tendermint/tendermint/mempool.(*Mempool).CheckTxWithInfo()
/go/src/github.com/tendermint/tendermint/mempool/mempool.go:387 +0xdc1
github.com/tendermint/tendermint/mempool.(*MempoolReactor).Receive()
/go/src/github.com/tendermint/tendermint/mempool/reactor.go:134 +0xb04
github.com/tendermint/tendermint/p2p.createMConnection.func1()
/go/src/github.com/tendermint/tendermint/p2p/peer.go:374 +0x25b
github.com/tendermint/tendermint/p2p/conn.(*MConnection).recvRoutine()
/go/src/github.com/tendermint/tendermint/p2p/conn/connection.go:599 +0xcce
Goroutine 183 (running) created at:
D[2019-02-27|10:10:49.058] Send switch=2 peer=1efafad5443abeea4b7a8155218e4369525d987e@127.0.0.1:46193 channel=48 conn=MConn{pipe} m
sgBytes=2B06579D0A146194480ADAE00C2836ED7125FEE65C1D9DD51049
github.com/tendermint/tendermint/mempool.(*MempoolReactor).AddPeer()
/go/src/github.com/tendermint/tendermint/mempool/reactor.go:105 +0x1b1
github.com/tendermint/tendermint/p2p.(*Switch).startInitPeer()
/go/src/github.com/tendermint/tendermint/p2p/switch.go:683 +0x13b
github.com/tendermint/tendermint/p2p.(*Switch).addPeer()
/go/src/github.com/tendermint/tendermint/p2p/switch.go:650 +0x585
github.com/tendermint/tendermint/p2p.(*Switch).addPeerWithConnection()
/go/src/github.com/tendermint/tendermint/p2p/test_util.go:145 +0x939
github.com/tendermint/tendermint/p2p.Connect2Switches.func2()
/go/src/github.com/tendermint/tendermint/p2p/test_util.go:109 +0x50
I[2019-02-27|10:10:49.058] Added good transaction validator=0 tx=43B4D1F0F03460BD262835C4AA560DB860CFBBE85BD02386D83DAC38C67B3AD7 res="&{CheckTx:gas_w
anted:1 }" height=0 total=375
Goroutine 190 (running) created at:
github.com/tendermint/tendermint/p2p/conn.(*MConnection).OnStart()
/go/src/github.com/tendermint/tendermint/p2p/conn/connection.go:210 +0x313
github.com/tendermint/tendermint/libs/common.(*BaseService).Start()
/go/src/github.com/tendermint/tendermint/libs/common/service.go:139 +0x4df
github.com/tendermint/tendermint/p2p.(*peer).OnStart()
/go/src/github.com/tendermint/tendermint/p2p/peer.go:179 +0x56
github.com/tendermint/tendermint/libs/common.(*BaseService).Start()
/go/src/github.com/tendermint/tendermint/libs/common/service.go:139 +0x4df
github.com/tendermint/tendermint/p2p.(*peer).Start()
<autogenerated>:1 +0x43
github.com/tendermint/tendermint/p2p.(*Switch).startInitPeer()
```
* explain the choice of a map DS for senders
* extract ids pool/mapper to a separate struct
* fix literal copies lock value from senders: sync.Map contains sync.Mutex
* use sync.Map#LoadOrStore instead of Load
* fixes after Ismail's review
* rename resCbNormal to resCbFirstTime
* Begin simple merkle compatibility PR
* Fix query_test
* Use trillian test vectors
* Change the split point per RFC 6962
* update spec
* refactor innerhash to match spec
* Update changelog
* Address @liamsi's comments
* Write the comment requested by @liamsi
* Enforce validators can only use the correct pubkey type
* adapt to variable renames
* Address comments from #2636
* separate updating and validation logic
* update spec
* Add test case for TestStringSliceEqual, clarify slice copying code
* Address @ebuchman's comments
* Split up testing validator update execution, and its validation
Within every tx in the mempool, we store a 64 bit counter,
as an index for when it was inserted into the mempool.
This counter doesn't really serve any purpose.
It was likely added for debugging at one point,
Removing the counter reclaims memory,
which enables greater mempool sizes / mitigates resources at the same size.
Closes#2835
This is a performance regression, but will also spare the types directory
from knowing about RFC 6962, which is a more correct abstraction. For txs
this performance hit will be fixed soon with #2603. For evidence, the
performance impact is negligible due to it being capped at a small number.
* crypto/merkle: Remove byter in favor of plain byte slices
This PR is fully backwards compatible in terms of function output!
(The Go API differs though) The only test case changes was to refactor
it to be table driven.
* Update godocs per review comments
* Switch nodeID to be tmhash.Size, add test names for net addr tests
Both of these came up when locally trying to change tmhash size.
* fix error introduced by merge
* Fix random distribution in bitArray.PickRandom
Previously it was very biased. 63 "_" followed by a single "x" had
much greater odds of being chosen. Additionally, the last element was
skewed. This fixes that by first preproccessing the set of all true
indices, and then randomly selecting a single element from there.
This commit also makes the code here significantly simpler, and
improves test cases.
* unlock mtx right after we select true indices
This also refactors the prior mempool to filter to be known as
"precheck filter" and this new filter is called "postcheck filter"
This PR also fixes a bug where the precheck filter previously didn't
account for the amino overhead, which could a maliciously sized tx to
halt blocks from getting any txs in them.
* Move maxGas outside of function definition to avoid race condition
* Type filter funcs and make public
* Use helper method for post check
* Remove superfluous Filter suffix
* Move default pre/post checks into package
* Fix broken references
* Fix typos
* Expand on examples for checks
These functions were not used anywhere within tendermint, or the
cosmos-sdk. (The functionality is already duplicated in the cosmos-sdk
types package)
* common: Delete unused functions within byteslice
* remove more unused code from strings.go and int.go
* Remove more unused code from int.go
* Fix testcase
We didn't use this code anywhere in the codebase. As such, we probably
should reduce the surface area we support. In the event that we do
in fact require 256 bit words inside of tendermint, we should adapt
the stdlibs' internal word representations, which also handles SIMD.
Inside of the SDK, a separate solution for big ints / larger words
is employed, which uses big ints. This in turn does utilize the stdlibs
SIMD support.
* Make mempool aware of MaxGas requirement
* update spec
* Add tests
* Switch GasWanted from kv store to persistent kv store
* Fix typo in test name
* switch back to using kvstore, not persistent kv store
* config: reduce default mempool size
This reduces the mempool size from 100k to 5k. Note that each secp256k1 sig
takes .5ms to compute. Therefore an adversary could previously delay every
node on the network's computation time upon receiving a block by 50 seconds.
This now reduces that ability to being able to only delay each node by 2.5
seconds. This change should be reverted once ABCI recheck is implemented.
* (squash this) fix test
* mempool: Store txs by hash inside of cache
This allows for large cachesizes, without fear of the memory
consumption growing rapidly.
* (squash this) rename hashedTx -> txHash
* crypto/secp256k1: Fix signature malleability, adopt more efficient encoding
This removes signature malleability per ADR 14, and makes secp match
the encoding in ADR 15.
* (squash this) add lock
* clist: Speedup detachNext() and detachPrev()
We used unnecessary function calls, defers, and extra mutexes.
These are not our friends for writing fast code in our libs.
* Remove more defers from clist functions
* Add more benchmarks
* mempool: Keep cache hashmap and linked list in sync
This removes bugs with the linked list being full, but hashmap empty
* address PR comments
* switch clist back to list
* ADR: Proposal for multisignature encoding
This proposal is partially tied to the resolution of #1957.
* Change title to Encoding standard for multisignatures
* ADR: Change multisigs ADR now that amino must be used for pubkeys
* Address PR comments
* p2p/pex: Allow configured seed nodes to be offline
Previously you couldn't startup tendermint if a seed node was offline.
This now allows you to startup tendermint, as long as all seed node addresses
are formatted correctly. In the event that all seed nodes are down,
and the address book is empty, then it crashes with an informative error msg.
(This case doesn't occur if no seeds were specified)
Closes#1716
* (Squash this) Address melekes' comments
* (squash this) fix package imports
* (squash this) fix pex_reactor comment
* (squash this) add a test case
This is to reduce wait times when initially connecting. This still runs checks
such as whether you still want additional peers.
A test case has been created, which fails if this change is not included.
This commit makes it such that circle CI only shows the module whose
tests it is currently running in the log, unless a test fails. For each
failing test, it will display the name of all failing tests, along with
their log output. This is done to make
our log output far more scrollable. We lose no information in debugging.
Previously we logged `Testing for i <i>` for all i in [0,100).
This was unnecessary. This changes it to just log the value for i on
error messages, to reduce the unnecessary verbosity in log files.
This now makes bit array functions which take in a second bit array, thread
safe. Previously there was a warning on bitarray.Update to be lock the
second parameter externally if thread safety wasrequired.
This was not done within the codebase, so it was fine to change here.
Closes#2080
* .github: Split the issue template into two seperate templates
Now we have different bug report and feature request templates.
* Forgot to add the name, and about fields
hkdfchachapoly was a construction we came up with. There is no longer any
reason to use it. We should instead just use xchacha for the remaining use
cases we have. (Symmetrically encrypting the keys in the sdk keys package)
* crypto: Add benchmarking code for signature schemes
This does a slight refactor for the key generation code. It now calls a
seperate unexported method to allow generation from a reader. I think this
will actually reduce time in generation, due to no longer initializing an
extra slice. This was needed in order to enable benchmarking.
This uses an internal package for the benchmarking code, so that this can
be standardized without being exported in the public API. The benchmarking
code is derived from agl/ed25519's benchmarking code, and has copied the
license over.
Closes#1984
This adds a new makefile command, which is used in CI linting, `make check_dep`.
This ensures the toml is in sync with the lock, and that were not pinning to a
branch in any repository.
This also adapts `make get_vendor_deps` to check the lock, in addition to
populating the vendor directory. This removes the need for `make ensure_deps`.
This makes `make get_vendor_deps` consistent between tendermint and the sdk.
The privkey.Generate method here was a custom-made method for deriving
a private key from another private key. This function is currently
not used anywhere in our codebase, and has not been reviewed enough
that it would be secure to use. This removes that method. We should
adopt the official ed25519 HD derivation once that has been standardized,
in order to fulfill this need.
closes#2000
* crypto/secp256k1: Add godocs, remove indirection in privkeys
The following was previously done for creating secp256k1 private keys:
First obtain privkey bytes. Then create a private key in the
underlying library, with scalar exponent equal to privKeyBytes.
(The method called was secp256k1.PrivKeyFromBytes,
https://github.com/btcsuite/btcd/blob/fb90c334dffc6c91843d8d36ac31e7f7eb6c7594/btcec/privkey.go#L21)
Then the private key was serialized using the underlying library, which just
returns back the bytes that comprised the scalar exponent, but padded to be
exactly 32 bytes.
https://github.com/btcsuite/btcd/blob/fb90c334dffc6c91843d8d36ac31e7f7eb6c7594/btcec/privkey.go#L70
Thus the entire indirection of calling the underlying library can be avoided
by just ensuring that we pass in a 32 byte value. A test case has even be written
to show this more clearly in review.
* crypto/secp256k1: Address PR comments
Squash this commit
* crypto: Remove note about re-registering amino paths when unnecessary.
This commit should be squashed.
This commit updates the godocs for the package, and adds an optimization
to the privkey.Pubkey() method.
The optimization is that in golang, the private key (due to interface
compatibility reasons) has a copy of the public key stored inside of it.
Therefore if this copy has already been computed, there is no need to
recompute it.
* tools: Remove redundant grep -v vendors/
This was used in conjunction with `go list <path>`, however `go list`
already ignores the vendor directory. This made this `grep -v` redundant.
* Missed an apostrophe
We were computing these functions incorrectly.
I'm not sure what distribution these numbers are, but it isn't the
normal exponential distribution. (We're making the probability of
getting a number of a particular bitlength equal, but the number in
that bitlength thats gets chosen is uniformly chosen)
We weren't using these functions anywhere in our codebase, and they
had a nomenclature error. (There aren't exponentially distributed
integers, instead they would be geometrically distributed)
This commit switches all usage of math/rand to cmn's rand. The only
exceptions are within the random file itself, the tools package, and the
crypto package. In tools you don't want it to lock between the go-routines.
The crypto package doesn't use it so the crypto package have no other
dependencies within tendermint/tendermint for easier portability.
Crypto/rand usage is unadjusted.
Closes#1343
* tools/tm-bench: Don't count the first block if its empty
* Try melekes suggestion
* Fix getting the start time so the first block is no longer empty
* Fix changelog entry
* Make code smell better
* tools/tmbench: Fix the end time being used for statistics calculation
Previously we were using the time at which all connections closed in statistics, not
the time after {duration} seconds.
* Use waitgroups for starting up
* tmbench: Make it more resilient to WSConn breaking
This commit changes the behavior of a broken connection from calling
os.Exit, to instead killing that connection. This also improves
the debug logging, by specifying connection index within errors.
* Rename connStatus to connsBroken
* change logging level
* crypto: Abstract pubkey / signature size when known to constants
* Created PubKeyEd25519Size as 32
* Created PubkeySecp256k1Size as 33
* Created SignatureEd25519Size as 64
* Remove extraneous message from changelog
* tmbench: Make sendloop act in one second segments
Previous behaviour was to make the sendloop send all its messages
and then time how long that took. (Possibly waiting if too fast)
This has the same operation when too fast, but stops the loop
after one second. This is useful if a large tx amount is specified
and the ping doesn't get executed. Tmbench no longer crashes on
large rates.
* Update Readme
* config: rename skip_upnp to upnp
Change default option to enable upnp.
Closes#1806
* doc updates
- fix comment and set UPNP to false in TestP2PConfig
- add UPNP to config template
- update changelog
* Switch ports 466xx to be 266xx
This is done so the default ports aren't in the linux kernel's default ephemeral port range.
* Update ABCI import
* Bump cache on circleci
* Get more verbose output for debugging
* Bump abci dependency
* Fix accidental change of a block header's hash
* pin abci release
