Merge remote-tracking branch 'origin/develop' into dev/adr_secp_signatures

docs/app-dev/abci-spec.md

@@ -160,9 +160,8 @@ See below for more details on the message types and how they are used.
 - **Request**:
   - `Hash ([]byte)`: The block's hash. This can be derived from the
     block header.
-  - `Header (struct{})`: The block header
-  - `Validators ([]SigningValidator)`: List of validators in the current validator
-    set and whether or not they signed a vote in the LastCommit
+  - `Header (struct{})`: The block header.
+  - `LastCommitInfo (LastCommitInfo)`: Info about the last commit.
   - `ByzantineValidators ([]Evidence)`: List of evidence of
     validators that acted maliciously
 - **Response**:
@@ -171,8 +170,9 @@ See below for more details on the message types and how they are used.
   - Signals the beginning of a new block. Called prior to
     any DeliverTxs.
   - The header is expected to at least contain the Height.
-  - The `Validators` and `ByzantineValidators` can be used to
-    determine rewards and punishments for the validators.
+  - The `LastCommitInfo` and `ByzantineValidators` can be used to determine
+    rewards and punishments for the validators. NOTE validators here do not
+    include pubkeys.
 
 ### CheckTx
 
@@ -326,3 +326,10 @@ See below for more details on the message types and how they are used.
     It is the proposer's local time when block was created.
   - `TotalVotingPower (int64)`: Total voting power of the validator set at
     height `Height`
+
+### LastCommitInfo
+
+- **Fields**:
+  - `CommitRound (int32)`: Commit round.
+  - `Validators ([]SigningValidator)`: List of validators in the current
+    validator set and whether or not they signed a vote.

docs/app-dev/app-development.md

@@ -365,14 +365,14 @@ ResponseBeginBlock requestBeginBlock(RequestBeginBlock req) {
 
 ### EndBlock
 
-The EndBlock request can be used to run some code at the end of every
-block. Additionally, the response may contain a list of validators,
-which can be used to update the validator set. To add a new validator or
-update an existing one, simply include them in the list returned in the
-EndBlock response. To remove one, include it in the list with a `power`
-equal to `0`. Tendermint core will take care of updating the validator
-set. Note the change in voting power must be strictly less than 1/3 per
-block if you want a light client to be able to prove the transition
+The EndBlock request can be used to run some code at the end of every block.
+Additionally, the response may contain a list of validators, which can be used
+to update the validator set. To add a new validator or update an existing one,
+simply include them in the list returned in the EndBlock response. To remove
+one, include it in the list with a `power` equal to `0`. Validator's `address`
+field can be left empty. Tendermint core will take care of updating the
+validator set. Note the change in voting power must be strictly less than 1/3
+per block if you want a light client to be able to prove the transition
 externally. See the [light client
 docs](https://godoc.org/github.com/tendermint/tendermint/lite#hdr-How_We_Track_Validators)
 for details on how it tracks validators.

docs/architecture/adr-015-crypto-encoding.md

@@ -0,0 +1,80 @@
+# ADR 015: Crypto encoding 
+
+## Context
+
+We must standardize our method for encoding public keys and signatures on chain. 
+Currently we amino encode the public keys and signatures.
+The reason we are using amino here is primarily due to ease of support in
+parsing for other languages.
+We don't need its upgradability properties in cryptosystems, as a change in
+the crypto that requires adapting the encoding, likely warrants being deemed
+a new cryptosystem.
+(I.e. using new public parameters)
+
+## Decision
+
+### Public keys
+
+For public keys, we will continue to use amino encoding on the canonical
+representation of the pubkey.
+(Canonical as defined by the cryptosystem itself)
+This has two significant drawbacks.
+Amino encoding is less space-efficient, due to requiring support for upgradability.
+Amino encoding support requires forking protobuf and adding this new interface support
+option in the langauge of choice.
+
+The reason for continuing to use amino however is that people can create code
+more easily in languages that already have an up to date amino library.
+It is possible that this will change in the future, if it is deemed that
+requiring amino for interacting with tendermint cryptography is unneccessary.
+
+The arguments for space efficiency here are refuted on the basis that there are
+far more egregious wastages of space in the SDK.
+The space requirement of the public keys doesn't cause many problems beyond
+increasing the space attached to each validator / account.
+
+The alternative to using amino here would be for us to create an enum type.
+Switching to just an enum type is worthy of investigation post-launch.
+For referrence, part of amino encoding interfaces is basically a 4 byte enum
+type definition.
+Enum types would just change that 4 bytes to be a varuint, and it would remove
+the protobuf overhead, but it would be hard to integrate into the existing API.
+
+### Signatures
+
+Signatures should be switched to be `[]byte`.
+Spatial efficiency in the signatures is quite important,
+as it directly affects the gas cost of every transaction,
+and the throughput of the chain.
+Signatures don't need to encode what type they are for (unlike public keys)
+since public keys must already be known.
+Therefore we can validate the signature without needing to encode its type.
+
+When placed in state, signatures will still be amino encoded, but it will be the
+primitive type `[]byte` getting encoded.
+
+#### Ed25519
+Use the canonical representation for signatures.
+
+#### Secp256k1
+There isn't a clear canonical representation here.
+Signatures have two elements `r,s`.
+We should encode these bytes as `r || s`, where `r` and `s` are both exactly
+32 bytes long.
+This is basically Ethereum's encoding, but without the leading recovery bit.
+
+## Status
+
+Proposed. The signature section seems to be agreed upon for the most part.
+Needs decision on Enum types.
+
+## Consequences
+
+### Positive
+* More space efficient signatures
+
+### Negative
+* We have an amino dependency for cryptography.
+
+### Neutral
+* No change to public keys

docs/spec/blockchain/blockchain.md

@@ -52,7 +52,8 @@ type Header struct {
     // application
     ResultsHash         []byte  // SimpleMerkle of []abci.Result from prevBlock
     AppHash             []byte  // Arbitrary state digest
-    ValidatorsHash      []byte  // SimpleMerkle of the ValidatorSet
+    ValidatorsHash      []byte  // SimpleMerkle of the current ValidatorSet
+    NextValidatorsHash  []byte  // SimpleMerkle of the next ValidatorSet
     ConsensusParamsHash []byte  // SimpleMerkle of the ConsensusParams
 
     // consensus
@@ -160,9 +161,12 @@ We refer to certain globally available objects:
 `block` is the block under consideration,
 `prevBlock` is the `block` at the previous height,
 and `state` keeps track of the validator set, the consensus parameters
-and other results from the application.
+and other results from the application. At the point when `block` is the block under consideration,
+the current version of the `state` corresponds to the state 
+after executing transactions from the `prevBlock`. 
 Elements of an object are accessed as expected,
-ie. `block.Header`. See [here](https://github.com/tendermint/tendermint/blob/master/docs/spec/blockchain/state.md) for the definition of `state`.
+ie. `block.Header`. 
+See [here](https://github.com/tendermint/tendermint/blob/master/docs/spec/blockchain/state.md) for the definition of `state`.
 
 ### Header
 
@@ -278,7 +282,14 @@ block.ValidatorsHash == SimpleMerkleRoot(state.Validators)
 
 Simple Merkle root of the current validator set that is committing the block.
 This can be used to validate the `LastCommit` included in the next block.
-May be updated by the application.
+
+### NextValidatorsHash
+
+```go
+block.NextValidatorsHash == SimpleMerkleRoot(state.NextValidators)
+```
+Simple Merkle root of the next validator set that will be the validator set that commits the next block.
+Modifications to the validator set are defined by the application.
 
 ### ConsensusParamsHash
 
@@ -407,25 +418,20 @@ set (TODO). Execute is defined as:
 
 ```go
 Execute(s State, app ABCIApp, block Block) State {
-    TODO: just spell out ApplyBlock here
-    and remove ABCIResponses struct.
-    abciResponses := app.ApplyBlock(block)
+    // Fuction ApplyBlock executes block of transactions against the app and returns the new root hash of the app state,
+    // modifications to the validator set and the changes of the consensus parameters.
+    AppHash, ValidatorChanges, ConsensusParamChanges := app.ApplyBlock(block)
 
     return State{
         LastResults: abciResponses.DeliverTxResults,
-        AppHash: abciResponses.AppHash,
-        Validators: UpdateValidators(state.Validators, abciResponses.ValidatorChanges),
+        AppHash: AppHash,
         LastValidators: state.Validators,
-        ConsensusParams: UpdateConsensusParams(state.ConsensusParams, abci.Responses.ConsensusParamChanges),
+        Validators: state.NextValidators,
+        NextValidators: UpdateValidators(state.NextValidators, ValidatorChanges), 
+        ConsensusParams: UpdateConsensusParams(state.ConsensusParams, ConsensusParamChanges),
     }
 }
 
-type ABCIResponses struct {
-    DeliverTxResults        []Result
-    ValidatorChanges        []Validator
-    ConsensusParamChanges   ConsensusParams
-    AppHash                 []byte
-}
 ```
 
 

docs/spec/blockchain/state.md

@@ -3,7 +3,7 @@
 ## State
 
 The state contains information whose cryptographic digest is included in block headers, and thus is
-necessary for validating new blocks. For instance, the set of validators and the results of
+necessary for validating new blocks. For instance, the validators set and the results of
 transactions are never included in blocks, but their Merkle roots are - the state keeps track of them.
 
 Note that the `State` object itself is an implementation detail, since it is never
@@ -18,8 +18,9 @@ type State struct {
     LastResults []Result
     AppHash []byte
 
-    Validators []Validator
     LastValidators []Validator
+    Validators []Validator
+    NextValidators []Validator
 
     ConsensusParams ConsensusParams
 }