mirrors/tendermint
1
0
Fork 0
mirror of https://github.com/tendermint/tendermint.git synced 2026-02-11 06:11:07 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' into wb/fix-state-no-initialized

Browse Source
This commit is contained in:
William Banfield
2022-05-19 10:31:25 -04:00
committed by GitHub
parent 0f53f94a34 850ae93a90
commit 02230b7683
4 changed files with 962 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
spec/abci++/README.md
View File

@@ -25,19 +25,15 @@ This allows Tendermint to run with applications written in many programming lang
This specification is split as follows:
- [Overview and basic concepts](./abci++_basic_concepts_002_draft.md) - interface's overview and concepts needed to understand other parts of this specification.
- [Overview and basic concepts](./abci++_basic_concepts_002_draft.md) - interface's overview and concepts
needed to understand other parts of this specification.
- [Methods](./abci++_methods_002_draft.md) - complete details on all ABCI++ methods
and message types.
- [Requirements for the Application](./abci++_app_requirements_002_draft.md) - formal requirements
on the Application's logic to ensure liveness of Tendermint. These requirements define what
Tendermint expects from the Application.
on the Application's logic to ensure Tendermint properties such as liveness. These requirements define what
Tendermint expects from the Application; second part on managing ABCI application state and related topics.
- [Tendermint's expected behavior](./abci++_tmint_expected_behavior_002_draft.md) - specification of
how the different ABCI++ methods may be called by Tendermint. This explains what the Application
is to expect from Tendermint.
>**TODO** Re-read these and remove redundant info
- [Applications](../abci/apps.md) - how to manage ABCI application state and other
details about building ABCI applications
- [Client and Server](../abci/client-server.md) - for those looking to implement their
own ABCI application servers

845
spec/abci++/abci++_app_requirements_002_draft.md
View File

@@ -5,6 +5,8 @@ title: Application Requirements
# Application Requirements
## Formal Requirements
This section specifies what Tendermint expects from the Application. It is structured as a set
of formal requirements that can be used for testing and verification of the Application's logic.
@@ -177,3 +179,846 @@ Likewise, `ExtendVote` can also be non-deterministic:
but may also depend on other values or operations.
* *w<sup>r</sup><sub>p</sub> = w<sup>r</sup><sub>q</sub> &#8655;
e<sup>r</sup><sub>p</sub> = e<sup>r</sup><sub>q</sub>*
## Managing the Application state and related topics
### Connection State
Tendermint maintains four concurrent ABCI++ connections, namely
[Consensus Connection](#consensus-connection),
[Mempool Connection](#mempool-connection),
[Info/Query Connection](#infoquery-connection), and
[Snapshot Connection](#snapshot-connection).
It is common for an application to maintain a distinct copy of
the state for each connection, which are synchronized upon `Commit` calls.
#### Concurrency
In principle, each of the four ABCI++ connections operates concurrently with one
another. This means applications need to ensure access to state is
thread safe. Up to v0.35.x, both the
[default in-process ABCI client](https://github.com/tendermint/tendermint/blob/v0.35.x/abci/client/local_client.go#L18)
and the
[default Go ABCI server](https://github.com/tendermint/tendermint/blob/v0.35.x/abci/server/socket_server.go#L32)
used a global lock to guard the handling of events across all connections, so they were not
concurrent at all. This meant whether your app was compiled in-process with
Tendermint using the `NewLocalClient`, or run out-of-process using the `SocketServer`,
ABCI messages from all connections were received in sequence, one at a
time.
This is no longer the case starting from v0.36.0: the global locks have been removed and it is
up to the Application to synchronize access to its state when handling
ABCI++ methods on all connections.
Nevertheless, as all ABCI calls are now synchronous, ABCI messages using the same connection are
still received in sequence.
#### FinalizeBlock
When the consensus algorithm decides on a block, Tendermint uses `FinalizeBlock` to send the
decided block's data to the Application, which uses it to transition its state.
The Application must remember the latest height from which it
has run a successful `Commit` so that it can tell Tendermint where to
pick up from when it recovers from a crash. See information on the Handshake
[here](#crash-recovery).
#### Commit
The Application should persist its state during `Commit`, before returning from it.
Before invoking `Commit`, Tendermint locks the mempool and flushes the mempool connection. This ensures that
no new messages
will be received on the mempool connection during this processing step, providing an opportunity to safely
update all four
connection states to the latest committed state at the same time.
When `Commit` returns, Tendermint unlocks the mempool.
WARNING: if the ABCI app logic processing the `Commit` message sends a
`/broadcast_tx_sync` or `/broadcast_tx` and waits for the response
before proceeding, it will deadlock. Executing `broadcast_tx` calls
involves acquiring the mempool lock that Tendermint holds during the `Commit` call.
Synchronous mempool-related calls must be avoided as part of the sequential logic of the
`Commit` function.
#### Candidate States
Tendermint calls `PrepareProposal` when it is about to send a proposed block to the network.
Likewise, Tendermint calls `ProcessProposal` upon reception of a proposed block from the
network. In both cases, the proposed block's data
is disclosed to the Application, in the same conditions as is done in `FinalizeBlock`.
The block data disclosed the to Application by these three methods are the following:
* the transaction list
* the `LastCommit` referring to the previous block
* the block header's hash (except in `PrepareProposal`, where it is not known yet)
* list of validators that misbehaved
* the block's timestamp
* `NextValidatorsHash`
* Proposer address
The Application may decide to *immediately* execute the given block (i.e., upon `PrepareProposal`
or `ProcessProposal`). There are two main reasons why the Application may want to do this:
* *Avoiding invalid transactions in blocks*.
In order to be sure that the block does not contain *any* invalid transaction, there may be
no way other than fully executing the transactions in the block as though it was the *decided*
block.
* *Quick `FinalizeBlock` execution*.
Upon reception of the decided block via `FinalizeBlock`, if that same block was executed
upon `PrepareProposal` or `ProcessProposal` and the resulting state was kept in memory, the
Application can simply apply that state (faster) to the main state, rather than reexecuting
the decided block (slower).
`PrepareProposal`/`ProcessProposal` can be called many times for a given height. Moreover,
it is not possible to accurately predict which of the blocks proposed in a height will be decided,
being delivered to the Application in that height's `FinalizeBlock`.
Therefore, the state resulting from executing a proposed block, denoted a *candidate state*, should
be kept in memory as a possible final state for that height. When `FinalizeBlock` is called, the Application should
check if the decided block corresponds to one of its candidate states; if so, it will apply it as
its *ExecuteTxState* (see [Consensus Connection](#consensus-connection) below),
which will be persisted during the upcoming `Commit` call.
Under adverse conditions (e.g., network instability), Tendermint might take many rounds.
In this case, potentially many proposed blocks will be disclosed to the Application for a given height.
By the nature of Tendermint's consensus algorithm, the number of proposed blocks received by the Application
for a particular height cannot be bound, so Application developers must act with care and use mechanisms
to bound memory usage. As a general rule, the Application should be ready to discard candidate states
before `FinalizeBlock`, even if one of them might end up corresponding to the
decided block and thus have to be reexecuted upon `FinalizeBlock`.
### States and ABCI++ Connections
#### Consensus Connection
The Consensus Connection should maintain an *ExecuteTxState* &mdash; the working state
for block execution. It should be updated by the call to `FinalizeBlock`
during block execution and committed to disk as the "latest
committed state" during `Commit`. Execution of a proposed block (via `PrepareProposal`/`ProcessProposal`)
**must not** update the *ExecuteTxState*, but rather be kept as a separate candidate state until `FinalizeBlock`
confirms which of the candidate states (if any) can be used to update *ExecuteTxState*.
#### Mempool Connection
The mempool Connection maintains *CheckTxState*. Tendermint sequentially processes an incoming
transaction (via RPC from client or P2P from the gossip layer) against *CheckTxState*.
If the processing does not return any error, the transaction is accepted into the mempool
and Tendermint starts gossipping it.
*CheckTxState* should be reset to the latest committed state
at the end of every `Commit`.
During the execution of a consensus instance, the *CheckTxState* may be updated concurrently with the
*ExecuteTxState*, as messages may be sent concurrently on the Consensus and Mempool connections.
At the end of the consensus instance, as described above, Tendermint locks the mempool and flushes
the mempool connection before calling `Commit`. This ensures that all pending `CheckTx` calls are
responded to and no new ones can begin.
After the `Commit` call returns, while still holding the mempool lock, `CheckTx` is run again on all
transactions that remain in the node's local mempool after filtering those included in the block.
Parameter `Type` in `RequestCheckTx`
indicates whether an incoming transaction is new (`CheckTxType_New`), or a
recheck (`CheckTxType_Recheck`).
Finally, after re-checking transactions in the mempool, Tendermint will unlock
the mempool connection. New transactions are once again able to be processed through `CheckTx`.
Note that `CheckTx` is just a weak filter to keep invalid transactions out of the mempool and,
utimately, ouf of the blockchain.
Since the transaction cannot be guaranteed to be checked against the exact same state as it
will be executed as part of a (potential) decided block, `CheckTx` shouldn't check *everything*
that affects the transaction's validity, in particular those checks whose validity may depend on
transaction ordering. `CheckTx` is weak because a Byzantine node need not care about `CheckTx`;
it can propose a block full of invalid transactions if it wants. The mechanism ABCI++ has
in place for dealing with such behavior is `ProcessProposal`.
##### Replay Protection
It is possible for old transactions to be sent again to the Application. This is typically
undesirable for all transactions, except for a generally small subset of them which are idempotent.
The mempool has a mechanism to prevent duplicated transactions from being processed.
This mechanism is nevertheless best-effort (currently based on the indexer)
and does not provide any guarantee of non duplication.
It is thus up to the Application to implement an application-specific
replay protection mechanism with strong guarantees as part of the logic in `CheckTx`.
#### Info/Query Connection
The Info (or Query) Connection should maintain a `QueryState`. This connection has two
purposes: 1) having the application answer the queries Tenderissued receives from users
(see section [Query](#query)),
and 2) synchronizing Tendermint and the Application at start up time (see
[Crash Recovery](#crash-recovery))
or after state sync (see [State Sync](#state-sync)).
`QueryState` is a read-only copy of *ExecuteTxState* as it was after the last
`Commit`, i.e.
after the full block has been processed and the state committed to disk.
#### Snapshot Connection
The Snapshot Connection is used to serve state sync snapshots for other nodes
and/or restore state sync snapshots to a local node being bootstrapped.
Snapshop management is optional: an Application may choose not to implement it.
For more information, see Section [State Sync](#state-sync).
### Transaction Results
The Application is expected to return a list of
[`ExecTxResult`](./abci%2B%2B_methods_002_draft.md#exectxresult) in
[`ResponseFinalizeBlock`](./abci%2B%2B_methods_002_draft.md#finalizeblock). The list of transaction
results must respect the same order as the list of transactions delivered via
[`RequestFinalizeBlock`](./abci%2B%2B_methods_002_draft.md#finalizeblock).
This section discusses the fields inside this structure, along with the fields in
[`ResponseCheckTx`](./abci%2B%2B_methods_002_draft.md#checktx),
whose semantics are similar.
The `Info` and `Log` fields are
non-deterministic values for debugging/convenience purposes. Tendermint logs them but they
are otherwise ignored.
#### Gas
Ethereum introduced the notion of *gas* as an abstract representation of the
cost of the resources consumed by nodes when processing a transaction. Every operation in the
Ethereum Virtual Machine uses some amount of gas.
Gas has a market-variable price based on which miners can accept or reject to execute a
particular operation.
Users propose a maximum amount of gas for their transaction; if the transaction uses less, they get
the difference credited back. Tendermint adopts a similar abstraction,
though uses it only optionally and weakly, allowing applications to define
their own sense of the cost of execution.
In Tendermint, the [ConsensusParams.Block.MaxGas](#consensus-parameters) limits the amount of
total gas that can be used by all transactions in a block.
The default value is `-1`, which means the block gas limit is not enforced, or that the concept of
gas is meaningless.
Responses contain a `GasWanted` and `GasUsed` field. The former is the maximum
amount of gas the sender of a transaction is willing to use, and the latter is how much it actually
used. Applications should enforce that `GasUsed <= GasWanted` &mdash; i.e. transaction execution
or validation should fail before it can use more resources than it requested.
When `MaxGas > -1`, Tendermint enforces the following rules:
* `GasWanted <= MaxGas` for every transaction in the mempool
* `(sum of GasWanted in a block) <= MaxGas` when proposing a block
If `MaxGas == -1`, no rules about gas are enforced.
In v0.35.x and earlier versions, Tendermint does not enforce anything about Gas in consensus,
only in the mempool.
This means it does not guarantee that committed blocks satisfy these rules.
It is the application's responsibility to return non-zero response codes when gas limits are exceeded
when executing the transactions of a block.
Since the introduction of `PrepareProposal` and `ProcessProposal` in v.0.36.x, it is now possible
for the Application to enforce that all blocks proposed (and voted for) in consensus &mdash; and thus all
blocks decided &mdash; respect the `MaxGas` limits described above.
Since the Application should enforce that `GasUsed <= GasWanted` when executing a transaction, and
it can use `PrepareProposal` and `ProcessProposal` to enforce that `(sum of GasWanted in a block) <= MaxGas`
in all proposed or prevoted blocks,
we have:
* `(sum of GasUsed in a block) <= MaxGas` for every block
The `GasUsed` field is ignored by Tendermint.
#### Specifics of `ResponseCheckTx`
If `Code != 0`, it will be rejected from the mempool and hence
not broadcasted to other peers and not included in a proposal block.
`Data` contains the result of the `CheckTx` transaction execution, if any. It does not need to be
deterministic since, given a transaction, nodes' Applications
might have a different *CheckTxState* values when they receive it and check their validity
via `CheckTx`.
Tendermint ignores this value in `ResponseCheckTx`.
`Events` include any events for the execution, though since the transaction has not
been committed yet, they are effectively ignored by Tendermint.
From v0.35.x on, there is a `Priority` field in `ResponseCheckTx` that can be
used to explicitly prioritize transactions in the mempool for inclusion in a block
proposal.
#### Specifics of `ExecTxResult`
`FinalizeBlock` is the workhorse of the blockchain. Tendermint delivers the decided block,
including the list of all its transactions synchronously to the Application.
The block delivered (and thus the transaction order) is the same at all correct nodes as guaranteed
by the Agreement property of Tendermint consensus.
In same block execution mode, field `LastResultsHash` in the block header refers to the results
of all transactions stored in that block. Therefore,
`PrepareProposal` must return `ExecTxResult` so that it can
be used to build the block to be proposed in the current height.
The `Data` field in `ExecTxResult` contains an array of bytes with the transaction result.
It must be deterministic (i.e., the same value must be returned at all nodes), but it can contain arbitrary
data. Likewise, the value of `Code` must be deterministic.
If `Code != 0`, the transaction will be marked invalid,
though it is still included in the block. Invalid transaction are not indexed, as they are
considered analogous to those that failed `CheckTx`.
Both the `Code` and `Data` are included in a structure that is hashed into the
`LastResultsHash` of the block header in the next height (next block execution mode), or the
header of the block to propose in the current height (same block execution mode, `ExecTxResult` as
part of `PrepareProposal`).
`Events` include any events for the execution, which Tendermint will use to index
the transaction by. This allows transactions to be queried according to what
events took place during their execution.
### Updating the Validator Set
The application may set the validator set during
[`InitChain`](./abci%2B%2B_methods_002_draft.md#initchain), and may update it during
[`FinalizeBlock`](./abci%2B%2B_methods_002_draft.md#finalizeblock)
(next block execution mode) or
[`PrepareProposal`](./abci%2B%2B_methods_002_draft.md#prepareproposal)/[`ProcessProposal`](./abci%2B%2B_methods_002_draft.md#processproposal)
(same block execution mode). In all cases, a structure of type
[`ValidatorUpdate`](./abci%2B%2B_methods_002_draft.md#validatorupdate) is returned.
The `InitChain` method, used to initialize the Application, can return a list of validators.
If the list is empty, Tendermint will use the validators loaded from the genesis
file.
If the list returned by `InitChain` is not empty, Tendermint will use its contents as the validator set.
This way the application can set the initial validator set for the
blockchain.
Applications must ensure that a single set of validator updates does not contain duplicates, i.e.
a given public key can only appear once within a given update. If an update includes
duplicates, the block execution will fail irrecoverably.
Structure `ValidatorUpdate` contains a public key, which is used to identify the validator:
The public key currently supports three types:
* `ed25519`
* `secp256k1`
* `sr25519`
Structure `ValidatorUpdate` also contains an `ìnt64` field denoting the validator's new power.
Applications must ensure that
`ValidatorUpdate` structures abide by the following rules:
* power must be non-negative
* if power is set to 0, the validator must be in the validator set; it will be removed from the set
* if power is greater than 0:
* if the validator is not in the validator set, it will be added to the
set with the given power
* if the validator is in the validator set, its power will be adjusted to the given power
* the total power of the new validator set must not exceed `MaxTotalVotingPower`, where
`MaxTotalVotingPower = MaxInt64 / 8`
Note the updates returned after processing the block at height `H` will only take effect
at block `H+2` (see Section [Methods](./abci%2B%2B_methods_002_draft.md)).
### Consensus Parameters
`ConsensusParams` are global parameters that apply to all validators in a blockchain.
They enforce certain limits in the blockchain, like the maximum size
of blocks, amount of gas used in a block, and the maximum acceptable age of
evidence. They can be set in
[`InitChain`](./abci%2B%2B_methods_002_draft.md#initchain), and updated in
[`FinalizeBlock`](./abci%2B%2B_methods_002_draft.md#finalizeblock)
(next block execution mode) or
[`PrepareProposal`](./abci%2B%2B_methods_002_draft.md#prepareproposal)/[`ProcessProposal`](./abci%2B%2B_methods_002_draft.md#processproposal)
(same block execution model).
These parameters are deterministically set and/or updated by the Application, so
all full nodes have the same value at a given height.
#### List of Parameters
These are the current consensus parameters (as of v0.36.x):
1. [BlockParams.MaxBytes](#blockparamsmaxbytes)
2. [BlockParams.MaxGas](#blockparamsmaxgas)
3. [EvidenceParams.MaxAgeDuration](#evidenceparamsmaxageduration)
4. [EvidenceParams.MaxAgeNumBlocks](#evidenceparamsmaxagenumblocks)
5. [EvidenceParams.MaxBytes](#evidenceparamsmaxbytes)
6. [SynchronyParams.MessageDelay](#synchronyparamsmessagedelay)
7. [SynchronyParams.Precision](#synchronyparamsprecision)
8. [TimeoutParams.Propose](#timeoutparamspropose)
9. [TimeoutParams.ProposeDelta](#timeoutparamsproposedelta)
10. [TimeoutParams.Vote](#timeoutparamsvote)
11. [TimeoutParams.VoteDelta](#timeoutparamsvotedelta)
12. [TimeoutParams.Commit](#timeoutparamscommit)
13. [TimeoutParams.BypassCommitTimeout](#timeoutparamsbypasscommittimeout)
##### BlockParams.MaxBytes
The maximum size of a complete Protobuf encoded block.
This is enforced by Tendermint consensus.
This implies a maximum transaction size that is this `MaxBytes`, less the expected size of
the header, the validator set, and any included evidence in the block.
Must have `0 < MaxBytes < 100 MB`.
##### BlockParams.MaxGas
The maximum of the sum of `GasWanted` that will be allowed in a proposed block.
This is *not* enforced by Tendermint consensus.
It is left to the Application to enforce (ie. if transactions are included past the
limit, they should return non-zero codes). It is used by Tendermint to limit the
transactions included in a proposed block.
Must have `MaxGas >= -1`.
If `MaxGas == -1`, no limit is enforced.
##### EvidenceParams.MaxAgeDuration
This is the maximum age of evidence in time units.
This is enforced by Tendermint consensus.
If a block includes evidence older than this (AND the evidence was created more
than `MaxAgeNumBlocks` ago), the block will be rejected (validators won't vote
for it).
Must have `MaxAgeDuration > 0`.
##### EvidenceParams.MaxAgeNumBlocks
This is the maximum age of evidence in blocks.
This is enforced by Tendermint consensus.
If a block includes evidence older than this (AND the evidence was created more
than `MaxAgeDuration` ago), the block will be rejected (validators won't vote
for it).
Must have `MaxAgeNumBlocks > 0`.
##### EvidenceParams.MaxBytes
This is the maximum size of total evidence in bytes that can be committed to a
single block. It should fall comfortably under the max block bytes.
Its value must not exceed the size of
a block minus its overhead ( ~ `BlockParams.MaxBytes`).
Must have `MaxBytes > 0`.
##### SynchronyParams.MessageDelay
This sets a bound on how long a proposal message may take to reach all
validators on a network and still be considered valid.
This parameter is part of the
[proposer-based timestamps](../consensus/proposer-based-timestamp)
(PBTS) algorithm.
##### SynchronyParams.Precision
This sets a bound on how skewed a proposer's clock may be from any validator
on the network while still producing valid proposals.
This parameter is part of the
[proposer-based timestamps](../consensus/proposer-based-timestamp)
(PBTS) algorithm.
##### TimeoutParams.Propose
Timeout in ms of the propose step of the Tendermint consensus algorithm.
This value is the initial timeout at every height (round 0).
The value in subsequent rounds is modified by parameter `ProposeDelta`.
When a new height is started, the `Propose` timeout value is reset to this
parameter.
If a node waiting for a proposal message does not receive one matching its
current height and round before this timeout, the node will issue a
`nil` prevote for the round and advance to the next step.
##### TimeoutParams.ProposeDelta
Increment in ms to be added to the `Propose` timeout every time the Tendermint
consensus algorithm advances one round in a given height.
When a new height is started, the `Propose` timeout value is reset.
##### TimeoutParams.Vote
Timeout in ms of the prevote and precommit steps of the Tendermint consensus
algorithm.
This value is the initial timeout at every height (round 0).
The value in subsequent rounds is modified by parameter `VoteDelta`.
When a new height is started, the `Vote` timeout value is reset to this
parameter.
The `Vote` timeout does not begin until a quorum of votes has been received.
Once a quorum of votes has been seen and this timeout elapses, Tendermint will
procced to the next step of the consensus algorithm. If Tendermint receives
all of the remaining votes before the end of the timeout, it will proceed
to the next step immediately.
##### TimeoutParams.VoteDelta
Increment in ms to be added to the `Vote` timeout every time the Tendermint
consensus algorithm advances one round in a given height.
When a new height is started, the `Vote` timeout value is reset.
##### TimeoutParams.Commit
This configures how long Tendermint will wait after receiving a quorum of
precommits before beginning consensus for the next height. This can be
used to allow slow precommits to arrive for inclusion in the next height
before progressing.
##### TimeoutParams.BypassCommitTimeout
This configures the node to proceed immediately to the next height once the
node has received all precommits for a block, forgoing the remaining commit timeout.
Setting this parameter to `false` (the default) causes Tendermint to wait
for the full commit timeout configured in `TimeoutParams.Commit`.
#### Updating Consensus Parameters
The application may set the `ConsensusParams` during
[`InitChain`](./abci%2B%2B_methods_002_draft.md#initchain),
and update them during
[`FinalizeBlock`](./abci%2B%2B_methods_002_draft.md#finalizeblock)
(next block execution mode) or
[`PrepareProposal`](./abci%2B%2B_methods_002_draft.md#prepareproposal)/[`ProcessProposal`](./abci%2B%2B_methods_002_draft.md#processproposal)
(same block execution mode).
If the `ConsensusParams` is empty, it will be ignored. Each field
that is not empty will be applied in full. For instance, if updating the
`Block.MaxBytes`, applications must also set the other `Block` fields (like
`Block.MaxGas`), even if they are unchanged, as they will otherwise cause the
value to be updated to the default.
##### `InitChain`
`ResponseInitChain` includes a `ConsensusParams` parameter.
If `ConsensusParams` is `nil`, Tendermint will use the params loaded in the genesis
file. If `ConsensusParams` is not `nil`, Tendermint will use it.
This way the application can determine the initial consensus parameters for the
blockchain.
##### `FinalizeBlock`, `PrepareProposal`/`ProcessProposal`
In next block execution mode, `ResponseFinalizeBlock` accepts a `ConsensusParams` parameter.
If `ConsensusParams` is `nil`, Tendermint will do nothing.
If `ConsensusParams` is not `nil`, Tendermint will use it.
This way the application can update the consensus parameters over time.
Likewise, in same block execution mode, `PrepareProposal` and `ProcessProposal` include
a `ConsensusParams` parameter. `PrepareProposal` may return a `ConsensusParams` to update
the consensus parameters in the block that is about to be proposed. If it returns `nil`
the consensus parameters will not be updated. `ProcessProposal` also accepts a
`ConsensusParams` parameter, which Tendermint will use it to calculate the corresponding
hashes and sanity-check them against those of the block that triggered `ProcessProposal`
at the first place.
Note the updates returned in block `H` will take effect right away for block
`H+1` (both in next block and same block execution mode).
### `Query`
`Query` is a generic method with lots of flexibility to enable diverse sets
of queries on application state. Tendermint makes use of `Query` to filter new peers
based on ID and IP, and exposes `Query` to the user over RPC.
Note that calls to `Query` are not replicated across nodes, but rather query the
local node's state - hence they may return stale reads. For reads that require
consensus, use a transaction.
The most important use of `Query` is to return Merkle proofs of the application state at some height
that can be used for efficient application-specific light-clients.
Note Tendermint has technically no requirements from the `Query`
message for normal operation - that is, the ABCI app developer need not implement
Query functionality if they do not wish to.
#### Query Proofs
The Tendermint block header includes a number of hashes, each providing an
anchor for some type of proof about the blockchain. The `ValidatorsHash` enables
quick verification of the validator set, the `DataHash` gives quick
verification of the transactions included in the block.
The `AppHash` is unique in that it is application specific, and allows for
application-specific Merkle proofs about the state of the application.
While some applications keep all relevant state in the transactions themselves
(like Bitcoin and its UTXOs), others maintain a separated state that is
computed deterministically *from* transactions, but is not contained directly in
the transactions themselves (like Ethereum contracts and accounts).
For such applications, the `AppHash` provides a much more efficient way to verify light-client proofs.
ABCI applications can take advantage of more efficient light-client proofs for
their state as follows:
* in next block executon mode, return the Merkle root of the deterministic application state in
`ResponseCommit.Data`. This Merkle root will be included as the `AppHash` in the next block.
* in same block execution mode, return the Merkle root of the deterministic application state
in `ResponsePrepareProposal.AppHash`. This Merkle root will be included as the `AppHash` in
the block that is about to be proposed.
* return efficient Merkle proofs about that application state in `ResponseQuery.Proof`
that can be verified using the `AppHash` of the corresponding block.
For instance, this allows an application's light-client to verify proofs of
absence in the application state, something which is much less efficient to do using the block hash.
Some applications (eg. Ethereum, Cosmos-SDK) have multiple "levels" of Merkle trees,
where the leaves of one tree are the root hashes of others. To support this, and
the general variability in Merkle proofs, the `ResponseQuery.Proof` has some minimal structure:
```protobuf
message ProofOps {
repeated ProofOp ops = 1
}
message ProofOp {
string type = 1;
bytes key = 2;
bytes data = 3;
}
```
Each `ProofOp` contains a proof for a single key in a single Merkle tree, of the specified `type`.
This allows ABCI to support many different kinds of Merkle trees, encoding
formats, and proofs (eg. of presence and absence) just by varying the `type`.
The `data` contains the actual encoded proof, encoded according to the `type`.
When verifying the full proof, the root hash for one ProofOp is the value being
verified for the next ProofOp in the list. The root hash of the final ProofOp in
the list should match the `AppHash` being verified against.
#### Peer Filtering
When Tendermint connects to a peer, it sends two queries to the ABCI application
using the following paths, with no additional data:
* `/p2p/filter/addr/<IP:PORT>`, where `<IP:PORT>` denote the IP address and
the port of the connection
* `p2p/filter/id/<ID>`, where `<ID>` is the peer node ID (ie. the
pubkey.Address() for the peer's PubKey)
If either of these queries return a non-zero ABCI code, Tendermint will refuse
to connect to the peer.
#### Paths
Queries are directed at paths, and may optionally include additional data.
The expectation is for there to be some number of high level paths
differentiating concerns, like `/p2p`, `/store`, and `/app`. Currently,
Tendermint only uses `/p2p`, for filtering peers. For more advanced use, see the
implementation of
[Query in the Cosmos-SDK](https://github.com/cosmos/cosmos-sdk/blob/v0.23.1/baseapp/baseapp.go#L333).
### Crash Recovery
On startup, Tendermint calls the `Info` method on the Info Connection to get the latest
committed state of the app. The app MUST return information consistent with the
last block it succesfully completed Commit for.
If the app succesfully committed block H, then `last_block_height = H` and `last_block_app_hash = <hash returned by Commit for block H>`. If the app
failed during the Commit of block H, then `last_block_height = H-1` and
`last_block_app_hash = <hash returned by Commit for block H-1, which is the hash in the header of block H>`.
We now distinguish three heights, and describe how Tendermint syncs itself with
the app.
```md
storeBlockHeight = height of the last block Tendermint saw a commit for
stateBlockHeight = height of the last block for which Tendermint completed all
block processing and saved all ABCI results to disk
appBlockHeight = height of the last block for which ABCI app succesfully
completed Commit
```
Note we always have `storeBlockHeight >= stateBlockHeight` and `storeBlockHeight >= appBlockHeight`
Note also Tendermint never calls Commit on an ABCI app twice for the same height.
The procedure is as follows.
First, some simple start conditions:
If `appBlockHeight == 0`, then call InitChain.
If `storeBlockHeight == 0`, we're done.
Now, some sanity checks:
If `storeBlockHeight < appBlockHeight`, error
If `storeBlockHeight < stateBlockHeight`, panic
If `storeBlockHeight > stateBlockHeight+1`, panic
Now, the meat:
If `storeBlockHeight == stateBlockHeight && appBlockHeight < storeBlockHeight`,
replay all blocks in full from `appBlockHeight` to `storeBlockHeight`.
This happens if we completed processing the block, but the app forgot its height.
If `storeBlockHeight == stateBlockHeight && appBlockHeight == storeBlockHeight`, we're done.
This happens if we crashed at an opportune spot.
If `storeBlockHeight == stateBlockHeight+1`
This happens if we started processing the block but didn't finish.
If `appBlockHeight < stateBlockHeight`
replay all blocks in full from `appBlockHeight` to `storeBlockHeight-1`,
and replay the block at `storeBlockHeight` using the WAL.
This happens if the app forgot the last block it committed.
If `appBlockHeight == stateBlockHeight`,
replay the last block (storeBlockHeight) in full.
This happens if we crashed before the app finished Commit
If `appBlockHeight == storeBlockHeight`
update the state using the saved ABCI responses but dont run the block against the real app.
This happens if we crashed after the app finished Commit but before Tendermint saved the state.
### State Sync
A new node joining the network can simply join consensus at the genesis height and replay all
historical blocks until it is caught up. However, for large chains this can take a significant
amount of time, often on the order of days or weeks.
State sync is an alternative mechanism for bootstrapping a new node, where it fetches a snapshot
of the state machine at a given height and restores it. Depending on the application, this can
be several orders of magnitude faster than replaying blocks.
Note that state sync does not currently backfill historical blocks, so the node will have a
truncated block history - users are advised to consider the broader network implications of this in
terms of block availability and auditability. This functionality may be added in the future.
For details on the specific ABCI calls and types, see the
[methods](abci%2B%2B_methods_002_draft.md) section.
#### Taking Snapshots
Applications that want to support state syncing must take state snapshots at regular intervals. How
this is accomplished is entirely up to the application. A snapshot consists of some metadata and
a set of binary chunks in an arbitrary format:
* `Height (uint64)`: The height at which the snapshot is taken. It must be taken after the given
height has been committed, and must not contain data from any later heights.
* `Format (uint32)`: An arbitrary snapshot format identifier. This can be used to version snapshot
formats, e.g. to switch from Protobuf to MessagePack for serialization. The application can use
this when restoring to choose whether to accept or reject a snapshot.
* `Chunks (uint32)`: The number of chunks in the snapshot. Each chunk contains arbitrary binary
data, and should be less than 16 MB; 10 MB is a good starting point.
* `Hash ([]byte)`: An arbitrary hash of the snapshot. This is used to check whether a snapshot is
the same across nodes when downloading chunks.
* `Metadata ([]byte)`: Arbitrary snapshot metadata, e.g. chunk hashes for verification or any other
necessary info.
For a snapshot to be considered the same across nodes, all of these fields must be identical. When
sent across the network, snapshot metadata messages are limited to 4 MB.
When a new node is running state sync and discovering snapshots, Tendermint will query an existing
application via the ABCI `ListSnapshots` method to discover available snapshots, and load binary
snapshot chunks via `LoadSnapshotChunk`. The application is free to choose how to implement this
and which formats to use, but must provide the following guarantees:
* **Consistent:** A snapshot must be taken at a single isolated height, unaffected by
concurrent writes. This can be accomplished by using a data store that supports ACID
transactions with snapshot isolation.
* **Asynchronous:** Taking a snapshot can be time-consuming, so it must not halt chain progress,
for example by running in a separate thread.
* **Deterministic:** A snapshot taken at the same height in the same format must be identical
(at the byte level) across nodes, including all metadata. This ensures good availability of
chunks, and that they fit together across nodes.
A very basic approach might be to use a datastore with MVCC transactions (such as RocksDB),
start a transaction immediately after block commit, and spawn a new thread which is passed the
transaction handle. This thread can then export all data items, serialize them using e.g.
Protobuf, hash the byte stream, split it into chunks, and store the chunks in the file system
along with some metadata - all while the blockchain is applying new blocks in parallel.
A more advanced approach might include incremental verification of individual chunks against the
chain app hash, parallel or batched exports, compression, and so on.
Old snapshots should be removed after some time - generally only the last two snapshots are needed
(to prevent the last one from being removed while a node is restoring it).
#### Bootstrapping a Node
An empty node can be state synced by setting the configuration option `statesync.enabled =
true`. The node also needs the chain genesis file for basic chain info, and configuration for
light client verification of the restored snapshot: a set of Tendermint RPC servers, and a
trusted header hash and corresponding height from a trusted source, via the `statesync`
configuration section.
Once started, the node will connect to the P2P network and begin discovering snapshots. These
will be offered to the local application via the `OfferSnapshot` ABCI method. Once a snapshot
is accepted Tendermint will fetch and apply the snapshot chunks. After all chunks have been
successfully applied, Tendermint verifies the app's `AppHash` against the chain using the light
client, then switches the node to normal consensus operation.
##### Snapshot Discovery
When the empty node joins the P2P network, it asks all peers to report snapshots via the
`ListSnapshots` ABCI call (limited to 10 per node). After some time, the node picks the most
suitable snapshot (generally prioritized by height, format, and number of peers), and offers it
to the application via `OfferSnapshot`. The application can choose a number of responses,
including accepting or rejecting it, rejecting the offered format, rejecting the peer who sent
it, and so on. Tendermint will keep discovering and offering snapshots until one is accepted or
the application aborts.
##### Snapshot Restoration
Once a snapshot has been accepted via `OfferSnapshot`, Tendermint begins downloading chunks from
any peers that have the same snapshot (i.e. that have identical metadata fields). Chunks are
spooled in a temporary directory, and then given to the application in sequential order via
`ApplySnapshotChunk` until all chunks have been accepted.
The method for restoring snapshot chunks is entirely up to the application.
During restoration, the application can respond to `ApplySnapshotChunk` with instructions for how
to continue. This will typically be to accept the chunk and await the next one, but it can also
ask for chunks to be refetched (either the current one or any number of previous ones), P2P peers
to be banned, snapshots to be rejected or retried, and a number of other responses - see the ABCI
reference for details.
If Tendermint fails to fetch a chunk after some time, it will reject the snapshot and try a
different one via `OfferSnapshot` - the application can choose whether it wants to support
restarting restoration, or simply abort with an error.
##### Snapshot Verification
Once all chunks have been accepted, Tendermint issues an `Info` ABCI call to retrieve the
`LastBlockAppHash`. This is compared with the trusted app hash from the chain, retrieved and
verified using the light client. Tendermint also checks that `LastBlockHeight` corresponds to the
height of the snapshot.
This verification ensures that an application is valid before joining the network. However, the
snapshot restoration may take a long time to complete, so applications may want to employ additional
verification during the restore to detect failures early. This might e.g. include incremental
verification of each chunk against the app hash (using bundled Merkle proofs), checksums to
protect against data corruption by the disk or network, and so on. However, it is important to
note that the only trusted information available is the app hash, and all other snapshot metadata
can be spoofed by adversaries.
Apps may also want to consider state sync denial-of-service vectors, where adversaries provide
invalid or harmful snapshots to prevent nodes from joining the network. The application can
counteract this by asking Tendermint to ban peers. As a last resort, node operators can use
P2P configuration options to whitelist a set of trusted peers that can provide valid snapshots.
##### Transition to Consensus
Once the snapshots have all been restored, Tendermint gathers additional information necessary for
bootstrapping the node (e.g. chain ID, consensus parameters, validator sets, and block headers)
from the genesis file and light client RPC servers. It also calls `Info` to verify the following:
* that the app hash from the snapshot it has delivered to the Application matches the apphash
stored in the next height's block (in next block execution), or the current block's height
(same block execution)
* that the version that the Application returns in `ResponseInfo` matches the version in the
current height's block header
Once the state machine has been restored and Tendermint has gathered this additional
information, it transitions to block sync (if enabled) to fetch any remaining blocks up the chain
head, and then transitions to regular consensus operation. At this point the node operates like
any other node, apart from having a truncated block history at the height of the restored snapshot.

102
spec/abci++/abci++_client_server_002_draft.md Normal file
View File

@@ -0,0 +1,102 @@
---
order: 5
title: Client and Server
---
# Client and Server
This section is for those looking to implement their own ABCI Server, perhaps in
a new programming language.
You are expected to have read all previous sections of ABCI++ specification, namely
[Basic Concepts](./abci%2B%2B_basic_concepts_002_draft.md),
[Methods](./abci%2B%2B_methods_002_draft.md),
[Application Requirements](./abci%2B%2B_app_requirements_002_draft.md), and
[Expected Behavior](./abci%2B%2B_tmint_expected_behavior_002_draft.md).
## Message Protocol and Synchrony
The message protocol consists of pairs of requests and responses defined in the
[protobuf file](../../proto/tendermint/abci/types.proto).
Some messages have no fields, while others may include byte-arrays, strings, integers,
or custom protobuf types.
For more details on protobuf, see the [documentation](https://developers.google.com/protocol-buffers/docs/overview).
As of v0.36 requests are synchronous. For each of ABCI++'s four connections (see
[Connections](./abci%2B%2B_app_requirements_002_draft.md)), when Tendermint issues a request to the
Application, it will wait for the response before continuing execution. As a side effect,
requests and responses are ordered for each connection, but not necessarily across connections.
## Server Implementations
To use ABCI in your programming language of choice, there must be an ABCI
server in that language. Tendermint supports four implementations of the ABCI server:
- in Tendermint's repository:
- In-process
- ABCI-socket
- GRPC
- [tendermint-rs](https://github.com/informalsystems/tendermint-rs)
- [tower-abci](https://github.com/penumbra-zone/tower-abci)
The implementations in Tendermint's repository can be tested using `abci-cli` by setting
the `--abci` flag appropriately.
See examples, in various stages of maintenance, in
[Go](https://github.com/tendermint/tendermint/tree/master/abci/server),
[JavaScript](https://github.com/tendermint/js-abci),
[C++](https://github.com/mdyring/cpp-tmsp), and
[Java](https://github.com/jTendermint/jabci).
### In Process
The simplest implementation uses function calls in Golang.
This means ABCI applications written in Golang can be linked with Tendermint Core and run as a single binary.
### GRPC
If you are not using Golang,
but [GRPC](https://grpc.io/) is available in your language, this is the easiest approach,
though it will have significant performance overhead.
Please check GRPC's documentation to know to set up the Application as an
ABCI GRPC server.
### Socket
Tendermint's socket-based ABCI interface is an asynchronous,
raw socket server which provides ordered message passing over unix or tcp.
Messages are serialized using Protobuf3 and length-prefixed with a [signed Varint](https://developers.google.com/protocol-buffers/docs/encoding?csw=1#signed-integers).
If GRPC is not available in your language, your application requires higher
performance, or otherwise enjoy programming, you may implement your own
ABCI server using the Tendermint's socket-based ABCI interface.
The first step is to auto-generate the relevant data
types and codec in your language using `protoc`.
In addition to being proto3 encoded, messages coming over
the socket are length-prefixed. proto3 doesn't have an
official length-prefix standard, so we use our own. The first byte in
the prefix represents the length of the Big Endian encoded length. The
remaining bytes in the prefix are the Big Endian encoded length.
For example, if the proto3 encoded ABCI message is `0xDEADBEEF` (4
bytes long), the length-prefixed message is `0x0104DEADBEEF` (`01` byte for encoding the length `04` of the message). If the proto3
encoded ABCI message is 65535 bytes long, the length-prefixed message
would start with 0x02FFFF.
Note that this length-prefixing scheme does not apply for GRPC.
Note that your ABCI server must be able to support multiple connections, as
Tendermint uses four connections.
## Client
There are currently two use-cases for an ABCI client. One is testing
tools that allow ABCI requests to be sent to the actual application via
command line. An example of this is `abci-cli`, which accepts CLI commands
to send corresponding ABCI requests.
The other is a consensus engine, such as Tendermint Core,
which makes ABCI requests to the application as prescribed by the consensus
algorithm used.

20
spec/abci++/abci++_methods_002_draft.md
View File

@@ -80,13 +80,15 @@ title: Methods
* **Usage**:
* Called once upon genesis.
* If ResponseInitChain.Validators is empty, the initial validator set will be the RequestInitChain.Validators
* If ResponseInitChain.Validators is not empty, it will be the initial
validator set (regardless of what is in RequestInitChain.Validators).
* If `ResponseInitChain.Validators` is empty, the initial validator set will be the `RequestInitChain.Validators`
* If `ResponseInitChain.Validators` is not empty, it will be the initial
validator set (regardless of what is in `RequestInitChain.Validators`).
* This allows the app to decide if it wants to accept the initial validator
set proposed by tendermint (ie. in the genesis file), or if it wants to use
a different one (perhaps computed based on some application specific
information in the genesis file).
set proposed by tendermint (ie. in the genesis file), or if it wants to use
a different one (perhaps computed based on some application specific
information in the genesis file).
* Both `ResponseInitChain.Validators` and `ResponseInitChain.Validators` are [ValidatorUpdate](#validatorupdate) structs.
So, technically, they both are _updating_ the set of validators from the empty set.
### Query
@@ -302,7 +304,7 @@ title: Methods
|-------------------------|--------------------------------------------------|---------------------------------------------------------------------------------------------|--------------|
| tx_records | repeated [TxRecord](#txrecord) | Possibly modified list of transactions that have been picked as part of the proposed block. | 2 |
| app_hash | bytes | The Merkle root hash of the application state. | 3 |
| tx_results | repeated [ExecTxResult](#txresult) | List of structures containing the data resulting from executing the transactions | 4 |
| tx_results | repeated [ExecTxResult](#exectxresult) | List of structures containing the data resulting from executing the transactions | 4 |
| validator_updates | repeated [ValidatorUpdate](#validatorupdate) | Changes to validator set (set voting power to 0 to remove). | 5 |
| consensus_param_updates | [ConsensusParams](#consensusparams) | Changes to consensus-critical gas, size, and other parameters. | 6 |
@@ -414,7 +416,7 @@ Note that, if _p_ has a non-`nil` _validValue_, Tendermint will use it as propos
|-------------------------|--------------------------------------------------|-----------------------------------------------------------------------------------|--------------|
| status | [ProposalStatus](#proposalstatus) | `enum` that signals if the application finds the proposal valid. | 1 |
| app_hash | bytes | The Merkle root hash of the application state. | 2 |
| tx_results | repeated [ExecTxResult](#txresult) | List of structures containing the data resulting from executing the transactions. | 3 |
| tx_results | repeated [ExecTxResult](#exectxresult) | List of structures containing the data resulting from executing the transactions. | 3 |
| validator_updates | repeated [ValidatorUpdate](#validatorupdate) | Changes to validator set (set voting power to 0 to remove). | 4 |
| consensus_param_updates | [ConsensusParams](#consensusparams) | Changes to consensus-critical gas, size, and other parameters. | 5 |
@@ -582,7 +584,7 @@ from this condition, but not sure), and _p_ receives a Precommit message for rou
| Name | Type | Description | Field Number |
|-------------------------|-------------------------------------------------------------|----------------------------------------------------------------------------------|--------------|
| events | repeated [Event](abci++_basic_concepts_002_draft.md#events) | Type & Key-Value events for indexing | 1 |
| tx_results | repeated [ExecTxResult](#txresult) | List of structures containing the data resulting from executing the transactions | 2 |
| tx_results | repeated [ExecTxResult](#exectxresult) | List of structures containing the data resulting from executing the transactions | 2 |
| validator_updates | repeated [ValidatorUpdate](#validatorupdate) | Changes to validator set (set voting power to 0 to remove). | 3 |
| consensus_param_updates | [ConsensusParams](#consensusparams) | Changes to consensus-critical gas, size, and other parameters. | 4 |
| app_hash | bytes | The Merkle root hash of the application state. | 5 |