diff --git a/spec/blocksync/communication.md b/spec/blocksync/communication.md index cfe424b96..9c2c8f40c 100644 --- a/spec/blocksync/communication.md +++ b/spec/blocksync/communication.md @@ -1,6 +1,6 @@ ## Communication between peers and components within the blocksync reactor -A newly joined or recovering node is connecting to a number of peers in order to sync up to the latest height in the blockchain. The peers do not neccessarily have to be validators but we assume that at least one of the peers is correct. The node requests the status of peers to learn the maximum and minimum heights for which the peer has blocks. Based on this, the node decides on the maximum height it needs to sync up to. **There is no additional check** on whether the peers were lying about their heights. +A newly joined or recovering node is connecting to a number of peers in order to sync up to the latest height in the blockchain. The peers do not neccessarily have to be validators but we assume that at least one of the peers is correct. The node requests the status of peers in ordeer to learn the maximum and minimum heights for which the peer has blocks. Based on this, the node decides on the maximum height it needs to sync up to. **There is no additional check** on whether the peers were lying about their heights. Each peer has an open p2p channel. The number of total requests in flight is limited (`maxPendingRequests` initially set to `maxTotalRequesters`). Additionally, there is an upper limit on requests **per** peer (20). @@ -9,13 +9,23 @@ Once a node receives messages via the p2p channel, they are propagated further v On startup, the reactor fires up four go routines: 1. Process requests 2. Pool routine -3. Handle block sync channel messages +3. Handle p2p channel messages 4. Process peer updates -The pool routine picks out blocks form the block pool and processes them. It also checks whether we should switch to consensus if we are caught up. ToDo - change wording (Remove we, discuss whether the pool routing should be the one checking the condition for consensus). +The pool routine picks out blocks form the block pool and processes them. It also checks whether the node should switch to consensus. ToDo - change wording (Remove we, discuss whether the pool routing should be the one checking the condition for consensus). -Process peer updates is in fact sending a response to a peer requesting our status update. +All messages that go through the p2p channel are processed within the `processBlockSyncCh` routine. +On peer update messages the reactor adds or removes the sending peer. + +**Note** There is currently a check for whether we have a message from an empty peer. + +``` go +// XXX: Pool#RedoRequest can sometimes give us an empty peer. + if len(peerUpdate.NodeID) == 0 { + return + } + ``` ### Communication channels @@ -26,31 +36,36 @@ Process peer updates is in fact sending a response to a peer requesting our stat - Message size: maximum size of a block + size of proto block messages (response message prefix and key size) Messages processed via the channel: - - - `BlockRequest {height int64, peerID types.NodeID}` : request block at height `height` from peer `peerID`. - - `BlockResponse {block types.Block} `: Send `block` to peer that requested it. - - `NoBlockResponse{height int64} `: Indicates that a peer does not have a block at `height`. - - `StatusRequest {} `: Sent to a peer to request its status. - - `StatusResponse {height int64, base int64} `: Send to a peer the lowest and heights height of blocks within it's store (`store.Height()`, `store.Base()`). - -`HeaderRequest{height: int64}` - -`HeaderResponse{header: Header}` +| Message name | Message fields| Description | +| --- | ---| ---| +| `BlockRequest`| `height int64, peerID types.NodeID` | request block at height `height` from peer `peerID`| +| `BlockResponse`| `block types.Block `| Send `block` to peer that requested it | +| `NoBlockResponse` |`height int64`|Indicates that a peer does not have a block at `height`| +|`StatusRequest`| `{} `| Sent to a peer to request its status| +|`StatusResponse` | `height int64, base int64`|Send to a peer the lowest and heights height of blocks within it's store (`store.Height()`, `store.Base()`)| + |`HeaderRequest` | `height: int64`| Request a header from peer for verification| +|`HeaderResponse` |`header: Header`| Return the header for the corresponding height| ### Reactor channels This section describes, per reactor component, the open channels and information they process. #### `BlockPool` -`requestsCh chan<- BlockRequest` The number of requests is capped by a fixed parameter `maxPendingRequestsPerPeer` (initially set to 20). -errorsCh chan<- peerError ; Channel buffer size is limited. + +| Channel name | Channel msg type| Description | +| --- | ---| ---| +|`requestsCh` | `BlockRequest` |The number of requests is capped by a fixed parameter `maxPendingRequestsPerPeer` (initially set to 20)| +|`errorsCh` |`peerError`| Channel buffer size is limited| + The reactor sends a p2p block request once it receives a signal via this channel from the block pool. The block pool will first pick a peer (in round robin) and assign it this particular height. Once this is done, the reactor can request a block. #### `bpRequester` -`gotBlock chan struct{}`; capped at 1; Here we simply register a received block and keep waiting for the reactor to terminate or a redo request. -**Note**. It is not clear why we need this. - -`redoCh chan types.NodeID`; capped at 1 ; Signals the requester to redo a request for aparticular block after replacing the peer for this height +| Channel name | Channel msg type| Description | +| --- | ---| ---| +`gotBlock` | `struct{}`| capped at 1; Here we simply register a received block and keep waiting for the reactor to terminate or a redo request. **Note**. It is not clear why we need this. | +|`redoCh`| `types.NodeID`| capped at 1 ; Signals the requester to redo a request for aparticular block after replacing the peer for this height| When a block is received by a requester, the requester does a number of checks on the received block. Before marking the block as available, the requester verifies the following: - that we expected a block at the particular height. @@ -63,10 +78,13 @@ In the code there is the following ToDo listed: If the checks pass, the `block` field of the requester is populated with the new block and i sthus made available to the blocksync reactor. #### `Reactor` -`requestsCh chan BlockRequest` :size `maxTotalRequesters` -`errorsCh chan peerError` : size `maxPeerErrBuffer` -`didProcessCh chan struct{}` : size `1`. +| Channel name | Channel msg type| Description | +| --- | ---| ---| +|`requestsCh`|`BlockRequest`|size `maxTotalRequesters`| +|`errorsCh`| `peerError`| size `maxPeerErrBuffer`| +|`didProcessCh`|`struct{}`| size `1`| + The channel is created within the pool routine of the reactor and is used to signal that the reactor should check the block pool for new blocks. A message is sent to the channel after a fixed timeout (`trySyncTicker`). As we need two blocks to verify one of them (this is more clearly defined in [verification](./verification.md), if we miss only on of them, we will not wait for the sync timer to time out, but rather try quickly again until we fetch both. `switchToConsensusTicker`. In addition to the sync timeout, in the same routine, the reactor checks periodically whether the conditions to switch to consensus are fullfilled. \ No newline at end of file diff --git a/spec/blocksync/data_structures.md b/spec/blocksync/data_structures.md index cad2429e3..170eaac6b 100644 --- a/spec/blocksync/data_structures.md +++ b/spec/blocksync/data_structures.md @@ -5,7 +5,46 @@ There are four core components of the blocksync reactor: the reactor itself, a b The reactor verifies received blocks, executes them against the application and commits them into the blockstore of the node. It also sends out requests to peers asking for more blocks and contains the logic to switch from blocksync to consenus. It contains a pointer to the block pool. -The block pool stores the last executed block(`height`), keeps track of peers connected to a node, assigns requests to peers (by creating `requesters`), the current height for each peer, along with the number of pending requestes for each peer. +```go +type Reactor struct { + service.BaseService + logger log.Logger + + // immutable + initialState sm.State + // store + stateStore sm.Store + + blockExec *sm.BlockExecutor + store sm.BlockStore + pool *BlockPool + consReactor consensusReactor + blockSync *atomicBool + + chCreator p2p.ChannelCreator + peerEvents p2p.PeerEventSubscriber + + requestsCh <-chan BlockRequest + errorsCh <-chan peerError + + metrics *consensus.Metrics + eventBus *eventbus.EventBus + + syncStartTime time.Time + + lastTrustedBlock *TrustedBlockData +} +``` + +`TrustedBlockData` contains the last synced and verified block along with the commit used to verify it. + +```go +type TrustedBlockData struct { + block *types.Block + commit *types.Commit +} +``` +The block pool stores the last executed block(`height`), keeps track of peers connected to a node, the current height for each peer, along with the number of pending requestes for each peer and assigns requests to peers (by creating `requesters`). ```go type BlockPool { diff --git a/spec/blocksync/readme.md b/spec/blocksync/readme.md index beee8d26d..144371b7b 100644 --- a/spec/blocksync/readme.md +++ b/spec/blocksync/readme.md @@ -10,7 +10,7 @@ In a proof of work blockchain, syncing with the chain is the same process as sta The Blocksync Reactor's high level responsibility is to enable peers who are far behind the current state of the consensus to quickly catch up by downloading -many blocks in parallel, verifying their commits, and executing them against the +many blocks (that have already been decided) in parallel, verifying their commits, and executing them against the ABCI application. Tendermint full nodes run the Blocksync Reactor as a service to provide blocks @@ -36,9 +36,9 @@ However, receiving messages via the p2p channel and sending status updates to ot ### Switching from blocksync to consensus Ideally, the switch to consensus is done either after we have caught up to the maximum height reported by a peer or we have not advanced our height for more than 60s. -The former id checked by calling `isCaughtUp` inside `poolRoutine` periodically. This period is set with `switchToConsensusTicker`. We consider a node to be caught up if it is 1 height away from the maximum height reported by its peers. The reason we **do not catch up until the maximum height** (`pool.maxPeerHeight`)is that we cannot verify block at `pool.maxPeerHeight` without the `lastCommit` of the block at `pool.maxPeerHeight + 1`. +The former is checked by calling `isCaughtUp` inside `poolRoutine` periodically. This period is set with `switchToConsensusTicker`. We consider a node to be caught up if it is 1 height away from the maximum height reported by its peers. The reason we **do not catch up until the maximum height** (`pool.maxPeerHeight`)is that we cannot verify block at `pool.maxPeerHeight` without the `lastCommit` of the block at `pool.maxPeerHeight + 1`. -BlockSync **does not** switch to consensus until we have synced at least one block as we need to have vote extensions in order to participate in consensus . Vote extensions are not provided to the blocksync reactor after state sync and we need to receive them from one of our peers. +Blocksync **does not** switch to consensus until we have synced at least one block. We need to have vote extensions in order to participate in consensus and they are not provided to the blocksync reactor after state sync. We therefore need to receive them from one of our peers. ## Architecture and algorithm diff --git a/spec/blocksync/verification.md b/spec/blocksync/verification.md index 1e0b93a7c..1a6637f35 100644 --- a/spec/blocksync/verification.md +++ b/spec/blocksync/verification.md @@ -9,7 +9,7 @@ Based on this state we verify subsequent blocks. ### Trusted state -The light client additionally relies on the notion of a **trusting period**. A trusting period is the time during which we assume we can trust validators because, if we do detect misbehaviour, we can slash them - they are still bonded. Beyond this period, the validators do not have to have any bonded assets and cannot be held accountable for they misbheaviour. Blocksync-ing blocks will most often be outside this trusting period for a particular block. Therefore, the trusting period assumptions as they are in the light client, cannot be applied here. +The light client additionally relies on the notion of a **trusting period**. A trusting period is the time during which we assume we can trust validators because, if we do detect misbehaviour, we can slash them - they are still bonded. Beyond this period, the validators do not have to have any bonded assets and cannot be held accountable for their misbheaviour. Blocksync-ing blocks will most often be outside this trusting period for a particular block. Therefore, the trusting period assumptions as they are in the light client, cannot be applied here. This has different implications based on how we obtain the initial trusted state. Currently there are two possibilities. : 1. If the node is blocksync-ing from genesis, we assume we trust the validators provided in the initial state and use them to verify the initial block received from peers. @@ -25,7 +25,9 @@ In this case we only verify that the validator hash of the block matches the val This is in contrast with the definition of trust period in the light client. The trusting period in the light client is always tied to present time. If we try and validate a block at a time outside this period, we will first find the first block within the trusting period and verify backwards from it. -It is worth noting that, running block sync from the first height is significantly slower than running statesync first. However, statesync does not keep the entire blockchain history and some operators might opt not to state sync. If we are starting from gensis we trust the validator set given to us in initially. +It is worth noting that, running block sync from the first height is significantly slower than running statesync first. However, statesync does not keep the entire blockchain history and some operators might opt not to state sync. The reason is that, if sufficiently many nodes state sync and other nodes who have historical data fail or leave the network, we have no history. + +If we are starting from genesis we trust the validator set given to us in initially. **Improvement** We expect to 'witness'verify this block against the blocks at the same height from multiple peers. @@ -109,5 +111,4 @@ We have no guarantees on the correctness of this peer. - Once we replace a peer due to it having timed out or not reporting a correct block, nothing prevents a node from reconnecting to it - *there is no notion of blacklisted peers*. - If we connect to subset of peers, they could feed the node faulty data. Eventually, when the node switches to consensus, it would realize there is something wrong, but then the node itself might be blacklisted.. - There is no check whether the maximum height reported by peers is true or not. A slow node could report a very distant height to the node - for example 2000, when the blockchain is at height 1000 in fact. This would lead to one part of the condition to switch to consensus never being true. To prevent the node switching due to not advancing, the malicious node sends a new block very slowly. Thus the node progresses but can never participate in consensus. This issue could potentially be mitigated if, instead of taking the maximum height reported by peers, we report the lowest of their maximums. The idea is that peers should be close enought to the top of the chain in any case. -- A blocksyncing node can flood peers with requests - constantly reporting that it has not synced up. At the moment the maximum amount of requests received is limited and protects peers to some extend against this attack. -- \ No newline at end of file +- A blocksyncing node can flood peers with requests - constantly reporting that it has not synced up. At the moment the maximum amount of requests received is limited and protects peers to some extend against this attack. \ No newline at end of file