tendermint/spec/consensus/light/verification.md

Core Verification

A lite client is a process that connects to Tendermint full node(s) and then tries to verify application data using the Merkle proofs.

Problem statement

We assume that the lite client knows a (base) header inithead it trusts (by social consensus or because the lite client has decided to trust the header before). The goal is to check whether another header newhead can be trusted based on the data in inithead.

The correctness of the protocol is based on the assumption that inithead was generated by an instance of Tendermint consensus.

Definitions

Data structures

In the following, only the details of the data structures needed for this specification are given.

  type Header struct {
       Height               int64
       Time                 Time          // the chain time when the header (block) was generated
       ValidatorsHash       []byte        // hash of the validators for the current block
       NextValidatorsHash   []byte        // hash of the validators for the next block
  }

  type SignedHeader struct {
       Header        Header
       Commit        Commit            // commit for the given header
  }

  type ValidatorSet struct {
       Validators         []Validator
       TotalVotingPower   int64
  }

  type Validator struct {
       Address       Address           // validator address (we assume validator's addresses are unique)
       VotingPower   int64             // validator's voting power
  }

  type TrustedState {
       SignedHeader   SignedHeader
       ValidatorSet   ValidatorSet
  }

Functions

For the purpose of this lite client specification, we assume that the Tendermint Full Node exposes the following functions over Tendermint RPC:

    // returns signed header: Header with Commit, for the given height
    func Commit(height int64) (SignedHeader, error)

    // returns validator set for the given height
    func Validators(height int64) (ValidatorSet, error)

Furthermore, we assume the following auxiliary functions:

    // returns true if the commit is for the header, ie. if it contains
    // the correct hash of the header; otherwise false
    func matchingCommit(header Header, commit Commit) bool

    // returns the set of validators from the given validator set that
    // committed the block (that correctly signed the block)
    // it assumes signature verification so it can be computationally expensive
    func signers(commit Commit, validatorSet ValidatorSet) []Validator

    // return the voting power the validators in v1 have according to their voting power in set v2
    // it does not assume signature verification
    func votingPowerIn(v1 []Validator, v2 ValidatorSet) int64

    // returns hash of the given validator set
    func hash(v2 ValidatorSet) []byte

Failure Model

For the purpose of model definitions we assume that there exists a function validators that returns the corresponding validator set for the given hash. The lite client specification is defined with respect to the following failure model:

Given a known bound TRUSTED_PERIOD, and a block b with header h generated at time Time (i.e. h.Time = Time), a set of validators that hold more than 2/3 of the voting power in validators(b.Header.NextValidatorsHash) is correct until time b.Header.Time + TRUSTED_PERIOD.

Assumption: "correct" is defined w.r.t. realtime (some Newtonian global notion of time, i.e., wall time), while Header.Time corresponds to the BFT time. In this note, we assume that clocks of correct processes are synchronized (for example using NTP), and therefore there is bounded clock drift (CLOCK_DRIFT) between local clocks and BFT time. More precisely, for every correct lite client process and every header.Time (i.e. BFT Time, for a header correctly generated by the Tendermint consensus), the following inequality holds: Header.Time < now + CLOCK_DRIFT, where now corresponds to the system clock at the lite client process.

Furthermore, we assume that TRUSTED_PERIOD is (several) order of magnitude bigger than CLOCK_DRIFT (TRUSTED_PERIOD >> CLOCK_DRIFT), as CLOCK_DRIFT (using NTP) is in the order of milliseconds and TRUSTED_PERIOD is in the order of weeks.

We expect a lite client process defined in this document to be used in the context in which there is some larger period during which misbehaving validators can be detected and punished (we normally refer to it as PUNISHMENT_PERIOD). Furthermore, we assume that TRUSTED_PERIOD < PUNISHMENT_PERIOD and that they are normally of the same order of magnitude, for example TRUSTED_PERIOD = PUNISHMENT_PERIOD / 2. Note that PUNISHMENT_PERIOD is often referred to as an unbonding period due to the "bonding" mechanism in modern proof of stake systems.

The specification in this document considers an implementation of the lite client under the Failure Model defined above. Mechanisms like fork accountability and evidence submission are defined in the context of PUNISHMENT_PERIOD and they incentivize validators to follow the protocol specification defined in this document. If they don't, and we have 1/3 (or more) faulty validators, safety may be violated. Our approach then is to detect these cases (after the fact), and take suitable repair actions (automatic and social). This is discussed in document on Fork accountability.

Remark: This failure model might change to a hybrid version that takes heights into account in the future.

Functions

In the functions below we will be using trustThreshold as a parameter. For simplicity we assume that trustThreshold is a float between 1/3 and 2/3 and we will not be checking it in the pseudo-code.

VerifySingle. The function VerifySingle attempts to validate given untrusted header and the corresponding validator sets based on a given trusted state. It ensures that the trusted state is still within its trusted period, and that the untrusted header is within assume clockDrift bound of the passed time now. Note that this function is not making external (RPC) calls to the full node; the whole logic is based on the local (given) state. This function is supposed to be used by the IBC handlers.

func VerifySingle(untrustedSh SignedHeader,
                  untrustedVs ValidatorSet,
                  untrustedNextVs ValidatorSet,
                  trustedState TrustedState,
                  trustThreshold float,
                  trustingPeriod Duration,
                  clockDrift Duration,
                  now Time) (TrustedState, error) {

    if untrustedSh.Header.Time > now + clockDrift {
        return (trustedState, ErrInvalidHeaderTime)
    }

    trustedHeader = trustedState.SignedHeader.Header
    if !isWithinTrustedPeriod(trustedHeader, trustingPeriod, now) {
        return (state, ErrHeaderNotWithinTrustedPeriod)
    }

    // we assume that time it takes to execute verifySingle function
    // is several order of magnitudes smaller than trustingPeriod
    error = verifySingle(
                trustedState,
                untrustedSh,
                untrustedVs,
                untrustedNextVs,
                trustThreshold)

    if error != nil return (state, error)

    // the untrusted header is now trusted
    newTrustedState = TrustedState(untrustedSh, untrustedNextVs)
    return (newTrustedState, nil)
}

// return true if header is within its lite client trusted period; otherwise returns false
func isWithinTrustedPeriod(header Header,
                           trustingPeriod Duration,
                           now Time) bool {

    return header.Time + trustedPeriod > now
}

Note that in case VerifySingle returns without an error (untrusted header is successfully verified) then we have a guarantee that the transition of the trust from trustedState to newTrustedState happened during the trusted period of trustedState.SignedHeader.Header.

verifySingle. The function verifySingle verifies a single untrusted header against a given trusted state. It includes all validations and signature verification. It is not publicly exposed since it does not check for header expiry (time constraints) and hence it's possible to use it incorrectly.

func verifySingle(trustedState TrustedState,
                  untrustedSh SignedHeader,
                  untrustedVs ValidatorSet,
                  untrustedNextVs ValidatorSet,
                  trustThreshold float) error {

    untrustedHeader = untrustedSh.Header
    untrustedCommit = untrustedSh.Commit

    trustedHeader = trustedState.SignedHeader.Header
    trustedVs = trustedState.ValidatorSet

    if trustedHeader.Height >= untrustedHeader.Height return ErrNonIncreasingHeight
    if trustedHeader.Time >= untrustedHeader.Time return ErrNonIncreasingTime

    // validate the untrusted header against its commit, vals, and next_vals
    error = validateSignedHeaderAndVals(untrustedSh, untrustedVs, untrustedNextVs)
    if error != nil return error

    // check for adjacent headers
    if untrustedHeader.Height == trustedHeader.Height + 1 {
        if trustedHeader.NextValidatorsHash != untrustedHeader.ValidatorsHash {
            return ErrInvalidAdjacentHeaders
        }
    } else {
        error = verifyCommitTrusting(trustedVs, untrustedCommit, untrustedVs, trustThreshold)
        if error != nil return error
    }

    // verify the untrusted commit
    return verifyCommitFull(untrustedVs, untrustedCommit)
}

// returns nil if header and validator sets are consistent; otherwise returns error
func validateSignedHeaderAndVals(signedHeader SignedHeader, vs ValidatorSet, nextVs ValidatorSet) error {
    header = signedHeader.Header
    if hash(vs) != header.ValidatorsHash return ErrInvalidValidatorSet
    if hash(nextVs) != header.NextValidatorsHash return ErrInvalidNextValidatorSet
    if !matchingCommit(header, signedHeader.Commit) return ErrInvalidCommitValue
    return nil
}

// returns nil if at least single correst signer signed the commit; otherwise returns error
func verifyCommitTrusting(trustedVs ValidatorSet,
                          commit Commit,
                          untrustedVs ValidatorSet,
                          trustLevel float) error {

    totalPower := trustedVs.TotalVotingPower
    signedPower := votingPowerIn(signers(commit, untrustedVs), trustedVs)

    // check that the signers account for more than max(1/3, trustLevel) of the voting power
    // this ensures that there is at least single correct validator in the set of signers
    if signedPower < max(1/3, trustLevel) * totalPower return ErrInsufficientVotingPower
    return nil
}

// returns nil if commit is signed by more than 2/3 of voting power of the given validator set
// return error otherwise
func verifyCommitFull(vs ValidatorSet, commit Commit) error {
    totalPower := vs.TotalVotingPower;
    signedPower := votingPowerIn(signers(commit, vs), vs)

    // check the signers account for +2/3 of the voting power
    if signedPower * 3 <= totalPower * 2 return ErrInvalidCommit
    return nil
}

VerifyHeaderAtHeight. The function VerifyHeaderAtHeight captures high level logic, i.e., application call to the lite client module to download and verify header for some height.

func VerifyHeaderAtHeight(untrustedHeight int64,
                          trustedState TrustedState,
                          trustThreshold float,
                          trustingPeriod Duration,
                          clockDrift Duration) (TrustedState, error)) {

    trustedHeader := trustedState.SignedHeader.Header

    now := System.Time()
    if !isWithinTrustedPeriod(trustedHeader, trustingPeriod, now) {
        return (trustedState, ErrHeaderNotWithinTrustedPeriod)
    }

    newTrustedState, err := VerifyBisection(untrustedHeight,
                                            trustedState,
                                            trustThreshold,
                                            trustingPeriod,
                                            clockDrift,
                                            now)

    if err != nil return (trustedState, err)

    now = System.Time()
    if !isWithinTrustedPeriod(trustedHeader, trustingPeriod, now) {
        return (trustedState, ErrHeaderNotWithinTrustedPeriod)
    }

    return (newTrustedState, err)
}

Note that in case VerifyHeaderAtHeight returns without an error (untrusted header is successfully verified) then we have a guarantee that the transition of the trust from trustedState to newTrustedState happened during the trusted period of trustedState.SignedHeader.Header.

VerifyBisection. The function VerifyBisection implements recursive logic for checking if it is possible building trust relationship between trustedState and untrusted header at the given height over finite set of (downloaded and verified) headers.

func VerifyBisection(untrustedHeight int64,
                     trustedState TrustedState,
                     trustThreshold float,
                     trustingPeriod Duration,
                     clockDrift Duration,
                     now Time) (TrustedState, error) {

    untrustedSh, error := Commit(untrustedHeight)
    if error != nil return (trustedState, ErrRequestFailed)

    untrustedHeader = untrustedSh.Header

    // note that we pass now during the recursive calls. This is fine as
    // all other untrusted headers we download during recursion will be
    // for a smaller heights, and therefore should happen before.
    if untrustedHeader.Time > now + clockDrift {
        return (trustedState, ErrInvalidHeaderTime)
    }

    untrustedVs, error := Validators(untrustedHeight)
    if error != nil return (trustedState, ErrRequestFailed)

    untrustedNextVs, error := Validators(untrustedHeight + 1)
    if error != nil return (trustedState, ErrRequestFailed)

    error = verifySingle(
             trustedState,
             untrustedSh,
             untrustedVs,
             untrustedNextVs,
             trustThreshold)

    if fatalError(error) return (trustedState, error)

    if error == nil {
        // the untrusted header is now trusted.
        newTrustedState = TrustedState(untrustedSh, untrustedNextVs)
        return (newTrustedState, nil)
    }

    // at this point in time we need to do bisection
    pivotHeight := ceil((trustedHeader.Height + untrustedHeight) / 2)

    error, newTrustedState = VerifyBisection(pivotHeight,
                                             trustedState,
                                             trustThreshold,
                                             trustingPeriod,
                                             clockDrift,
                                             now)
    if error != nil return (newTrustedState, error)

    return VerifyBisection(untrustedHeight,
                           newTrustedState,
                           trustThreshold,
                           trustingPeriod,
                           clockDrift,
                           now)
}

func fatalError(err) bool {
    return err == ErrHeaderNotWithinTrustedPeriod OR
           err == ErrInvalidAdjacentHeaders OR
           err == ErrNonIncreasingHeight OR
           err == ErrNonIncreasingTime OR
           err == ErrInvalidValidatorSet OR
           err == ErrInvalidNextValidatorSet OR
           err == ErrInvalidCommitValue OR
           err == ErrInvalidCommit
}

The case untrusted_h.Header.height < trusted_h.Header.height

In the use case where someone tells the lite client that application data that is relevant for it can be read in the block of height k and the lite client trusts a more recent header, we can use the hashes to verify headers "down the chain." That is, we iterate down the heights and check the hashes in each step.

Remark. For the case were the lite client trusts two headers i and j with i < k < j, we should discuss/experiment whether the forward or the backward method is more effective.

func Backwards(trusted_h,untrusted_h) error {
  assert (untrusted_h.Header.height < trusted_h.Header.height)
  if !isWithinTrustedPeriod(trusted_h) return ErrHeaderNotTrusted(trusted_h)

  old := trusted_h
  for i := trusted_h.Header.height - 1; i > untrusted_h.Header.height; i-- {
    new := Commit(i)
    if (hash(new) != old.Header.hash) {
      return ErrInvalidAdjacentHeaders
    }
    old := new
    if !isWithinTrustedPeriod(trusted_h) return ErrHeaderNotTrusted(trusted_h)
  }
  if hash(untrusted_h) != old.Header.hash return ErrInvalidAdjacentHeaders
  return nil
 }

In order to incentivize correct behavior of validators that run Tendermint consensus protocol, fork detection protocol (it will be explained in different document) is executed in case of a fork (conflicting headers are detected). As detecting conflicting headers, its propagation through the network (by the gossip protocol) and execution of the fork accountability protocol on the chain takes time, the lite client logic assumes conservative value for trusted period. More precisely, in the context of lite client we always operate with a smaller trusted period that we call lite client trusted period (LITE_CLIENT_TRUSTED_PERIOD). If we assume that upper bound for fork detection, propagation and processing on the chain is denoted with fork procession period (FORK_PROCESSING_PERIOD), then the following formula holds: LITE_CLIENT_TRUSTED_PERIOD + FORK_PROCESSING_PERIOD < TRUSTED_PERIOD, where TRUSTED_PERIOD comes from the Tendermint Failure Model.

Assumption: In the following, we assume that untrusted_h.Header.height > trusted_h.Header.height. We will quickly discuss the other case in the next section.

We consider the following set-up:

• the lite client communicates with one full node
• the lite client locally stores all the headers that has passed basic verification and that are within lite client trust period. In the pseudo code below we write Store.Add(header) for this. If a header failed to verify, then the full node we are talking to is faulty and we should disconnect from it and reinitialise with new peer.
• If CanTrust returns error, then the lite client has seen a forged header or the trusted header has expired (it is outside its trusted period).
  • In case of forged header, the full node is faulty so lite client should disconnect and reinitialise with new peer. If the trusted header has expired, we need to reinitialise lite client with new trusted header (that is within its trusted period), but we don't necessarily need to disconnect from the full node we are talking to (as we haven't observed full node misbehavior in this case).

Context of this document

In order to make sure that full nodes have the incentive to follow the protocol, we have to address the following three Issues

1. The lite client needs a method to verify headers it obtains from a full node it connects to according to trust assumptions -- this document.

2. The lite client must be able to connect to other full nodes to detect and report on failures in the trust assumptions (i.e., conflicting headers) -- a future document (see #4215).

3. In the event the trust assumption fails (i.e., a lite client is fooled by a conflicting header), the Tendermint fork accountability protocol must account for the evidence -- a future document (see #3840).

The term "trusting" above indicates that the correctness of the protocol depends on this assumption. It is in the responsibility of the user that runs the lite client to make sure that the risk of trusting a corrupted/forged inithead is negligible.

• For each header h it has locally stored, the lite client stores whether it trusts h. We write trust(h) = true, if this is the case.

• signed header fields: contains a header and a commit for the current header; a "seen commit". In Tendermint consensus the "canonical commit" is stored in header height + 1.

  • Validator fields. We will write a validator as a tuple (v,p) such that
    • v is the identifier (we assume identifiers are unique in each validator set)
    • p is its voting power

Definitions

• TRUSTED_PERIOD: trusting period
• for realtime t, the predicate correct(v,t) is true if the validator v follows the protocol until time t (we will see about recovery later).

Tendermint Failure Model

If a block b is generated at time Time (and this time is stored in the block), then a set of validators that hold more than 2/3 of the voting power in validators(b.Header.NextValidatorsHash) is correct until time b.Header.Time + TRUSTED_PERIOD.

Formally, [ \sum_{(v,p) \in h.Header.NextV \wedge correct(v,h.Header.bfttime + TRUSTED_PERIOD)} p > 2/3 \sum_{(v,p) \in h.Header.NextV} p ]

Lite Client Trusting Spec

The lite client communicates with a full node and learns new headers. The goal is to locally decide whether to trust a header. Our implementation needs to ensure the following two properties:

• Lite Client Completeness: If header h was correctly generated by an instance of Tendermint consensus (and its age is less than the trusting period), then the lite client should eventually set trust(h) to true.

• Lite Client Accuracy: If header h was not generated by an instance of Tendermint consensus, then the lite client should never set trust(h) to true.

Remark: If in the course of the computation, the lite client obtains certainty that some headers were forged by adversaries (that is were not generated by an instance of Tendermint consensus), it may submit (a subset of) the headers it has seen as evidence of misbehavior.

Remark: In Completeness we use "eventually", while in practice trust(h) should be set to true before h.Header.bfttime + tp. If not, the block cannot be trusted because it is too old.

Remark: If a header h is marked with trust(h), but it is too old (its bfttime is more than tp ago), then the lite client should set trust(h) to false again.

Assumption: Initially, the lite client has a header inithead that it trusts correctly, that is, inithead was correctly generated by the Tendermint consensus.

To reason about the correctness, we may prove the following invariant.

Verification Condition: Lite Client Invariant. For each lite client l and each header h: if l has set trust(h) = true, then validators that are correct until time h.Header.bfttime + tp have more than two thirds of the voting power in h.Header.NextV.

Formally, [ \sum_{(v,p) \in h.Header.NextV \wedge correct(v,h.Header.bfttime + tp)} p > 2/3 \sum_{(v,p) \in h.Header.NextV} p ]

Remark. To prove the invariant, we will have to prove that the lite client only trusts headers that were correctly generated by Tendermint consensus, then the formula above follows from the Tendermint failure model.

High Level Solution

Upon initialization, the lite client is given a header inithead it trusts (by social consensus). It is assumed that inithead satisfies the lite client invariant. (If inithead has been correctly generated by Tendermint consensus, the invariant follows from the Tendermint Failure Model.) Note that the inithead should be within its trusted period during initialization.

When a lite clients sees a signed new header snh, it has to decide whether to trust the new header. Trust can be obtained by (possibly) the combination of three methods.

1. Uninterrupted sequence of proof. If a block is appended to the chain, where the last block is trusted (and properly committed by the old validator set in the next block), and the new block contains a new validator set, the new block is trusted if the lite client knows all headers in the prefix. Intuitively, a trusted validator set is assumed to only chose a new validator set that will obey the Tendermint Failure Model.

2. Trusting period. Based on a trusted block h, and the lite client invariant, which ensures the fault assumption during the trusting period, we can check whether at least one validator, that has been continuously correct from h.Header.bfttime until now, has signed snh. If this is the case, similarly to above, the chosen validator set in snh does not violate the Tendermint Failure Model.

3. Bisection. If a check according to the trusting period fails, the lite client can try to obtain a header hp whose height lies between h and snh in order to check whether h can be used to get trust for hp, and hp can be used to get trust for snh. If this is the case we can trust snh; if not, we may continue recursively.

How to use it

We consider the following use case: the lite client wants to verify a header for some given height k. Thus:

• it requests the signed header for height k from a full node
• it tries to verify this header with the methods described here.

This can be used in several settings:

• someone tells the lite client that application data that is relevant for it can be read in the block of height k.
• the lite clients wants the latest state. It asks a full nude for the current height, and uses the response for k.
• in case of inter-blockchain communication protocol (IBC) the light client runs on a chain and someone feeds it signed headers as input and it computes whether it can trust it.

Details

Observation 1. If h.Header.bfttime + tp > now, we trust the old validator set h.Header.NextV.

When we say we trust h.Header.NextV we do not trust that each individual validator in h.Header.NextV is correct, but we only trust the fact that less than 1/3 of them are faulty (more precisely, the faulty ones have less than 1/3 of the total voting power).

Correctness arguments

Towards Lite Client Accuracy:

• Assume by contradiction that untrusted_h was not generated correctly and the lite client sets trust to true because CheckSupport returns true.
• trusted_h is trusted and sufficiently new
• by Tendermint Fault Model, less than 1/3 of voting power held by faulty validators => at least one correct validator v has signed untrusted_h.
• as v is correct up to now, it followed the Tendermint consensus protocol at least up to signing untrusted_h => untrusted_h was correctly generated, we arrive at the required contradiction.

Towards Lite Client Completeness:

• The check is successful if sufficiently many validators of trusted_h are still validators in untrusted_h and signed untrusted_h.
• If untrusted_h.Header.height = trusted_h.Header.height + 1, and both headers were generated correctly, the test passes

Verification Condition: We may need a Tendermint invariant stating that if untrusted_h.Header.height = trusted_h.Header.height + 1 then signers(untrusted_h.Commit) \subseteq trusted_h.Header.NextV.

Remark: The variable trustThreshold can be used if the user believes that relying on one correct validator is not sufficient. However, in case of (frequent) changes in the validator set, the higher the trustThreshold is chosen, the more unlikely it becomes that CheckSupport returns true for non-adjacent headers.

Correctness arguments (sketch)

Lite Client Accuracy:

• Assume by contradiction that untrusted_h was not generated correctly and the lite client sets trust to true because CanTrustBisection returns nil.
• CanTrustBisection returns true only if all calls to CheckSupport in the recursion return nil.
• Thus we have a sequence of headers that all satisfied the CheckSupport
• again a contradiction

Lite Client Completeness:

This is only ensured if upon Commit(pivot) the lite client is always provided with a correctly generated header.

Stalling

With CanTrustBisection, a faulty full node could stall a lite client by creating a long sequence of headers that are queried one-by-one by the lite client and look OK, before the lite client eventually detects a problem. There are several ways to address this:

• Each call to Commit could be issued to a different full node
• Instead of querying header by header, the lite client tells a full node which header it trusts, and the height of the header it needs. The full node responds with the header along with a proof consisting of intermediate headers that the light client can use to verify. Roughly, Bisection would then be executed at the full node.
• We may set a timeout how long bisection may take.