# Creating a built-in application in Go ## Guide assumptions This guide is designed for beginners who want to get started with a Tendermint Core application from scratch. It does not assume that you have any prior experience with Tendermint Core. Tendermint Core is a service that provides a Byzantine fault tolerant consensus engine for state-machine replication. The replicated state-machine, or "application", can be written in any language that can send and receive protocol buffer messages. This tutorial is written for Go and uses Tendermint as a library, but applications not written in Go can use Tendermint to drive state-machine replication in a client-server model. This tutorial expects some understanding of the Go programming language. If you have never written Go, you may want to go through [Learn X in Y minutes Where X=Go](https://learnxinyminutes.com/docs/go/) first to familiarize yourself with the syntax. By following along with this guide, you'll create a Tendermint Core application called kvstore, a (very) simple distributed BFT key-value store. > Note: please use a released version of Tendermint with this guide. The guides will work with the latest released version. > Be aware that they may not apply to unreleased changes on master. > We strongly advise against using unreleased commits for your development. ## 1.1 Installing Go Please refer to [the official guide for installing Go](https://golang.org/doc/install). Verify that you have the latest version of Go installed: ```bash $ go version go version go1.17.5 darwin/amd64 ``` Note that the exact patch number may differ as Go releases come out. ## 1.2 Creating a new Go project We'll start by creating a new Go project. ```bash mkdir kvstore cd kvstore go mod init github.com// ``` Inside the example directory create a `main.go` file with the following content: > Note: there is no need to clone or fork Tendermint in this tutorial. ```go package main import ( "fmt" ) func main() { fmt.Println("Hello, Tendermint Core") } ``` When run, this should print "Hello, Tendermint Core" to the standard output. ```bash $ go run main.go Hello, Tendermint Core ``` ## 1.3 Writing a Tendermint Core application Tendermint Core communicates with an application through the Application BlockChain Interface (ABCI) protocol. All of the message types Tendermint uses for communicating with the application can be found in the ABCI [protobuf file](https://github.com/tendermint/spec/blob/b695d30aae69933bc0e630da14949207d18ae02c/proto/tendermint/abci/types.proto). We will begin by creating the basic scaffolding for an ABCI application in a new `app.go` file. The first step is to create a new type, `KVStoreApplication` with methods that implement the abci `Application` interface. Create a file called `app.go` and add the following contents: ```go package main import ( abcitypes "github.com/tendermint/tendermint/abci/types" ) type KVStoreApplication struct {} var _ abcitypes.Application = (*KVStoreApplication)(nil) func NewKVStoreApplication() *KVStoreApplication { return &KVStoreApplication{} } func (app *KVStoreApplication) Info(req abcitypes.RequestInfo) abcitypes.ResponseInfo { return abcitypes.ResponseInfo{} } func (app *KVStoreApplication) DeliverTx(req abcitypes.RequestDeliverTx) abcitypes.ResponseDeliverTx { return abcitypes.ResponseDeliverTx{Code: 0} } func (app *KVStoreApplication) CheckTx(req abcitypes.RequestCheckTx) abcitypes.ResponseCheckTx { return abcitypes.ResponseCheckTx{Code: 0} } func (app *KVStoreApplication) Commit() abcitypes.ResponseCommit { return abcitypes.ResponseCommit{} } func (app *KVStoreApplication) Query(req abcitypes.RequestQuery) abcitypes.ResponseQuery { return abcitypes.ResponseQuery{Code: 0} } func (app *KVStoreApplication) InitChain(req abcitypes.RequestInitChain) abcitypes.ResponseInitChain { return abcitypes.ResponseInitChain{} } func (app *KVStoreApplication) BeginBlock(req abcitypes.RequestBeginBlock) abcitypes.ResponseBeginBlock { return abcitypes.ResponseBeginBlock{} } func (app *KVStoreApplication) EndBlock(req abcitypes.RequestEndBlock) abcitypes.ResponseEndBlock { return abcitypes.ResponseEndBlock{} } func (app *KVStoreApplication) ListSnapshots(abcitypes.RequestListSnapshots) abcitypes.ResponseListSnapshots { return abcitypes.ResponseListSnapshots{} } func (app *KVStoreApplication) OfferSnapshot(abcitypes.RequestOfferSnapshot) abcitypes.ResponseOfferSnapshot { return abcitypes.ResponseOfferSnapshot{} } func (app *KVStoreApplication) LoadSnapshotChunk(abcitypes.RequestLoadSnapshotChunk) abcitypes.ResponseLoadSnapshotChunk { return abcitypes.ResponseLoadSnapshotChunk{} } func (app *KVStoreApplication) ApplySnapshotChunk(abcitypes.RequestApplySnapshotChunk) abcitypes.ResponseApplySnapshotChunk { return abcitypes.ResponseApplySnapshotChunk{} } ``` ### 1.3.1 Add a persistent data store Our application will need to write its state out to persistent storage so that it can stop and start without losing all of its data. For this tutorial, we will use [BadgerDB](https://github.com/dgraph-io/badger). Badger is a fast embedded key-value store. First, add Badger as a dependency of your go module using the `go get` command: `go get github.com/dgraph-io/badger/v3` Next, let's update the application and its constructor to receive a handle to the database. Update the application struct as follows: ```go type KVStoreApplication struct { db *badger.DB pendingBlock *badger.Txn } ``` And change the constructor to set the appropriate field when creating the application: ```go func NewKVStoreApplication(db *badger.DB) *KVStoreApplication { return &KVStoreApplication{db: db} } ``` The `pendingBlock` keeps track of the transactions that will update the application's state when a block is completed. Don't worry about it for now, we'll get to that later. Finally, update the `import` stanza at the top to include the `Badger` library: ```go import( "github.com/dgraph-io/badger/v3" abcitypes "github.com/tendermint/tendermint/abci/types" ) ``` ### 1.3.1 CheckTx When Tendermint Core receives a new transaction, Tendermint asks the application if the transaction is acceptable. In our new application, let's implement some basic validation for the transactions it will receive. For our KV store application, a transaction is a string with the form `key=value`, indicating a key and value to write to the store. Add the following helper method to `app.go`: ```go func (app *KVStoreApplication) validateTx(tx []byte) uint32 { parts := bytes.SplitN(tx, []byte("="), 2) // check that the transaction is not malformed if len(parts) != 2 || len(parts[0]) == 0 { return 1 } return 0 } ``` And call it from within your `CheckTx` method: ```go func (app *KVStoreApplication) CheckTx(req abcitypes.RequestCheckTx) abcitypes.ResponseCheckTx { code := app.validateTx(req.Tx) return abcitypes.ResponseCheckTx{Code: code} } ``` Any response with a non-zero code will be considered invalid by Tendermint. Our `CheckTx` logic returns `0` to Tendermint when a transaction passes its validation checks. The specific value of the code is meaningless to Tendermint. Non-zero codes are logged by Tendermint so applications can provide more specific information on why the transaction was rejected. Note that `CheckTx` _does not execute_ the transaction, it only verifies that that the transaction _could_ be executed. We do not know yet if the rest of the network has agreed to accept this transaction into a block. Finally, make sure to add the `bytes` package to the your import stanza at the top of `app.go`: ```go import( "bytes" "github.com/dgraph-io/badger/v3" abcitypes "github.com/tendermint/tendermint/abci/types" ) ``` While this `CheckTx` is simple and only validates that the transaction is well-formed, it is very common for `CheckTx` to make more complex use of the state of an application. ### 1.3.2 BeginBlock -> DeliverTx -> EndBlock -> Commit When the Tendermint consensus engine has decided on the block, the block is transferred to the application over three ABCI method calls: `BeginBlock`, `DeliverTx`, and `EndBlock`. `BeginBlock` is called once to indicate to the application that it is about to receive a block. `DeliverTx` is called repeatedly, once for each `Tx` that was included in the block. `EndBlock` is called once to indicate to the application that no more transactions will be delivered to the application. To implement these calls in our application we're going to make use of Badger's transaction mechanism. Bagder uses the term _transaction_ in the context of databases, be careful not to confuse it with _blockchain transactions_. First, let's create a new Badger `Txn` during `BeginBlock`: ```go func (app *KVStoreApplication) BeginBlock(req abcitypes.RequestBeginBlock) abcitypes.ResponseBeginBlock { app.pendingBlock = app.db.NewTransaction(true) return abcitypes.ResponseBeginBlock{} } ``` Next, let's modify `DeliverTx` to add the `key` and `value` to the database `Txn` every time our application receives a new `RequestDeliverTx`. ```go func (app *KVStoreApplication) DeliverTx(req abcitypes.RequestDeliverTx) abcitypes.ResponseDeliverTx { if code := app.validateTx(req.Tx); code != 0 { return abcitypes.ResponseDeliverTx{Code: code} } parts := bytes.SplitN(req.Tx, []byte("="), 2) key, value := parts[0], parts[1] if err := app.pendingBlock.Set(key, value); err != nil { log.Panicf("Error reading database, unable to verify tx: %v", err) } return abcitypes.ResponseDeliverTx{Code: 0} } ``` Note that we check the validity of the transaction _again_ during `DeliverTx`. Transactions are not guaranteed to be valid when they are delivered to an application. This can happen if the application state is used to determine transaction validity. Application state may have changed between when the `CheckTx` was initially called and when the transaction was delivered in `DeliverTx` in a way that rendered the transaction no longer valid. Also note that we don't commit the Badger `Txn` we are building during `DeliverTx`. Other methods, such as `Query`, rely on a consistent view of the application's state. The application should only update its state when the full block has been delivered. The `Commit` method indicates that the full block has been delivered. During `Commit`, the application should persist the pending `Txn`. Let's modify our `Commit` method to persist the new state to the database: ```go func (app *KVStoreApplication) Commit() abcitypes.ResponseCommit { if err := app.pendingBlock.Commit(); err != nil { log.Panicf("Error writing to database, unable to commit block: %v", err) } return abcitypes.ResponseCommit{Data: []byte{}} } ``` Finally, make sure to add the `log` library to the `import` stanza as well: ```go import ( "bytes" "log" "github.com/dgraph-io/badger/v3" abcitypes "github.com/tendermint/tendermint/abci/types" ) ``` You may have noticed that the application we are writing will _crash_ if it receives an unexpected error from the database during the `DeliverTx` or `Commit` methods. This is not an accident. If the application received an error from the database, there is no deterministic way for it to make progress so the only safe option is to terminate. ### 1.3.3 Query Method We'll want to be able to determine if a transaction was committed to the state-machine. To do this, let's implement the `Query` method in `app.go`: ```go func (app *KVStoreApplication) Query(req abcitypes.RequestQuery) abcitypes.ResponseQuery { resp := abcitypes.ResponseQuery{Key: req.Data} dbErr := app.db.View(func(txn *badger.Txn) error { item, err := txn.Get(req.Data) if err != nil { if err != badger.ErrKeyNotFound { return err } resp.Log = "key does not exist" return nil } return item.Value(func(val []byte) error { resp.Log = "exists" resp.Value = val return nil }) }) if dbErr != nil { log.Panicf("Error reading database, unable to verify tx: %v", dbErr) } return resp } ``` ## 1.3.4 Additional Methods You'll notice that we left several methods unchanged. Specifically, we have yet to implement the `Info` and `InitChain` methods and we did not implement any of the `*Snapthot` methods. These methods are all important for running Tendermint applications in production but are not required for getting a very simple application up and running. To better understand these methods and why they are useful, check out the Tendermint [specification on ABCI](https://github.com/tendermint/spec/tree/20b2abb5f9a83c2d9d97b53e555e4ea5a6bd7dc4/spec/abci). ## 1.4 Starting an application and a Tendermint Core instance in the same process Now that we have the basic functionality of our application in place, let's put it all together inside of our `main.go` file. Add the following code to your `main.go` file: ```go package main import ( "flag" "fmt" "log" "os" "os/signal" "path/filepath" "syscall" "github.com/dgraph-io/badger/v3" "github.com/spf13/viper" abciclient "github.com/tendermint/tendermint/abci/client" cfg "github.com/tendermint/tendermint/config" tmlog "github.com/tendermint/tendermint/libs/log" nm "github.com/tendermint/tendermint/node" "github.com/tendermint/tendermint/types" ) var homeDir string func init() { flag.StringVar(&homeDir, "tm-home", "", "Path to the tendermint config directory (if empty, uses $HOME/.tendermint)") } func main() { flag.Parse() if homeDir == "" { homeDir = os.ExpandEnv("$HOME/.tendermint") } config := cfg.DefaultValidatorConfig() config.SetRoot(homeDir) viper.SetConfigFile(fmt.Sprintf("%s/%s", homeDir, "config/config.toml")) if err := viper.ReadInConfig(); err != nil { log.Fatalf("Reading config: %v", err) } if err := viper.Unmarshal(config); err != nil { log.Fatalf("Decoding config: %v", err) } if err := config.ValidateBasic(); err != nil { log.Fatalf("Invalid configuration data: %v", err) } gf, err := types.GenesisDocFromFile(config.GenesisFile()) if err != nil { log.Fatalf("Loading genesis document: %v", err) } dbPath := filepath.Join(homeDir, "badger") db, err := badger.Open(badger.DefaultOptions(dbPath)) if err != nil { log.Fatalf("Opening database: %v", err) } defer func() { if err := db.Close(); err != nil { log.Fatalf("Closing database: %v", err) } }() app := NewKVStoreApplication(db) acc := abciclient.NewLocalCreator(app) logger := tmlog.MustNewDefaultLogger(tmlog.LogFormatPlain, tmlog.LogLevelInfo, false) node, err := nm.New(config, logger, acc, gf) if err != nil { log.Fatalf("Creating node: %v", err) } node.Start() defer func() { node.Stop() node.Wait() }() c := make(chan os.Signal, 1) signal.Notify(c, os.Interrupt, syscall.SIGTERM) <-c } ``` This is a huge blob of code, so let's break down what it's doing. First, we load in the Tendermint Core configuration files: ```go ... config := cfg.DefaultValidatorConfig() config.SetRoot(homeDir) viper.SetConfigFile(fmt.Sprintf("%s/%s", homeDir, "config/config.toml")) if err := viper.ReadInConfig(); err != nil { log.Fatalf("Reading config: %v", err) } if err := viper.Unmarshal(config); err != nil { log.Fatalf("Decoding config: %v", err) } if err := config.ValidateBasic(); err != nil { log.Fatalf("Invalid configuration data: %v", err) } gf, err := types.GenesisDocFromFile(config.GenesisFile()) if err != nil { log.Fatalf("Loading genesis document: %v", err) } ... ``` Next, we create a database handle and use it to construct our ABCI application: ```go ... dbPath := filepath.Join(homeDir, "badger") db, err := badger.Open(badger.DefaultOptions(dbPath).WithTruncate(true)) if err != nil { log.Fatalf("Opening database: %v", err) } defer func() { if err := db.Close(); err != nil { log.Fatalf("Error closing database: %v", err) } }() app := NewKVStoreApplication(db) acc := abciclient.NewLocalCreator(app) ... ``` Then we construct a logger: ```go ... logger := tmlog.MustNewDefaultLogger(tmlog.LogFormatPlain, tmlog.LogLevelInfo, false) ... ``` Now we have everything setup to run the Tendermint node. We construct a node by passing it the configuration, the logger, a handle to our application and the genesis file: ```go ... node, err := nm.New(config, logger, acc, gf) if err != nil { log.Fatalf("Creating node: %v", err) } ... ``` Finally, we start the node: ```go ... node.Start() defer func() { node.Stop() node.Wait() }() ... ``` The additional logic at the end of the file allows the program to catch `SIGTERM`. This means that the node can shutdown gracefully when an operator tries to kill the program: ```go ... c := make(chan os.Signal, 1) signal.Notify(c, os.Interrupt, syscall.SIGTERM) <-c ... ``` ## 1.5 Getting Up and Running Our application is almost ready to run. Let's install the latest release version of the Tendermint library. From inside of the project directory, run: ```sh go get github.com/tendermint/tendermint@v0.35.0 ``` Next, we'll need to populate the Tendermint Core configuration files. This command will create a `tendermint-home` directory in your project and add a basic set of configuration files in `tendermint-home/config/`. For more information on what these files contain see [the configuration documentation](https://github.com/tendermint/tendermint/blob/v0.35.0/docs/nodes/configuration.md). From the root of your project, run: ```bash go run github.com/tendermint/tendermint/cmd/tendermint@v0.35.0 init validator --home ./tendermint-home ``` Next, build the application: ```bash go build -mod=mod -o my-app # use -mod=mod to automatically update go.sum ``` Everything is now in place to run your application. Run: ```bash $ ./my-app -tm-home ./tendermint-home ``` The application will begin producing blocks and you can see this reflected in the log output. You now have successfully started running an application using Tendermint Core 🎉🎉. ## 1.6 Using the application Your application is now running and emitting logs to the terminal. Now it's time to see what this application can do! Let's try submitting a transaction to our new application. Open another terminal window and run the following curl command: ```bash $ curl -s 'localhost:26657/broadcast_tx_commit?tx="tendermint=rocks"' ``` If everything went well, you should see a response indicating which height the transaction was included in the blockchain. Finally, let's make sure that transaction really was persisted by the application. Run the following command: ```bash $ curl -s 'localhost:26657/abci_query?data="tendermint"' ``` Let's examine the response object that this request returns. The request returns a `json` object with a `key` and `value` field set. ```json ... "key": "dGVuZGVybWludA==", "value": "cm9ja3M=", ... ``` Those values don't look like the `key` and `value` we sent to Tendermint, what's going on here? The response contain a `base64` encoded representation of the data we submitted. To get the original value out of this data, we can use the `base64` command line utility. Run: ``` echo cm9ja3M=" | base64 -d ``` ## Outro I hope everything went smoothly and your first, but hopefully not the last, Tendermint Core application is up and running. If not, please [open an issue on Github](https://github.com/tendermint/tendermint/issues/new/choose). To dig deeper, read [the docs](https://docs.tendermint.com/master/).