Commit Graph

756 Commits

Author SHA1 Message Date
M. J. Fromberger
7fe3e78a38 Update the psql indexer schema and implementation (#6868)
Update the schema and implementation of the Postgres event indexer to improve 
certain types of queries against the index. These changes address the use cases
raised by #6843, and are partly inspired by the prototype schema in that issue.

In the old schema, events were flattened, making it difficult to find all the events 
associated with a particular block or transaction. In addition, events with no key/value
attributes were entirely lost, since entries were generated only for attributes.

To address these issues, this new schema records blocks, transactions, events,
and attributes in separate tables, and provides views that join these tables to
give a more convenient query surface for block and transaction events.

- All events for a given block can be queried from the `block_events` view.
- All events for a given transaction can be queried from the `tx_events` view.
- Multiple events for the same key can be indexed for both blocks and transactions.

The tests have been reworked, but all of the existing test cases for the old schema 
still pass with the new implementation. Various other minor cleanups are included,

ADR-065 is also updated to reflect the updated schema.
2021-08-31 18:35:07 -04:00
M. J. Fromberger
41a361ed8d psql: add documentation and simplify constructor API (#6856)
Add documentation comments to the psql event sink package, and simplify the
constructor function so that it does not return the SQL database handle.  The
handle is needed for testing, so expose that via a separate method on the
concrete type.

Update the tests and existing usage for the change. This change does not affect
the behaviour of the sink, so there are no functional changes, only syntactic
updates.
2021-08-24 17:18:27 -04:00
William Banfield
bc2b529b95 inspect: add inspect mode for debugging crashed tendermint node (#6785)
EDIT: Updated, see [comment below]( https://github.com/tendermint/tendermint/pull/6785#issuecomment-897793175)

This change adds a sketch of the `Debug` mode. 

This change adds a `Debug` struct to the node package. This `Debug` struct is intended to be created and started by a command in the `cmd` directory. The `Debug` struct runs the RPC server on the data directories: both the state store and the block store.

This change required a good deal of refactoring. Namely, a new `rpc.go` file was added to the `node` package. This file encapsulates functions for starting RPC servers used by nodes. A potential additional change is to further factor this code into shared code _in_ the `rpc` package. 

Minor API tweaks were also made that seemed appropriate such as the mechanism for fetching routes from the `rpc/core` package.

Additional work is required to register the `Debug` service as a command in the `cmd` directory but I am looking for feedback on if this direction seems appropriate before diving much further.

closes: #5908
2021-08-24 18:12:06 +00:00
Sam Kleinman
d7c3a8f682 time: make median time library type private (#6853)
This is a very minor change, but I was looking through the code, and
this seems like it shouldn't be exported or used more broadly, so I've
moved it out.
2021-08-24 15:43:13 +00:00
Aleksandr Bezobchuk
cce0a3c171 docs: add package godoc for indexer (#6839)
* docs: add indexer godoc

* docs++

* docs++

* docs++

* docs++

* docs++

* Update state/indexer/doc.go

Co-authored-by: M. J. Fromberger <fromberger@interchain.io>

* Update state/indexer/doc.go

Co-authored-by: M. J. Fromberger <fromberger@interchain.io>

* Update state/indexer/doc.go

Co-authored-by: M. J. Fromberger <fromberger@interchain.io>

* Update state/indexer/doc.go

Co-authored-by: M. J. Fromberger <fromberger@interchain.io>

* Update state/indexer/doc.go

Co-authored-by: M. J. Fromberger <fromberger@interchain.io>

* Update state/indexer/doc.go

Co-authored-by: M. J. Fromberger <fromberger@interchain.io>

* docs++

Co-authored-by: M. J. Fromberger <fromberger@interchain.io>
2021-08-19 20:25:17 +00:00
William Banfield
4e96c6b234 tools: add mockery to tools.go and remove mockery version strings (#6787)
This change aims to keep versions of mockery consistent across developer laptops.

This change adds mockery to the `tools.go` file so that its version can be managed consistently in the `go.mod` file.

Additionally, this change temporarily disables adding mockery's version number to generated files. There is an outstanding issue against the mockery project related to the version string behavior when running from `go get`. I have created a pull request to fix this issue in the mockery project.
see: https://github.com/vektra/mockery/issues/397
2021-07-30 20:47:15 +00:00
Callum Waters
02f8e4c0bd blockstore: fix problem with seen commit (#6782) 2021-07-30 17:37:04 +02:00
M. J. Fromberger
6dd8984fef Fix and clarify breaks from select cases. (#6781)
Update those break statements inside case clauses that are intended to reach an
enclosing for loop, so that they correctly exit the loop.

The candidate files for this change were located using:

    % staticcheck -checks SA4011 ./... | cut -d: -f-2

This change is intended to preserve the intended semantics of the code, but
since the code as-written did not have its intended effect, some behaviour may
change. Specifically: Some loops may have run longer than they were supposed
to, prior to this change.

In one case I was not able to clearly determine the intended outcome. That case
has been commented but otherwise left as-written.

Fixes #6780.
2021-07-29 22:28:32 -04:00
M. J. Fromberger
8f06e0c9e7 cleanup: remove redundant error plumbing (#6778)
This is a mostly-automated fixup using Comby (https://comby.dev) to remove 
lexically-obvious redundant error checks. No functional changes are intended.

To reproduce the core change:

    # Collapse redundant error check conditionals
    % comby -in-place 'if err != nil {
       return err
    }
    return nil' 'return err' .go

    # Fold out unnecessary error temporaries
    % comby -in-place ':[spc~^\s*]err :[~:?]= :[any]
       return err' ':[spc]return :[any]' .go

Fixes #6479 and related cases.
2021-07-28 15:38:46 -04:00
JayT106
e87b0391cb cli/indexer: Reindex events (#6676) 2021-07-28 00:04:54 +02:00
William Banfield
f70396c6fd add and run make target for generating existing mocks (#6732)
There are many `//go:generate mockery` lines in the source code.
This change adds a make target to invoke these mock generations.
This change also invokes the mock invocations and adds the resulting mocks to the repo.

Related to #5274
2021-07-18 00:46:04 +00:00
William Banfield
3ccfb26137 psql: close opened rows in tests (#6669) 2021-07-07 11:37:42 -04:00
JayT106
e850863296 state/indexer: close row after query (#6664)
Closes: #6661 

Note: see another error during the events indexing, guess the raw tx size exceeds the limitation?
```
3:17PM ERR failed to index block txs err="pq: index row size 2768 exceeds btree version 4 maximum 2704 for index \"tx_results_tx_result_key\"" height=5205112 module=txindex
2021-07-07 10:28:54 +00:00
Aleksandr Bezobchuk
4f8bcb1cce docs: update events (#6658)
* docs: update events

* lint++

* lint++
2021-07-06 12:48:05 +00:00
JayT106
9d0817b308 Revert "rpc: re-index missing events (#6535)" (#6631)
This reverts commit 167fa738a3.
2021-06-29 15:48:24 -04:00
JayT106
167fa738a3 rpc: re-index missing events (#6535) 2021-06-25 09:14:37 -04:00
Sam Kleinman
a6b30faf35 libs/time: move types/time into libs (#6595) 2021-06-17 16:02:31 -04:00
JayT106
10fa151749 state/types: refactor makeBlock, makeBlocks and makeTxs (#6567)
Closes #3584
2021-06-11 15:17:20 +00:00
JayT106
a456b71f1f state: move pruneBlocks from consensus/state to state/execution (#6541)
state: move pruneBlocks function from consensus/state to state/execution

Closes #5414
2021-06-10 14:24:47 +00:00
Callum Waters
32bc399bdd node: fix genesis on start up (#6563) 2021-06-10 10:22:35 +02:00
JayT106
d2b78ec09b indexer: use INSERT ... ON CONFLICT in the psql eventsink insert functions (#6556) 2021-06-09 09:22:43 -04:00
Callum Waters
d6b4bc267b node/tests: clean up use of genesis doc and surrounding tests (#6554) 2021-06-08 19:32:15 +02:00
Callum Waters
6f6ac5c04e state sync: reverse sync implementation (#6463) 2021-06-08 19:23:52 +02:00
Sam Kleinman
a855f96946 p2p: renames for reactors and routing layer internal moves (#6547) 2021-06-08 08:17:09 -04:00
Marko
1709e49813 version: revert version through ldflag only (#6494)
## Description

Add version back to versions, but allow it to be overridden via a ldflag.

Reason:

Many users are not setting the ldflag causing issues with tooling that relies on it (cosmjs)

closes #6488

cc @webmaster128
2021-05-28 14:22:31 +00:00
JayT106
711a718162 config/indexer: custom event indexing (#6411) 2021-05-27 10:44:46 -04:00
Greg Morrison
692f23d589 events: Add block_id to NewBlockEvent (#6478)
Adds `block_id` to the `newblock` websocket event

Closes #6028
2021-05-27 07:43:24 +00:00
Marko
719e028e00 libs: internalize some packages (#6366)
## Description

Internalize some libs. This reduces the amount ot public API tendermint is supporting. The moved libraries are mainly ones that are used within Tendermint-core.
2021-05-25 16:25:31 +00:00
JayT106
e334555393 node/state: graceful shutdown in the consensus state (#6370) 2021-05-10 21:45:26 +02:00
Callum Waters
a91680efee test: create common functions for easily producing tm data structures (#6435) 2021-05-07 17:00:02 +02:00
Aleksandr Bezobchuk
09a6ad7b1e types: Refactor EventAttribute (#6408) 2021-04-30 09:27:51 -04:00
Callum Waters
20610be98c state: keep a cache of block verification results (#6402) 2021-04-29 08:31:38 +02:00
Ismail Khoffi
ee70430255 libs: remove most of libs/rand (#6364) 2021-04-23 16:00:35 +02:00
JayT106
43eacd159f state/indexer: reconstruct indexer, move txindex into the indexer package (#6382) 2021-04-21 16:37:44 -04:00
Aleksandr Bezobchuk
46e06c9732 state: fix block event indexing reserved key check (#6314)
copy 🍝  bug when porting similar logic from the tx indexing code.
2021-04-03 16:18:54 +00:00
Marko
32ee737d42 change index block log to info (#6290)
## Description

Change log from error to info for indexing blocks
2021-03-29 13:50:12 +00:00
Aleksandr Bezobchuk
d43a79ddf6 state: cleanup block indexing logs and null (#6263) 2021-03-22 13:25:42 -04:00
Aleksandr Bezobchuk
003f394512 rpc: index block events to support block event queries (#6226) 2021-03-22 11:47:35 -04:00
Marko
efd2fde474 privval: add ctx to privval interface (#6240)
## Description

- Add `context.Context` to Privval interface

This pr does not introduce context into our custom privval connection protocol because this will be removed in the next release. When this pr is released.
2021-03-16 14:41:03 +00:00
colin axnér
5a7ac0abdb update outdated comment in state.go (#6214)
I couldn't find any references to `state.NextState(...)`. Looks like `updateState` replaced that function
2021-03-05 13:36:07 +00:00
Marko
0d0181856b indexer: remove info log (#6194)
## Description

Replace info log with Debug.
2021-03-02 17:18:39 +00:00
Aleksandr Bezobchuk
c21f4c6fde state executor: groom logs (#6152)
ref: #5912
2021-02-23 15:57:27 +00:00
githubsands
d5cf7831f1 state: save in batches within the state store (#6067) 2021-02-23 16:37:46 +01:00
Silas Davis
9498cd80bd Normalise GenesisDoc before saving to state (#6059)
Co-authored-by: Callum <cmwaters19@gmail.com>
2021-02-23 13:27:42 +01:00
Callum Waters
162f67cf26 correct spelling to US english (#6077) 2021-02-11 18:59:18 +01:00
Callum Waters
90d3f56797 store: fix deadlock in pruning (#6007) 2021-02-02 11:36:52 +01:00
Marko
2a2279e010 types: cleanup protobuf.go (#6023)
## Description

- remove unused functions
- remove a function used in tests.

Closes: #XXX
2021-02-01 12:02:45 +00:00
Marko
1f01e5d726 params: remove blockTimeIota (#5987)
## Description

- removes blocktimeiota 
- merges block params in abci and core state
- spec change: https://github.com/tendermint/spec/pull/248


Closes: #5939
2021-01-28 13:47:24 +00:00
Marko
70bb8cc8b7 proto: seperate native and proto types (#5994)
## Description

Separate protobuf and domain types. We should avoid using protobuf in our core logic. 

ref #5460
2021-01-27 20:14:27 +00:00
Anton Kaliaev
d76add65a6 libs/log: format []byte as hexidecimal string (uppercased) (#5960)
Closes: #5806 

Co-authored-by: Lanie Hei <heixx011@umn.edu>
2021-01-25 16:25:29 +04:00