## NOTE: this pr exclusively runs commands from the makefile found here
This PR ONLY runs `make format` ... then `make mockery`
Its purpose is to ensure that the review scope of other PR's, which changed .go files and thus triggered the linter that only runs conditionally, have smaller review
scopes, and should be merged before:
https://github.com/tendermint/tendermint/pull/9738https://github.com/tendermint/tendermint/pull/9739https://github.com/tendermint/tendermint/pull/9742
---
#### PR checklist
- [x] Tests written/updated, or no tests needed
- [x] `CHANGELOG_PENDING.md` updated, or no changelog entry needed
- [x] Updated relevant documentation (`docs/`) and code comments, or no
documentation updates needed
* Performance improvements for the event query API (#7319)
Rework the implementation of event query parsing and execution to
improve performance and reduce memory usage.
Previous memory and CPU profiles of the pubsub service showed query
processing as a significant hotspot. While we don't have evidence that
this is visibly hurting users, fixing it is fairly easy and self-contained.
Updates #6439.
Typical benchmark results comparing the original implementation (PEG) with the reworked implementation (Custom):
```
TEST TIME/OP BYTES/OP ALLOCS/OP SPEEDUP MEM SAVING
BenchmarkParsePEG-12 51716 ns 526832 27
BenchmarkParseCustom-12 2167 ns 4616 17 23.8x 99.1%
BenchmarkMatchPEG-12 3086 ns 1097 22
BenchmarkMatchCustom-12 294.2 ns 64 3 10.5x 94.1%
```
* libs/pubsub/query: specify peg version in go.mod
The code to generate the pubsub queries was dependent on an unspecified
version of the peg tool. This brings peg into go.mod so it is on a fixed
version. This should also enable dependabot to notify us of future
updates to peg.
The version of query.peg.go generated from the current version of peg
correctly contains the special "Code generated by..." line to indicate
to other tools that the file is automatically generated and should
therefore be excluded from linters, etc.
I removed the make target as there were no git grep results referencing
"gen_query_parser"; directly running "go generate" is a reasonable
expectation in Go projects.
Now that "go run" is module aware, I would typically use "go run" inside
the go:generate directive, but in this case we go build to a gitignore-d
directory in order to work around the nondeterministic output detailed
in pointlander/peg#129.
* libs/pubsub/query: check error from (*QueryParser).Init()
The newly generated peg code returns an error from Init(); the previous
version was niladic.
Co-authored-by: Sam Kleinman <garen@tychoish.com>
Closes#4603
Commands used (VIM):
```
:args `rg -l errors.Wrap`
:argdo normal @q | update
```
where q is a macros rewriting the `errors.Wrap` to `fmt.Errorf`.
* Rename Tag(s) to Event(s)
- tag was replaced with event, but in some places it still mentions tag, would be easier to understand if we tried to replace it with event to not confuse people.
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* more changes from tag -> event
* rename events to compositeKeys and keys
* Apply suggestions from code review
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* add minor documentation on how composite keys are constructed
* rename eventkey to compositekey
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* add changelog entry & add info to regenerate confid to changelog entry
## Issue:
This PR adds an "EXISTS" condition to the event query grammar. It enables querying for the occurrence of an event without having to provide a condition for one of its attributes.
As an example, someone interested in all slashing events might currently catch them with a query such as slash.power > 0.
With this PR the event can be captured with slash.power EXISTS or just slash EXISTS to catch by event type.
## Examples:
`slash EXISTS`
## Commits:
* Add EXISTS condition to query grammar
* Gofmt files
* Move PEG instructions out of auto-generated file to prevent overwrite
* Update libs/pubsub/query/query.go
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* Update changelog and add test case
* Merge with other changes in PR #4070
* Add EXISTS to Conditions() func
* Apply gofmt
* Addressing PR comments
Some linting/cleanup missed from the initial events refactor
Don't panic; instead, return false, error when matching breaks unexpectedly
Strip non-numeric chars from values when attempting to match against query values
Have the server log during send upon error
* cleanup/lint Query#Conditions and do not panic
* cleanup/lint Query#Matches and do not panic
* cleanup/lint matchValue and do not panic
* rever to panic in Query#Conditions
* linting
* strip alpha chars when attempting to match
* add pending log entries
* Update libs/pubsub/query/query.go
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* build: update variable names
* update matchValue to return an error
* update Query#Matches to return an error
* update TestMatches
* log error in send
* Fix tests
* Fix TestEmptyQueryMatchesAnything
* fix linting
* Update libs/pubsub/query/query.go
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* Update libs/pubsub/query/query.go
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* Update libs/pubsub/query/query.go
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* Update libs/pubsub/query/query.go
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* Update libs/pubsub/query/query.go
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* Update libs/pubsub/pubsub.go
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* add missing errors pkg import
* update Query#Conditions to return an error
* update query pkg unit tests
* update TxIndex#Search
* update pending changelog
* Fix long line errors in abci, crypto, and libs packages
* Fix long lines in p2p and rpc packages
* Fix long lines in abci, state, and tools packages
* Fix long lines in behaviour and blockchain packages
* Fix long lines in cmd and config packages
* Begin fixing long lines in consensus package
* Finish fixing long lines in consensus package
* Add lll exclusion for lines containing URLs
* Fix long lines in crypto package
* Fix long lines in evidence package
* Fix long lines in mempool and node packages
* Fix long lines in libs package
* Fix long lines in lite package
* Fix new long line in node package
* Fix long lines in p2p package
* Ignore gocritic warning
* Fix long lines in privval package
* Fix long lines in rpc package
* Fix long lines in scripts package
* Fix long lines in state package
* Fix long lines in tools package
* Fix long lines in types package
* Enable lll linter
## PR
This PR introduces a fundamental breaking change to the structure of ABCI response and tx tags and the way they're processed. Namely, the SDK can support more complex and aggregated events for distribution and slashing. In addition, block responses can include duplicate keys in events.
Implement new Event type. An event has a type and a list of KV pairs (ie. list-of-lists). Typical events may look like:
"rewards": [{"amount": "5000uatom", "validator": "...", "recipient": "..."}]
"sender": [{"address": "...", "balance": "100uatom"}]
The events are indexed by {even.type}.{even.attribute[i].key}/.... In this case a client would subscribe or query for rewards.recipient='...'
ABCI response types and related types now include Events []Event instead of Tags []cmn.KVPair.
PubSub logic now publishes/matches against map[string][]string instead of map[string]string to support duplicate keys in response events (from #1385). A match is successful if the value is found in the slice of strings.
closes: #1859closes: #2905
## Commits:
* Implement Event ABCI type and updates responses to use events
* Update messages_test.go
* Update kvstore.go
* Update event_bus.go
* Update subscription.go
* Update pubsub.go
* Update kvstore.go
* Update query logic to handle slice of strings in events
* Update Empty#Matches and unit tests
* Update pubsub logic
* Update EventBus#Publish
* Update kv tx indexer
* Update godocs
* Update ResultEvent to use slice of strings; update RPC
* Update more tests
* Update abci.md
* Check for key in validateAndStringifyEvents
* Fix KV indexer to skip empty keys
* Fix linting errors
* Update CHANGELOG_PENDING.md
* Update docs/spec/abci/abci.md
Co-Authored-By: Federico Kunze <31522760+fedekunze@users.noreply.github.com>
* Update abci/types/types.proto
Co-Authored-By: Ethan Buchman <ethan@coinculture.info>
* Update docs/spec/abci/abci.md
Co-Authored-By: Ethan Buchman <ethan@coinculture.info>
* Update libs/pubsub/query/query.go
Co-Authored-By: Ethan Buchman <ethan@coinculture.info>
* Update match function to match if ANY value matches
* Implement TestSubscribeDuplicateKeys
* Update TestMatches to include multi-key test cases
* Update events.go
* Update Query interface godoc
* Update match godoc
* Add godoc for matchValue
* DRY-up tx indexing
* Return error from PublishWithEvents in EventBus#Publish
* Update PublishEventNewBlockHeader to return an error
* Fix build
* Update events doc in ABCI
* Update ABCI events godoc
* Implement TestEventBusPublishEventTxDuplicateKeys
* Update TestSubscribeDuplicateKeys to be table-driven
* Remove mod file
* Remove markdown from events godoc
* Implement TestTxSearchDeprecatedIndexing test
* green pubsub tests :OK:
* get rid of clientToQueryMap
* Subscribe and SubscribeUnbuffered
* start adapting other pkgs to new pubsub
* nope
* rename MsgAndTags to Message
* remove TagMap
it does not bring any additional benefits
* bring back EventSubscriber
* fix test
* fix data race in TestStartNextHeightCorrectly
```
Write at 0x00c0001c7418 by goroutine 796:
github.com/tendermint/tendermint/consensus.TestStartNextHeightCorrectly()
/go/src/github.com/tendermint/tendermint/consensus/state_test.go:1296 +0xad
testing.tRunner()
/usr/local/go/src/testing/testing.go:827 +0x162
Previous read at 0x00c0001c7418 by goroutine 858:
github.com/tendermint/tendermint/consensus.(*ConsensusState).addVote()
/go/src/github.com/tendermint/tendermint/consensus/state.go:1631 +0x1366
github.com/tendermint/tendermint/consensus.(*ConsensusState).tryAddVote()
/go/src/github.com/tendermint/tendermint/consensus/state.go:1476 +0x8f
github.com/tendermint/tendermint/consensus.(*ConsensusState).handleMsg()
/go/src/github.com/tendermint/tendermint/consensus/state.go:667 +0xa1e
github.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine()
/go/src/github.com/tendermint/tendermint/consensus/state.go:628 +0x794
Goroutine 796 (running) created at:
testing.(*T).Run()
/usr/local/go/src/testing/testing.go:878 +0x659
testing.runTests.func1()
/usr/local/go/src/testing/testing.go:1119 +0xa8
testing.tRunner()
/usr/local/go/src/testing/testing.go:827 +0x162
testing.runTests()
/usr/local/go/src/testing/testing.go:1117 +0x4ee
testing.(*M).Run()
/usr/local/go/src/testing/testing.go:1034 +0x2ee
main.main()
_testmain.go:214 +0x332
Goroutine 858 (running) created at:
github.com/tendermint/tendermint/consensus.(*ConsensusState).startRoutines()
/go/src/github.com/tendermint/tendermint/consensus/state.go:334 +0x221
github.com/tendermint/tendermint/consensus.startTestRound()
/go/src/github.com/tendermint/tendermint/consensus/common_test.go:122 +0x63
github.com/tendermint/tendermint/consensus.TestStateFullRound1()
/go/src/github.com/tendermint/tendermint/consensus/state_test.go:255 +0x397
testing.tRunner()
/usr/local/go/src/testing/testing.go:827 +0x162
```
* fixes after my own review
* fix formatting
* wait 100ms before kicking a subscriber out
+ a test for indexer_service
* fixes after my second review
* no timeout
* add changelog entries
* fix merge conflicts
* fix typos after Thane's review
Co-Authored-By: melekes <anton.kalyaev@gmail.com>
* reformat code
* rewrite indexer service in the attempt to fix failing test
https://github.com/tendermint/tendermint/pull/3227/#issuecomment-462316527
* Revert "rewrite indexer service in the attempt to fix failing test"
This reverts commit 0d9107a098.
* another attempt to fix indexer
* fixes after Ethan's review
* use unbuffered channel when indexing transactions
Refs https://github.com/tendermint/tendermint/pull/3227#discussion_r258786716
* add a comment for EventBus#SubscribeUnbuffered
* format code
