As a small developer quality of life improvement, I found many individual unit tests that take longer than around a second to complete, and set them to skip when run under `go test -short`.
On my machine, the wall timings for tests (with `go test -count=1 ./...` and optionally `-short` and `-race`) are roughly:
- Long tests, no race detector: about 1m42s
- Short tests, no race detector: about 17s
- Long tests, race detector enabled: about 2m1s
- Short tests, race detector enabled: about 28s
This PR is split into many commits each touching a single package, with commit messages detailing the approximate timing change per package.
shout out to @joeabbey for the inspiration. This makes the lazy
functions internal by default to prevent potential misuse by external
callers.
Should backport cleanly into 0.36 and I'll handle a messy merge into 0.35
* abci:mempoolError from ResponseCheckTx
* responseCheckTx returns an error if Tendermint decides not to accept an app after CheckTx
*updated spec, upgrading.md and changelog.md
Fixes#8583.
The block results include validator updates in ABCI protobuf format, which does
not encode "correctly" according to the expected Amino style RPC clients expect.
- Write a regression test for this issue.
- Add JSON marshaling overrides for ABCI ValidatorUpdate messages.
* rpc: rework timeouts to be per-method instead of global
Prior to this change, we set a 10-second global timeout for all RPC methods
using the net/http Server type's WriteTimeout. This meant that any request
whose handler did not return within that period would simply drop the
connection to the client.
This timeout is too short for a default, as evidenced by issues like [1] and
[2]. In addition, the mode of failure on the client side is confusing; it
shows up as a dropped connection (EOF) rather than a meaningful error from the
service. More importantly, various methods have diffent constraints: Some
should be able to return quickly, others may need to adjust based on the
application workload.
This is a first step toward supporting configurable timeouts. This change:
- Removes the server-wide default global timeout, and instead:
- Wires up a default context timeout for all RPC handlers.
- Increases the default timeout from 10s to 60s.
- Adds a hook to override this per-method as needed.
This does NOT expose the timeouts in the configuration file (yet).
[1] https://github.com/osmosis-labs/osmosis/issues/1391
[2] https://github.com/tendermint/tendermint/issues/8465
Update the static documentation about event subscription to include the new
/events API, and to add more details about how queries work. Mention that the
streaming API is deprecated.
In #8397 I tried to remove all the cases where we needed to keep track of the
target type of parameters for JSON encoding, but there is one case still left:
When decoding parameters from URL query terms, there is no way to tell whether
or not we need base64 encoding without knowing whether the underlying type of
the target is string or []byte.
To fix this, keep track of parameters that are []byte valued when RPCFunc is
compiling its argument map, and use that when parsing URL query terms. Update
the tests accordingly.
Pass all parameters from JSON-RPC requests to their corresponding handlers
using struct types instead of positional parameters. This allows us to control
encoding of arguments using only the standard library, and to eliminate the
remaining special-purpose JSON encoding hooks in the server.
To support existing use, the server still allows arguments to be encoded in
JSON as either an array or an object.
Related changes:
- Rework the RPCFunc constructor to reduce reflection during RPC call service.
- Add request parameter wrappers for each RPC service method.
- Update the RPC Environment methods to use these types.
- Update the interfaces and shims derived from Environment to the new
signatures.
- Update and extend test cases.
I was digging around over here, and thought it'd be good to
cleanup/standardize the line formating on a few of these methods. Also
found a few cases where we could use contexts better so did a little
bit of cleanup there too!
Bumps [github.com/vektra/mockery/v2](https://github.com/vektra/mockery) from 2.10.2 to 2.10.4.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/vektra/mockery/releases">github.com/vektra/mockery/v2's releases</a>.</em></p>
<blockquote>
<h2>v2.10.4</h2>
<h2>Changelog</h2>
<ul>
<li>c943e69 Merge pull request <a href="https://github-redirect.dependabot.com/vektra/mockery/issues/441">#441</a> from cfstras/fix/support-more-env-keys</li>
<li>ed87cf6 fix: allow configuring flags with "-" as Env var</li>
<li>17abd96 fix: unused config field <code>Tags</code></li>
<li>53114cf test: add test for env var configurations</li>
</ul>
<h2>v2.10.3</h2>
<h2>Changelog</h2>
<ul>
<li>ee25bcf Add/update mocks</li>
<li>4703d1a Merge pull request <a href="https://github-redirect.dependabot.com/vektra/mockery/issues/444">#444</a> from vektra/remove_need_deps</li>
<li>ba1f213 Remove packages.NeedDeps</li>
<li>ed38b20 Update go.sum</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="c943e696a0"><code>c943e69</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/vektra/mockery/issues/441">#441</a> from cfstras/fix/support-more-env-keys</li>
<li><a href="4703d1a42f"><code>4703d1a</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/vektra/mockery/issues/444">#444</a> from vektra/remove_need_deps</li>
<li><a href="ed38b20a49"><code>ed38b20</code></a> Update go.sum</li>
<li><a href="ee25bcfcb5"><code>ee25bcf</code></a> Add/update mocks</li>
<li><a href="ba1f213e17"><code>ba1f213</code></a> Remove packages.NeedDeps</li>
<li><a href="17abd96c57"><code>17abd96</code></a> fix: unused config field <code>Tags</code></li>
<li><a href="53114cfce4"><code>53114cf</code></a> test: add test for env var configurations</li>
<li><a href="ed87cf671a"><code>ed87cf6</code></a> fix: allow configuring flags with "-" as Env var</li>
<li>See full diff in <a href="https://github.com/vektra/mockery/compare/v2.10.2...v2.10.4">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
</details>
This change set implements the most recent version of `FinalizeBlock`.
# What does this change actually contain?
* This change set is rather large but fear not! The majority of the files touched and changes are renaming `ResponseDeliverTx` to `ExecTxResult`. This should be a pretty inoffensive change since they're effectively the same type but with a different name.
* The `execBlockOnProxyApp` was totally removed since it served as just a wrapper around the logic that is now mostly encapsulated within `FinalizeBlock`
* The `updateState` helper function has been made a public method on `State`. It was being exposed as a shim through the testing infrastructure, so this seemed innocuous.
* Tests already existed to ensure that the application received the `ByzantineValidators` and the `ValidatorUpdates`, but one was fixed up to ensure that `LastCommitInfo` was being sent across.
* Tests were removed from the `psql` indexer that seemed to search for an event in the indexer that was not being created.
# Questions for reviewers
* We store this [ABCIResponses](5721a13ab1/proto/tendermint/state/types.pb.go (L37)) type in the data base as the block results. This type has changed since v0.35 to contain the `FinalizeBlock` response. I'm wondering if we need to do any shimming to keep the old data retrieveable?
* Similarly, this change is exposed via the RPC through [ResultBlockResults](5721a13ab1/rpc/coretypes/responses.go (L69)) changing. Should we somehow shim or notify for this change?
closes: #7658
Since the goal of reading events at the head of the event log is to satisfy a
subscription style interface, there is no point in allowing head polling with
no wait interval. The pagination case already bypasses long polling, so the
extra option is unneessary.
Set a minimum default long-polling interval for the head case.
Add a test for minimum delay.
This allows the caller to stream events. It handles the bookkeeping for cursors
and pagination, and delivers items to a callback.
Handle missed items by reporting a structured error. The caller can use the
Reset method to "catch up" to head after this happens.
Add a manual test CLI to probe a running node. Requires the node to be
configured with the event log settings.
Add a unit test that scripts input to the stream to exercise it.
- Update documentation to deprecate the old methods.
- Add Events methods to HTTP, WS, and Local clients.
- Add Events method to the light client wrapper.
- Rename legacy events client to SubscriptionClient.
* testing: reduce usage of the MustDefualtLogger constructor
* Apply suggestions from code review
Co-authored-by: M. J. Fromberger <michael.j.fromberger@gmail.com>
* cleanup tests
Co-authored-by: M. J. Fromberger <michael.j.fromberger@gmail.com>
This method implements the eventlog extension interface to expose ABCI metadata
to the log for query processing. Only the types that have ABCI events need to
implement this.
- Add an event log to the environment
- Add a sketch of the handler method
- Add an /events RPCFunc to the route map
- Implement query logic
- Subscribe to pubsub if confingured, handle termination
This is the first step in removing the mutex from ABCI applications:
making our test applications hold mutexes, which this does, hopefully
with zero impact. If this lands well, then we can explore deleting the
other mutexes (in the ABCI server and the clients.) While this change
is not user impacting at all, removing the other mutexes *will* be.
In persuit of this, I've changed the KV app somewhat, to put almost
all of the logic in the base application and make the persistent
application mostly be a wrapper on top of that with a different
storage layer.
* Update OpenAPI docs.
- Add an Events tag for event methods.
- Add schema entries for event request/response types.
- Clarify the documentation for broadcast methods.
- Note that websocket will be deprecated in v0.36.
* Rebased and git-squashed the commits in PR #6546
migrate abci to finalizeBlock
work on abci, proxy and mempool
abciresponse, blok events, indexer, some tests
fix some tests
fix errors
fix errors in abci
fix tests amd errors
* Fixes after rebasing PR#6546
* Restored height to RequestFinalizeBlock & other
* Fixed more UTs
* Fixed kvstore
* More UT fixes
* last TC fixed
* make format
* Update internal/consensus/mempool_test.go
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
* Addressed @williambanfield's comments
* Fixed UTs
* Addressed last comments from @williambanfield
* make format
Co-authored-by: marbar3778 <marbar3778@yahoo.com>
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
Our test cases spew a lot of files and directories around $TMPDIR. Make more
thorough use of the testing package's TempDir methods to ensure these are
cleaned up.
In a few cases, this required plumbing test contexts through existing helper
code. In a couple places an explicit path was required, to work around cases
where we do global setup during a TestMain function. Those cases probably
deserve more thorough cleansing (preferably with fire), but for now I have just
worked around it to keep focused on the cleanup.
We have this one solitary metric from the go-metrics package. In principle
this statistic could be useful, but the way we have it hooked up, nothing can
observe the value: We don't export it, we don't log it, and it does not auto
publish anywhere.
Given that this state of affairs has not changed since the metric was first
added in 2017 (c08618f), I think we can safely discard it. No one is now or has
ever gotten any data out of this metric.
Extend the decoding rules for URL query parameters so that if the target type
implements encoding.TextUnmarshaler, the decoder will use it.
This is a non-breaking change.
* rpc/client: remove the placeholder RunState type.
I added the RunState type in #6971 to disconnect clients from the service
plumbing, which they do not need. Now that we have more complete context
plumbing, the lifecycle of a client no longer depends on this type: It serves
as a carrier for a logger, and a Boolean flag for "running" status, neither of
which is used outside of tests.
Logging in particular is defaulted to a no-op logger in all production use.
Arguably we could just remove the logging calls, since they are never invoked
except in tests. To defer the question of whether we should do that or make the
logging go somewhere more productive, I've preserved the existing use here.
Remove use of the IsRunning method that was provided by the RunState, and use
the Start method and context to govern client lifecycle.
Remove the one test that exercised "unstarted" clients. I would like to remove
that method entirely, but that will require updating the constructors for all
the client types to plumb a context and possibly other options. I have deferred
that for now.
* rpc: fix layout of endpoint list
The output of the default endpoint-list query was not correctly segregating
methods with and without arguments. Fix this, and also clean up the output to
be easier to read (both in code and in generated source).
Fixes#3618.
* rpc: simplify the handling of JSON-RPC request and response IDs
Replace the ID wrapper interface with plain JSON. Internally, the client
libraries use only integer IDs, and the server does not care about the ID
structure apart from checking its validity.
Basic structure of this change:
- Remove the jsonrpcid interface and its helpers.
- Unexport the ID field of request and response.
- Add helpers for constructing requests and responses.
- Fix up usage and tests.
Update the interface of the batch decoder to match the type signature of the
single-response case. The caller provides the outputs, so there is no need to
return them as well.
No functional changes.
Responses are constructed from requests using MakeResponse, MakeError, and
MakeErrorf. This ensures the response is always paired with the correct ID,
makes cases where there is no ID more explicit at the usage site, and
consolidates the handling of error introspection across transports.
The logic for unpacking errors and assigning JSON-RPC response types was
previously duplicated in three places. Consolidate it in the types package for
the RPC subsystem.
* update test cases
Instead of having the exported Request and Response type expose the version
directly, delegate version injection to an unexported shim. We already had the
shim; this just boosts it to the top level and does a bit more checking.
* Check JSON-RPC version marker.
These are only ever used with the defaults, except in our own tests. A search
of cs.github.com shows no other callers.
The use in the test was solely to bug out the go-metrics package so its
goroutines don't trigger the leak checker. Use the package's own flag for that
purpose instead. Note that calling "Stop" on the metric helps, but is not
sufficient -- the Stop does not wait for its goroutine to exit.
This is the interface shared by types that can be used as event data in, for
example, subscriptions via the RPC.
To be compatible with the RPC service, data need to support JSON encoding.
Require this as part of the interface.
