This pull request completes the change to the `metricsgen` metrics. It adds `go generate` directives to all of the files containing the `Metrics` structs.
Using the outputs of `metricsdiff` between these generated metrics and `main`, we can see that there is a minimal diff between the two sets of metrics when run locally. The diff here stems from removal of the word 'message' which was done in v0.36+ and is ultimately a better phrasing. This metric has not yet been released, so this phrasing is preferred.
```
./metricsdiff old new
Metric changes:
+++ tendermint_consensus_full_prevote_delay
+++ tendermint_consensus_quorum_prevote_delay
--- tendermint_consensus_full_prevote_message_delay
--- tendermint_consensus_quorum_prevote_message_delay
```
This change also adds parsing for a `metrics:` key in a field comment. If a comment line begins with `//metrics:` the rest of the line is interpreted to be the metric help text. Additionally, a bug where lists of labels were not properly quoted in the `metricsgen` rendered output was fixed.
In my view, docs and tests are not needed for this internal only change.
---
#### PR checklist
- [ ] Tests written/updated, or no tests needed
- [ ] `CHANGELOG_PENDING.md` updated, or no changelog entry needed
- [ ] Updated relevant documentation (`docs/`) and code comments, or no
documentation updates needed
Ports #8488 to main
Update the nightly E2E workflows to fix the notifications for the
v0.34.x branch while also simplifying the messages and making them more
readable.
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* internal/proxy: add initial set of abci metrics (#7115)
This PR adds an initial set of metrics for use ABCI. The initial metrics enable the calculation of timing histograms and call counts for each of the ABCI methods. The metrics are also labeled as either 'sync' or 'async' to determine if the method call was performed using ABCI's `*Async` methods.
An example of these metrics is included here for reference:
```
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="0.0001"} 0
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="0.0004"} 5
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="0.002"} 12
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="0.009"} 13
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="0.02"} 13
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="0.1"} 13
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="0.65"} 13
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="2"} 13
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="6"} 13
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="25"} 13
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="+Inf"} 13
tendermint_abci_connection_method_timing_sum{chain_id="ci",method="commit",type="sync"} 0.007802058000000001
tendermint_abci_connection_method_timing_count{chain_id="ci",method="commit",type="sync"} 13
```
These metrics can easily be graphed using prometheus's `histogram_quantile(...)` method to pick out a particular quantile to graph or examine. I chose buckets that were somewhat of an estimate of expected range of times for ABCI operations. They start at .0001 seconds and range to 25 seconds. The hope is that this range captures enough possible times to be useful for us and operators.
* fixup
* fixups
* docs: add abci timing metrics to the metrics docs (#7311)
* format table
This is a port of #8470 and #8501 from the former mainline branch, in
support of #2600 and #9076.
No additional changes other than checking out the code have been
implemented in this PR. A subsequent PR will incorporate this code.
* docs/rfc: add testnet RFC
Following several discussions internal to the Tendermint engineering
team, I am posting an RFC discussing the high-level details of the
Tendermint team owning and operating a long-lived testnet in order to
build experience running Tendermint, and to demonstrate that Tendermint
is stable under production workloads.
The outcome of this RFC will be a new track of work to begin building
and maintaining a testnet associated with the main branch of tendermint.
See the "Testnet MVP" section specifically for some of the first
milestones.
Note, I added the RFC where it would live once #9115 is merged to
restore the RFC layout from the v0.36.x branch. docs/rfc/README.md will
need to be updated to include this RFC once #9115 is merged.
This RFC is related to #9078.
* docs/rfc: minor updates to testnet rfc
* docs/rfc: respond to more feedback on testnet RFC
* docs/rfc: add RFC 023 to rfc index
Bumps [github.com/bufbuild/buf](https://github.com/bufbuild/buf) from 1.4.0 to 1.7.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/bufbuild/buf/releases">github.com/bufbuild/buf's releases</a>.</em></p>
<blockquote>
<h2>v1.7.0</h2>
<ul>
<li>Support protocol and encoding client options based on content-type in Studio Agent.</li>
<li>Add <code>--draft</code> flag to <code>buf push</code>.</li>
<li>Add <code>buf beta registry draft {list,delete}</code> commands.</li>
</ul>
<h2>v1.6.0</h2>
<ul>
<li>Fix issue where <code>// buf:lint:ignore</code> comment ignores did not work for the
<code>ENUM_FIRST_VALUE_ZERO</code> rule.</li>
<li>Add <code>buf beta studio-agent</code> command to support the upcoming Buf Studio.</li>
</ul>
<h2>v1.5.0</h2>
<ul>
<li>Upgrade to <code>protoc</code> 3.20.1 support.</li>
<li>Fix an issue where <code>buf</code> would fail if two or more roots contained
a file with the same name, but with different file types (i.e. a
regular file vs. a directory).</li>
<li>Fix check for <code>PACKAGE_SERVICE_NO_DELETE</code> to detect deleted services.</li>
<li>Remove <code>buf beta registry track</code>.</li>
<li>Remove <code>buf beta registry branch</code>.</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/bufbuild/buf/blob/main/CHANGELOG.md">github.com/bufbuild/buf's changelog</a>.</em></p>
<blockquote>
<h2>[v1.7.0] - 2022-06-27</h2>
<ul>
<li>Support protocol and encoding client options based on content-type in Studio Agent.</li>
<li>Add <code>--draft</code> flag to <code>buf push</code>.</li>
<li>Add <code>buf beta registry draft {list,delete}</code> commands.</li>
</ul>
<h2>[v1.6.0] - 2022-06-21</h2>
<ul>
<li>Fix issue where <code>// buf:lint:ignore</code> comment ignores did not work for the
<code>ENUM_FIRST_VALUE_ZERO</code> rule.</li>
<li>Add <code>buf beta studio-agent</code> command to support the upcoming Buf Studio.</li>
</ul>
<h2>[v1.5.0] - 2022-05-30</h2>
<ul>
<li>Upgrade to <code>protoc</code> 3.20.1 support.</li>
<li>Fix an issue where <code>buf</code> would fail if two or more roots contained
a file with the same name, but with different file types (i.e. a
regular file vs. a directory).</li>
<li>Fix check for <code>PACKAGE_SERVICE_NO_DELETE</code> to detect deleted services.</li>
<li>Remove <code>buf beta registry track</code>.</li>
<li>Remove <code>buf beta registry branch</code>.</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="028fdd557d"><code>028fdd5</code></a> Update to v1.7.0</li>
<li><a href="c6a545ecc0"><code>c6a545e</code></a> Run <code>make upgrade</code> (<a href="https://github-redirect.dependabot.com/bufbuild/buf/issues/1302">#1302</a>)</li>
<li><a href="65503feaf3"><code>65503fe</code></a> allow Dockerfile.plugin or Dockerfile (<a href="https://github-redirect.dependabot.com/bufbuild/buf/issues/1301">#1301</a>)</li>
<li><a href="075e294d6a"><code>075e294</code></a> add CLI support for repository draft commit (<a href="https://github-redirect.dependabot.com/bufbuild/buf/issues/1269">#1269</a>)</li>
<li><a href="3d2ca26216"><code>3d2ca26</code></a> ModuleReferenceForString support reference with slash (<a href="https://github-redirect.dependabot.com/bufbuild/buf/issues/1298">#1298</a>)</li>
<li><a href="72b51ab871"><code>72b51ab</code></a> rename CuratedPlugin options to default_options (<a href="https://github-redirect.dependabot.com/bufbuild/buf/issues/1300">#1300</a>)</li>
<li><a href="65c921b0b7"><code>65c921b</code></a> rename opts/options to default_opts/default_options (<a href="https://github-redirect.dependabot.com/bufbuild/buf/issues/1299">#1299</a>)</li>
<li><a href="0a8aec22c7"><code>0a8aec2</code></a> BSR-379: document version can be optional (<a href="https://github-redirect.dependabot.com/bufbuild/buf/issues/1297">#1297</a>)</li>
<li><a href="33033b8fc3"><code>33033b8</code></a> BSR-369: update plugin dependencies to a struct (<a href="https://github-redirect.dependabot.com/bufbuild/buf/issues/1296">#1296</a>)</li>
<li><a href="613283806d"><code>6132838</code></a> Add webhook delete and list commands to beta and fixup create (<a href="https://github-redirect.dependabot.com/bufbuild/buf/issues/1293">#1293</a>)</li>
<li>Additional commits viewable in <a href="https://github.com/bufbuild/buf/compare/v1.4.0...v1.7.0">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
</details>
Bumps [minimist](https://github.com/substack/minimist) from 1.2.5 to 1.2.6.
<details>
<summary>Commits</summary>
<ul>
<li><a href="7efb22a518"><code>7efb22a</code></a> 1.2.6</li>
<li><a href="ef88b9325f"><code>ef88b93</code></a> security notice for additional prototype pollution issue</li>
<li><a href="c2b981977f"><code>c2b9819</code></a> isConstructorOrProto adapted from PR</li>
<li><a href="bc8ecee438"><code>bc8ecee</code></a> test from prototype pollution PR</li>
<li>See full diff in <a href="https://github.com/substack/minimist/compare/1.2.5...1.2.6">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/tendermint/tendermint/network/alerts).
</details>
Import the readme, contributing guidelines, code of conduct, security
guide and releases guide. Format all of these documents, replacing
references to `master` by references to `main`, and use Markdown link
references instead of embedded links wherever it improves legibility.
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Update CODEOWNERS to use teams
Update the `CODEOWNERS` file to use the
@tendermint/tendermint-engineering and @tendermint/tendermint-research
teams as opposed to adding people one by one. This makes repository
administration somewhat easier to manage, especially when
onboarding/offboarding people.
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* Add Ethan as superuser
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* libs/pubsub/query: specify peg version in go.mod
The code to generate the pubsub queries was dependent on an unspecified
version of the peg tool. This brings peg into go.mod so it is on a fixed
version. This should also enable dependabot to notify us of future
updates to peg.
The version of query.peg.go generated from the current version of peg
correctly contains the special "Code generated by..." line to indicate
to other tools that the file is automatically generated and should
therefore be excluded from linters, etc.
I removed the make target as there were no git grep results referencing
"gen_query_parser"; directly running "go generate" is a reasonable
expectation in Go projects.
Now that "go run" is module aware, I would typically use "go run" inside
the go:generate directive, but in this case we go build to a gitignore-d
directory in order to work around the nondeterministic output detailed
in pointlander/peg#129.
* libs/pubsub/query: check error from (*QueryParser).Init()
The newly generated peg code returns an error from Init(); the previous
version was niladic.
Co-authored-by: Sam Kleinman <garen@tychoish.com>
This test would fail if run with "go test -count=2" because it uses a
fixed address and was not closing the server, so the subsequent run
could not bind to the address.
While closing the server is correct, it would probably be better if the
API was able to report the bound address so that we could pass
"localhost:0" for an anonymous port. But I am currently focusing on test
cleanup, not ready to change any existing APIs.
In the v0.34 line, the socket and gRPC clients require explicit flushes to
ensure that the client and server have received an async request. Add these
calls explicitly where required in the backport of the priority mempool.
In addition, the gRPC client's flush plumbing was not fully hooked up in the
v0.34 line, so this change includes that update as well.
This is a manual cherry-pick of commit b94470a6a4.
In the original implementation transactions evicted for priority were also
removed from the cache. In addition, remove expired transactions from the
cache.
Related:
- Add Has method to cache implementations.
- Update tests to exercise this condition.
A manual backport of #8984.
This case is symmetric to what we did for CheckTx calls, where we release the
mempool mutex to ensure callbacks can fire during call setup. We also need
this behaviour for application flush, for the same reason: The caller holds the
lock by contract from the Mempool interface.
A manual cherry-pick of 9e64c95.
The way this was originally structured, we reacquired the lock after issuing
the initial ABCI CheckTx call, only to immediately release it. Restructure the
code so that this redundant acquire is no longer necessary.
- Remove warning log for issue #8775.
- Fix call to FlushAsync (no error is reported).
- Don't log on rechecks, it's the default (manual backport of #8969).
