rfc: event system (#6957)

docs/rfc/README.md

@@ -42,5 +42,6 @@ sections.
 - [RFC-002: Interprocess Communication](./rfc-002-ipc-ecosystem.md)
 - [RFC-003: Performance Taxonomy](./rfc-003-performance-questions.md)
 - [RFC-004: E2E Test Framework Enhancements](./rfc-004-e2e-framework.md)
+- [RFC-005: Event System](./rfc-005-event-system.rst)
 
 <!-- - [RFC-NNN: Title](./rfc-NNN-title.md) -->

docs/rfc/rfc-005-event-system.rst

@@ -0,0 +1,122 @@
+=====================
+RFC 005: Event System
+=====================
+
+Changelog
+---------
+
+- 2021-09-17: Initial Draft (@tychoish)
+
+Abstract
+--------
+
+The event system within Tendermint, which supports a lot of core
+functionality, also represents a major infrastructural liability. As part of
+our upcoming review of the RPC interfaces and our ongoing thoughts about
+stability and performance, as well as the preparation for Tendermint 1.0, we
+should revisit the design and implementation of the event system. This
+document discusses both the current state of the system and potential
+directions for future improvement.
+
+Background
+----------
+
+Current State of Events
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The event system makes it possible for clients, both internal and external,
+to receive notifications of state replication events, such as new blocks,
+new transactions, validator set changes, as well as intermediate events during
+consensus. Because the event system is very cross cutting, the behavior and
+performance of the event publication and subscription system has huge impacts
+for all of Tendermint.
+
+The subscription service is exposed over the RPC interface, but also powers
+the indexing (e.g. to an external database,) and is the mechanism by which
+`BroadcastTxCommit` is able to wait for transactions to land in a block.
+
+The current pubsub mechanism relies on a couple of buffered channels,
+primarily between all event creators and subscribers, but also for each
+subscription. The result of this design is that, in some situations with the
+right collection of slow subscription consumers the event system can put
+backpressure on the consensus state machine and message gossiping in the
+network, thereby causing nodes to lag.
+
+Improvements
+~~~~~~~~~~~~
+
+The current system relies on implicit, bounded queues built by the buffered channels,
+and though threadsafe, can force all activity within Tendermint to serialize,
+which does not need to happen. Additionally, timeouts for subscription
+consumers related to the implementation of the RPC layer, may complicate the
+use of the system.
+
+References
+~~~~~~~~~~
+
+- Legacy Implementation
+  - `publication of events <https://github.com/tendermint/tendermint/blob/master/libs/pubsub/pubsub.go#L333-L345>`_ 
+  - `send operation <https://github.com/tendermint/tendermint/blob/master/libs/pubsub/pubsub.go#L489-L527>`_ 
+  - `send loop <https://github.com/tendermint/tendermint/blob/master/libs/pubsub/pubsub.go#L381-L402>`_
+- Related RFCs 
+  - `RFC 002: IPC Ecosystem <./rfc-002-ipc-ecosystem.md>`_ 
+  - `RFC 003: Performance Questions <./rfc-003-performance-questions.md>`_ 
+
+Discussion
+----------
+
+Changes to Published Events
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+As part of this process, the Tendermint team should do a study of the existing
+event types and ensure that there are viable production use cases for
+subscriptions to all event types. Instinctively it seems plausible that some
+of the events may not be useable outside of tendermint, (e.g. ``TimeoutWait``
+or ``NewRoundStep``) and it might make sense to remove them. Certainly, it
+would be good to make sure that we don't maintain infrastructure for unused or
+un-useful message indefinitely.
+
+Blocking Subscription
+~~~~~~~~~~~~~~~~~~~~~
+
+The blocking subscription mechanism makes it possible to have *send*
+operations into the subscription channel be un-buffered (the event processing
+channel is still buffered.) In the blocking case, events from one subscription
+can block processing that event for other non-blocking subscriptions. The main
+case, it seems for blocking subscriptions is ensuring that a transaction has
+been committed to a block for ``BroadcastTxCommit``. Removing blocking
+subscriptions entirely, and potentially finding another way to implement
+``BroadcastTxCommit``, could lead to important simplifications and
+improvements to throughput without requiring large changes.
+
+Subscription Identification
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Before `#6386 <https://github.com/tendermint/tendermint/pull/6386>`_, all
+subscriptions were identified by the combination of a client ID and a query,
+and with that change, it became possible to identify all subscription given
+only an ID, but compatibility with the legacy identification means that there's a
+good deal of legacy code as well as client side efficiency that could be
+improved. 
+
+Pubsub Changes
+~~~~~~~~~~~~~~
+
+The pubsub core should be implemented in a way that removes the possibility of
+backpressure from the event system to impact the core system *or* for one
+subscription to impact the behavior of another area of the
+system. Additionally, because the current system is implemented entirely in
+terms of a collection of buffered channels, the event system (and large
+numbers of subscriptions) can be a source of memory pressure. 
+
+These changes could include: 
+
+- explicit cancellation and timeouts promulgated from callers (e.g. RPC end
+  points, etc,) this should be done using contexts.
+
+- subscription system should be able to spill to disk to avoid putting memory
+  pressure on the core behavior of the node (consensus, gossip).
+  
+- subscriptions implemented as cursors rather than channels, with either
+  condition variables to simulate the existing "push" API or a client side
+  iterator API with some kind of long polling-type interface.