Enable basic authentication in the RPC client using the https://username:password@node:port format.
Issue #4248 has details about what was refactored/enhanced (it's not as bad as it looks.)
I'm open to suggestions on where/how the documentation should be updated.
Please note that PR #4284 is superseded with this PR. The reason for this is because both PR makes changes to the same code.
Fixes#4051
Function `parseRemoteAddr` is forcing protocol HTTP and protocol HTTPs to tcp. This causes the bug in the issue #4051.
I find that the tcp is only needed where `net.Dial`. So I moved the switch to makeHTTPDialer.
This is a backport to the v0.32 branch.
Co-authored-by: yk <tankhoon@gmail.com>
Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com>
This commit contains commit messages from the 52 commits from Tendermint 0.32.0 to 0.32.7. This is a result of creating releases from our security advisories, rather than merging these advisories back into the main repo before creating releases. In the future, we will adopt a git workflow that will reduce these commits to only the commits that make up RC2 for (for example) Tendermint 0.32.8.
* docs: fix consensus spec formatting (#3804)
* abci/server: recover from app panics in socket server (#3809)
fixes#3800
* abci/client: fix DATA RACE in gRPC client (#3798)
* Remove go func {}()
closes#357
- Remove go func(){}() that caused race condiditon
- To reproduce
- add -race in make file to `install_abci`
- Remove `CGO_ENABLED=0` & add -race to `install`
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* remove -race
* fix data race
also, reorder callbacks similarly to socket client
* docs: "Writing a built-in Tendermint Core application in Go" guide (#3608)
* docs: go built-in guide
* fix package imports, add badger db, simplify Query
* newTendermint function
* working example
* finish the first guide
* add one more note
* add the second Golang guide - external ABCI app
* fix typos
* libs: Remove db from tendermint in favor of tendermint/tm-cmn (#3811)
* Remove db from tendemrint in favor of tendermint/tm-cmn
- remove db from `libs`
- update dependancy, there have been no breaking changes in the updated deps
- https://github.com/grpc/grpc-go/releases
- https://github.com/golang/protobuf/releases
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* changelog add
* gofmt
* more gofmt
* docs: add A TOC to the Readme.md of ADR Section (#3820)
* ADR TOC in readme.md
* Added A TOC to the Readme.md of ADR Section
- Added table of contents to the Readme of the architecture section.
- Easier to traverse and when you know what is there.
- If the Adr's become viewable online it would help guide the user
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* add tm-cmn to subprojects
* normalize word
* rpc: make max_body_bytes and max_header_bytes configurable (#3818)
* rpc: make max_body_bytes and max_header_bytes configurable
* update changelog pending
* p2p/conn: Add Bufferpool (#3664)
* use byte buffer pool to decreass allocs
* wrap to put buffer in defer
* wapper defer
* add dependency
* remove Gopkg,*
* add change log
* rpc: /broadcast_evidence (#3481)
* implement broadcast_duplicate_vote endpoint
* fix test_cover
* address comments
* address comments
* Update abci/example/kvstore/persistent_kvstore.go
Co-Authored-By: mossid <torecursedivine@gmail.com>
* Update rpc/client/main_test.go
Co-Authored-By: mossid <torecursedivine@gmail.com>
* address comments in progress
* reformat the code
* make linter happy
* make tests pass
* replace BroadcastDuplicateVote with BroadcastEvidence
* fix test
* fix endpoint name
* improve doc
* fix TestBroadcastEvidenceDuplicateVote
* Update rpc/core/evidence.go
Co-Authored-By: Thane Thomson <connect@thanethomson.com>
* add changelog entry
* fix TestBroadcastEvidenceDuplicateVote
* mempool: make max_msg_bytes configurable (#3826)
* mempool: make max_msg_bytes configurable
* apply suggestions from code review
* update changelog pending
* apply suggestions from code review again
* rpc: return err if page is incorrect (less than 0 or greater than tot… (#3825)
* rpc: return err if page is incorrect (less than 0 or greater than total pages)
Fixes#3813
* fix rpc_test
* blockchain: Reorg reactor (#3561)
* go routines in blockchain reactor
* Added reference to the go routine diagram
* Initial commit
* cleanup
* Undo testing_logger change, committed by mistake
* Fix the test loggers
* pulled some fsm code into pool.go
* added pool tests
* changes to the design
added block requests under peer
moved the request trigger in the reactor poolRoutine, triggered now by a ticker
in general moved everything required for making block requests smarter in the poolRoutine
added a simple map of heights to keep track of what will need to be requested next
added a few more tests
* send errors to FSM in a different channel than blocks
send errors (RemovePeer) from switch on a different channel than the
one receiving blocks
renamed channels
added more pool tests
* more pool tests
* lint errors
* more tests
* more tests
* switch fast sync to new implementation
* fixed data race in tests
* cleanup
* finished fsm tests
* address golangci comments :)
* address golangci comments :)
* Added timeout on next block needed to advance
* updating docs and cleanup
* fix issue in test from previous cleanup
* cleanup
* Added termination scenarios, tests and more cleanup
* small fixes to adr, comments and cleanup
* Fix bug in sendRequest()
If we tried to send a request to a peer not present in the switch, a
missing continue statement caused the request to be blackholed in a peer
that was removed and never retried.
While this bug was manifesting, the reactor kept asking for other
blocks that would be stored and never consumed. Added the number of
unconsumed blocks in the math for requesting blocks ahead of current
processing height so eventually there will be no more blocks requested
until the already received ones are consumed.
* remove bpPeer's didTimeout field
* Use distinct err codes for peer timeout and FSM timeouts
* Don't allow peers to update with lower height
* review comments from Ethan and Zarko
* some cleanup, renaming, comments
* Move block execution in separate goroutine
* Remove pool's numPending
* review comments
* fix lint, remove old blockchain reactor and duplicates in fsm tests
* small reorg around peer after review comments
* add the reactor spec
* verify block only once
* review comments
* change to int for max number of pending requests
* cleanup and godoc
* Add configuration flag fast sync version
* golangci fixes
* fix config template
* move both reactor versions under blockchain
* cleanup, golint, renaming stuff
* updated documentation, fixed more golint warnings
* integrate with behavior package
* sync with master
* gofmt
* add changelog_pending entry
* move to improvments
* suggestion to changelog entry
* Renamed wire.go to codec.go (#3827)
* Renamed wire.go to codec.go
- Wire was the previous name of amino
- Codec describes the file better than `wire` & `amino`
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* ide error
* rename amino.go to codec.go
* docs: add guides to docs (#3830)
* add staticcheck linting (#3828)
cleanup to add linter
grpc change:
https://godoc.org/google.golang.org/grpc#WithContextDialerhttps://godoc.org/google.golang.org/grpc#WithDialer
grpc/grpc-go#2627
prometheous change:
due to UninstrumentedHandler, being deprecated in the future
empty branch = empty if or else statement
didn't delete them entirely but commented
couldn't find a reason to have them
could not replicate the issue #3406
but if want to keep it commented then we should comment out the if statement as well
* types: move MakeVote / MakeBlock functions (#3819)
to the types package
Paritally Fixes#3584
* p2p: Fix error logging for connection stop (#3824)
* p2p: fix false-positive error logging when stopping connections
This changeset fixes two types of false-positive errors occurring during
connection shutdown.
The first occurs when the process invokes FlushStop() or Stop() on a
connection. While the previous behavior did properly wait for the sendRoutine
to finish, it did not notify the recvRoutine that the connection was shutting
down. This would cause the recvRouting to receive and error when reading and
log this error. The changeset fixes this by notifying the recvRoutine that
the connection is shutting down.
The second occurs when the connection is terminated (gracefully) by the other side.
The recvRoutine would get an EOF error during the read, log it, and stop the connection
with an error. The changeset detects EOF and gracefully shuts down the connection.
* bring back the comment about flushing
* add changelog entry
* listen for quitRecvRoutine too
* we have to call stopForError
Otherwise peer won't be removed from the peer set and maybe readded
later.
* p2p: Do not write 'Couldn't connect to any seeds' if there are no seeds (#3834)
* Do not write 'Couldn't connect to any seeds' if there are no seeds
* changelog
* remove privValUpgrade
* Fix typo in changelog
* Update CHANGELOG_PENDING.md
Co-Authored-By: Marko <marbar3778@yahoo.com>
I'm setting up all peers dynamically by calling dial_peers, so p2p.seeds in configs is empty, and I'm seeing error log a lot in logs.
* docs: add a footer to guides (#3835)
* docs: "Writing a Tendermint Core application in Kotlin (gRPC)" guide (#3838)
* add abci grpc kotlin guide
* Update docs/guides/kotlin.md
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* Update docs/guides/kotlin.md
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* Update docs/guides/kotlin.md
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* Update kotlin.md
* node: allow replacing existing p2p.Reactor(s) (#3846)
* node: allow replacing existing p2p.Reactor(s)
using [`CustomReactors`
option](https://godoc.org/github.com/tendermint/tendermint/node#CustomReactors).
Warning: beware of accidental name clashes. Here is the list of existing
reactors: MEMPOOL, BLOCKCHAIN, CONSENSUS, EVIDENCE, PEX.
* check the absence of "CUSTOM" prefix
* merge 2 tests
* add doc.go to node package
* gocritic (1/2) (#3836)
Add gocritic as a linter
The linting is not complete, but should i complete in this PR or in a following.
23 files have been touched so it may be better to do in a following PR
Commits:
* Add gocritic to linting
- Added gocritic to linting
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* gocritic
* pr comments
* remove switch in cmdBatch
* tm-cmn to tm-db (#3850)
* tm-cmn to tm-db
* go.mod changes
* go.mod changes
* more go.mod
* fix tm-db
* ci fix, pending change
* version tmdb (#3854)
* txindexer: Refactor Tx Search Aggregation (#3851)
- Replace the previous intersect call, which was called at each query condition, with a map intersection.
- Replace fmt.Sprintf with string()
closes: #3076
Benchmarks
```
Old
goos: darwin
goarch: amd64
pkg: github.com/tendermint/tendermint/state/txindex/kv
BenchmarkTxSearch-4 200 103641206 ns/op 7998416 B/op 71171 allocs/op
PASS
ok github.com/tendermint/tendermint/state/txindex/kv 26.019s
New
goos: darwin
goarch: amd64
pkg: github.com/tendermint/tendermint/state/txindex/kv
BenchmarkTxSearch-4 1000 38615024 ns/op 13515226 B/op 166460 allocs/op
PASS
ok github.com/tendermint/tendermint/state/txindex/kv 53.618s
```
~62% performance improvement
Commits:
* Refactor tx search
* Add pending changelog entry
* Add tx search benchmarking
* remove intermediate hashes list
also reset timer in BenchmarkTxSearch
and fix other benchmark
* fix import
* Add test cases
* Fix searching
* Replace fmt.Sprintf with string
* Update state/txindex/kv/kv.go
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* Rename params
* Cleanup
* Check error in benchmarks
* release for v0.32.2
* Merge PR #3860: Update log v0.32.2
* changelog updates
* pr comments
* Fix for panic in signature verification if a peer sends a nil public key.
* update version.go
* Changelog update
* Update CHANGELOG.md
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* update changelog
* p2p: only allow ed25519 pubkeys when connecting
also, recover from any possible failures in acceptPeers
Refs #4030
* update changelog and bump version to v0.32.6
* set the date to today
* cs: panic only when WAL#WriteSync fails
- modify WAL#Write and WAL#WriteSync to return an error
* types: validate Part#Proof
add ValidateBasic to crypto/merkle/SimpleProof
* cs: limit max bit array size and block parts count
* cs: test new limits
* cs: only assert important stuff
* update changelog and bump version to 0.32.7
* fixes after Ethan's review
* align max wal msg and max consensus msg sizes
* fix tests
* fix test
* Rc2 v0.32.8
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* move issue to big fix
* Fix long line errors in abci, crypto, and libs packages
* Fix long lines in p2p and rpc packages
* Fix long lines in abci, state, and tools packages
* Fix long lines in behaviour and blockchain packages
* Fix long lines in cmd and config packages
* Begin fixing long lines in consensus package
* Finish fixing long lines in consensus package
* Add lll exclusion for lines containing URLs
* Fix long lines in crypto package
* Fix long lines in evidence package
* Fix long lines in mempool and node packages
* Fix long lines in libs package
* Fix long lines in lite package
* Fix new long line in node package
* Fix long lines in p2p package
* Ignore gocritic warning
* Fix long lines in privval package
* Fix long lines in rpc package
* Fix long lines in scripts package
* Fix long lines in state package
* Fix long lines in tools package
* Fix long lines in types package
* Enable lll linter
* New lint version upgrade
- linter was upgraded
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* enable-a;; is deprecated
* minor change
* another try
* some more changes
* some more changes
* reenable prealloc
* add version till bot is fixed
* Pin range scope vars
* Don't disable scopelint
This PR repairs linter errors seen when running the following commands:
golangci-lint run --no-config --disable-all=true --enable=scopelint
Contributes to #3262
cleanup to add linter
grpc change:
https://godoc.org/google.golang.org/grpc#WithContextDialerhttps://godoc.org/google.golang.org/grpc#WithDialer
grpc/grpc-go#2627
prometheous change:
due to UninstrumentedHandler, being deprecated in the future
empty branch = empty if or else statement
didn't delete them entirely but commented
couldn't find a reason to have them
could not replicate the issue #3406
but if want to keep it commented then we should comment out the if statement as well
So we can have an error code when rpc fails.
* Update CHANGELOG_PENDING.md
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* wrap correct error
* format code
Continues from #3280 in building support for batched requests/responses in the JSON RPC (as per issue #3213).
* Add JSON RPC batching for client and server
As per #3213, this adds support for [JSON RPC batch requests and
responses](https://www.jsonrpc.org/specification#batch).
* Add additional checks to ensure client responses are the same as results
* Fix case where a notification is sent and no response is expected
* Add test to check that JSON RPC notifications in a batch are left out in responses
* Update CHANGELOG_PENDING.md
* Update PR number now that PR has been created
* Make errors start with lowercase letter
* Refactor batch functionality to be standalone
This refactors the batching functionality to rather act in a standalone
way. In light of supporting concurrent goroutines making use of the same
client, it would make sense to have batching functionality where one
could create a batch of requests per goroutine and send that batch
without interfering with a batch from another goroutine.
* Add examples for simple and batch HTTP client usage
* Check errors from writer and remove nolinter directives
* Make error strings start with lowercase letter
* Refactor examples to make them testable
* Use safer deferred shutdown for example Tendermint test node
* Recompose rpcClient interface from pre-existing interface components
* Rename WaitGroup for brevity
* Replace empty ID string with request ID
* Remove extraneous test case
* Convert first letter of errors.Wrap() messages to lowercase
* Remove extraneous function parameter
* Make variable declaration terse
* Reorder WaitGroup.Done call to help prevent race conditions in the face of failure
* Swap mutex to value representation and remove initialization
* Restore empty JSONRPC string ID in response to prevent nil
* Make JSONRPCBufferedRequest private
* Revert PR hard link in CHANGELOG_PENDING
* Add client ID for JSONRPCClient
This adds code to automatically generate a randomized client ID for the
JSONRPCClient, and adds a check of the IDs in the responses (if one was
set in the requests).
* Extract response ID validation into separate function
* Remove extraneous comments
* Reorder fields to indicate clearly which are protected by the mutex
* Refactor for loop to remove indexing
* Restructure and combine loop
* Flatten conditional block for better readability
* Make multi-variable declaration slightly more readable
* Change for loop style
* Compress error check statements
* Make function description more generic to show that we support different protocols
* Preallocate memory for request and result objects
* Fixed accepting integer IDs in requests for Tendermint RPC server (#2366)
* added a wrapper interface `jsonrpcid` that represents both string and int IDs in JSON-RPC requests/responses + custom JSON unmarshallers
* changed client-side code in RPC that uses it
* added extra tests for integer IDs
* updated CHANGELOG_PENDING, as suggested by PR instructions
* addressed PR comments
* added table driven tests for request type marshalling/unmarshalling
* expanded handler test to check IDs
* changed pending changelog note
* changed json rpc request/response unmarshalling to use empty interfaces and type switches on ID
* some cleanup
* use increment and decrement operators.
* remove unnecessary else branches.
* fix package comment with leading space.
* fix receiver names.
* fix error strings.
* remove omittable code.
* remove redundant return statement.
* Revert changes (code is generated.)
* use cfg as receiver name for all config-related types.
* use lsi as the receiver name for the LastSignedInfo type.
* linter: address gosimple lints
* linter: make deterministic & a rebase fix
* lint/rpc: fix a gosimple lint
* run linter in CI
* fix rebase mistake
* fix makefile
* ugh
* revert Makefile
* add metalinter to CI
* try this
* linter: last little fix
* need glide
* better
* okayy circle, have it your way
* lints: gosimple
* pr comments
comment out failing consensus tests for now
rewrite rpc httpclient to use new pubsub package
import pubsub as tmpubsub, query as tmquery
make event IDs constants
EventKey -> EventTypeKey
rename EventsPubsub to PubSub
mempool does not use pubsub
rename eventsSub to pubsub
new subscribe API
fix channel size issues and consensus tests bugs
refactor rpc client
add missing discardFromChan method
add mutex
rename pubsub to eventBus
remove IsRunning from WSRPCConnection interface (not needed)
add a comment in broadcastNewRoundStepsAndVotes
rename registerEventCallbacks to broadcastNewRoundStepsAndVotes
See https://dave.cheney.net/2014/03/19/channel-axioms
stop eventBuses after reactor tests
remove unnecessary Unsubscribe
return subscribe helper function
move discardFromChan to where it is used
subscribe now returns an err
this gives us ability to refuse to subscribe if pubsub is at its max
capacity.
use context for control overflow
cache queries
handle err when subscribing in replay_test
rename testClientID to testSubscriber
extract var
set channel buffer capacity to 1 in replay_file
fix byzantine_test
unsubscribe from single event, not all events
refactor httpclient to return events to appropriate channels
return failing testReplayCrashBeforeWriteVote test
fix TestValidatorSetChanges
refactor code a bit
fix testReplayCrashBeforeWriteVote
add comment
fix TestValidatorSetChanges
fixes from Bucky's review
update comment [ci skip]
test TxEventBuffer
update changelog
fix TestValidatorSetChanges (2nd attempt)
only do wg.Done when no errors
benchmark event bus
create pubsub server inside NewEventBus
only expose config params (later if needed)
set buffer capacity to 0 so we are not testing cache
new tx event format: key = "Tx" plus a tag {"tx.hash": XYZ}
This should allow to subscribe to all transactions! or a specific one
using a query: "tm.events.type = Tx and tx.hash = '013ABF99434...'"
use TimeoutCommit instead of afterPublishEventNewBlockTimeout
TimeoutCommit is the time a node waits after committing a block, before
it goes into the next height. So it will finish everything from the last
block, but then wait a bit. The idea is this gives it time to hear more
votes from other validators, to strengthen the commit it includes in the
next block. But it also gives it time to hear about new transactions.
waitForBlockWithUpdatedVals
rewrite WAL crash tests
Task:
test that we can recover from any WAL crash.
Solution:
the old tests were relying on event hub being run in the same thread (we
were injecting the private validator's last signature).
when considering a rewrite, we considered two possible solutions: write
a "fuzzy" testing system where WAL is crashing upon receiving a new
message, or inject failures and trigger them in tests using something
like https://github.com/coreos/gofail.
remove sleep
no cs.Lock around wal.Save
test different cases (empty block, non-empty block, ...)
comments
add comments
test 4 cases: empty block, non-empty block, non-empty block with smaller part size, many blocks
fixes as per Bucky's last review
reset subscriptions on UnsubscribeAll
use a simple counter to track message for which we panicked
also, set a smaller part size for all test cases
* Updated code with feedback from @melekes, @ebuchman and @silasdavis.
* Added Makefile clause `release` to only run the test on seeing tag
`release` during releases i.e
```shell
make release
```
which will run the comprehensive and long integration-ish tests.
Fixes https://github.com/tendermint/tendermint/issues/751.
Adds jitter to our exponential backoff to mitigate a self DDOS
vector. The jitter is a randomly picked percentage of a second
whose purpose is to ensure that each exponential backoff retry
occurs within (1<<attempts) == 2**attempts, but with the delay
each client will have a random buffer time before it tries to
reconnect instead of all at once reconnections that might even
bring back the previous conditions that might have caused the
dial to the WSServer to have failed e.g
* Network outage
* File descriptor exhaustion
* False positives from firewalls
etc
