227 lines
12 KiB
Markdown
227 lines
12 KiB
Markdown
# Ports and protocols in Scylla
|
|
|
|
Scylla is a distributed system, with multiple Scylla nodes communicating
|
|
with each other and with client nodes making requests. All these different
|
|
nodes use many different protocols and TCP ports (currently, all use TCP)
|
|
to communicate with each other. Some of these protocols have an external
|
|
specification - for example Scylla supports Cassandra's CQL protocol and
|
|
Amazon's DynamoDB protocol for client requests. Other protocols, both
|
|
client-facing and internal (between Scylla nodes), are Scylla-defined.
|
|
|
|
The goal of this document is to survey all these different protocols -
|
|
what goes over which protocol, which port each protocol uses (and how this
|
|
can be configured), how to disable or enable some of the protocols, and what
|
|
part of the Scylla source code handles which protocol.
|
|
|
|
Unfortunately, the long evolution of some of these protocols resulted in
|
|
some of them getting confusing names, and often different names in Scylla's
|
|
code, documentation and configuration. This document aims to clear up some
|
|
of this confusion.
|
|
|
|
The Wireshark tool can be used to inspect and understand the messages sent
|
|
over many of the following protocols. This includes even Scylla's internal
|
|
inter-node protocol - for instructions and examples check out the blog post:
|
|
https://www.scylladb.com/2020/05/21/dissecting-scylla-packets-with-wireshark/
|
|
|
|
# Internal communication
|
|
|
|
All inter-node communiction - communication between the Scylla nodes
|
|
themselves - happen over one protocol and one TCP port - by default port 7000
|
|
(unencrypted) or 7001 (encrypted). These are the same TCP ports used by
|
|
Cassandra for this purpose, but Scylla's internal protocol is completely
|
|
different from Cassandra, and incompatible with it (one cannot mix Scylla
|
|
and Cassandra nodes in a single cluster).
|
|
|
|
Scylla's internal protocol is built on top of Seastar's "RPC" messaging
|
|
mechanism, itself built on top of TCP. This protocol includes messages of
|
|
different types ("verbs") - which are all listed in the source file
|
|
`message/messaging_service.cc`.
|
|
|
|
Here is a non-exhaustive list of things that Scylla does with these messages:
|
|
|
|
* "Gossip" between the nodes to maintain knowledge of the cluster topology:
|
|
Which nodes belong to the cluster, their state and token ownership.
|
|
|
|
* Read and write (mutation) messages sent from a node receiving a client's
|
|
request (a so-called coordinator) to a replica which holds relevant data.
|
|
|
|
* In particular, mutations to special system tables are used to maintain
|
|
cluster-wide agreement on schemas (table definitions).
|
|
|
|
* Streaming data to new nodes and repairing data between nodes.
|
|
|
|
* Messages used to implement LWT (lightweight transactions).
|
|
|
|
Although all these different types of messages use the same RPC protocol and
|
|
the same destination port on the receiver, we don't want to multiplex all of
|
|
these messages over a _single_ TCP connection. If we do, this would allow
|
|
messages of one type to severely delay messages of other types (head-of-line
|
|
blocking). So Scylla opens several sockets to the same destination port
|
|
and uses a different socket for different message types. The function
|
|
`do_get_rpc_client_idx` determines which types of messages get bunched
|
|
together over one socket. As of this writing, the different messages are
|
|
split into four sockets.
|
|
|
|
As everything in Scylla, the messaging service (providing this internal
|
|
communication) is a sharded service, so each shard keeps its own sockets
|
|
to each remote node. The remote node is also sharded. We cannot know for
|
|
sure which shard on the remote node will handle the messages, but we make
|
|
an effort (which is not a guarantee!) that if the two nodes have the same
|
|
number of shards, messages from shard N in the source node arrive to shard
|
|
N in the destination node.
|
|
|
|
Port 7000 is the default port for Scylla's internal communication. This choice
|
|
can be overriden by the `storage_port` configuration option. This awkward
|
|
name, `storage_port`, was kept for backward compatibility with Cassandra's
|
|
YAML configuration file. In very early Cassandra versions, this port was
|
|
used _only_ for "storage" messages (read and write), and other messages such
|
|
as gossip were sent to a different port. So today we are still stuck with
|
|
this outdated name of this configuration option.
|
|
|
|
There is also a `listen_address` configuration option to set the IP address
|
|
(and therefore network interface) on which Scylla should listen for the
|
|
internal protocol. This address defaults to `localhost`, but in any
|
|
setup except a one-node test, should be overriden.
|
|
|
|
TODO: there is also `listen_interface` option... Which wins? What's the default?
|
|
TODO: mention SSL, how it is configured, and `ssl_storage_port` (default 7001).
|
|
|
|
# CQL client protocol
|
|
|
|
The CQL binary protocol is Cassandra's and Scylla's main client-facing
|
|
protocol. Scylla supports several Scylla-only extensions to this protocol,
|
|
described in [protocol-extensions.md](protocol-extensions.md).
|
|
|
|
By default, Scylla listens to the CQL protocol on port 9042, which can be
|
|
configured via the `native_transport_port` configuration option. Scylla also
|
|
supports the CQL protocol via TLS/SSL encryption, which can be enabled via the
|
|
`native_transport_port_ssl` configuration option (default port is 9142). Users
|
|
that want to only support the encrypted CQL protocol can disable the
|
|
unencrypted default CQL protocol by setting the `native_transport_port` to 0.
|
|
The CQL protocol support can be disabled altogether by setting the
|
|
`start_native_transport` option to `false`. These option names were chosen for
|
|
backward-compatibility with Cassandra configuration files: They refers to CQL
|
|
as the "native transport", to contrast with the older Thrift protocol
|
|
(described below) which wasn't native to Cassandra.
|
|
|
|
There is also a `rpc_address` configuration option to set the IP address
|
|
(and therefore network interface) on which Scylla should listen for the
|
|
CQL protocol. This address defaults to `localhost`, but in any setup except
|
|
a one-node test, should be overriden. Note that the same option `rpc_address`
|
|
applies to both CQL and Thrift protocols.
|
|
|
|
TODO: there is also `rpc_interface` option... Which wins? What's the default?
|
|
TODO: again mention SSL, and `native_transport_port_ssl` (default 9142).
|
|
|
|
# Thrift client protocol
|
|
|
|
The Apache Thrift protocol was early Cassandra's client protocol, until
|
|
it was superceded in Cassandra 1.2 with the binary CQL protocol. Thrift
|
|
was still nominally supported by both Cassandra and Scylla for many years,
|
|
but was recently dropped in Cassandra (version 4.0) and is likely to be
|
|
dropped by Scylla in the future as well, so it is not recommended for new
|
|
applications.
|
|
|
|
By default scylla listens to the Thirft protocol on port 9160, which can be
|
|
configured via the `rpc_port` configuration option. Again, this confusing name
|
|
was used for backward-compatibility with Cassandra's configuration files.
|
|
Cassandra used the term "rpc" because Apache Thrift is a remote procedure
|
|
call (RPC) framework. In Scylla, this name is especially confusing, because
|
|
as mentioned above, Scylla's internal communication protocol is based on
|
|
Seastar's RPC, which has nothing to do with the "`rpc_port`" described here.
|
|
|
|
There is also a `rpc_address` configuration option to set the IP address
|
|
(and therefore network interface) on which Scylla should listen for the
|
|
Thrift protocol. This address defaults to `localhost`, but in any
|
|
setup except a one-node test, should be overriden. Note that the same
|
|
option `rpc_address` applies to both CQL and Thrift protocols.
|
|
|
|
TODO: there is also `rpc_interface` option... Which wins? What's the default?
|
|
TODO: is there an SSL version of Thrift?
|
|
|
|
# DynamoDB client protocol
|
|
|
|
Scylla also supports, as an experimental feature, Amazon's DynamoDB API.
|
|
The DynamoDB API is a JSON over HTTP (unencrypted) or HTTPS (encrypted)
|
|
protocol. Because Scylla's support for this protocol is experimental,
|
|
it is not turned on by default, and must be turned on manually by setting
|
|
the `alternator_port` and/or `alternator_https_port` configuration option.
|
|
"Alternator" is the codename of Scylla's DynamoDB API support, and is
|
|
documented in more detail in [alternator.md](alternator/alternator.md).
|
|
|
|
The standard ports that DynamoDB uses are the standard HTTP and HTTPS
|
|
ports (80 and 443, respectively), but in tests we usually use the
|
|
unprivileged port numbers 8000 and 8043 instead.
|
|
|
|
There is also an `alternator_address` configuration option to set the IP
|
|
address (and therefore network interface) on which Scylla should listen
|
|
for the DynamoDB protocol. This address defaults to 0.0.0.0.
|
|
|
|
# Redis client protocol
|
|
|
|
Scylla also has partial and experimental support for the Redis API.
|
|
Again, because this support is experimental, it is not turned on by
|
|
default, and must be turned on manually by setting the `redis_port`
|
|
and/or `redis_ssl_port` configuration option.
|
|
|
|
The traditional port used for Redis is 6379. Regular Redis does not
|
|
support SSL, so there is no traditional choice of port for it.
|
|
|
|
See [redis.md](redis/redis.md) for more information about Scylla's
|
|
support for the Redis protocol.
|
|
|
|
# Metrics protocol
|
|
|
|
Scylla provides an HTTP-based protocol to fetch performance and activity
|
|
metrics from Scylla, which is described in detail in [metrics.md](metrics.md).
|
|
|
|
Scylla listens by default on port 9180 for metric requests. This port number
|
|
can be configured with the `prometheus_port` configuration option, named
|
|
after the Prometheus protocol - and the Prometheus server which is usually
|
|
used to collect these metrics.
|
|
|
|
There is also a `prometheus_address` configuration option to set the IP
|
|
address (and therefore network interface) on which Scylla should listen
|
|
for the metrics protocol. This address defaults to 0.0.0.0.
|
|
|
|
# REST API protocol
|
|
|
|
The CQL client protocol mentioned above is useful mostly for data and table
|
|
requests, but does not offer commands for _administrative_ operations such
|
|
as repair, compact, adding or removing nodes, and so on. Cassandra uses
|
|
nodetool and JMX for these (see below), but Scylla's native approach is a
|
|
RESTful API (HTTP requests).
|
|
|
|
Scylla listens for this REST API by default on port 10000, which can be
|
|
configured with the `api_port` configuration option.
|
|
|
|
The REST API has no notion of autentication of authorization, and allows
|
|
anyone connecting to it to perform destructive operations. Therefore, it
|
|
only listens for connection on the localhost (127.0.0.1) interface. This
|
|
default can be overridden by the `api_address` option - but shouldn't.
|
|
|
|
The available REST API commands are listed in JSON files in the `api/api-doc/`
|
|
directory of the Scylla source. A user can explore this API interactively
|
|
by pointing a browser to http://localhost:10000/ui/.
|
|
|
|
There is an ongoing, but incomplete, effort to replace this REST API by
|
|
a newer, "v2", API. See [api_v2.md](api_v2.md). When complete, this v2
|
|
API will be available on the same port. You can explore the little it
|
|
offers now in the aforementioned UI by replacing the URL in the box with
|
|
"http://localhost:10000/v2".
|
|
|
|
# JMX
|
|
|
|
The "nodetool" management command connects to Scylla using Java's JMX
|
|
protocol, not the REST API described above. This protocol was kept for
|
|
backward compatibility with Cassandra and its unmodified "nodetool" command.
|
|
To implement the JMX protocol, we have a separate project,
|
|
[scylla-jmx](https://github.com/scylladb/scylla-jmx), which runs a Java
|
|
program which accepts the JMX requests supported by Cassandra, and translates
|
|
them to requests to our own REST API. These REST API requests are sent
|
|
to Scylla's REST API port over the loopback (localhost) interface.
|
|
|
|
The port on which scylla-jmx listens is by default port 7199. This port,
|
|
and the listen address, can be overridden with the `-jp` and `-ja` options
|
|
(respectively) of the `scylla-jmx` script.
|