Boot failed when loading sstable with missing summary because a
internal procedure failed to take into account that a sstable
can have its summary recreated from index. Make it work again
by making that procedure aware of that.
Fixes#3057.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
When recreating summary, data length was passed as data offset to
procedure that decides whether to sample or not. The problem is
that the procedure decides to sample index entry if data offset
is beyond a threshold. So the resulting summary will contain
only N sequential indexes entries starting from the first one,
which makes it quite inefficient. What should be done instead
is to pass position of current index entry, so summary content
will be as if it was created by a regular sstable write.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
stream_session.cc:417:62: error: cannot call member function ‘utils::UUID streaming::stream_session::plan_id()’ without object
sslog.warn("[Stream #{}] Failed to send: {}", plan_id(), ep);
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <20171214022621.19442-1-raphaelsc@scylladb.com>
There are some users used original default cluster_name 'Test Cluster',
they will fail to start the node for cluster_name change if they use
new scylla.yaml.
'ScyllaDB Cluster' isn't more beautiful than 'Test Cluster', reset back
to original old to avoid problem for users.
Fixes#3060
Signed-off-by: Amos Kong <amos@scylladb.com>
Message-Id: <8c9dab8a64d0f4ab3a5d6910b87af696c60e5076.1513072453.git.amos@scylladb.com>
"While aa8c2cbc16 'Merge "Migrate sstables
to flat_mutation_reader" from Piotr' has converted the low-level sstable
reader to the new flat_mutation_reader interface there were still
multiple readers related to sstables that required converting,
including:
- restricted reader
- filtering reader
- single partition sstable reader
This series completes their conversion to the flat stream interface."
* tag 'flat_mutation_reader-sstable-readers/v2' of https://github.com/pdziepak/scylla:
db: convert single_key_sstalbe_reader to flat streams
db: fully convert incremental_reader_selector to flat readers
db: make make_range_sstable_reader() return flat reader
db: make column_family::make_reader() return flat reader
db: make column_family::make_sstable_reader() return a flat reader
filtering_reader: switch to flat mutation fragment streams
filtering_reader: pass a const dht::decorated_key& to the callback
mutation_reader: drop make_restricted_reader()
db: use make_restricted_flat_reader
mutation_reader: convert restricted reader to flat streams
Before flat mutation readers sstable::read_row() returned a
future<streamed_mutation>. That required a helper reader that would wait
for the streamed_mutations from all relevant sstables to be created and
then construct a mutation merger.
With flat mutation readers sstable::read_row_flat() returns a
flat_mutation_reader (no futures) so that the code can be simplified by
collecting all the relevant readers and creating a combined reader
without suspension points.
The unfortunate disadvantage of the flat_mutation_reader-based approach
is the fact that combined reader now needlessly compares the partition
keys even though we know that we read only a single partition, but
optimising that is out of scope of this patch.
All users of the filtering reader need only the decorated key of a
partition, but currently the predicate is given a reference to
streamed_mutations which are obsolete now.
In the case there are large number of column families, the sender will
send all the column families in parallel. We allow 20% of shard memory
for streaming on the receiver, so each column family will have 1/N, N is
the number of in-flight column families, memory for memtable. Large N
causes a lot of small sstables to be generated.
It is possible there are multiple senders to a single receiver, e.g.,
when a new node joins the cluster, the maximum in-flight column families
is number of peer node. The column families are sent in the order of
cf_id. It is not guaranteed that all peers has the same speed so they
are sending the same cf_id at the same time, though. We still have
chance some of the peers are sending the same cf_id.
Fixes#3065
Message-Id: <46961463c2a5e4f1faff232294dc485ac4f1a04e.1513159678.git.asias@scylladb.com>
We have had an issue recently where failed SSTable writes left the
generated SSTables dangling in a potentially invalid state. If the write
had, for instance, started and generated tmp TOCs but not finished,
those files would be left for dead.
We had fixed this in commit b7e1575ad4,
but streaming memtables still have the same isse.
Note that we can't fix this in the common function
write_memtable_to_sstable because different flushers have different
retry policies.
Fixes#3062
Signed-off-by: Glauber Costa <glauber@scylladb.com>
Message-Id: <20171213011741.8156-1-glauber@scylladb.com>
"This patch series refines the security model for the upcoming switch to
roles-based access control. Roles are still do not have any function,
but CQL statements related to roles manipulate metdata. The next major
patch series after this one will switch the system to roles.
Previously, most operations around roles required superuser, but this
violates an important idea in security called the "principal of least
privilege": that a user should have only the minimum access possible to
resources in order to achieve their objective.
To that end, this patch series introduces permissions on role resources.
For example, to grant a role to a user, the performing user must have
been granted AUTHORIZE on the role being granted.
In the table below, a user (role) that has been granted the permission
in the left-most column can perform the CQL query in the right columns
depending on if the permission has been granted to the root role
resource (all roles), or a particular role resource.
Perm. All roles Specific role (r)
---------------------------------------------------------
CREATE CREATE ROLE
ALTER ALTER ROLE * ALTER ROLE r
DROP DROP ROLE * DROP ROLE r
AUTHORIZE GRANT ROLE */REVOKE GRANT ROLE r/
ROLE * REVOKE ROLE r
DESCRIBE LIST ROLES
The following restrictions around superuser exist:
- CREATE ROLE: Only a superuser can create a superuser role.
- ALTER ROLE: Only a superuser can alter the superuser status of a role.
- ALTER ROLE: You cannot alter the superuser status of yourself or of a
role granted to you.
- DROP ROLE: Only a superuser can drop a role that has superuser.
The following additional "escape hatches" apply:
- ALTER ROLE: You can alter yourself (except to give yourself
superuser).
- LIST ROLES: You can list your own roles and list the roles of any role
granted to you.
Finally, a note on terminology: I like to say a role (or user) "is"
superuser if the role (user) has directly been marked as a superuser. A
role (user) "has" superuser if they have been granted a role that is a
superuser. The second statement encompasses the first, since a role can
always be said to have been granted to itself.
Fixes #2988."
* 'jhk/role_permissions/v2' of https://github.com/hakuch/scylla: (24 commits)
auth: Move permissions cache instance to service
auth: Add roles query function to service
cql3: Update access checks for `revoke_role_statement`
cql3: Update access checks in `grant_role_statement`
cql3: Update access checks in `list_roles_statement`
cql3: Update access checks in `drop_role_statement`
cql3: Update access checks in `alter_role_statement`
cql3: Update access checks in `create_role_statement`
tests: Switch to dedicated testing superuser
auth: Publicize enforcing check for service
tests: Expose client state from test env
Allow checking permissions from `client_state`
auth: Support querying for granted superuser
auth/service.hh: Document the class
cql3: Change `create_role_statement` base
cql3/Cql.g: Add role resources to grammar
cql3/Cql.g: Avoid extra copy of `auth::resource`
auth:resource.cc: Use `string_view` in reverse map
auth: Add `role` resource kind
auth: Add the DESCRIBE permission
...
Instead of a single sharded service shared all by all instances of
`auth::service`, it makes more sense for each instance of
`auth::service` to own its own instance of the permissions cache.
While it just calls into the underlying role manager, this level of
indirection allows us to add a roles cache in the future (which is
consistent with the behavior of Apache Cassandra).
A user with DESCRIBE on the root role resource can list any roles of any
roles, and also the roles in the system.
Otherwise, a user can list all the roles it has been granted and can
list all roles granted to those roles.
A role can be dropped if the performer has DROP permission on the role.
A role that has superuser (either directly or through another role
it has been granted) cannot be dropped except by a superuser.
Only superusers can alter superuser status, but only to roles not
granted to them. You can always alter your own role. You can alter
another role if you have ALTER permission on the role.
CREATE ROLE requires CREATE on <ALL ROLES>. Creating a superuser role
requires that the performer is a superuser.
This change also forms the beginning of a test suite for the CQL
interface to roles. We start with verifying access-control properties of
CREATE ROLE as written in this patch.
The auth service will eventually add the default
superuser ("cassandra"), but the current code does so after a delay.
Using a dedicated superuser for unit tests side-steps the issue and
allows the user to be created immediately.
Previously, this function was private and only `ensure_has_permission`
was public. `ensure_has_permission` throws in the absence of a
permission, but it can also be useful to query a permission without it
being an error.
This functionality is useful for implementing CQL statements and will
replace `auth::is_super_user` once roles have replaced users in Scylla.
Since eventually the auth service will have a roles cache, this function
is here rather than a part `role_manager`.
When a user is granted DESCRIBE on all roles (a resource kind that
doesn't exist yet in the code, but will exist soon), they gain the
ability to execute LIST ROLES queries.
Different kinds of resources support different permissions. For example,
a keyspace supports the CREATE permission, which allows a user to
create tables in that keyspace. However, a table does not have an
applicable CREATE permission.
If a non-applicable permission is requested, an
`invalid_request_exception` is thrown.
"Fixes #2866Fixes#2894
Changes gossip propagation to allow "atomic" grouping of values to ensure
their respective order.
Modifies gossip bootstrap startup to potentially wait longer in cases
where stabilization (messages done) takes time, to avoid data loss
in repair."
* 'calle/gossip' of github.com:scylladb/seastar-dev:
gossip: wait for stabilized gossip on bootstrap
gossiper: Prevent race condition in propagation
utils::to_string: Add printers for pairs+maps
utils::in: Add helper type for perfect forwarding initializer lists
This set is equal to `permissions::ALL`. When we switch over to
resource-specific permission sets, we will filter the set of all
permissions to only those that are applicable for the resource in
question.
Applicable permission sets will soon be specific to each kind of
resource. This change prepares us for dynamic querying of permission
sets by resource.
* seastar ac78eec...2b23547 (10):
> Merge "update shares for I/O classes" from Glauber
> Merge "Resumable tasks" from Avi
> input_stream: un-unroll input_stream::consume()
> net: adding yaml-based parser for network configuration supporting multiple interfaces
> scripts: perftune.py: don't attempt to set IRQs' affinity when IRQs list is empty
> tutorial: fix example code
> http: api_docs add swagger 2.0 support
> Support custom function for reading of config-files.
> Revert "provide an interface for updating the shares of an I/O class"
> provide an interface for updating the shares of an I/O class
The encoding logic was incorrect for big endian systems (shift needed
to be in the opposite direction). Rather than fix that issue I have
re-written the relevant code to restrict the storage format to little
endian byte order on all systems. My hope is that this will be a bit
easier to maintain.
Message-Id: <20171211124454.77488-1-mike.munday@ibm.com>
Most distros on s390x don't currently have gold installed by default.
Rather than disable gold on the platform add a check to see if gold
is installed and switch back to using the default system linker if it
isn't. The try_compile_and_link functionality is copied from the
seastar project.
Message-Id: <20171211122156.77385-1-mike.munday@ibm.com>
"The changes in this series fall into one of the following:
1) improve unit tests
2) improve code reuse in mvcc so that later cahnges will be easier
3) fix minor issues which were exposed by the above"
* tag 'tgrabiec/improve-and-fix-mvcc-tests-v4' of github.com:scylladb/seastar-dev:
tests: mvcc: Add more tests for consistency of continuity merging
tests: mvcc: Fix test_apply_is_atomic()
tests: mvcc: Do not assume that continuity of current row is updated on partition_snapshot_row_cursor::maybe_refresh()
mvcc: Reuse partition_snapshot_row_cursor in apply_to_incomplete()
mvcc: Propagate region reference to partition_entry::apply_to_incomplete()
mvcc: Introduce partition_snapshot_row_cursor::ensure_entry_if_complete()
mvcc: partition_snapshot_row_cursor: Extract prepare_heap()
mvcc: Add const-qualified partition_version_ref::operator*()
tests: mvcc: Use mutation_partition_assertions
tests: Introduce mutation_partition_assertions
tests: Randomize static row continuity in random_mutation_generator
tests: mutation_assertion: Introduce is_continuous()
mvcc: Introduce partition_snapshot_row_cursor::read_partition()
mutation_partition: Introduce deletable_row::apply() from a clustering_row fragment
mutation_partition: Extract sliced() from mutation into mutation_partition
mvcc: Introduce partition_snapshot::static_row_continuous()
mvcc: Introduce partition_snapshot::range_tombstones() for full range
mvcc: Don't require external schema in parition_snapshot::range_tombstones()
mutation_partition: Define equal_continuity() using get_continuity()
mutation_partition: Make check_continuity() const-qualified
mutation_partition: Make check_continuity() public
mutation_partition: Introduce mutation_partition::get_continuity()
Introduce clustering_interval_set
mutation_partition: Leave moved-from row in an empty state
mutation_partition: Fix upgrade() not preserving static row continuity
