Currently, Scylla logs a warning when it writes a cell, row or partition which are larger than certain configured sizes. These warnings contain the partition key and in case of rows and cells also the cluster key which allow the large row or partition to be identified. However, these keys can contain user-private, sensitive information. The information which identifies the partition/row/cell is also inserted into tables system.large_partitions, system.large_rows and system.large_cells respectivelly.
This change removes the partition and cluster keys from the log messages, but still inserts them into the system tables.
The logged data will look like this:
Large cells:
WARN 2024-04-02 16:49:48,602 [shard 3: mt] large_data - Writing large cell ks_name/tbl_name: cell_name (SIZE bytes) to sstable.db
Large rows:
WARN 2024-04-02 16:49:48,602 [shard 3: mt] large_data - Writing large row ks_name/tbl_name: (SIZE bytes) to sstable.db
Large partitions:
WARN 2024-04-02 16:49:48,602 [shard 3: mt] large_data - Writing large partition ks_name/tbl_name: (SIZE bytes) to sstable.db
Fixes#18041Closesscylladb/scylladb#18166
(cherry picked from commit f1cc6252fd)
This commit updates the Upgrade ScyllaDB Image page.
- It removes the incorrect information that updating underlying OS packages is mandatory.
- It adds information about the extended procedure for non-official images.
(cherry picked from commit fc90112b97)
Closesscylladb/scylladb#17886
Before this PR, writes to the previous CDC generations would
always be rejected. After this PR, they will be accepted if the
write's timestamp is greater than `now - generation_leeway`.
This change was proposed around 3 years ago. The motivation was
to improve user experience. If a client generates timestamps by
itself and its clock is desynchronized with the clock of the node
the client is connected to, there could be a period during
generation switching when writes fail. We didn't consider this
problem critical because the client could simply retry a failed
write with a higher timestamp. Eventually, it would succeed. This
approach is safe because these failed writes cannot have any side
effects. However, it can be inconvenient. Writing to previous
generations was proposed to improve it.
The idea was rejected 3 years ago. Recently, it turned out that
there is a case when the client cannot retry a write with the
increased timestamp. It happens when a table uses CDC and LWT,
which makes timestamps permanent. Once Paxos commits an entry
with a given timestamp, Scylla will keep trying to apply that entry
until it succeeds, with the same timestamp. Applying the entry
involves writing to the CDC log table. If it fails, we get stuck.
It's a major bug with an unknown perfect solution.
Allowing writes to previous generations for `generation_leeway` is
a probabilistic fix that should solve the problem in practice.
Apart from this change, this PR adds tests for it and updates
the documentation.
This PR is sufficient to enable writes to the previous generations
only in the gossiper-based topology. The Raft-based topology
needs some adjustments in loading and cleaning CDC generations.
These changes won't interfere with the changes introduced in this
PR, so they are left for a follow-up.
Fixesscylladb/scylladb#7251Fixesscylladb/scylladb#15260Closesscylladb/scylladb#17134
* github.com:scylladb/scylladb:
docs: using-scylla: cdc: remove info about failing writes to old generations
docs: dev: cdc: document writing to previous CDC generations
test: add test_writes_to_previous_cdc_generations
cdc: generation: allow increasing generation_leeway through error injection
cdc: metadata: allow sending writes to the previous generations
(cherry picked from commit 9bb4482ad0)
Backport note: in tests, replaced `servers_add` with loop of `server_add`
With this commit:
- The information about ScyllaDB Enterprise OS support
is removed from the Open Source documentation.
- The information about ScyllaDB Open Source OS support
is moved to the os-support-info file in the _common folder.
- The os-support-info file is included in the os-support page
using the scylladb_include_flag directive.
This update employs the solution we added with
https://github.com/scylladb/scylladb/pull/16753.
It allows to dynamically add content to a page
depending on the opensource/enterprise flag.
Refs https://github.com/scylladb/scylladb/issues/15484Closesscylladb/scylladb#17310
(cherry picked from commit ef1468d5ec)
This commit adds the missing redirections
to the pages whose source files were
previously stored in the install-scylla folder
and were moved to another location.
Closesscylladb/scylladb#17367
(cherry picked from commit e132ffdb60)
This PR:
- Adds the upgrade guide from ScyllaDB Open Source 5.4 to ScyllaDB Enterprise 2024.1. Note: The need to include the "Restore system tables" step in rollback has been confirmed; see https://github.com/scylladb/scylladb/issues/11907#issuecomment-1842657959.
- Removes the 5.1-to-2022.2 upgrade guide (unsupported versions).
Fixes https://github.com/scylladb/scylladb/issues/16445Closesscylladb/scylladb#16887
* github.com:scylladb/scylladb:
doc: fix the OSS version number
doc: metric updates between 2024.1. and 5.4
doc: remove the 5.1-to-2022.2 upgrade guide
doc: add the 5.4-to-2024.1 upgrade guide
(cherry picked from commit edb983d165)
This commit adds the information that
ScyllaDB Enterprise 2024.1 is based
on ScyllaDB Open Source 5.4
to the OSS vs. Enterprise matrix.
Closesscylladb/scylladb#16880
(cherry picked from commit a462b914cb)
This commit removes the upgrade guides
from ScyllaDB Open Source to Enterprise
for versions we no longer support.
In addition, it removes a link to
one of the removed pages from
the Troubleshooting section (the link is
redundant).
(cherry picked from commit 0ad3ef4c55)
Closesscylladb/scylladb#16913
This commit removes the information about ScyllaDB Cloud Serverless,
which is no longer valid.
(cherry picked from commit 758284318a)
Closesscylladb/scylladb#16805
Major compaction already flushes each table to make
sure it considers any mutations that are present in the
memtable for the purpose of tombstone purging.
See 64ec1c6ec6
However, tombstone purging may be inhibited by data
in commitlog segments based on `gc_time_min` in the
`tombstone_gc_state` (See f42eb4d1ce).
Flushing all sstables in the database release
all references to commitlog segments and there
it maximizes the potential for tombstone purging,
which is typically the reason for running major compaction.
However, flushing all tables too frequently might
result in tiny sstables. Since when flushing all
keyspaces using `nodetool flush` the `force_keyspace_compaction`
api is invoked for keyspace successively, we need a mechanism
to prevent too frequent flushes by major compaction.
Hence a `compaction_flush_all_tables_before_major_seconds` interval
configuration option is added (defaults to 24 hours).
In the case that not all tables are flushed prior
to major compaction, we revert to the old behavior of
flushing each table in the keyspace before major-compacting it.
Fixesscylladb/scylladb#15777Closesscylladb/scylladb#15820
to address the confliction, following change is also included in this changeset:
tools/scylla-nodetool: implement the cleanup command
The --jobs command-line argument is accepted but ignored, just like the
current nodetool does.
Refs: scylladb/scylladb#15588Closesscylladb/scylladb#16756
* github.com:scylladb/scylladb:
docs: nodetool: flush: enrich examples
docs: nodetool: compact: fix example
api: add /storage_service/compact
api: add /storage_service/flush
tools/scylla-nodetool: implement the flush command
compaction_manager: flush_all_tables before major compaction
database: add flush_all_tables
api: compaction: add flush_memtables option
test/nodetool: jmx: fix path to scripts/scylla-jmx
scylla-nodetool, docs: improve optional params documentation
tools/scylla-nodetool: extract keyspace/table parsing
tools/scylla-nodetool: implement the cleanup command
test/nodetool: rest_api_mock: add more options for multiple requests
This commit removes support for CentOS 7
from the docs.
The change applies to version 5.4,so it
must be backported to branch-5.4.
Refs https://github.com/scylladb/scylla-enterprise/issues/3502
In addition, this commit removes the information
about Amazon Linux and Oracle Linux, unnecessarily added
without request, and there's no clarity over which versions
should be documented.
Closesscylladb/scylladb#16279
(cherry picked from commit af1405e517)
Provide 3 examples, like in the nodetool/compact page:
global, per-keyspace, per-table.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
(cherry picked from commit 310ff20e1e)
It looks like `nodetool compact standard1` is meant
to show how to compact a specified table, not a keyspace.
Note that the previous example like is for a keyspace.
So fix the table compaction example to:
`nodetool compact keyspace1 standard1`
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
(cherry picked from commit d32b90155a)
Document the behavior if no keyspace is specified
or no table(s) are specified for a given keyspace.
Fixesscylladb/scylladb#16032
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
(cherry picked from commit 9324363e55)
This commit replaces the link to the OSS-only page
(the 5.2-to-5.4 upgrade guide not present in
the Enterprise docs) on the Raft page.
While providing the link to the specific upgrade
guide is more user-friendly, it causes build failures
of the Enterprise documentation. I've replaced
it with the link to the general Upgrade section.
The ".. only:: opensource" directive used to wrap
the OSS-only content correctly excludes the content
form the Enterprise docs - but it doesn't prevent
build warnings.
This commit must be backported to branch-5.4 to
prevent errors in all versions.
Closesscylladb/scylladb#16176
(cherry picked from commit 24d5dbd66f)
This commit adds a short paragraph to the Raft
page to explain how to enable consistent
topology updates with Raft - an experimental
feature in version 5.4.
The paragraph should satisfy the requirements
for version 5.4. The Raft page will be
rewritten in the next release when consistent
topology changes with Raft will be GA.
Fixes https://github.com/scylladb/scylladb/issues/15080
Requires backport to branch-5.4.
Closesscylladb/scylladb#16273
(cherry picked from commit 409e20e5ab)
This commit fixes the rollback procedure in
the 4.6-to-5.0 upgrade guide:
- The "Restore system tables" step is removed.
- The "Restore the configuration file" command
is fixed.
- The "Gracefully shutdown ScyllaDB" command
is fixed.
In addition, there are the following updates
to be in sync with the tests:
- The "Backup the configuration file" step is
extended to include a command to backup
the packages.
- The Rollback procedure is extended to restore
the backup packages.
- The Reinstallation section is fixed for RHEL.
Refs https://github.com/scylladb/scylladb/issues/11907
This commit must be backported to branch-5.4, branch-5.2, and branch-5.1
Closesscylladb/scylladb#16155
(cherry picked from commit 1e80bdb440)
This commit fixes the rollback procedure in
the 5.0-to-5.1 upgrade guide:
- The "Restore system tables" step is removed.
- The "Restore the configuration file" command
is fixed.
- The "Gracefully shutdown ScyllaDB" command
is fixed.
In addition, there are the following updates
to be in sync with the tests:
- The "Backup the configuration file" step is
extended to include a command to backup
the packages.
- The Rollback procedure is extended to restore
the backup packages.
- The Reinstallation section is fixed for RHEL.
Also, I've the section removed the rollback
section for images, as it's not correct or
relevant.
Refs https://github.com/scylladb/scylladb/issues/11907
This commit must be backported to branch-5.4, branch-5.2, and branch-5.1
Closesscylladb/scylladb#16154
(cherry picked from commit 7ad0b92559)
This commit fixes the rollback procedure in
the 5.1-to-5.2 upgrade guide:
- The "Restore system tables" step is removed.
- The "Restore the configuration file" command
is fixed.
- The "Gracefully shutdown ScyllaDB" command
is fixed.
In addition, there are the following updates
to be in sync with the tests:
- The "Backup the configuration file" step is
extended to include a command to backup
the packages.
- The Rollback procedure is extended to restore
the backup packages.
- The Reinstallation section is fixed for RHEL.
Also, I've the section removed the rollback
section for images, as it's not correct or
relevant.
Refs https://github.com/scylladb/scylladb/issues/11907
This commit must be backported to branch-5.4 and branch-5.2.
Closesscylladb/scylladb#16152
(cherry picked from commit 91cddb606f)
This commit adds information on how to enable
object storage for a keyspace.
The "Keyspace storage options" section already
existed in the doc, but it was not valid as
the support was only added in version 5.4
The scope of this commit:
- Update the "Keyspace storage options" section.
- Add the information about object storage support
to the Data Definition> CREATE KEYSPACE section
* Marked as "Experimental".
* Excluded from the Enterprise docs with the
.. only:: opensource directive.
This commit must be backported to branch-5.4,
as support for object storage was added
in version 5.4.
Closesscylladb/scylladb#16081
(cherry picked from commit bfe19c0ed2)
This commit fixes the rollback procedure in
the 5.2-to-5.4 upgrade guide:
- The "Restore system tables" step is removed.
- The "Restore the configuration file" command
is fixed.
- The "Gracefully shutdown ScyllaDB" command
is fixed.
In addition, there are the following updates
to be in sync with the tests:
- The "Backup the configuration file" step is
extended to include a command to backup
the packages.
- The Rollback procedure is extended to restore
the backup packages.
- The Reinstallation section is fixed for RHEL.
Also, I've removed the optional step to enable
consistent schema management from the list of
steps - the appropriate section has already
been removed, but it remained in the procedure
description, which was misleading.
Refs https://github.com/scylladb/scylladb/issues/11907
This commit must be backported to branch-5.4
Closesscylladb/scylladb#16114
(cherry picked from commit 3751acce42)
This commit adds the .. only:: opensource directive
to the Raft page to exclude the link to the 5.2-to-5.4
upgrade guide from the Enterprise documentation.
The Raft page belongs to both OSS and Enterprise
documentation sets, while the upgrade guide
is OSS-only. This causes documentation build
issues in the Enterprise repository, for example,
https://github.com/scylladb/scylla-enterprise/pull/3242.
As a rule, all OSS-only links should be provided
by using the .. only:: opensource directive.
This commit must be backported to branch-5.4
to prevent errors in the documentation for
ScyllaDB Enterprise 2024.1
(backport)
Closesscylladb/scylladb#16064
(cherry picked from commit ca22de4843)
This commit updates the Repair-Based Node
Operations page. In particular:
- Information about RBNO enabled for all
node operations is added (before 5.4, RBNO
was enabled for the replace operation, while
it was experimental for others).
- The content is rewritten to remove redundant
information about previous versions.
The improvement is part of the 5.4 release.
This commit must be backported to branch-5.4
Closesscylladb/scylladb#16015
(cherry picked from commit 8a4a8f077a)
since we use `sphinx_multiversion` for building multiple versions of document. and in #15860, we changed the way how options are rendered, so the same change should be applied to the branch which includes the option list.
to address the conflicts, in addition to #15860, the depended PRs are also backported. so, in this pull request, following PRs are backported:
- #15827
- #15782
- #15860Closesscylladb/scylladb#16030
* github.com:scylladb/scylladb:
docs: add divider using CSS
docs: extract _clean_description as a filter
docs: render option with role
docs: update cofig params design
docs: parse source files right into rst
instead of hardwiring the formatting in the html code, do this using
CSS, more flexible this way.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
(cherry picked from commit ff12f1f678)
would be better to split the parser from the formatter. in future,
we can apply more filter on top of the exiting one.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
(cherry picked from commit 1694a7addc)
so we can cross-reference them with the syntax like
:confval:`alternator_timeout_in_ms`.
or even render an option like:
.. confval:: alternator_timeout_in_ms
in order to make the headerlink of the option visible,
a new CSS rule is added.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
(cherry picked from commit 9ddc639237)
This commit fixes the information about
Raft-based consistent cluster management
in the 5.2-to-5.4 upgrade guide.
This a follow-up to https://github.com/scylladb/scylladb/pull/15880 and must be backported to branch-5.4.
In addition, it adds information about removing
DateTieredCompactionStrategy to the 5.2-to-5.4
upgrade guide, including the guideline to
migrate to TimeWindowCompactionStrategy.
Closesscylladb/scylladb#15988
(cherry picked from commit ca0f5f39b5)
This commit updates the cqlsh compatibility
with Python to Python 3.
In addition it:
- Replaces "Cassandra" with "ScyllaDB" in
the description of cqlsh.
The previous description was outdated, as
we no longer can talk about using cqlsh
released with Cassandra.
- Replaces occurrences of "Scylla" with "ScyllaDB".
- Adds additional locations of cqlsh (Docker Hub
and PyPI), as well as the link to the scylla-cqlsh
repository.
Closesscylladb/scylladb#16016
(cherry picked from commit 8d618bbfc6)
This PR adds the 5.2-5.4 upgrade guide.
In addition, it removes the redundant upgrade guide from 5.2 to 5.3 (as 5.3 was skipped), as well as some mentions of version 5.3.
This PR must be backported to branch-5.4.
Closesscylladb/scylladb#15880
* github.com:scylladb/scylladb:
doc: add the upgrade guide from 5.2 to 5.4
doc: remove version "5.3" from the docs
doc: remove the 5.2-to-5.3 upgrade guide
(cherry picked from commit 74f68a472f)
This commit updates the package installation
instructions in version 5.4.
- It updates the variables to include "5.4"
as the version name.
- It adds the information for the newly supported
Rocky/RHEL 9 - a new EPEL download link is required.
Closesscylladb/scylladb#15963
(cherry picked from commit 1e0cbfe522)
This commit adds OS support information
in version 5.4 (removing the non-released
version 5.3).
In particular, it adds support for Oracle Linux
and Amazon Linux.
Also, it removes support for outdated versions.
Closesscylladb/scylladb#15923
(cherry picked from commit 3756705520)
This commit fixes the layout of the Reference
page. Previously, the toctree level was "2",
which made the page hard to navigate.
This PR changes the level to "1".
In addition, the capitalization of page
titles is fixed.
This is a follow-up PR to the ones that
created and updated the Reference section.
It must be backported to branch-5.4.
Closesscylladb/scylladb#15830
(cherry picked from commit e223624e2e)
This commit:
- Removes upgrade guides for versions older than 5.0.
The oldest one is from version 4.6 to 5.0.
- Adds the redirections for the removed pages.
Closesscylladb/scylladb#15709
This commit removes the information about the recommended way of upgrading ScyllaDB images - by updating ScyllaDB and OS packages in one step. This upgrade procedure is not supported (it was implemented, but then reverted).
The scope of this commit:
- Remove the information from the 5.0-to.-5.1 upgrade guide and replace with general info.
- Remove the information from the 4.6-to.-5.1 upgrade guide and replace with general info.
- Remove the information from the 5.x.y-to.-5.x.z upgrade guide and replace with general info.
- Remove the following files as no longer necessary (they were only created to incorporate the (invalid) information about image upgrade into the upgrade guides.
/upgrade/_common/upgrade-image-opensource.rst
/upgrade/_common/upgrade-guide-v5-patch-ubuntu-and-debian-p1.rst
/upgrade/_common/upgrade-guide-v5-patch-ubuntu-and-debian-p2.rst
/upgrade/_common/upgrade-guide-v5-patch-ubuntu-and-debian.rst
This PR is a continuation of https://github.com/scylladb/scylladb/pull/15739.
**This PR must be backported to branch-5.2 and branch-5.1.**
Closesscylladb/scylladb#15740
* github.com:scylladb/scylladb:
doc: remove wrong image upgrade info (5.x.y-to-5.x.y)
doc: remove wrong image upgrade info (4.6-to-5.0)
doc: remove wrong image upgrade info (5.0-to-5.1)
This commit removes the information about
the recommended way of upgrading ScyllaDB
images - by updating ScyllaDB and OS packages
in one step.
This upgrade procedure is not supported
(it was implemented, but then reverted).
The scope of this commit:
- Remove the information from the 5.1-to.-5.2
upgrade guide and replace with general info.
- Remove the information from the Image Upgrade
page.
- Remove outdated info (about previous releases)
from the Image Upgrade page.
- Rename "AMI Upgrade" as "Image Upgrade"
in the page tree.
Refs: https://github.com/scylladb/scylladb/issues/15733Closesscylladb/scylladb#15739
