Just like 4a8ed4cc6f, we also need to wait for udev event completion to
create /dev/disk/by-uuid/$UUID for newly formatted disk, to mount the
disk just after formatting.
Also added code to check make sure uuid and uuid based device path are valid.
Fixes#11359Closes#11399
* github.com:scylladb/scylladb:
scylla_raid_setup: prevent mount failed for /var/lib/scylla
scylla_raid_setup: check uuid and device path are valid
this setting was removed back in
dcdd207349, so despite that we are still
passing `storage_service_config` to the ctor of `storage_service`,
`storage_service::storage_service()` just drops it on the floor.
in this change, `storage_service_config` class is removed, and all
places referencing it are updated accordingly.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
Closes#11415
Google Groups recently started rewriting the From: header, garbaging
our git log. This script rewrites it back, using the Reply-To header
as a still working source.
Closes#11416
It was pointed out to me that our description of the synchronous_updates
materialized-view option does not make it clear enough what is the
default setting, or why a user might want to use this option.
This patch changes the description to (I hope) better address these
issues.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Closes#11404
* github.com:scylladb/scylladb:
doc: cql-extensions.md: replace "Scylla" by "ScyllaDB"
doc: cql-extensions.md: improve description of synchronous views
This is a very important aspect of the tool that was completely missing from the document before. Also add a comparison with SStableDump.
Fixes: https://github.com/scylladb/scylladb/issues/11363Closes#11390
* github.com:scylladb/scylladb:
docs: scylla-sstable.rst: add comparison with SStableDump
docs: scylla-sstable.rst: add section about providing the schema
It was recently decided that the database should be referred to as
"ScyllaDB", not "Scylla". This patch changes existing references
in docs/cql/cql-extensions.md.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
It was pointed out to me that our description of the synchronous_updates
materialized-view option does not make it clear enough what is the
default setting, or why a user might want to use this option.
This patch changes the description to (I hope) better address these
issues.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
"
The topology object maintains all sort of node/DC/RACK mappings on
board. When new entries are added to it the DC and RACK are taken
from the global snitch instance which, in turn, checks gossiper,
system keyspace and its local caches.
This set make topology population API require DC and RACK via the
call argument. In most of the cases the populating code is the
storage service that knows exactly where to get those from.
After this set it will be possible to remove the dependency knot
consiting of snitch, gossiper, system keyspace and messaging.
"
* 'br-topology-dc-rack-info' of https://github.com/xemul/scylla:
toplogy: Use the provided dc/rack info
test: Provide testing dc/rack infos
storage_service: Provide dc/rack for snitch reconfiguration
storage_service: Provide dc/rack from system ks on start
storage_service: Provide dc/rack from gossiper for replacement
storage_service: Provide dc/rack from gossiper for remotes
storage_service,dht,repair: Provide local dc/rack from system ks
system_keyspace: Cache local dc-rack on .start()
topology: Some renames after previous patch
topology: Require entry in the map for update_normal_tokens()
topology: Make update_endpoint() accept dc-rack info
replication_strategy: Accept dc-rack as get_pending_address_ranges argument
dht: Carry dc-rack over boot_strapper and range_streamer
storage_service: Make replacement info a real struct
* seastar f9f5228b74...f2d70c4a17 (51):
> cmake: attach property to Valgrind not to hwloc
> Create the seastar_memory logger in all builds
> drop unused parameters
> Merge "Unify pollable_fd shutdown and abort_{reader|writer}" from Pavel E
> > pollable_fd: Replace two booleans with a mask
> > pollable_fd: Remove abort_reader/_writer
> Merge "Improve Rx channels assignment" from Vlad
> > perftune.py: fix comments of IRQ ordering functors
> > perftune.py: add VIRTIO fast path IRQs ordering functor
> > perftune.py: reduce number of Rx channels to the number of IRQ CPUs
> > perftune.py: introduce a --num-rx-queues parameter
> program_options: enable optional selection_value
> .gitignore: ignore the directories generated by VS Code and CLion.
> httpd: compare the Connection header value in a case-insensitive manner.
> httpd: move the logic of keepalive to a separate method.
> register one default priority class for queue
> Reset _total_stats before each run
> log: add colored logging support
> Merge "perftune.py: add NUMA aware auto-detection for big machines" from Vlad
> > perftune.py: mention 'irq_cpu_mask' in the description of the script operation
> > perftune.py: NetPerfTuner: fix bits counting in self.irqs_cpu_mask wider than 32 bits
> > perftune.py: PerfTuneBase.cpu_mask_is_zero(cpu_mask): cosmetics: fix a comment and a variable name
> > perftune.py: PerfTuneBase.cpu_mask_is_zero(cpu_mask): take into account omitted zero components of the mask
> > perftune.py: PerfTuneBase.compute_cpu_mask_for_mode(): cosmetics: fix a variable name
> > perftune.py: stop printing 'mode' in --dump-options-file
> > perftune.py: introduce a generic auto_detect_irq_mask(cpu_mask) function
> > perftune.py: DiskPerfTuner: use self.irqs_cpu_mask for tuning non-NVME disks
> > perftune.py: stop auto-detecting and using 'mode' internally
> > perftune.py: introduce --get-irq-cpu-mask command line parameter
> > perftune.py: introduce --irq-core-auto-detection-ratio parameter
> build: add a space after function name
> Update HACKING.md
> log: do not inherit formatter<seastar::log_level> from formatter<string_view>
> Merge "Mark connected_socket::shutdown_...'s internals noexcept" from Pavel E
> > native-stack: Mark tcp::in_state() (and its wrappers) const noexcept
> > native-stack: Mark tcb::close and tcb::abort_reader noexcept
> > native-stack: Mark tcp::connection::close_{read|write}() noexcept
> > native-stack: Mark tcb::clear_delayed_ack() and tcb::stop_retransmit_timer() noexcept
> > tls: Mark session::close() noexcept
> > file_desc: Add fdinfo() helper
> > posix-stack: Mark posix_connected_socket_impl::shutdown_{input|output}() noexcept
> > tests: Mark loopback_buffer::shutdown() noexcept
> Merge "Enhance RPC connection error injector" from Pavel E
> > loopback_socket: Shuffle error injection
> > loopback_socket: Extend error injection
> > loopback_socket: Add one-shot errors
> > loopback_socket: Add connection error injection
> > rpc_test: Extend error injector with kind
> > rpc_test: Inject errors on all paths
> > rpc_test: Use injected connect error
> > rpc_test: De-duplicate test socket creation
> Merge 'tls: vec_push: handle async errors rather than throwing on_internal_error' from Benny Halevy
> > tls: do_handshake: handle_output_error of gnutls_handshake
> > tls: session: vec_push: return output_pending error
> > tls: session: vec_push: reindent
> log: disambiguate formatter<log_level> from operator<<
> tls_test: Fix spurious fail in test_x509_client_with_builder_system_trust_multiple (et al)
Fixes scylladb/scylladb#11252
Closes#11401
This PR is related to https://github.com/scylladb/scylla-docs/issues/4124 and https://github.com/scylladb/scylla-docs/issues/4123.
**New Enterprise Upgrade Guide from 2021.1 to 2022.2**
I've added the upgrade guide for ScyllaDB Enterprise image. In consists of 3 files:
/upgrade/_common/upgrade-guide-v2022-ubuntu-and-debian-p1.rst
upgrade/_common/upgrade-image.rst
/upgrade/_common/upgrade-guide-v2022-ubuntu-and-debian-p2.rst
**Modified Enterprise Upgrade Guides 2021.1 to 2022.2**
I've modified the existing guides for Ubuntu and Debian to use the same files as above, but exclude the image-related information:
/upgrade/_common/upgrade-guide-v2022-ubuntu-and-debian-p1.rst + /upgrade/_common/upgrade-guide-v2022-ubuntu-and-debian-p2.rst = /upgrade/_common/upgrade-guide-v2022-ubuntu-and-debian.rst
To make things simpler and remove duplication, I've replaced the guides for Ubuntu 18 and 20 with a generic Ubuntu guide.
**Modified Enterprise Upgrade Guides from 4.6 to 5.0**
These guides included a bug: they included the image-related information (about updating OS packages), because a file that includes that information was included by mistake. What's worse, it was duplicated. After the includes were removed, image-related information is no longer included in the Ubuntu and Debian guides (this fixes https://github.com/scylladb/scylla-docs/issues/4123).
I've modified the index file to be in sync with the updates.
Closes#11285
* github.com:scylladb/scylladb:
doc: reorganize the content to list the recommended way of upgrading the image first
doc: update the image upgrade guide for ScyllaDB image to include the location of the manifest file
doc: fix the upgrade guides for Ubuntu and Debian by removing image-related information
doc: update the guides for Ubuntu and Debian to remove image information and the OS version number
doc: add the upgrade guide for ScyllaDB image from 2021.1 to 2022.1
Having an error while pinging a peer is not a critical error. The code
retires and move on. Lets log the message with less severity since
sometimes those error may happen (for instance during node replace
operation some nodes refuse to answer to pings) and dtest complains that
there are unexpected errors in the logs.
Message-Id: <Ywy5e+8XVwt492Nc@scylladb.com>
on_compaction_completion() is not very descriptive. let's rename
it, following the example of
update_sstable_lists_on_off_strategy_completion().
Also let's coroutinize it, so to remove the restriction of running
it inside a thread only.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Closes#11407
Test teardown involves dropping the test keyspace. If there are stopped servers occasionally we would see timeouts.
Start stopped servers after a test is finished (and passed).
Revert previous commit making teardown async again.
Closes#11412
* github.com:scylladb/scylladb:
test.py: restart stopped servers before teardown...
Revert "test.py: random tables make DDL queries async"
Said command is broken since 4.6, as the type of `reader_concurrency_semaphore::_permit_list` was changed without an accompanying update to this command. This series updates said command and adds it to the list of tested commands so we notice if it breaks in the future.
Closes#11389
* github.com:scylladb/scylladb:
test/scylla-gdb: test scylla read-stats
scylla-gdb.py: read_stats: update w.r.t. post 4.5 code
scylla-gdb.py: improve string_view_printer implementation
This PR change the CentOS 8 support to Rocky, and add 5.1 and 2022.1, 2022.2 rows to the list of Scylla releases
Closes#11383
* github.com:scylladb/scylladb:
OS support page: use CentOS not Centos
OS support page: add 5.1, 2022.1 and 2022.2
OS support page: Update CentOS 8 to Rocky 8
for topology tests
Test teardown involves dropping the test keyspace. If there are stopped
servers occasionally we would see timeouts.
Start stopped servers after a test is finished.
Signed-off-by: Alejo Sanchez <alejo.sanchez@scylladb.com>
Issuing two CREATE TABLE statements with a different name for one of
the partition key columns leads to the following assertion failure on
all replicas:
scylla: schema.cc:363: schema::schema(const schema::raw_schema&, std::optional<raw_view_info>): Assertion `!def.id || def.id == id - column_offset(def.kind)' failed.
The reason is that once the create table mutations are merged, the
columns table contains two entries for the same position in the
partition key tuple.
If the schemas were the same, or not conflicting in a way which leads
to abort, the current behavior would be to drop the older table as if
the last CREATE TABLE was preceded by a DROP TABLE.
The proposed fix is to make CREATE TABLE mutation include a tombstone
for all older schema changes of this table, effectively overriding
them. The behavior will be the same as if the schemas were not
different, older table will be dropped.
Fixes#11396Closes#11398
* github.com:scylladb/scylladb:
db: schema_tables: Make table creation shadow earlier concurrent changes
db: schema_tables: Fix formatting
db: schema_mutations: Make operator<<() print all mutations
schema_mutations: Make it a monoid by defining appropriate += operator
Issuing two CREATE TABLE statements with a different name for one of
the partition key columns leads to the following assertion failure on
all replicas:
scylla: schema.cc:363: schema::schema(const schema::raw_schema&, std::optional<raw_view_info>): Assertion `!def.id || def.id == id - column_offset(def.kind)' failed.
The reason is that once the create table mutations are merged, the
columns table contains two entries for the same position in the
partition key tuple.
If the schemas were the same, or not conflicting in a way which leads
to abort, the current behavior would be to drop the older table as if
the last CREATE TABLE was preceded by a DROP TABLE.
The proposed fix is to make CREATE TABLE mutation include a tombstone
for all older schema changes of this table, effectively overriding
them. The behavior will be the same as if the schemas were not
different, older table will be dropped.
Fixes#11396
Patch 765d2f5e46 did not
evaluate the #if SCYLLA_BUILD_MODE directives properly
and it always matched SCYLLA_BULD_MODE == release.
This change fixes that by defining numerical codes
for each build mode and using macro expansion to match
the define SCYLLA_BUILD_MODE against these codes.
Also, ./configure.py was changes to pass SCYLLA_BUILD_MODE
to all .cc source files, and makes sure it is defined
in build_mode.hh.
Support was added for coverage build mode,
and an #error was added if SCYLLA_BUILD_MODE
was not recognized by the #if ladder directives.
Additional checks verifying the expected SEASTAR_DEBUG
against SCYLLA_BUILD_MODE were added as well,
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Closes#11387
There are async timeouts for ALTER queries. Seems related to othe issues
with the driver and async.
Make these queries synchronous for now.
Signed-off-by: Alejo Sanchez <alejo.sanchez@scylladb.com>
Closes#11394
This commit introduces the following changes to Alternator compability doc:
* As of https://github.com/scylladb/scylladb/pull/11298 Alternator will return ProvisionedThroughput in DescribeTable API calls. We add the fact that tables will default to a BillingMode of PAY_PER_REQUEST (this wasn't made explicit anywhere in the docs), and that the values for RCUs/WCUs are hardcoded to 0.
* Mention the fact that ScyllaDB (thus Alternator) hashing function is different than AWS proprietary implementation for DynamoDB. This is mostly of an implementation aspect rather than a bug, but it may cause user confusion when/if comparing the ResultSet between DynamoDB and Alternator returned from Table Scans.
Refs: https://github.com/scylladb/scylladb/issues/11222
Fixes: https://github.com/scylladb/scylladb/issues/11315Closes#11360
Just like 4a8ed4c, we also need to wait for udev event completion to
create /dev/disk/by-uuid/$UUID for newly formatted disk, to mount the
disk just after formatting.
Fixes#11359
Commitlog imposes a limit on the size of mutations
and throws an exception if it's exceeded. In case of
schema changes before raft this exception was delivered
to the client. Now it happens while saving the raft
command in io_fiber in persistence->store_log_entries
and what the client gets is just a timeout exception,
which doesn't say much about the cause of the problem.
This patch introduces an explicit command size limit
and provides a clear error message in this case.
Closes#11318
* github.com:scylladb/scylladb:
raft, use max_command_size to satisfy commitlog limit
raft, limit for command size
Previous patches made all the callers of topology.update_endpoint()
(via token_metadata.update_topology()) provide correct dc/rack info
for the endpoint. It's now possible to stop using global snitch by
topology and just rely on the dc/rack argument.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
There's a test that's sensitive to correct dc/rack info for testing
entries. To populate them it uses global rack-inferring snitch instance
or a special "testing" snitch. To make it continue working add a helper
that would populate the topology properly (spoiler: next branch will
replace it with explicitly populated topology object).
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
When snitch reconfigures (gossiper-property-file one) it kicks storage
service so that it updates itself. This place also needs to update the
dc/rack info about itself, the correct (new) values are taken from the
snitch itself.
There's a bug here -- system.local table it not update with new data
until restart.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
When a node starts it loads the information about peers from
system.peers table and populates token metadata and topology with this
information. The dc/rack are taken from the sys-ks cache here.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
When a node it started to replace another node it updates token metadata
and topology with the target information eary. The tokens are now taken
from gossiper shadow round, this patch makes the same for dc/rack info.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
When a node is notified about other nodes state change it may want to
update the topology information about it. In all those places the
dc/rack into about the peer is provided by the gossiper.
Basically, these updates mirror the relevant updates of tokens on the
token metadata object.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
