Just like test/alternator, make redis-test runnable from test.py.
For this we move the redis tests into a subdirectory of tests/,
and create a script to run them: tests/redis/run.
These tests currently fail, so we did not yet modify test.py to actually
run them automatically.
Fixes#6331
"
There's last call for global storage service left in compaction code, it
comes from cleanup_compaction to get local token ranges for filtering.
The call in question is a pure wrapper over database, so this set just
makes use of the database where it's already available (perform_cleanup)
and adds it where it's needed (perform_sstable_upgrade).
tests: unit(dev), nodetool upgradesstables
"
* 'br-remove-ss-from-compaction-3' of https://github.com/xemul/scylla:
storage_service: Remove get_local_ranges helper
compaction: Use database from options to get local ranges
compaction: Keep database reference on upgrade options
compaction: Keep database reference on cleanup options
db: Factor out get_local_ranges helper
This patch causes orphaned hints (hints that were written towards a node
that is no longer their replica) to be sent with a default write
timeout. This is what is currently done for non-orphaned hints.
Previously, the timeout was hardcoded to one hour. This could cause a
long delay while shutting down, as hints manager waits until all ongoing
hint sending operation finish before stopping itself.
Fixes: #7051
This reader was probably created in ancient times, when readers didn't
yet have a _schema member of their own. But now that they do, it is not
necessary to store the schema in the reader implementation, there is one
available in the parent class.
While at it also move the schema into the class when calling the
constructor.
Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Reviewed-by: Benny Halevy <bhalevy@scylladb.com>
Message-Id: <20200821070358.33937-1-bdenes@scylladb.com>
"
We keep refrences to locator::token_metadata in many places.
Most of them are for read-only access and only a few want
to modify the token_metadata.
Recently, in 94995acedb,
we added yielding loops that access token_metadata in order
to avoid cpu stalls. To make that possible we need to make
sure they token_metadata object they are traversing won't change
mid-loop.
This series is a first step in ensuring the serialization of
updates to shared token metadata to reading it.
Test: unit(dev)
Dtest: bootstrap_test:TestBootstrap.start_stop_test{,_node}, update_cluster_layout_tests.py -a next-gating(dev)
"
* tag 'constify-token-metadata-access-v2' of github.com:bhalevy/scylla:
api/http_context: keep a const sharded<locator::token_metadata>&
gossiper: keep a const token_metadata&
storage_service: separate get_mutable_token_metadata
range_streamer: keep a const token_metadata&
storage_proxy: delete unused get_restricted_ranges declaration
storage_proxy: keep a const token_metadata&
storage_proxy: get rid of mutable get_token_metadata getter
database: keep const token_metadata&
database: keyspace_metadata: pass const locator::token_metadata& around
everywhere_replication_strategy: move methods out of line
replication_strategy: keep a const token_metadata&
abstract_replication_strategy: get_ranges: accept const token_metadata&
token_metadata: rename calculate_pending_ranges to update_pending_ranges
token_metadata: mark const methods
token_ranges: pending_endpoints_for: return empty vector if keyspace not found
token_ranges: get_pending_ranges: return empty vector if keyspace not found
token_ranges: get rid of unused get_pending_ranges variant
replication_strategy: calculate_natural_endpoints: make token_metadata& param const
token_metadata: add get_datacenter_racks() const variant
The cleanup compaction wants to keep local tokens on-board and gets
them from storage_service.get_local_ranges().
This method is the wrapper around database.get_keyspace_local_ranges()
created in previous patch, the live database reference is already
available on the descriptor's options, so we can short-cut the call.
This allows removing the last explicit call for global storage_service
instance from compaction code.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
The only place that creates them is the API upgrade_sstables call.
The created options object doesn't over-survive the returned
future, so it's safe to keep this reference there.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
The database is available at both places that create the options --
tests and API perform_cleanup call.
Options object doesn't over-survive the returned future, so it's
safe to keep the reference on it.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Storage service and repair code have identical helpers to get local
ranges for keyspace. Move this helper's code onto database, later it
will be reused by one more place.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Use a different getter for a token_metadata& that
may be changed so we can better synchronize readers
and writers of token_metadata and eventually allow
them to yield in asynchronous loops.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
We'd like to strictly control who can modify token metadata
and nobody currently needs a mutable reference to storage_proxy::_token_metadata.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
No need to modify token_metadata form database code.
Also, get rid of mutable get_token_metadata variant.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Move methods depending on token_metadata to source file
so we can avoid including token_metadata.hh in header files
where spossible.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Many token_metadata methods do not modify the object
and can be marked as const.
The motivation is to better control who may modify
token_metadata.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
When user defines a build mode with configure.py, the build, check, and
test targets fail as follows:
./configure.py --mode=dev && ninja build
ninja: error: 'debug-build', needed by 'build', missing and no known rule to make it
Fix the issue by making the targets depend on build targets for
specified build modes, not all available modes.
Message-Id: <20200813105639.1641090-1-penberg@scylladb.com>
We already have a test, test_streams.py::test_streams_updateitem_old_image,
for issue #6935: It tests that the OLD_IMAGE in Alternator Streams should
contain the item's key.
However this test was missing one corner case, which is the first solution
for this issue did incorrectly. So in this patch we add a test for this
corner case, test_streams_updateitem_old_image_empty_item:
This corner case about the item existing, but *empty*, i.e., having just
the key but no other attribute. In this case, OLD_IMAGE should return that
empty item - including its key. Not nothing.
As usual, this test passes on DynamoDB and xfails on Alternator, and the
"xfail" mark will be removed when issue #6935 is fixed.
Refs #6935.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20200819155229.34475-1-nyh@scylladb.com>
When binding prepared statement it is possible that values being binded
are not correct. Unfortunately before this patch, the error message
was only saying what type got a wrong value. This was not very helpful
because there could be multiple columns with the same type in the table.
We also support collections so sometimes error was saying that there
is a wrong value for a type T but the affected column was actually of
type collection<T>.
This patch adds information about a column name that got the wrong
value so that it's easier to find and fix the problem.
Tests: unit(dev)
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
Message-Id: <90b70a7e5144d7cb3e53876271187e43fd8eee25.1597832048.git.piotr@scylladb.com>
"
The messaging service is (as many other services) present in
the global namespace and is widely accessed from where needed
with global get(_local)?_messaging_service() calls. There's a
long-term task to get rid of this globality and make services
and componenets reference each-other and, for and due-to this,
start and stop in specific order. This set makes this for the
messaging service.
The service is very low level and doesn't depend on anything.
It's used by gossiper, streaming, repair, migration manager,
storage proxy, storage service and API. According to this
dependencies the set consists of several parts:
patches 1-9 are preparatory, they encapsulate messaging service
init/fini stuff in its own module and decouple it from the
db::config
patch 10-12 introduce local service reference in main and set
its init/fini calls at the early stage so that this reference
can later be passed to those depending on it
patches 13-42 replace global referencing of messaging service
from other subsystems with local references initialized from
main.
patch 43 finalizes tests.
patch 44 wraps things up with removing global messaiging service
instance along with get(_local)?_messaging_service calls.
The service's stopping part is deliberately left incomplete (as
it is now), the sharded service remains alive, only the instance's
stop() method is called (and is empty for a while). Since the
messaging service's users still do not stop cleanly, its instances
should better continue leaking on exit.
Once (if) the seastar gets the helper rpc::has_handlers() method
merged the messaging_service::stop() will be able to check if all
the verbs had been unregistered (spoiler: not yet, more fixes to
come).
For debugging purposes the pointer on now-local messaging service
instance is kept in service::debug namespace.
tests: unit(dev)
dtest(dev: simple_boot_shutdown, repair, update_cluster_layout)
manual start-stop
"
* 'br-unglobal-messaging-service-2' of https://github.com/xemul/scylla: (44 commits)
messaging_service: Unglobal messaging service instance
tests: Use own instances of messaging_service
storage_service: Use local messaging reference
storage_service: Keep reference on sharded messaging service
migration_manager: Add messaging service as argument to get_schema_definition
migration_manager: Use local messaging reference in simple cases
migration_manager: Keep reference on messaging
migration_manager: Make push_schema_mutation private non-static method
migration_manager: Move get_schema_version verb handling from proxy
repair: Stop using global messaging_service references
repair: Keep sharded messaging service reference on repair_meta
repair: Keep sharded messaging service reference on repair_info
repair: Keep reference on messaging in row-level code
repair: Keep sharded messaging service in API
repair: Unset API endpoints on stop
repair: Setup API endpoints in separate helper
repair: Push the sharded<messaging_service> reference down to sync_data_using_repair
repair: Use existing sharded db reference
repair: Mark repair.cc local functions as static
streaming: Keep messaging service on send_info
...
Remove the global messaging_service, keep it on the main stack.
But also store a pointer on it in debug namespace for debugging.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
The global one is going away, no core code uses it, so all tests
can be safely switched to use their own instances.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
All the places the are (and had become such with previous patches) using
the global messaging service and the storage service methods, so they
can access the local reference on the messaging service.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
There are 4 places that call this helper:
- storage proxy. Callers are rpc verb handlers and already have the proxy
at hands from which they can get the messaging service instance
- repair. There's local-global messaging instance at hands, and the caller
is in verb handler too
- streaming. The caller is verb handler, which is unregistered on stop, so
the messaging service instance can be captured
- migration manager itself. The caller already uses "this", so the messaging
service instance can be get from it
The better approach would be to make get_schema_definition be the method of
migration_manager, but the manager is stopped for real on shutdown, thus
referencing it from the callers might not be safe and needs revisiting. At
the same time the messaging service is always alive, so using its reference
is safe.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Most of those places are either non-static migration_manager methods.
Plus one place where the local service instance is already at hands.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
The local migration manager instance is already available at caller, so
we can call a method on it. This is to facilitate next patching.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
The user of this verb is migration manager, so the handler must be it as well.
The hander code now explicitly gets global proxy. This call is safe, as proxy
is not stopped nowadays. In the future we'll need to revisit the relation
between migration - proxy - stats anyway.
The use of local migration manager is safe, as it happens in verb handler which
is unregistered and is waited to be completed on migration manager stop.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Now all the users of messaging service have the needed reference.
Again, the messaging service is not really stopped at the end, so its usage
is safe regardless of whether repair stuff itself leaks on stop or not.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
