In much of our column_families APIs, we need to pass a pointer to the database.
The only reason we do that, is so we can properly handle the commit log entries
after we seal the current memtables into sstables.
Now that we store a pointer to the commit log in the CF itself at the time it
is created, we no longer have to do it. As a result, the APIs are a lot
cleaner, with no gratuitous parameters.
My motivation for this was the flush method, but as a result, apply() also gets
cleaner.
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
"Some of the system tables will set gc_grace_seconds, or default_time_to_live
but right now we are commenting them out.
The best way to do it would have been to somehow save the fields so we don't need to
set it all the time - as this function is effectively doing.
However, those fields would feel very spurious in the constructor, and it is not
like there isn't a lot of other things for us to set. More importantly, calling
those schema functions is a very rare event. We usually call it once and store
the pointer somewhere.
With that, we're very close to implementing everything that the system tables
needs set, missing only COMPACT STORAGE and information about compaction
strategy."
The function is currently just a simple wrapper over
storage_proxy::query(). The comment has description of details which
are much lower level than this interface.
There are two versions of update_tokens: one for the tokens used by this node,
which goes to the local table, and another for the remote tokens, used by
remote nodes, which goes to the peers table.
The former was implemented, the latter was not. Implement it.
One node: Origin does not issue a flush here, at least in the version of the
code we imported. However, a flush is present in all other variants, and won't
hurt, aside from creating an extra, probably very small, sstable. So I'm
flushing.
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
We ended up with two different implementations of force_blocking_flush,
none of them ideal.
This patch merges both in one that makes more sense, getting rid of the
duplication.
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
To implement that, we will resort to a cache mechanism, instead of doing the
query all the time. This is mainly because we want to avoid overfuturization
of the callers, that are usually just interested in passing simple strings
around.
We will be able to intercept all updates to it, and maintain consistency with our
internal cache. The updates are not done in this patchset.
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
This function is called at bootstrap, to make sure the system tables exist in
the keyspace list. I honestly don't know why do we have to force a delete +
reconstruct. But let's keep consistency with Origin here.
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
"As previously said, there were some unidentified bugs that prevented update_tokens from
working properly. The first one was sent alongside the series, the second one took me more
time, but it is fixed here."
At this point, users of the interface are futurized already, so we
just need to make sure they call the right function.
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
This function is called at startup and makes sure that the cluster_name field
in system.local exists, and if it exists, that it matches the expected value.
To simplifly things, I am leaving the sstable check out. For us, that would be
a map-reduce operation, and if the sstables are indeed corrupted, we would have
caught that already for sure.
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
This patch provide a function to store the current schema version.
Currently, it is called every time the node boots, with a random schema.
That is incorrect and will be fixed shortly. But for now, cqlsh needs
to see a valid value here, so this will do.
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
We won't create our own versions of database and query_proxy, so we need
some setup to be done. The current code will capture those variables and
setup the structure used to conduct the queries. Later on, it will also
carry on the database's basic setup.
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
This is meant to resolve to dependecy loop between token_metadata.hh
and system_keyspace.hh.
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Snitch class semantics defined to be per-Node. To make it so we
introduce here a static member in an i_endpoint_snitch class that
has to contain the pointer to the relevant snitch class instance.
Since the snitch contents are not always pure const it has to be per
shard, therefore we'll make it a "distributed". All the I/O is going
to take place on a single shard and if there are changes - they are going
to be propagated to the rest of the shards.
The application is responsible to initialize this distributed<shnitch>
before it's used for the first time.
This patch effectively reverts most of the "locator: futurize
snitch creation" a2594015f9 patch - the part that modifies the
code that was creating the snitch instance. Since snitch is
created explicitly by the application and all the rest of the code
simply assumes that the above global is initialized we won't need
all those changes any more and the code will get back to be nice and simple
as it was before the patch above.
So, to summarize, this patch does the following:
- Reverts the changes introduced by a2594015f9 related to the fact that
every time a replication strategy was created there should have been created
a snitch that would have been stored in this strategy object. More specifically,
methods like keyspace::create_replication_strategy() do not return a future<>
any more and this allows to simplify the code that calls it significantly.
- Introduce the global distributed<snitch_ptr> object:
- It belongs to the i_endpoint_snitch class.
- There has been added a corresponding interface to access both global and
shard-local instances.
- locator::abstract_replication_strategy::create_replication_strategy() does
not accept snitch_ptr&& - it'll get and pass the corresponding shard-local
instance of the snitch to the replication strategy's constructor by itself.
- Adjusted the existing snitch infrastructure to the new semantics:
- Modified the create_snitch() to create and start all per-shard snitch
instances and update the global variable.
- Introduced a static i_endpoint_snitch::stop_snitch() function that properly
stops the global distributed snitch.
- Added the code to the gossiping_property_file_snitch that distributes the
changed data to all per-shard snitch objects.
- Made all existing snitches classes properly maintain their state in order
to be able to shut down cleanly.
- Patched both urchin and cql_query_test to initialize a snitch instance before
all other services.
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
New in v6:
- Rebased to the current master.
- Extended a commit message a little - the summary.
New in v5:
- database::create_keyspace(): added a missing _keyspaces.emplace()
New in v4:
- Kept the database::create_keyspace() to return future<> by Glauber's request
and added a description to this method that needs to be changed when Glauber
adds his bits that require this interface.
A lot of our tests run in memory only, but now that our write path is complete,
we may start running into problems soon, as we write down the sstables.
It would be nice to force the database to run in-memory only in some situations.
Even in the real world, some scenarios may benefit from that in the future.
This patch forces durable_writes to be always false in case we force the data
directory to be an empty list.
For system tables, the patch also fixes a bug. Because system tables were
forceably initialized with durable_writes = false, we would never write them to
disk, even when we were supposed to.
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Fix keyspace strategy options to preserve key-value ordering by
switching to std::map. We need this to be able to store the map in
database as JSON because unordered maps can cause the schema merging
code to attempt a keyspace update, which we don't support, even though
the values did not change.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
New in v2:
- storage_service: add a non-const version of get_token_metadata().
- get_broadcast_address(): check if net::get_messaging_service().local_is_initialized()
before calling net::get_local_messaging_service().listen_address().
- get_broadcast_address(): return an inet_address by value.
- system_keyspace: introduce db::system_keyspace::endpoint_dc_rack
- fb_utilities: use listen_address as broadcast_address for now
Store a lw_shared_ptr<keyspace_metadata> in struct keyspace so callers
in migration manager, for example, can look it up.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Use the keyspace_metadata type for keyspace creation functions. This is
needed to be able to have a mapping from keyspace name to keyspace
metadata for various call-sites.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
* database now holds all keyspace + column family object
* column families are mapped by uuid, either generated or explicit
* lookup by name tuples or uuid
* finder functions now return refs + throws on missing obj
