Some tests (eg murmur_hash_test) need only byte manipulation
functions. By specifying dependencies precisely we can drastically
reduce recompilation times, which speeds up development cycle.
I managed to reduce recompilation time for murmur_hash_test from 5
minutes to 4 seconds by breaking dependency on whole urchin object
set.
Use commit log in database, from Calle:
"Initial" usage of the commitlog in database mutation path.
A commitlog is created in "work" dirs when initing the db
from a datadir. However, since we have neither disk data storage,
nor replay capability yet (and no real db config), the settings
are basically to just write in-memory serialization, write them to
disk and then discard them. So in fact, pointless. But at least using
the log...
* A commitlog is created in "work" dirs when initing the db
from a datadir. However, since we have neither disk data storage,
nor replay capability yet (and no real db config), the settings
are basically to just write in-memory serialization, write them to
disk and then discard them. So in fact, pointless. But at least using
the log...
* Moved the actual "apply" of mutation into database. If a commitlog
is active, add an entry to it before applying mutation.
Partitions should be ordered using Origin's ordering, which is first
by token, then by Origin's representation of the key. That is the
natural ordering of decorated_key.
This also changes mutation class to hold decorated_key, to avoid
decoration overhead at different layers.
This gives about 30% increase in tps in:
build/release/tests/perf/perf_simple_query -c1 --query-single-key
This patch switches query result format from a structured one to a
serialized one. The problems with structured format are:
- high level of indirection (vector of vectors of vectors of blobs), which
is not CPU cache friendly
- high allocation rate due to fine-grained object structure
On replica side, the query results are probably going to be serialized
in the transport layer anyway, so this change only subtracts
work. There is no processing of the query results on replica other
than concatenation in case of range queries. If query results are
collected in serialized form from different cores, we can concatenate
them without copying by simply appending the fragments into the
packet. This optimization is not implemented yet.
On coordinator side, the query results would have to be parsed from
the transport layer buffers anyway, so this also doesn't add work, but
again saves allocations and copying. The CQL server doesn't need
complex data structures to process the results, it just goes over it
linearly consuming it. This patch provides views, iterators and
visitors for consuming query results in serialized form. Currently the
iterators assume that the buffer is contiguous but we could easily
relax this in future so that we can avoid linearization of data
received from seastar sockets.
The coordinator side could be optimized even further for CQL queries
which do not need processing (eg. select * from cf where ...) we
could make the replica send the query results in the format which is
expected by the CQL binary protocol client. So in the typical case the
coordinator would just pass the data using zero-copy to the client,
prepending a header.
We do need structure for prefetched rows (needed by list
manipulations), and this change adds query result post-processing
which converts serialized query result into a structured one, tailored
particularly for prefetched rows needs.
This change also introduces partition_slice options. In some queries
(maybe even in typical ones), we don't need to send partition or
clustering keys back to the client, because they are already specified
in the query request, and not queried for. The query results hold now
keys as optional elements. Also, meta-data like cell timestamp and
ttl is now also optional. It is only needed if the query has
writetime() or ttl() functions in it, which it typically won't have.
To be more precise, do not take schema_ptr by value.
Fixes crashes in running smp > 1 where mutations applied across shards
(i.e. foreign memory) would cause schema_ptr:s to get out of sync (using
other shards ptr)
bytes and sstring are distinct types, since their internal buffers are of
different length, but bytes_view is an alias of sstring_view, which makes
it possible of objects of different types to leak across the abstraction
boundary.
Fix this by making bytes a basic_sstring<int8_t, ...> instead of using char.
int8_t is a 'signed char', which is a distinct type from char, so now
bytes_view is a distinct type from sstring_view.
uint8_t would have been an even better choice, but that diverges from Origin
and would have required an audit.
This patch converts (for very small value of 'converts') some
replication related classes. Only static topology is supported (it is
created in keyspace::create_replication_strategy()). During mutation
no replication is done, since messaging service is not ready yet,
only endpoints are calculated.
From Avi:
This patchsets completes the conversion of scalar functions (TOKEN is still
missing, and maybe others, but the infrastructure is there).
Conflicts:
database.cc
* database now holds all keyspace + column family object
* column families are mapped by uuid, either generated or explicit
* lookup by name tuples or uuid
* finder functions now return refs + throws on missing obj
In CQL a row is considered as present if its row marker is live or it
has any cells live. The 'insert' statement creates a row
marker. Internally Origin handles that by inserting a special cell
whose name shares the prefix with other cells in that row.
One consequence of this way of things is that when we query a column
slice from sstables we will have to read the whole CQL row, even if
not all columns are queried. We won't have to include the data, but we
will need liveness information in order to commute it with other
mutations, so that we can finally determine if the row is live or not.
From Pekka:
This series adds support for creating keyspaces. We already have the CQL
front-end implemented so all that remains is converging mutations in
legacy_schema_tables.cc as well as parts of migration_manager.hh and
wiring that up to the CQL execution path.
System keyspace is used for things like keyspace and table metadata.
Initialize it in database constructor so that they're always available.
Needed for CQL create keyspace test case, for example.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
