With global indexes, target column name is always the same as the string
kept in 'options[target]' field. It's not the case for local indexes,
and so a proper extracting function is used to get the value.
When (re)creating a local index, the target string needs to be used
to parse out the actual indexed column:
"(base_pk_part1,base_pk_part2,base_pk_part3),actual_indexed_column".
This column is later used to deterine if an index should be applied
to a SELECT statement.
With local indexes it's not sufficient to check if a single
restriction is supported by an index in order to decide
that in can be used, because local indexes can be leveraged
only when full partition key is properly restricted.
(It also serves as a great example why restrictions code
would greatly benefit from a facelift! :) )
Index restrictions that match a given index were recomputed
during execution stage, which is redundant and prone to errors.
Now, used index restrictions are cached in a prepare statement.
Instead of choosing the first index that we find (in column def order),
the index with highest score is picked. Currently local indexes
score higher than global ones if restrictions allow local indexing
to be applied.
When computing paging state for local indexes, the partition
and clustering keys are different than with global ones:
- partition key is the same as base's
- clustering key starts with the indexed column
It already accepts several arguments that can be extracted from 'this',
and more will be added in the future.
New parameters include lambdas prepared during prepare stage
that define how to extract partition/clustering key ranges depending
on which index is used, so keeping it a static function will result
in unbounded number of parameters with complex types, which will
in turn make the function header almost illegible for a reader.
Hence, read_posting_list becomes a member function with easy access
to any data prepared during prepare stage.
Instead of having just one column definition, index target is now
a variant of either single column definition or a vector of them.
The vector is expected to be used when part of a target definition
is enclosed in parentheses:
$ CREATE INDEX ON t((p),v);
or
$ CREATE INDEX ON t((p1,p2), v);
etc.
This feature will allow providing (possibly composite) base partition key
to CREATE INDEX statement, which will result in creating a local index.
When the index is local, its partition key in underlying materialized
view is the the same as base's, and the indexed column is a first
clustering key. This implementation ensures that view and base rows
will reside on the same partition, while querying the indexed column
will be possible by putting it as a first clustering key part.
Our guidelines dictate that each header is self-sufficient, i.e.
after including it into an empty .cc file, the .cc file can be compiled
without having to include any other header file.
Currently we don't have any tool to check that a header is self
sufficient. This patch aims to remedy that by adding a target to check
each header, as well as a target to check all the headers.
For each header a target is generated that does the equivalent of
including the header into an empty .cc file, then compiling the
resulting .cc file.This targetis called {header_name}.o, so for
given the header `myheader.hh` this will be `build/dev/myheader.hh.o`
(if the dev build-mode is used).
Also a target, `checkheaders` is added which validates all headers in
the project. This currently fails as we have many headers that are not
self-sufficient.
Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Message-Id: <fdf550dc71203417252f1d8144e7a540eec074a1.1552636812.git.bdenes@scylladb.com>
Tomek and I recently had a discussion about whether or not a commitlog
replay would be safe after we dropped or truncated a table that is not
flushed (durable, but auto_snapshots being false).
While we agreed that would be the safe, we both agreed we would feel
better with a unit test covering that.
This patch adds such a test (btw, it passes)
Signed-off-by: Glauber Costa <glauber@scylladb.com>
Message-Id: <20190318223811.6862-1-glauber@scylladb.com>
On some environment dh_strip fails at libreloc/ld.so, so it's better to
skip too just like libprotobuf.so.15.
error message is:
dh_strip -Xlibprotobuf.so.15 --dbg-package=scylla-server-dbg
strip:debian/scylla-server/opt/scylladb/libreloc/ld.so[.gnu.build.attributes]: corrupt GNU build attribute note: bad description size: Bad value
dh_strip: strip --remove-section=.comment --remove-section=.note --strip-unneeded debian/scylla-server/opt/scylladb/libreloc/ld.so returned exit code 1
0
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20190319005153.26506-1-syuu@scylladb.com>
n1, n2, n3 in the cluster,
shutdown n1, n2, n3
start n1, n2
start n3, we saw features are enabled using the system table while n1 and n2 are already up and running in the cluster.
INFO 2019-02-27 09:24:41,023 [shard 0] gossip - Feature check passed. Local node 127.0.0.3 features = {CORRECT_COUNTER_ORDER, CORRECT_NON_COMPOUND_RANGE_TOMBSTONES, COUNTERS, DIGEST_MULTIPARTITION_READ, INDEXES, LARGE_PARTITIONS, LA_SSTABLE_FORMAT, MATERIALIZED_VIEWS, MC_SSTABLE_FORMAT, RANGE_TOMBSTONES, ROLES, ROW_LEVEL_REPAIR, SCHEMA_TABLES_V3, STREAM_WITH_RPC_STREAM, TRUNCATION_TABLE, WRITE_FAILURE_REPLY, XXHASH}, Remote common_features = {CORRECT_COUNTER_ORDER, CORRECT_NON_COMPOUND_RANGE_TOMBSTONES, COUNTERS, DIGEST_MULTIPARTITION_READ, INDEXES, LARGE_PARTITIONS, LA_SSTABLE_FORMAT, MATERIALIZED_VIEWS, MC_SSTABLE_FORMAT, RANGE_TOMBSTONES, ROLES, ROW_LEVEL_REPAIR, SCHEMA_TABLES_V3, STREAM_WITH_RPC_STREAM, TRUNCATION_TABLE, WRITE_FAILURE_REPLY, XXHASH}
INFO 2019-02-27 09:24:41,025 [shard 0] storage_service - Starting up server gossip
INFO 2019-02-27 09:24:41,063 [shard 0] gossip - Node 127.0.0.1 does not contain SUPPORTED_FEATURES in gossip, using features saved in system table, features={CORRECT_COUNTER_ORDER, CORRECT_NON_COMPOUND_RANGE_TOMBSTONES, COUNTERS, DIGEST_MULTIPARTITION_READ, INDEXES, LARGE_PARTITIONS, LA_SSTABLE_FORMAT, MATERIALIZED_VIEWS, MC_SSTABLE_FORMAT, RANGE_TOMBSTONES, ROLES, ROW_LEVEL_REPAIR, SCHEMA_TABLES_V3, STREAM_WITH_RPC_STREAM, TRUNCATION_TABLE, WRITE_FAILURE_REPLY, XXHASH}
INFO 2019-02-27 09:24:41,063 [shard 0] gossip - Node 127.0.0.2 does not contain SUPPORTED_FEATURES in gossip, using features saved in system table, features={CORRECT_COUNTER_ORDER, CORRECT_NON_COMPOUND_RANGE_TOMBSTONES, COUNTERS, DIGEST_MULTIPARTITION_READ, INDEXES, LARGE_PARTITIONS, LA_SSTABLE_FORMAT, MATERIALIZED_VIEWS, MC_SSTABLE_FORMAT, RANGE_TOMBSTONES, ROLES, ROW_LEVEL_REPAIR, SCHEMA_TABLES_V3, STREAM_WITH_RPC_STREAM, TRUNCATION_TABLE, WRITE_FAILURE_REPLY, XXHASH}
The problem is we enable the features too early in the start up process.
We should enable features after gossip is settled.
Fixes#4289
Message-Id: <04f2edb25457806bd9e8450dfdcccc9f466ae832.1551406991.git.asias@scylladb.com>
This brings the version check up-to-date with README.md and HACKING.md,
which were updated by commit fa2b03 ("Replace std::experimental types
with C++17 std version.") to say that minimum GCC 8.1.1 is required.
Tests: manually run configure.py with various `--compiler` values.
Signed-off-by: Dejan Mircevski <dejan@scylladb.com>
Message-Id: <20190318130543.24982-1-dejan@scylladb.com>
Three nodes in the cluster node1, node2, node3
Shutdown the whole cluster
Start node1
Start node2, node2 sees empty remote common_features.
gossip - Feature check passed. Local node 127.0.0.2 features =
{CORRECT_COUNTER_ORDER, CORRECT_NON_COMPOUND_RANGE_TOMBSTONES, COUNTERS,
DIGEST_MULTIPARTITION_READ, INDEXES, LARGE_PARTITIONS,
LA_SSTABLE_FORMAT, MATERIALIZED_VIEWS, MC_SSTABLE_FORMAT,
RANGE_TOMBSTONES, ROLES, ROW_LEVEL_REPAIR, SCHEMA_TABLES_V3,
STREAM_WITH_RPC_STREAM, WRITE_FAILURE_REPLY, XXHASH},
Remote common_features = {}
The problem is node3 hasn't started yet, node1 sees node3 has empty
features. In get_supported_features(), an empty common features will be
returned if an empty features of a node is seen. To fix, we should
fallback to use the features saved in system table.
Start node3, node3 sees empty remote common_features.
gossip - Feature check passed. Local node 127.0.0.3 features =
{CORRECT_COUNTER_ORDER, CORRECT_NON_COMPOUND_RANGE_TOMBSTONES, COUNTERS,
DIGEST_MULTIPARTITION_READ, INDEXES, LARGE_PARTITIONS,
LA_SSTABLE_FORMAT, MATERIALIZED_VIEWS, MC_SSTABLE_FORMAT,
RANGE_TOMBSTONES, ROLES, ROW_LEVEL_REPAIR, SCHEMA_TABLES_V3,
STREAM_WITH_RPC_STREAM, WRITE_FAILURE_REPLY, XXHASH},
Remote common_features = {}
The problem is node3 hasn't inserted its own features into gossip
endpoint_state_map. get_supported_features() returns the common features
of all nodes in endpoint_state_map. To fix, we should fallback to use
the features stored in the system table for such node in this case.
Fixes#4225Fixes#4341
* dev asias/fix_check_knows_remote_features.upstream.v4.1:
gossiper: Remove unused register_feature and unregister_feature
gossiper: Remove unused wait_for_feature_on_all_node and
wait_for_feature_on_node
gossiper: Log feature is enabled only if the feature is not enabled
previously
gossiper: Fix empty remote common_features in
check_knows_remote_features
Three nodes in the cluster node1, node2, node3
Shutdown the whole cluster
Start node1
Start node2, node2 sees empty remote common_features.
gossip - Feature check passed. Local node 127.0.0.2 features =
{CORRECT_COUNTER_ORDER, CORRECT_NON_COMPOUND_RANGE_TOMBSTONES, COUNTERS,
DIGEST_MULTIPARTITION_READ, INDEXES, LARGE_PARTITIONS,
LA_SSTABLE_FORMAT, MATERIALIZED_VIEWS, MC_SSTABLE_FORMAT,
RANGE_TOMBSTONES, ROLES, ROW_LEVEL_REPAIR, SCHEMA_TABLES_V3,
STREAM_WITH_RPC_STREAM, WRITE_FAILURE_REPLY, XXHASH},
Remote common_features = {}
The problem is node3 hasn't started yet, node1 sees node3 has empty
features. In get_supported_features(), an empty common features will be
returned if an empty features of a node is seen. To fix, we should
fallback to use the features saved in system table.
Start node3, node3 sees empty remote common_features.
gossip - Feature check passed. Local node 127.0.0.3 features =
{CORRECT_COUNTER_ORDER, CORRECT_NON_COMPOUND_RANGE_TOMBSTONES, COUNTERS,
DIGEST_MULTIPARTITION_READ, INDEXES, LARGE_PARTITIONS,
LA_SSTABLE_FORMAT, MATERIALIZED_VIEWS, MC_SSTABLE_FORMAT,
RANGE_TOMBSTONES, ROLES, ROW_LEVEL_REPAIR, SCHEMA_TABLES_V3,
STREAM_WITH_RPC_STREAM, WRITE_FAILURE_REPLY, XXHASH},
Remote common_features = {}
The problem is node3 hasn't inserted its own features into gossip
endpoint_state_map. get_supported_features() returns the common features
of all nodes in endpoint_state_map. To fix, we should fallback to use
the features stored in the system table for such node in this case.
Fixes#4225
We saw the log "Feature FOO is enabled" more than once like below. It is
better to log it only when the feature is not enabled previously.
gossip - InetAddress 127.0.0.1 is now UP, status = NORMAL
gossip - Feature CORRECT_COUNTER_ORDER is enabled
gossip - Feature CORRECT_NON_COMPOUND_RANGE_TOMBSTONES is enabled
gossip - Feature COUNTERS is enabled
gossip - Feature DIGEST_MULTIPARTITION_READ is enabled
gossip - Feature INDEXES is enabled
gossip - Feature LARGE_PARTITIONS is enabled
gossip - Feature LA_SSTABLE_FORMAT is enabled
gossip - Feature MATERIALIZED_VIEWS is enabled
gossip - Feature MC_SSTABLE_FORMAT is enabled
gossip - Feature RANGE_TOMBSTONES is enabled
gossip - Feature ROLES is enabled
gossip - Feature ROW_LEVEL_REPAIR is enabled
gossip - Feature SCHEMA_TABLES_V3 is enabled
gossip - Feature STREAM_WITH_RPC_STREAM is enabled
gossip - Feature TRUNCATION_TABLE is enabled
gossip - Feature WRITE_FAILURE_REPLY is enabled
gossip - Feature XXHASH is enabled
gossip - Feature CORRECT_COUNTER_ORDER is enabled
gossip - Feature CORRECT_NON_COMPOUND_RANGE_TOMBSTONES is enabled
gossip - Feature COUNTERS is enabled
gossip - Feature DIGEST_MULTIPARTITION_READ is enabled
gossip - Feature INDEXES is enabled
gossip - Feature LARGE_PARTITIONS is enabled
gossip - Feature LA_SSTABLE_FORMAT is enabled
gossip - Feature MATERIALIZED_VIEWS is enabled
gossip - Feature MC_SSTABLE_FORMAT is enabled
gossip - Feature RANGE_TOMBSTONES is enabled
gossip - Feature ROLES is enabled
gossip - Feature ROW_LEVEL_REPAIR is enabled
gossip - Feature SCHEMA_TABLES_V3 is enabled
gossip - Feature STREAM_WITH_RPC_STREAM is enabled
gossip - Feature TRUNCATION_TABLE is enabled
gossip - Feature WRITE_FAILURE_REPLY is enabled
gossip - Feature XXHASH is enabled
gossip - InetAddress 127.0.0.2 is now UP, status = NORMAL
It works accidentally but it just expanded by bash to use mached files
in current directory, not correctly recognized by tar.
Need to use full file name instead.
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20190312172243.5482-2-syuu@scylladb.com>
We get following link error when running reloc/build_reloc.sh in dbuild,
need to enable DPDK on Seastar:
g++: error: /usr/lib64/librte_cfgfile.so: No such file or directory
g++: error: /usr/lib64/librte_cmdline.so: No such file or directory
g++: error: /usr/lib64/librte_ethdev.so: No such file or directory
g++: error: /usr/lib64/librte_hash.so: No such file or directory
g++: error: /usr/lib64/librte_kvargs.so: No such file or directory
g++: error: /usr/lib64/librte_mbuf.so: No such file or directory
g++: error: /usr/lib64/librte_eal.so: No such file or directory
g++: error: /usr/lib64/librte_mempool.so: No such file or directory
g++: error: /usr/lib64/librte_mempool_ring.so: No such file or directory
g++: error: /usr/lib64/librte_pmd_bnxt.so: No such file or directory
g++: error: /usr/lib64/librte_pmd_e1000.so: No such file or directory
g++: error: /usr/lib64/librte_pmd_ena.so: No such file or directory
g++: error: /usr/lib64/librte_pmd_enic.so: No such file or directory
g++: error: /usr/lib64/librte_pmd_fm10k.so: No such file or directory
g++: error: /usr/lib64/librte_pmd_qede.so: No such file or directory
g++: error: /usr/lib64/librte_pmd_i40e.so: No such file or directory
g++: error: /usr/lib64/librte_pmd_ixgbe.so: No such file or directory
g++: error: /usr/lib64/librte_pmd_nfp.so: No such file or directory
g++: error: /usr/lib64/librte_pmd_ring.so: No such file or directory
g++: error: /usr/lib64/librte_pmd_sfc_efx.so: No such file or directory
g++: error: /usr/lib64/librte_pmd_vmxnet3_uio.so: No such file or directory
g++: error: /usr/lib64/librte_ring.so: No such file or directory
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20190312172243.5482-1-syuu@scylladb.com>
Since we removed dist/common/bin/scyllatop we are getting a build error
on .deb package build (1bb65a0888).
To fix the error we need to create a symlink for /usr/bin/scyllatop.
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20190316162105.28855-1-syuu@scylladb.com>
"
Refuse to accept SSTables that were created with partitioner
different than the one used by the Scylla server.
Fixes#4331
"
* 'haaawk/4331/v4' of github.com:scylladb/seastar-dev:
sstables: Add test for sstable::validate_partitioner
sstables: Add sstable::validate_partitioner and use it
Make sure the exception is thrown when Scylla
tries to load an SSTable created with non-compatible partitioner.
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
Scylla server can't read sstables that were created
with different partitioner than the one being used by Scylla.
We should make sure that Scylla identifies such mismatch
and refuses to use such SSTables.
We can use partitioner information stored in validation metadata
(Statistics.db file) for each SSTable and compare it against
partitioner used by Scylla.
Fixes#4331
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
