Commit Graph

15083 Commits

Author SHA1 Message Date
Duarte Nunes
bd3cedd240 db/view: Process base updates to column unselected by its views
When a view's PK only contains the columns that form the base's PK,
then the liveness of a particular view row is determined not only by
the base row's marker, but also by the selected and, more importantly,
unselected columns. So, process base updates to columns unselected by
any of its views.

Refs #3362

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2018-04-23 09:32:02 +01:00
Duarte Nunes
ac9b93eb89 db/view: Consider partition tombstone when generating updates
Not adding the partition tombstone to the current list of tombstones
may cause updates to be incorrectly generated.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2018-04-23 09:32:02 +01:00
Duarte Nunes
e6467f46b7 tests/view_schema_test: Remove unneeded test
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2018-04-23 09:32:02 +01:00
Duarte Nunes
b0cb5480d5 mutation_fragment: Allow querying if row is live
For clustering_row and static_row, allow querying whether they are
live or not.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2018-04-23 09:32:02 +01:00
Duarte Nunes
164f043768 view_info: Add view_column() overload
For when we already have the base's column_definition.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2018-04-23 09:32:02 +01:00
Duarte Nunes
31370fd7b1 view_info: Explicitly initialize base-dependent fields
Instead of lazily-initializing the regular base column in the view's
PK field, explicitly initialize it. This will be used by future
patches that don't have access to the schema when wanting to obtain
that column.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2018-04-23 09:32:02 +01:00
Duarte Nunes
b77b71436d cql3/alter_table_statement: Forbid dropping columns of MV base tables
When a view's PK only contains the columns that form the base's PK,
then the liveness of a particular view row is determined not only by
the base row's marker, but also by the selected and, more importantly,
unselected columns.

The fact that unselected columns can keep a view row alive also
requires that users cannot drop columns of base tables with
materialized views, which this patch implements.

Refs #3362

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2018-04-23 09:32:02 +01:00
Avi Kivity
28be4ff5da Revert "Merge "Implement loading sstables in 3.x format" from Piotr"
This reverts commit 513479f624, reversing
changes made to 01c36556bf. It breaks
booting.

Fixes #3376.
2018-04-23 06:47:00 +03:00
Avi Kivity
513479f624 Merge "Implement loading sstables in 3.x format" from Piotr
"
Pass sstable version to parse, write and describe_type methods to make it possible to handle different versions.
For now serialization header from 3.x format is ignored.

Tests: units (release)
"

* 'haaawk/sstables3/loading_v3' of ssh://github.com/scylladb/seastar-dev:
  Add test for loading the whole sstable
  Add test for loading statistics
  Add support for 3_x stats metadata
  Pass sstable version to describe_type
  Pass sstable version to write methods
  metadata_type: add Serialization type
  Pass sstable_version_types to parse methods
  Add test for reading filter
  Add test for read_summary
  sstables 3.x: Add test for reading TOC
  sstable: Make component_map version dependent
  sstable::component_type: add operator<<
  Extract sstable::component_type to separete header
  Remove unused sstable::get_shared_components
  sstable_version_types: add mc version
2018-04-22 16:18:39 +03:00
Piotr Jastrzebski
0288121c0a Add test for loading the whole sstable
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2018-04-22 15:07:03 +02:00
Piotr Jastrzebski
fbe9ee72d6 Add test for loading statistics
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2018-04-22 15:07:03 +02:00
Piotr Jastrzebski
b683870644 Add support for 3_x stats metadata
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2018-04-22 15:06:51 +02:00
Takuya ASADA
01c36556bf dist/debian: use --configfile to specify pbuilderrc
Use --configfile to specify pbuilderrc, instead of copying it to home directory.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20180420024624.9661-1-syuu@scylladb.com>
2018-04-22 16:06:42 +03:00
Piotr Jastrzebski
26ab3056ae Pass sstable version to describe_type
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2018-04-22 14:41:11 +02:00
Piotr Jastrzebski
0022c309ee Pass sstable version to write methods
This will allow writing different versions differently

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2018-04-22 14:41:10 +02:00
Piotr Jastrzebski
65fe564cd2 metadata_type: add Serialization type
Ignore it while reading sstable 3_x and throw
if it's present when reading 2_x.

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2018-04-22 14:40:04 +02:00
Piotr Jastrzebski
d68f3b328f Pass sstable_version_types to parse methods
Parsing will depend on the sstable version when
we have support for both 2_x and 3_x formats.

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2018-04-22 13:46:12 +02:00
Piotr Jastrzebski
9b448b9082 Add test for reading filter
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2018-04-22 13:46:12 +02:00
Piotr Jastrzebski
6bb5468ba0 Add test for read_summary
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2018-04-22 13:46:12 +02:00
Piotr Jastrzebski
6c2cf40ce8 sstables 3.x: Add test for reading TOC
Make sure DigestCRC32 is handled correctly.

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2018-04-22 13:46:12 +02:00
Piotr Jastrzebski
00756582ca sstable: Make component_map version dependent
Introduce sstable_version_constants that will be a proxy
serving correct constants depending on the format version.

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2018-04-22 13:46:12 +02:00
Piotr Jastrzebski
94fbec788e sstable::component_type: add operator<<
Make it possible to print out component_type.

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2018-04-22 13:46:12 +02:00
Piotr Jastrzebski
82d483a1d3 Extract sstable::component_type to separete header
It will be used in other places which won't depend on
sstable.

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2018-04-22 13:45:29 +02:00
Avi Kivity
70220d8f85 tests: sstable_datafile_test: peel off redundant parentheses around compression_parameters initializer
The compression_parameter constructor is called with an extra level of
parentheses. Presumably this caused a temporary object to be constructed
and then moved into the argument being initialized, but gcc 8 complains
about ambiguity.

Make it happy by stripping off the redundant parentheses.
Message-Id: <20180421121854.12314-1-avi@scylladb.com>
2018-04-21 13:53:29 +01:00
Avi Kivity
7a141c0240 tests: network_topology_strategy_test: peel off redundant parentheses around token initializer
The token constructor is called with an extra level of parentheses. Presumably
this caused a temporary object to be constructed and then moved into the
variable being initialized, but gcc 8 complains about ambiguity.

Make it happy by stripping off the redundant parentheses.
Message-Id: <20180421121736.12136-1-avi@scylladb.com>
2018-04-21 13:53:29 +01:00
Avi Kivity
7c54e8559c mutation_fragment: fix concept for mutation_fragment::consume()
The parameters to the MutationFragmentConsumer concept must be concrete
types, not decltype(auto).

Reported by gcc 8.
Message-Id: <20180421110738.7574-1-avi@scylladb.com>
2018-04-21 13:53:29 +01:00
Duarte Nunes
6eeb6514f1 Merge 'Introduce "scylla active-sstables" command' from Tomasz
"
Prints info about sstables used by readers

Example:

  (gdb) scylla active-sstables
  sstable "keyspace1"."standard1"#5, readers=3 data_file_size=39393952
  sstable "keyspace1"."standard1"#6, readers=3 data_file_size=127513304
  sstable_count=2, total_index_lists_size=0
"

* 'tgrabiec/gdb-scylla-active-sstables' of github.com:tgrabiec/scylla:
  gdb: Introduce "scylla active-sstables" command
  gdb: Make list_unordered_map() more general
  gdb: Improve compatibility with python2.7
2018-04-19 19:04:59 +01:00
Tomasz Grabiec
fb126abdc5 gdb: Introduce "scylla active-sstables" command
Prints info about sstables used by readers

Example:

  (gdb) scylla active-sstables
  sstable "keyspace1"."standard1"#5, readers=3 data_file_size=39393952
  sstable "keyspace1"."standard1"#6, readers=3 data_file_size=127513304
  sstable_count=2, total_index_lists_size=0
2018-04-19 19:45:52 +02:00
Tomasz Grabiec
68dd61a0e7 gdb: Make list_unordered_map() more general
1) vt.name returns None for some types, use str() instead
 2) some unorderd_maps use 'false' as the second Hash_node template parameter
 3) some consumers will prefer a reference to the value instead of its address
2018-04-19 19:06:00 +02:00
Tomasz Grabiec
309257ddda gdb: Improve compatibility with python2.7
Which is still used in some builds of GDB
2018-04-19 19:04:26 +02:00
Piotr Jastrzebski
0c96573807 Remove unused sstable::get_shared_components
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2018-04-18 10:24:57 +02:00
Piotr Jastrzebski
4f1528192f sstable_version_types: add mc version
This is the latest version of 3.x SSTable format.

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2018-04-18 10:24:57 +02:00
Duarte Nunes
1db6d7d6e2 cql3/functions: Add some missing functions
Fixes #3368

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <20180417170638.12625-1-duarte@scylladb.com>
2018-04-17 21:15:14 +03:00
Duarte Nunes
17917e12ce db/view: Wait for schema agreement in background upon view building
Waiting for schema agreement in the foreground may cause the node to
not boot in useful time.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <20180417125915.11262-1-duarte@scylladb.com>
2018-04-17 18:03:43 +03:00
Duarte Nunes
b5e7d5fa2c column_family: Make reader without going through mutation source
When doing the read before write for a materialized view update, call
make_reader directly.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <20180417091918.10043-1-duarte@scylladb.com>
2018-04-17 12:22:36 +03:00
Takuya ASADA
e99f43ef43 dist/debian: call lsb_release after command existance check
Fixes #3364

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1523907917-13188-1-git-send-email-syuu@scylladb.com>
2018-04-17 10:54:39 +03:00
Avi Kivity
2c2175ab34 Merge "Add support for reading variant integers from SSTables" from Piotr
"
Enhance continuous_data_consumer to use existing vint serialization for reading
variant integers from SSTables.

Also available at:
https://github.com/scylladb/seastar-dev/commits/haaawk/sstables3/unsigned-vint-v6

Tests: units (release)
"

* 'haaawk/sstables3/unsigned-vint-v6' of ssh://github.com/scylladb/seastar-dev:
  sstables: add test for continuous_data_consumer::read_unsigned_vint
  buffer_input_stream: make it possible to specify chunk size
  Add tests for make_limiting_data_source
  Introduce make_limiting_data_source
  sstables: add continuous_data_consumer::read_unsigned_vint
  Cover serialized_size_from_first_byte in tests
  core: add unsigned_vint::serialized_size_from_first_byte
  sstables: add all dependant headers to consumer.hh
  sstables: add all dependant headers to exceptions.hh
  core: add #pragma once to vint-serialization.hh
2018-04-17 10:09:38 +03:00
Takuya ASADA
ace44784e8 dist/debian: use ~root as HOME to place .pbuilderrc
When 'always_set_home' is specified on /etc/sudoers pbuilder won't read
.pbuilderrc from current user home directory, and we don't have a way to change
the behavor from sudo command parameter.

So let's use ~root/.pbuilderrc and switch to HOME=/root when sudo executed,
this can work both environment which does specified always_set_home and doesn't
specified.

Fixes #3366

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1523926024-3937-1-git-send-email-syuu@scylladb.com>
2018-04-17 09:37:16 +03:00
Takuya ASADA
5a71d4f814 dist/debian: use apt-get instead of apt
To suppress following warning, use apt-get instead of apt:
"WARNING: apt does not have a stable CLI interface. Use with caution in scripts."

Fixes #3365

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1523909727-13343-1-git-send-email-syuu@scylladb.com>
2018-04-17 09:29:16 +03:00
Piotr Jastrzebski
c5dda1c0c9 sstables: add test for continuous_data_consumer::read_unsigned_vint
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2018-04-16 21:14:34 +02:00
Piotr Jastrzebski
fdad8eba97 buffer_input_stream: make it possible to specify chunk size
This will allow to force input stream to return its data
in chunks of a specified size.

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2018-04-16 21:11:13 +02:00
Piotr Jastrzebski
4406d11095 Add tests for make_limiting_data_source
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2018-04-16 21:00:35 +02:00
Piotr Jastrzebski
cc6e619aa9 Introduce make_limiting_data_source
This method takes a data_source and returns another data_source
that returns data from the input source but in chunks of limited
size.

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2018-04-16 20:56:30 +02:00
Piotr Jastrzebski
b68d1fa5bd sstables: add continuous_data_consumer::read_unsigned_vint
This allows reading unsigned variant integers from
SSTable format 3.x.

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2018-04-16 20:30:10 +02:00
Piotr Jastrzebski
4431c1bbe7 Cover serialized_size_from_first_byte in tests
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2018-04-16 20:26:44 +02:00
Piotr Jastrzebski
e423529077 core: add unsigned_vint::serialized_size_from_first_byte
This method takes first byte and determins how many bytes
are used to represent an unsigned variant integer.

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2018-04-16 20:12:03 +02:00
Botond Dénes
07fb2e9c4d make_foreign_reader: don't wrap local readers
If the to-be-wrapped reader is local (lives on the same shard where
make_foreign_reader() is called) there is no need to wrap it with
foreign_reader. Just return it as is.

Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Message-Id: <886ed883b707f163603a40b56b8823f2bb6c47c6.1523873224.git.bdenes@scylladb.com>
2018-04-16 15:11:20 +03:00
Piotr Jastrzebski
20705c4536 sstables: add all dependant headers to consumer.hh
Before it was depending on byteorder.hh that just happend
to be included in all compilation units that were using consumer.hh
This change makes the header compile when used in new compilation units.

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2018-04-16 11:02:49 +02:00
Piotr Jastrzebski
9288074d02 sstables: add all dependant headers to exceptions.hh
Before it was depending on print.hh that just happend
to be included in all compilation units that were using
exceptions.hh. This change makes the header compile
 when used in new compilation units.

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2018-04-16 11:02:33 +02:00
Avi Kivity
7c01e66d53 cql3: query_processor: store and use just local shard reference of storage_proxy
Since storage_proxy provides access to the entire cluster, a local shard
reference is sufficient.  Adjust query_processor to store a reference to
just the local shard, rather than a seastar::sharded<storage_proxy> and
adjust callers.

This simplifies the code a little.
Message-Id: <20180415142656.25370-3-avi@scylladb.com>
2018-04-16 10:20:50 +02:00