Commit Graph

14065 Commits

Author SHA1 Message Date
Tomasz Grabiec
b0a56a91c2 Merge "Remove memtable::make_reader" from Piotr
Migrate all the places that used memtable::make_reader to use
memtable::make_flat_reader and remove memtable::make_reader.

* seastar-dev.git haaawk/remove_memtable_make_reader_v2_rebased:
  Remove memtable::make_reader
  Stop using memtable::make_reader in row_cache_stress_test
  Stop using memtable::make_reader in row_cache_test
  Stop using memtable::make_reader in mutation_test
  Stop using memtable::make_reader in streamed_mutation_test
  Stop using memtable::make_reader in memtable_snapshot_source.hh
  Stop using memtable::make_reader in memtable::apply
  Add consume_partitions(flat_mutation_reader& reader, Consumer consumer)
  Add default parameter values in make_combined_reader
  Migrate test_virtual_dirty_accounting_on_flush to flat reader
  Migrate test_adding_a_column_during_reading_doesnt_affect_read_result
  Simplify flat_reader_assertions& produces(const mutation& m)
  Migrate test_partition_version_consistency_after_lsa_compaction_happens
  flat_mutation_reader: Allow setting buffer capacity
  Add next_mutation() to flat_mutation_reader_assertions
  cf::for_all_partitions::iteration_state: don't store schema_ptr
  read_mutation_from_flat_mutation_reader: don't take schema_ptr
  Migrate test_fast_forward_to_after_memtable_is_flushed to flat reader
2017-12-21 14:02:56 +01:00
Piotr Jastrzebski
17f2eb8ff7 Remove memtable::make_reader
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Piotr Jastrzebski
85d2b24415 Stop using memtable::make_reader in row_cache_stress_test
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Piotr Jastrzebski
129a282cbf Stop using memtable::make_reader in row_cache_test
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Piotr Jastrzebski
dc75df6353 Stop using memtable::make_reader in mutation_test
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Piotr Jastrzebski
660086f2d6 Stop using memtable::make_reader in streamed_mutation_test
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Piotr Jastrzebski
2a9cd5bffe Stop using memtable::make_reader in memtable_snapshot_source.hh
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Piotr Jastrzebski
6bcee5976b Stop using memtable::make_reader in memtable::apply
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Piotr Jastrzebski
a67d6bef29 Add consume_partitions(flat_mutation_reader& reader, Consumer consumer)
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Piotr Jastrzebski
ff718d6573 Add default parameter values in make_combined_reader
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Piotr Jastrzebski
b1676db658 Migrate test_virtual_dirty_accounting_on_flush to flat reader
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Piotr Jastrzebski
b90677272f Migrate test_adding_a_column_during_reading_doesnt_affect_read_result
to flat reader

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Piotr Jastrzebski
20e31e462e Simplify flat_reader_assertions& produces(const mutation& m)
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Piotr Jastrzebski
ddecd385c1 Migrate test_partition_version_consistency_after_lsa_compaction_happens
to flat reader

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Piotr Jastrzebski
5f8fba8a61 flat_mutation_reader: Allow setting buffer capacity
Needed in tests to limit amount of prefetching done by readers, so
that it's easier to test interleaving of various events.

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Piotr Jastrzebski
b18c075470 Add next_mutation() to flat_mutation_reader_assertions
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Piotr Jastrzebski
308ec43ea5 cf::for_all_partitions::iteration_state: don't store schema_ptr
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Piotr Jastrzebski
570703a169 read_mutation_from_flat_mutation_reader: don't take schema_ptr
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Piotr Jastrzebski
681dc26dd1 Migrate test_fast_forward_to_after_memtable_is_flushed to flat reader
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Tomasz Grabiec
71cc63dfa6 Merge "Fixes for multi_range_reader" from Paweł
The following patches contain fixes for skipping to the next parititon
in multi_range_reader and completelty dissable support for fast
forwarding inside a single partition, which is not needed and would only
add unnecessary complexity.

* https://github.com/pdziepak/scylla.git fix-multi_range_reader/v1:
  flat_multi_range_mutation_reader: disallow
    streamed_mutation::forwarding
  flat_multi_range_mutation_reader: clear buffer on next_partition()
  tests/flat_multi_range_mutation_reader: test skipping to next
    partition
2017-12-21 11:06:57 +01:00
George Tavares
ceecd542cd db/view: Consume updated rows regardless of static row
Using Materialized Views, if the base table has static columns,
and the update in base table mutates static and non static rows,
the streamed_mutation is stopped before process non static row.
The patch avoids stopping the stream_mutation and adds a test case.

Message-Id: <20171220173434.25091-1-tavares.george@gmail.com>
2017-12-21 00:49:15 +01:00
Paweł Dziepak
da0655ab3c tests/flat_multi_range_mutation_reader: test skipping to next partition 2017-12-20 16:08:09 +00:00
Paweł Dziepak
5d72acac0c flat_multi_range_mutation_reader: clear buffer on next_partition() 2017-12-20 16:08:09 +00:00
Paweł Dziepak
3cf46a31a6 flat_multi_range_mutation_reader: disallow streamed_mutation::forwarding
Properly implementing streamed_mutation::forwarding::yes in multi range
reader would noticeably increase its complexity and is not needed.
2017-12-20 14:50:11 +00:00
Tomasz Grabiec
dfe48bbbc7 range_tombstone_list: Fix insert_from()
end_bound was not updated in one of the cases in which end and
end_kind was changed, as a result later merging decision using
end_bound were incorrect. end_bound was using the new key, but the old
end_kind.

Fixes #3083.
Message-Id: <1513772083-5257-1-git-send-email-tgrabiec@scylladb.com>
2017-12-20 12:20:20 +00:00
Raphael S. Carvalho
daaadfd515 compaction_manager: remove dead sstable rewrite submission function
this rewrite submission was used by old resharding, but it's no longer
needed, so let's remove it.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <20171219191052.13689-1-raphaelsc@scylladb.com>
2017-12-20 09:29:43 +02:00
Avi Kivity
772d1f47d7 Merge "Fix read amplification in sstable reads" from Paweł
"4b9a34a85425d1279b471b2ff0b0f2462328929c "Merge sstable_data_source
into sstable_mutation_reader" has introduced unintentional changes, some
of them causing excessive read amplification during empty range reads.
The following patches restore the previous behaviour."

* tag 'fix-read-amplification/v1' of https://github.com/pdziepak/scylla:
  sstables: set _read_enabled to false if possible
  sstables: set _single_partition_read for single parititon reads
2017-12-19 18:17:14 +02:00
Tomasz Grabiec
6a6bf58b98 flat_mutation_reader: Fix make_nonforwardable()
It emitted end-of-stream prematurely if buffer was full.
Message-Id: <1513697716-32634-1-git-send-email-tgrabiec@scylladb.com>
2017-12-19 15:56:49 +00:00
Avi Kivity
2137d753b3 Merge "Serialize compaction of same size tier for different cfs" from Raphael
"Currently, compaction manager will serialize compaction of same size tier
(or weight) if they belong to the same column family. However, it fails to
do so if the compaction jobs belong to different column families.
That can lead to an ungodly amount of running compaction which gets worse
the higher the number of shards and active column families. The problem
is that it may affect overall system performance due to excessive resource
usage. It's easy to trigger it during bootstraping after loading node with
new sstables or repairing, or if lots of cfs are being actively written."

Fixes #1295.

* 'similar_sized_compaction_serialization_v4' of github.com:raphaelsc/scylla:
  sstables: remove column_family from compaction_weight_registration
  compaction_manager: serialize compaction of same size tier for different cfs
  sstables: introduces deregister() and weight() to compaction_weight_registration
  sstables: move compaction_weight_registration to its own header
  sstables: improve compact_sstables() interface
2017-12-19 16:32:27 +02:00
Tomasz Grabiec
7b36c8423c row_cache: Fix single_partition_populating_reader not waiting on create_underlying() to resolve
Results in undefined behavior.
Message-Id: <1513691679-27081-1-git-send-email-tgrabiec@scylladb.com>
2017-12-19 16:12:11 +02:00
Paweł Dziepak
574c6006f6 sstables: set _read_enabled to false if possible 2017-12-19 13:59:13 +00:00
Paweł Dziepak
1beb3552fc sstables: set _single_partition_read for single parititon reads 2017-12-19 13:59:13 +00:00
Piotr Jastrzebski
570fc5afed Use row_cache::make_flat_reader in column_family::make_reader
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
Message-Id: <ba1659ceed8676f45942ce6e7506158026947345.1513687259.git.piotr@scylladb.com>
2017-12-19 14:42:32 +02:00
Raphael S. Carvalho
928beae242 Fix compilation of db/hints/manager.cc and row_cache.cc
compiler: gcc (GCC) 6.3.1 20161221 (Red Hat 6.3.1-1)

Problems introduced in f6a461c7a4
and 37b19ae6ba, respectively.

They both fail to compile due to use of method in lambda without
explicit mention of this. Some of failure is fixed by not using
auto in lambda parameter.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <20171218222144.12297-1-raphaelsc@scylladb.com>
2017-12-19 11:15:45 +01:00
Avi Kivity
d97ea6b0f4 Merge seastar upstream
* seastar 2b23547...adaca37 (7):
  > Merge "Support for skipping over bytes from input stream in input_stream::consume" from Vladimir
  > build: enforce Boost >= 1.58 during configuration.
  > Tutorial: beginning of documentation of CPU scheduling et al.
  > circular_buffer: make move-constructor noexcept
  > circular_buffer: convert existing documentation to doxygen format
  > build: fix detection of membarrier syscall support
  > Merge "Improve systemwide_memory_barrier() on newer Linuces" from Avi
2017-12-19 11:21:35 +02:00
Avi Kivity
8dbc6bbcdc Update scylla-ami submodule
* dist/ami/files/scylla-ami be90a3f...3366c93 (1):
  > scylla_install_ami: skip ec2_check while building AMI
2017-12-19 10:10:22 +02:00
Takuya ASADA
77fbdd487c dist/ami: Switch to official CentOS base image
We had switched our own CentOS base image since we couldn't make built AMI to
public due to base image settings, it's probably because the image provided via
AWS market place.
However, I've found an official image outside of market place, and I succeeded
making built AMI to public based on the image.
URL: https://wiki.centos.org/Cloud/AWS

Once we could able to use official image, we probably should use official one.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1513659164-28029-1-git-send-email-syuu@scylladb.com>
2017-12-19 10:07:14 +02:00
Tomasz Grabiec
37b19ae6ba Merge "Migrate cache to use flat_mutation_reader" from Piotr 2017-12-18 17:53:20 +01:00
Piotr Jastrzebski
d756c49baf Rename cache_streamed_mutation_test to cache_flat_mutation_reader_test
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-18 16:37:57 +01:00
Piotr Jastrzebski
14d98aaa0b Rename row_cache::create_underlying_flat_reader to
create_underlying_reader

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-18 16:37:57 +01:00
Piotr Jastrzebski
49993e56a9 Remove unused row_cache::create_underlying_reader
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-18 16:37:57 +01:00
Piotr Jastrzebski
b976872c1a Rename all *_underlying_flat methods in read_context to *_underlying.
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-18 16:37:57 +01:00
Piotr Jastrzebski
1457a3d771 Rename cache_entry::*read_flat to cache_entry::*read
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-18 16:37:57 +01:00
Piotr Jastrzebski
8b796a884f Rename read_context::enter_flat_partition to enter_partition
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-18 16:37:57 +01:00
Piotr Jastrzebski
8d37b71843 Rename autoupdating_underlying_flat_reader to autoupdating_underlying_reader
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-18 16:37:57 +01:00
Piotr Jastrzebski
9789c37e9d Remove autoupdating_underlying_reader
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-18 16:37:56 +01:00
Piotr Jastrzebski
893e434207 Stop using autoupdating_underlying_reader in read_context
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-18 16:37:56 +01:00
Piotr Jastrzebski
6e9b54cc77 Remove unused cache_streamed_mutation
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-18 16:37:56 +01:00
Piotr Jastrzebski
df17bad13b Remove unused cache_entry::read and do_read
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-18 16:37:56 +01:00
Piotr Jastrzebski
003670c3cd Remove unused read_directly_from_underlying overload
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-18 16:37:56 +01:00