Commit Graph

14076 Commits

Author SHA1 Message Date
Piotr Jastrzebski
b3b6db4f50 Remove unused make_combined_reader overload.
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 17:00:43 +01:00
Piotr Jastrzebski
202c562f68 Migrate test_combining_two_partially_overlapping_readers to flat reader
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 17:00:43 +01:00
Piotr Jastrzebski
6c62454076 Migrate test_combining_two_non_overlapping_readers to flat reader
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 17:00:43 +01:00
Piotr Jastrzebski
bef2cf8ed9 Migrate combined_mutation_reader_test to flat reader
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 17:00:43 +01:00
Piotr Jastrzebski
19d4bce624 Migrate test_sm_fast_forwarding_combining_reader to flat reader
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 17:00:43 +01:00
Piotr Jastrzebski
17e6f6b089 Migrate test_combining_one_empty_reader to flat reader
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 17:00:43 +01:00
Piotr Jastrzebski
1f77370d9e Migrate test_combining_two_empty_readers to flat reader
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 17:00:43 +01:00
Piotr Jastrzebski
a702d0ec3f Migrate test_combining_two_readers_with_one_reader_empty to flat reader
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 17:00:43 +01:00
Piotr Jastrzebski
9a5d6bd8af Migrate test_combining_one_reader_with_many_partitions to flat reader
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 17:00:43 +01:00
Piotr Jastrzebski
13551e6f50 Migrate test_combining_two_readers_with_the_same_row to flat reader
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 17:00:43 +01:00
Piotr Jastrzebski
b1c1709127 Migrate make_combined_mutation_source to flat reader
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 17:00:42 +01:00
Piotr Jastrzebski
024e01ad9e mutation_source: Add constructors for sources that ignore forwarding
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 16:59:57 +01:00
Piotr Jastrzebski
17f2eb8ff7 Remove memtable::make_reader
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Piotr Jastrzebski
85d2b24415 Stop using memtable::make_reader in row_cache_stress_test
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Piotr Jastrzebski
129a282cbf Stop using memtable::make_reader in row_cache_test
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Piotr Jastrzebski
dc75df6353 Stop using memtable::make_reader in mutation_test
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Piotr Jastrzebski
660086f2d6 Stop using memtable::make_reader in streamed_mutation_test
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Piotr Jastrzebski
2a9cd5bffe Stop using memtable::make_reader in memtable_snapshot_source.hh
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Piotr Jastrzebski
6bcee5976b Stop using memtable::make_reader in memtable::apply
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Piotr Jastrzebski
a67d6bef29 Add consume_partitions(flat_mutation_reader& reader, Consumer consumer)
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Piotr Jastrzebski
ff718d6573 Add default parameter values in make_combined_reader
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Piotr Jastrzebski
b1676db658 Migrate test_virtual_dirty_accounting_on_flush to flat reader
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Piotr Jastrzebski
b90677272f Migrate test_adding_a_column_during_reading_doesnt_affect_read_result
to flat reader

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Piotr Jastrzebski
20e31e462e Simplify flat_reader_assertions& produces(const mutation& m)
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Piotr Jastrzebski
ddecd385c1 Migrate test_partition_version_consistency_after_lsa_compaction_happens
to flat reader

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Piotr Jastrzebski
5f8fba8a61 flat_mutation_reader: Allow setting buffer capacity
Needed in tests to limit amount of prefetching done by readers, so
that it's easier to test interleaving of various events.

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Piotr Jastrzebski
b18c075470 Add next_mutation() to flat_mutation_reader_assertions
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Piotr Jastrzebski
308ec43ea5 cf::for_all_partitions::iteration_state: don't store schema_ptr
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Piotr Jastrzebski
570703a169 read_mutation_from_flat_mutation_reader: don't take schema_ptr
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Piotr Jastrzebski
681dc26dd1 Migrate test_fast_forward_to_after_memtable_is_flushed to flat reader
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-21 11:47:07 +01:00
Tomasz Grabiec
71cc63dfa6 Merge "Fixes for multi_range_reader" from Paweł
The following patches contain fixes for skipping to the next parititon
in multi_range_reader and completelty dissable support for fast
forwarding inside a single partition, which is not needed and would only
add unnecessary complexity.

* https://github.com/pdziepak/scylla.git fix-multi_range_reader/v1:
  flat_multi_range_mutation_reader: disallow
    streamed_mutation::forwarding
  flat_multi_range_mutation_reader: clear buffer on next_partition()
  tests/flat_multi_range_mutation_reader: test skipping to next
    partition
2017-12-21 11:06:57 +01:00
George Tavares
ceecd542cd db/view: Consume updated rows regardless of static row
Using Materialized Views, if the base table has static columns,
and the update in base table mutates static and non static rows,
the streamed_mutation is stopped before process non static row.
The patch avoids stopping the stream_mutation and adds a test case.

Message-Id: <20171220173434.25091-1-tavares.george@gmail.com>
2017-12-21 00:49:15 +01:00
Paweł Dziepak
da0655ab3c tests/flat_multi_range_mutation_reader: test skipping to next partition 2017-12-20 16:08:09 +00:00
Paweł Dziepak
5d72acac0c flat_multi_range_mutation_reader: clear buffer on next_partition() 2017-12-20 16:08:09 +00:00
Paweł Dziepak
3cf46a31a6 flat_multi_range_mutation_reader: disallow streamed_mutation::forwarding
Properly implementing streamed_mutation::forwarding::yes in multi range
reader would noticeably increase its complexity and is not needed.
2017-12-20 14:50:11 +00:00
Tomasz Grabiec
dfe48bbbc7 range_tombstone_list: Fix insert_from()
end_bound was not updated in one of the cases in which end and
end_kind was changed, as a result later merging decision using
end_bound were incorrect. end_bound was using the new key, but the old
end_kind.

Fixes #3083.
Message-Id: <1513772083-5257-1-git-send-email-tgrabiec@scylladb.com>
2017-12-20 12:20:20 +00:00
Raphael S. Carvalho
daaadfd515 compaction_manager: remove dead sstable rewrite submission function
this rewrite submission was used by old resharding, but it's no longer
needed, so let's remove it.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <20171219191052.13689-1-raphaelsc@scylladb.com>
2017-12-20 09:29:43 +02:00
Avi Kivity
772d1f47d7 Merge "Fix read amplification in sstable reads" from Paweł
"4b9a34a85425d1279b471b2ff0b0f2462328929c "Merge sstable_data_source
into sstable_mutation_reader" has introduced unintentional changes, some
of them causing excessive read amplification during empty range reads.
The following patches restore the previous behaviour."

* tag 'fix-read-amplification/v1' of https://github.com/pdziepak/scylla:
  sstables: set _read_enabled to false if possible
  sstables: set _single_partition_read for single parititon reads
2017-12-19 18:17:14 +02:00
Tomasz Grabiec
6a6bf58b98 flat_mutation_reader: Fix make_nonforwardable()
It emitted end-of-stream prematurely if buffer was full.
Message-Id: <1513697716-32634-1-git-send-email-tgrabiec@scylladb.com>
2017-12-19 15:56:49 +00:00
Avi Kivity
2137d753b3 Merge "Serialize compaction of same size tier for different cfs" from Raphael
"Currently, compaction manager will serialize compaction of same size tier
(or weight) if they belong to the same column family. However, it fails to
do so if the compaction jobs belong to different column families.
That can lead to an ungodly amount of running compaction which gets worse
the higher the number of shards and active column families. The problem
is that it may affect overall system performance due to excessive resource
usage. It's easy to trigger it during bootstraping after loading node with
new sstables or repairing, or if lots of cfs are being actively written."

Fixes #1295.

* 'similar_sized_compaction_serialization_v4' of github.com:raphaelsc/scylla:
  sstables: remove column_family from compaction_weight_registration
  compaction_manager: serialize compaction of same size tier for different cfs
  sstables: introduces deregister() and weight() to compaction_weight_registration
  sstables: move compaction_weight_registration to its own header
  sstables: improve compact_sstables() interface
2017-12-19 16:32:27 +02:00
Tomasz Grabiec
7b36c8423c row_cache: Fix single_partition_populating_reader not waiting on create_underlying() to resolve
Results in undefined behavior.
Message-Id: <1513691679-27081-1-git-send-email-tgrabiec@scylladb.com>
2017-12-19 16:12:11 +02:00
Paweł Dziepak
574c6006f6 sstables: set _read_enabled to false if possible 2017-12-19 13:59:13 +00:00
Paweł Dziepak
1beb3552fc sstables: set _single_partition_read for single parititon reads 2017-12-19 13:59:13 +00:00
Piotr Jastrzebski
570fc5afed Use row_cache::make_flat_reader in column_family::make_reader
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
Message-Id: <ba1659ceed8676f45942ce6e7506158026947345.1513687259.git.piotr@scylladb.com>
2017-12-19 14:42:32 +02:00
Raphael S. Carvalho
928beae242 Fix compilation of db/hints/manager.cc and row_cache.cc
compiler: gcc (GCC) 6.3.1 20161221 (Red Hat 6.3.1-1)

Problems introduced in f6a461c7a4
and 37b19ae6ba, respectively.

They both fail to compile due to use of method in lambda without
explicit mention of this. Some of failure is fixed by not using
auto in lambda parameter.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <20171218222144.12297-1-raphaelsc@scylladb.com>
2017-12-19 11:15:45 +01:00
Avi Kivity
d97ea6b0f4 Merge seastar upstream
* seastar 2b23547...adaca37 (7):
  > Merge "Support for skipping over bytes from input stream in input_stream::consume" from Vladimir
  > build: enforce Boost >= 1.58 during configuration.
  > Tutorial: beginning of documentation of CPU scheduling et al.
  > circular_buffer: make move-constructor noexcept
  > circular_buffer: convert existing documentation to doxygen format
  > build: fix detection of membarrier syscall support
  > Merge "Improve systemwide_memory_barrier() on newer Linuces" from Avi
2017-12-19 11:21:35 +02:00
Avi Kivity
8dbc6bbcdc Update scylla-ami submodule
* dist/ami/files/scylla-ami be90a3f...3366c93 (1):
  > scylla_install_ami: skip ec2_check while building AMI
2017-12-19 10:10:22 +02:00
Takuya ASADA
77fbdd487c dist/ami: Switch to official CentOS base image
We had switched our own CentOS base image since we couldn't make built AMI to
public due to base image settings, it's probably because the image provided via
AWS market place.
However, I've found an official image outside of market place, and I succeeded
making built AMI to public based on the image.
URL: https://wiki.centos.org/Cloud/AWS

Once we could able to use official image, we probably should use official one.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1513659164-28029-1-git-send-email-syuu@scylladb.com>
2017-12-19 10:07:14 +02:00
Tomasz Grabiec
37b19ae6ba Merge "Migrate cache to use flat_mutation_reader" from Piotr 2017-12-18 17:53:20 +01:00
Piotr Jastrzebski
d756c49baf Rename cache_streamed_mutation_test to cache_flat_mutation_reader_test
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2017-12-18 16:37:57 +01:00