mirrors/scylladb
1
0
Fork 0
mirror of https://github.com/scylladb/scylladb.git synced 2026-05-01 21:55:50 +00:00
Code Issues Packages Projects Releases Wiki Activity
14,716 Commits 241 Branches 541 Tags
2a0ece52051bc3b487ff1fc14dd280bc69aa486c
Commit Graph

153 Commits

Author SHA1 Message Date
Tomasz Grabiec
d9f0c1f097 tests: cache: Fix invalidate() not being waited for 
Probably responsible for occasional failures of subsequent assertion.
Didn't mange to reproduce.

Message-Id: <1520330967-584-1-git-send-email-tgrabiec@scylladb.com>
 2018-03-06 12:14:04 +02:00
Tomasz Grabiec
9c3e56fb16 tests: row_cache: Improve test for snapshot consistency on eviction 
Reproduces https://github.com/scylladb/scylla/issues/3215.
Message-Id: <1518710592-21925-1-git-send-email-tgrabiec@scylladb.com>
 2018-02-15 16:48:23 +00:00
Tomasz Grabiec
b3415880b2 tests: row_cache: Add test for exception safety of updates from memtable 2018-02-15 10:13:02 +01:00
Avi Kivity
404172652e Merge "Use xxHash for digest instead of MD5" from Duarte 
"This series changes digest calculation to use a faster algorithm
(xxHash) and to also cache calculated cell hashes that can be kept in
memory to speed up subsequent digest requests.

The MD5 hash function has proved to be slow for large cell values:

size = 256; elapsed = 4us
size = 512; elapsed = 8us
size = 1024; elapsed = 14us
size = 2048; elapsed = 21us
size = 4096; elapsed = 33us
size = 8192; elapsed = 51us
size = 16384; elapsed = 86us
size = 32768; elapsed = 150us
size = 65536; elapsed = 278us
size = 131072; elapsed = 531us
size = 262144; elapsed = 1032us
size = 524288; elapsed = 2026us
size = 1048576; elapsed = 4004us
size = 2097152; elapsed = 7943us
size = 4194304; elapsed = 15800us
size = 8388608; elapsed = 31731us
size = 16777216; elapsed = 64681us
size = 33554432; elapsed = 130752us
size = 67108864; elapsed = 263154us

The xxHash is a non-cryptographic, 64bit (there's work in progress on
the 128 version) hash that can be used to replace MD5. It performs much
better:

size = 256; elapsed = 2us
size = 512; elapsed = 1us
size = 1024; elapsed = 1us
size = 2048; elapsed = 2us
size = 4096; elapsed = 2us
size = 8192; elapsed = 3us
size = 16384; elapsed = 5us
size = 32768; elapsed = 8us
size = 65536; elapsed = 14us
size = 131072; elapsed = 28us
size = 262144; elapsed = 59us
size = 524288; elapsed = 116us
size = 1048576; elapsed = 226us
size = 2097152; elapsed = 456us
size = 4194304; elapsed = 935us
size = 8388608; elapsed = 1848us
size = 16777216; elapsed = 4723us
size = 33554432; elapsed = 10507us
size = 67108864; elapsed = 21622us

Performance was tested using a 3 node cluster with 1 cpu and 8GB,
and with the following cassandra-stress loaders. Measurements are for
the read workload.

sudo taskset -c 4-15 ./cassandra-stress write cl=ALL n=5000000 -schema 'replication(factor=3)' -col 'size=FIXED(1024) n=FIXED(4)' -mode native cql3 -rate threads=100
sudo taskset -c 4-15 ./cassandra-stress mixed cl=ALL 'ratio(read=1)' n=10000000 -pop 'dist=gauss(1..5000000,5000000,500000)' -col 'size=FIXED(1024) n=FIXED(4)' -mode native cql3 -rate threads=100

xxhash + caching:

Results:
op rate                   : 32699 [READ:32699]
partition rate            : 32699 [READ:32699]
row rate                  : 32699 [READ:32699]
latency mean              : 3.0 [READ:3.0]
latency median            : 3.0 [READ:3.0]
latency 95th percentile   : 3.9 [READ:3.9]
latency 99th percentile   : 4.5 [READ:4.5]
latency 99.9th percentile : 6.6 [READ:6.6]
latency max               : 24.0 [READ:24.0]
Total partitions          : 10000000 [READ:10000000]
Total errors              : 0 [READ:0]
total gc count            : 0
total gc mb               : 0
total gc time (s)         : 0
avg gc time(ms)           : NaN
stdev gc time(ms)         : 0
Total operation time      : 00:05:05
END

md5:

Results:
op rate                   : 25241 [READ:25241]
partition rate            : 25241 [READ:25241]
row rate                  : 25241 [READ:25241]
latency mean              : 3.9 [READ:3.9]
latency median            : 3.9 [READ:3.9]
latency 95th percentile   : 5.1 [READ:5.1]
latency 99th percentile   : 5.8 [READ:5.8]
latency 99.9th percentile : 8.0 [READ:8.0]
latency max               : 24.8 [READ:24.8]
Total partitions          : 10000000 [READ:10000000]
Total errors              : 0 [READ:0]
total gc count            : 0
total gc mb               : 0
total gc time (s)         : 0
avg gc time(ms)           : NaN
stdev gc time(ms)         : 0
Total operation time      : 00:06:36
END

This translates into a 21% improvoment for this workload.

Bigger cell values were also tested:

sudo taskset -c 4-15 ./cassandra-stress write cl=ALL n=1000000 -schema 'replication(factor=3)' -col 'size=FIXED(4096) n=FIXED(4)' -mode native cql3 -rate threads=100
sudo taskset -c 4-15 ./cassandra-stress mixed cl=ALL 'ratio(read=1)' n=10000000 -pop 'dist=gauss(1..1000000,500000,100000)' -col 'size=FIXED(4096) n=FIXED(4)' -mode native cql3 -rate threads=100

xxhash + caching:

Results:
op rate                   : 19964 [READ:19964]
partition rate            : 19964 [READ:19964]
row rate                  : 19964 [READ:19964]
latency mean              : 4.9 [READ:4.9]
latency median            : 4.6 [READ:4.6]
latency 95th percentile   : 7.2 [READ:7.2]
latency 99th percentile   : 11.5 [READ:11.5]
latency 99.9th percentile : 13.6 [READ:13.6]
latency max               : 29.2 [READ:29.2]
Total partitions          : 10000000 [READ:10000000]
Total errors              : 0 [READ:0]
total gc count            : 0
total gc mb               : 0
total gc time (s)         : 0
avg gc time(ms)           : NaN
stdev gc time(ms)         : 0
Total operation time      : 00:08:20
END

md5:

Results:
op rate                   : 12773 [READ:12773]
partition rate            : 12773 [READ:12773]
row rate                  : 12773 [READ:12773]
latency mean              : 7.7 [READ:7.7]
latency median            : 7.3 [READ:7.3]
latency 95th percentile   : 10.2 [READ:10.2]
latency 99th percentile   : 16.8 [READ:16.8]
latency 99.9th percentile : 19.2 [READ:19.2]
latency max               : 71.5 [READ:71.5]
Total partitions          : 10000000 [READ:10000000]
Total errors              : 0 [READ:0]
total gc count            : 0
total gc mb               : 0
total gc time (s)         : 0
avg gc time(ms)           : NaN
stdev gc time(ms)         : 0
Total operation time      : 00:13:02
END

This translates into a 37% improvoment for this workload.

Fixes #2884

Tests: unit-tests (release), dtests (smp=2)

Note: dtests are kinda broken in master (> 30 failures), so take the
tests tag with a grain of himalayan salt."

* 'xxhash/v5' of https://github.com/duarten/scylla: (29 commits)
  tests/row_cache_test: Test hash caching
  tests/memtable_test: Test hash caching
  tests/mutation_test: Use xxHash instead of MD5 for some tests
  tests/mutation_test: Test xx_hasher alongside md5_hasher
  schema: Remove unneeded include
  service/storage_proxy: Enable hash caching
  service/storage_service: Add and use xxhash feature
  message/messaging_service: Specify algorithm when requesting digest
  storage_proxy: Extract decision about digest algorithm to use
  cache_flat_mutation_reader: Pre-calculate cell hash
  partition_snapshot_reader: Pre-calculate cell hash
  query::partition_slice: Add option to specify when digest is requested
  row: Use cached hash for hash calculation
  mutation_partition: Replace hash_row_slice with appending_hash
  mutation_partition: Allow caching cell hashes
  mutation_partition: Force vector_storage internal storage size
  test.py: Increase memory for row_cache_stress_test
  atomic_cell_hash: Add specialization for atomic_cell_or_collection
  query-result: Use digester instead of md5_hasher
  range_tombstone: Replace feed_hash() member function with appending_hash
  ...
 2018-02-08 18:24:58 +02:00
Tomasz Grabiec
c1b82e60e3 tests: row_cache: Add test for memtable readers surviving flush and eviction 
Reproduces https://github.com/scylladb/scylla/issues/3186
 2018-02-06 14:24:19 +01:00
Duarte Nunes
992de302a2 tests/row_cache_test: Test hash caching 
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
 2018-02-01 01:02:50 +00:00
Piotr Jastrzebski
7729bc5e7b Remove unused mutation_reader_assertions 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:56:48 +01:00
Piotr Jastrzebski
39ec13133f row_cache: rename make_flat_reader to make_reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:45 +01:00
Piotr Jastrzebski
0d76091a28 test_mvcc: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:45 +01:00
Piotr Jastrzebski
425c1624cd test_cache_population_and_clear_race: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:45 +01:00
Piotr Jastrzebski
dc97acb778 test_cache_population_and_update_race: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:45 +01:00
Piotr Jastrzebski
1bead9747a test_continuity_flag_and_invalidate_race: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:45 +01:00
Piotr Jastrzebski
4266b9759e test_update_failure: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:45 +01:00
Piotr Jastrzebski
d5366026b1 row_cache_test: use flat reader in verify_has 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:45 +01:00
Piotr Jastrzebski
56b0157831 row_cache_test: use flat reader in has_key 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:45 +01:00
Piotr Jastrzebski
06bca9f4d5 test_sliced_read_row_presence: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:45 +01:00
Piotr Jastrzebski
6c3d9cdb9f test_lru: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:45 +01:00
Piotr Jastrzebski
a979869a15 test_update_invalidating: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:45 +01:00
Piotr Jastrzebski
781d9a324d test_scan_with_partial_partitions: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:45 +01:00
Piotr Jastrzebski
f199aab1ad test_cache_populates_partition_tombstone: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:45 +01:00
Piotr Jastrzebski
9755f7677c test_tombstone_merging_in_partial_partition: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:45 +01:00
Piotr Jastrzebski
2e1b12b6ce consume_all,populate_range: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:45 +01:00
Piotr Jastrzebski
d08f4a40b2 test_readers_get_all_data_after_eviction: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:45 +01:00
Piotr Jastrzebski
f99992261f test_tombstones_are_not_missed_when_range_is_invalidated: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:45 +01:00
Piotr Jastrzebski
50fb2a57b6 test_exception_safety_of_reads: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:45 +01:00
Piotr Jastrzebski
f0af5a1321 test_exception_safety_of_transitioning_from_underlying_read_to_read_from_cache: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:45 +01:00
Piotr Jastrzebski
98b97be19a test_exception_safety_of_partition_scan: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:45 +01:00
Piotr Jastrzebski
5010c082f6 test_concurrent_population_before_latest_version_iterator: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:44 +01:00
Piotr Jastrzebski
f8964f3aff test_concurrent_populating_partition_range_reads: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:44 +01:00
Piotr Jastrzebski
3e1da7525e test_random_row_population: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:44 +01:00
Piotr Jastrzebski
e6cf785829 test_continuity_is_populated_when_read_overlaps_with_older_version: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:44 +01:00
Piotr Jastrzebski
2b61411c7b test_continuity_population_with_multicolumn_clustering_key: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:44 +01:00
Piotr Jastrzebski
561f5fbb5a test_continuity_is_populated_for_single_row_reads: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:44 +01:00
Piotr Jastrzebski
a1b6557877 test_concurrent_setting_of_continuity_on_read_upper_bound: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:44 +01:00
Piotr Jastrzebski
6bbd0c7301 test_reading_from_random_partial_partition: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:44 +01:00
Piotr Jastrzebski
327eb8fbbd test_tombstone_merging_of_overlapping_tombstones_in_many_versions: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:44 +01:00
Piotr Jastrzebski
07df1a6f87 test_concurrent_reads_and_eviction: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:44 +01:00
Piotr Jastrzebski
63f45d522e test_eviction: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:44 +01:00
Piotr Jastrzebski
57d19a390a test_random_partition_population: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:44 +01:00
Piotr Jastrzebski
e9e8121ffe test_single_key_queries_after_population_in_reverse_order: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:44 +01:00
Piotr Jastrzebski
9acbb1e0f4 test_query_of_incomplete_range_goes_to_underlying: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:44 +01:00
Piotr Jastrzebski
7456c31e10 test_cache_delegates_to_underlying_only_once_with_single_partition: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:44 +01:00
Piotr Jastrzebski
4a3f5249ce test_cache_uses_continuity_info_for_single_partition_query: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:44 +01:00
Piotr Jastrzebski
869443e11f test_cache_delegates_to_underlying_only_once_empty_single_partition_query: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:44 +01:00
Piotr Jastrzebski
4cc9a0d852 test_cache_delegates_to_underlying_only_once_empty_full_range: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:44 +01:00
Piotr Jastrzebski
5cdc77b66e test_cache_works_after_clearing: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:44 +01:00
Piotr Jastrzebski
5091474f14 test_cache_delegates_to_underlying: use flat reader 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:54:44 +01:00
Piotr Jastrzebski
c0c88b3d4e Fix master 
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:53:11 +01:00
Piotr Jastrzebski
c394dd9288 row_cache_test: add tests for small_buffer 
When a buffer of a flat reader is small then the reader can't
handle range_tombstones correctly.

This is not a problem on a production when the buffer is large.

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
 2018-01-24 20:09:11 +01:00
José Guilherme Vanz
380bc0aa0d Swap arguments order of mutation constructor 
Swap arguments in the mutation constructor keeping the same standard
from the constructor variants. Refs #3084

Signed-off-by: José Guilherme Vanz <guilherme.sft@gmail.com>
Message-Id: <20180120000154.3823-1-guilherme.sft@gmail.com>
 2018-01-21 12:58:42 +02:00