Commit Graph

16859 Commits

Author SHA1 Message Date
Avi Kivity
71fc5fb738 exceptions: convert sprint() to format()
sprint() recently became more strict, throwing on sprint("%s", 5). Replace
with the more modern format().

Mechanically converted with https://github.com/avikivity/unsprint.
2018-11-01 13:16:17 +00:00
Avi Kivity
7ae23d8f9b index: convert sprint() to format()
sprint() recently became more strict, throwing on sprint("%s", 5). Replace
with the more modern format().

Mechanically converted with https://github.com/avikivity/unsprint.
2018-11-01 13:16:17 +00:00
Avi Kivity
fd513c42ad streaming: convert sprint() to format()
sprint() recently became more strict, throwing on sprint("%s", 5). Replace
with the more modern format().

Mechanically converted with https://github.com/avikivity/unsprint.
2018-11-01 13:16:17 +00:00
Avi Kivity
8501e2a45d streaming: progress_info: fix format string
We try to escape % as \%, but the correct escape is %%.
2018-11-01 13:16:17 +00:00
Avi Kivity
da17c29bd3 api: convert sprint() to format()
sprint() recently became more strict, throwing on sprint("%s", 5). Replace
with the more modern format().

Mechanically converted with https://github.com/avikivity/unsprint.
2018-11-01 13:16:17 +00:00
Avi Kivity
82818758ca dht: convert sprint() to format()
sprint() recently became more strict, throwing on sprint("%s", 5). Replace
with the more modern format().

Mechanically converted with https://github.com/avikivity/unsprint.
2018-11-01 13:16:17 +00:00
Avi Kivity
7a125c6634 thrift: convert sprint() to format()
sprint() recently became more strict, throwing on sprint("%s", 5). Replace
with the more modern format().

Mechanically converted with https://github.com/avikivity/unsprint.
2018-11-01 13:16:17 +00:00
Avi Kivity
0c33d13165 locator: convert sprint() to format()
sprint() recently became more strict, throwing on sprint("%s", 5). Replace
with the more modern format().

Mechanically converted with https://github.com/avikivity/unsprint.
2018-11-01 13:16:17 +00:00
Avi Kivity
e096fa2fde gms: convert sprint() to format()
sprint() recently became more strict, throwing on sprint("%s", 5). Replace
with the more modern format().

Mechanically converted with https://github.com/avikivity/unsprint.
2018-11-01 13:16:17 +00:00
Avi Kivity
d77e044cde db: convert sprint() to format()
sprint() recently became more strict, throwing on sprint("%s", 5). Replace
with the more modern format().

Mechanically converted with https://github.com/avikivity/unsprint.
2018-11-01 13:16:17 +00:00
Avi Kivity
5f79ff0f54 transport: convert sprint() to format()
sprint() recently became more strict, throwing on sprint("%s", 5). Replace
with the more modern format().

Mechanically converted with https://github.com/avikivity/unsprint.
2018-11-01 13:16:17 +00:00
Avi Kivity
be99101f36 utils: convert sprint() to format()
sprint() recently became more strict, throwing on sprint("%s", 5). Replace
with the more modern format().

Mechanically converted with https://github.com/avikivity/unsprint.
2018-11-01 13:16:17 +00:00
Avi Kivity
455f00e993 sstables: convert sprint() to format()
sprint() recently became more strict, throwing on sprint("%s", 5). Replace
with the more modern format().

Mechanically converted with https://github.com/avikivity/unsprint.
2018-11-01 13:16:17 +00:00
Avi Kivity
eb74fe784d auth: convert sprint() to format()
sprint() recently became more strict, throwing on sprint("%s", 5). Replace
with the more modern format().

Mechanically converted with https://github.com/avikivity/unsprint.
2018-11-01 13:16:17 +00:00
Avi Kivity
cb7ee5c765 cql3: convert sprint() to format()
sprint() recently became more strict, throwing on sprint("%s", 5). Replace
with the more modern format().

Mechanically converted with https://github.com/avikivity/unsprint.
2018-11-01 13:16:17 +00:00
Avi Kivity
8cca3b2879 row_cache: fix bad format string syntax
Some sprint() calls use the fmt language instead of the printf syntax. Convert
them all the way to format().
2018-11-01 13:16:17 +00:00
Avi Kivity
6488b017c3 repair: fix bad format string syntax
Some sprint() calls use the fmt language instead of the printf syntax. Convert
them all the way to format().
2018-11-01 13:16:17 +00:00
Avi Kivity
bceff1550c tests: fix bad format string syntax
Some sprint() calls use the fmt language instead of the printf syntax. Convert
them all the way to format().
2018-11-01 13:16:17 +00:00
Avi Kivity
7ff5569ee8 dht: fix bad format string syntax
Some sprint() calls use the fmt language instead of the printf syntax. Convert
them all the way to format().
2018-11-01 13:16:17 +00:00
Avi Kivity
738e713edf sstables: fix bad format string syntax
Some sprint() calls use the fmt language instead of the printf syntax. Convert
them all the way to format().
2018-11-01 13:16:17 +00:00
Avi Kivity
3cf434b863 utils: estimated_histogram: convert generated format strings to fmt
Convert printf games to format games.

Note that fmt supports specifying the field width as an argument, but that
is left to a dedicated change.
2018-11-01 13:16:17 +00:00
Avi Kivity
8ca4b7abea tests: perf_fast_forward: rename "format" variable
The format local variable will soon alias with the format function which we
intend to use in the same context. Rename it away to avoid a clash.
2018-11-01 13:16:17 +00:00
Avi Kivity
7908f09148 tests: perf_fast_forward: massage result of sprint() into std::string
sprint() returns std::string(), but the new format() returns an sstring. Usually
an sstring is wanted but in this case an sstring will fail as it is added to
an std::string.

Fix the failure (after spring->format conversion) by converting to an std::string.
2018-11-01 13:16:17 +00:00
Avi Kivity
7726ce23b7 utils: i_filter: rename "format" variable
The format variable hides the format function, which we'll soon want to use
here. Rename the format variable to unhide the function.
2018-11-01 13:16:17 +00:00
Avi Kivity
04b70a2ff8 system_keyspace: simplify complicated sprint()
update_peer_info() uses two sprint()s where one would do, which confuses
the sprint-to-fmt translator. Simplify the code by using just one call.
2018-11-01 13:16:17 +00:00
Avi Kivity
23e05a045b cql: convert Cql.g sprint()s to fmt
The only sprint() call had an extra complication due to quoting, which can be
removed now.
2018-11-01 13:16:16 +00:00
Avi Kivity
8db8c01fbe types: get rid of PRId64 formatting
It's not needed for out sprint() implementation, and gets in the way of
converting all formatting to fmt.
2018-11-01 13:16:16 +00:00
Avi Kivity
96173e81e0 Update seastar submodule
* seastar c1e0e5d...c02150e (5):
  > prometheus: pass names as query parameter instead of part of the URL
  > treewide: convert printf() style formatting to fmt
  > print: add fmt_print()
  > build: Remove experimental CMake support
  > Merge "Correct and clean-up `signal_test`" from Jesse
2018-11-01 13:13:48 +00:00
Yibo Cai (Arm Technology China)
79136e895f utils/crc: calculate crc in parallel
It achieves 2.0x speedup on intel E5 and 1.1x to 2.5x speedup on
various arm64 microarchitectures.

The algorithm cuts data into blocks of 1024 bytes and calculates crc
for each block, which is furthur divided into three subblocks of 336
bytes(42 uint64) each, and 16 remaining bytes(2 uint64).

For each iteration, three independent crc are caculated for one uint64
from each subgroup. It increases IPC(instructions per cycle) much.
After subblocks are done, three crc and remaining two uint64 are
combined using carry-less multiplication to reach the final result
for one block of 1024 bytes.

Signed-off-by: Yibo Cai <yibo.cai@arm.com>
Message-Id: <1541042759-24767-1-git-send-email-yibo.cai@arm.com>
2018-11-01 10:19:32 +02:00
Takuya ASADA
5bf9a03d65 dist/debian: skip running dh_strip_nondeterminism
On some Fedora environment dh build tries to run
dh_strip_nondeterminism, and fails sice Fedora does not provide such
command.
(see:
http://jenkins.cloudius-systems.com/view/master/job/scylla-master/job/unified-deb/3/console)

To prevent the build error we need to skip it.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20181030062935.9930-1-syuu@scylladb.com>
2018-10-31 10:23:54 +02:00
Tomasz Grabiec
62c7685b0d Merge "Proper support for static rows in SSTables 3.x" from Vladimir
This patchset addresses two issues with static rows support in SSTables
3.x. ('mc' format):

1. Since collections are allowed in static rows, we need to check for
complex deletion, set corresponding flag and write tombstones, if any.
2. Column indices need to be partitioned for static columns the same way
they are partitioned for regular ones.

 * github.com/argenet/scylla.git projects/sstables-30/columns-proper-order-followup/v1:
  sstables: Partition static columns by atomicity when reading/writing
    SSTables 3.x.
  sstables: Use std::reference_wrapper<> instead of a helper structure.
  sstables: Check for complex deletion when writing static rows.
  tests: Add/fix comments to
    test_write_interleaved_atomic_and_collection_columns.
  tests: Add test covering inverleaved atomic and collection cells in
    static row.
2018-10-30 10:36:46 +01:00
Vladimir Krivopalov
d82ac02fad tests: Add test covering inverleaved atomic and collection cells in static row.
Signed-off-by: Vladimir Krivopalov <vladimir@scylladb.com>
2018-10-29 15:01:34 -07:00
Vladimir Krivopalov
7bd95399ed tests: Add/fix comments to test_write_interleaved_atomic_and_collection_columns.
Signed-off-by: Vladimir Krivopalov <vladimir@scylladb.com>
2018-10-29 15:00:55 -07:00
Vladimir Krivopalov
6bd738ceb1 sstables: Check for complex deletion when writing static rows.
It is possible to have collections in a static row so we need to check
for collection-wide tombstones like with clustering rows.

Signed-off-by: Vladimir Krivopalov <vladimir@scylladb.com>
2018-10-29 14:59:19 -07:00
Vladimir Krivopalov
6b7003088a sstables: Use std::reference_wrapper<> instead of a helper structure.
No need to store column_id separately as it can be accessed from the
column_definition.

Signed-off-by: Vladimir Krivopalov <vladimir@scylladb.com>
2018-10-29 14:58:08 -07:00
Vladimir Krivopalov
8592b834d1 sstables: Partition static columns by atomicity when reading/writing SSTables 3.x.
Collections are permitted in static rows so same partitioning as for
regular columns is required.

Signed-off-by: Vladimir Krivopalov <vladimir@scylladb.com>
2018-10-29 10:32:02 -07:00
Takuya ASADA
2ac14dcf25 dist/redhat: prevent build error on older Fedora/CentOS
Current scylla.spec fails build on Fedora 27, since python2-pystache is
new package name that renamed on Fedora 28.
But Fedora 28's python2-pystache has tag "Provides: pystache",
so we can depends on old package name, this way we can build scylla.spec both
on Fedora 27/28.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20181028175450.31156-1-syuu@scylladb.com>
2018-10-29 11:36:40 +02:00
Yibo Cai (Arm Technology China)
1c48e3fbec utils/crc: leverage arm64 crc extension
It achieves 6.7x to 11x speedup on various arm64 microarchitectures.

Signed-off-by: Yibo Cai <yibo.cai@arm.com>
Message-Id: <1540781879-15465-1-git-send-email-yibo.cai@arm.com>
2018-10-29 10:50:48 +02:00
Nadav Har'El
b8337f8c9d Materalized views: fix race condition in resharding while view building
When a node reshards (i.e., restarts with a different number of CPUs), and
is in the middle of building a view for a pre-existing table, the view
building needs to find the right token from which to start building on all
shards. We ran the same code on all shards, hoping they would all make
the same decision on which token to continue. But in some cases, one
shard might make the decision, start building, and make progress -
all before a second shard goes to make the decision, which will now
be different.

This resulted, in some rare cases, in the new materialized view missing
a few rows when the build was interrupted with a resharding.

The fix is to add the missing synchronization: All shards should make
the same decision on whether and how to reshard - and only then should
start building the view.

Fixes #3890
Fixes #3452

Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20181028140549.21200-1-nyh@scylladb.com>
2018-10-28 17:20:10 +00:00
Avi Kivity
75dbff984c Merge "Re-order columns when reading/writing SSTables 3.x" from Vladimir
"
In Cassandra, row columns are stored in a BTree that uses the following
ordering on them:
    - all atomic columns go first, then all multi-cell ones
    - columns of both types (atomic and multi-cell) are
      lexicographically ordered by name regarding each other

Scylla needs to store columns and their respective indices using the
same ordering as well as when reading them back.

Fixes #3853

Tests: unit {release}

+

Checked that the following SSTables are dumped fine using Cassandra's
sstabledump:

cqlsh:sst3> CREATE TABLE atomic_and_collection3 ( pk int, ck int, rc1 text, rc2 list<text>, rc3 text, rc4 list<text>, rc5 text, rc6 list<text>, PRIMARY KEY (pk, ck)) WITH compression = {'sstable_compression': ''};
cqlsh:sst3> INSERT INTO atomic_and_collection3 (pk, ck, rc1, rc4, rc5) VALUES (0, 0, 'hello', ['beautiful','world'], 'here');
<< flush >>

sstabledump:

[
  {
    "partition" : {
      "key" : [ "0" ],
      "position" : 0
    },
    "rows" : [
      {
        "type" : "row",
        "position" : 96,
        "clustering" : [ 0 ],
        "liveness_info" : { "tstamp" : "1540599270139464" },
        "cells" : [
          { "name" : "rc1", "value" : "hello" },
          { "name" : "rc5", "value" : "here" },
          { "name" : "rc4", "deletion_info" : { "marked_deleted" : "1540599270139463", "local_delete_time" : "1540599270" } },
          { "name" : "rc4", "path" : [ "45e22cb0-d97d-11e8-9f07-000000000000" ], "value" : "beautiful" },
          { "name" : "rc4", "path" : [ "45e22cb1-d97d-11e8-9f07-000000000000" ], "value" : "world" }
        ]
      }
    ]
  }
]
"

* 'projects/sstables-30/columns-proper-order/v1' of https://github.com/argenet/scylla:
  tests: Test interleaved atomic and multi-cell columns written to SSTables 3.x.
  sstables: Re-order columns (atomic first, then collections) for SSTables 3.x.
  sstables: Use a compound structure for storing information used for reading columns.
2018-10-28 10:56:09 +02:00
Rafi Einstein
32525f2694 Space-Saving Top-k algorithm for handling stream summary statistics
Based on the following implementation ([2]) for the Space-Saving algorithm from [1].
[1] http://www.cse.ust.hk/~raywong/comp5331/References/EfficientComputationOfFrequentAndTop-kElementsInDataStreams.pdf
[2] https://github.com/addthis/stream-lib/blob/master/src/main/java/com/clearspring/analytics/stream/StreamSummary.java

The algorithm keeps a map between keys seen and their counts, keeping a bound on the number of tracked keys.
Replacement policy evicts the key with the lowest count while inheriting its count, and recording an estimation
of the error which results from that.
This error estimation can be later used to prove if the distribution we arrived at corresponds to the real top-K,
which we can display alongside the results.
Accuracy depends on the number of tracked keys.

Introduced as part of 'nodetool toppartition' query implementation.

Refs #2811
Message-Id: <20181027220937.58077-1-rafie@scylladb.com>
2018-10-28 10:10:28 +02:00
Vladimir Krivopalov
f3dc2a4927 tests: Test interleaved atomic and multi-cell columns written to SSTables 3.x.
Signed-off-by: Vladimir Krivopalov <vladimir@scylladb.com>
2018-10-26 16:58:34 -07:00
Vladimir Krivopalov
7e56e9fca6 sstables: Re-order columns (atomic first, then collections) for SSTables 3.x.
In Cassandra, row columns are stored in a BTree that uses the following
ordering on them:
    - all atomic columns go first, then all multi-cell ones
    - columns of both types (atomic and multi-cell) are
      lexicographically ordered by name regarding each other

Since schema already has all columns lexicographically sorted by name,
we only need to stably partition them by atomicity for that.

Fixes #3853

Signed-off-by: Vladimir Krivopalov <vladimir@scylladb.com>
2018-10-26 15:58:33 -07:00
Vladimir Krivopalov
210507b867 sstables: Use a compound structure for storing information used for reading columns.
This representation makes it easier to operate with compound structures
instead of separate values that were stored in multiple containers.

Signed-off-by: Vladimir Krivopalov <vladimir@scylladb.com>
2018-10-26 11:32:44 -07:00
Tomasz Grabiec
cf2d5c19fb Merge "Properly write static rows missing columns for SSTables 3.x." from Vladimir
Before this fix, write_missing_columns() helper would always deal with
regular columns even when writing static rows.

This would cause errors on reading those files.

Now, the missing columns are written correctly for regular and static
rows alike.

* github.com/argenet/scylla.git projects/sstables-30/fix-writing-static-missing-columns/v1:
  schema: Add helper method returning the count of columns of specified
    kind.
  sstables: Honour the column kind when writing missing columns in 'mc'
    format.
  tests: Add test for a static row with missing columns (SStables 3.x.).
2018-10-26 09:06:01 +02:00
Vladimir Krivopalov
9843343ad8 tests: Add test for a static row with missing columns (SStables 3.x.).
This is a test case for #3892.

Signed-off-by: Vladimir Krivopalov <vladimir@scylladb.com>
2018-10-25 17:16:31 -07:00
Vladimir Krivopalov
44043cfd44 sstables: Honour the column kind when writing missing columns in 'mc' format.
Previously, we've been writing the wrong missing columns indices for
static rows because write_missing_columns() explicitly used regular
columns internally.

Now, it takes the proper column kind into account.

Fixes #3892

Signed-off-by: Vladimir Krivopalov <vladimir@scylladb.com>
2018-10-25 17:09:09 -07:00
Vladimir Krivopalov
399f815a89 schema: Add helper method returning the count of columns of specified kind.
Signed-off-by: Vladimir Krivopalov <vladimir@scylladb.com>
2018-10-25 17:07:20 -07:00
Tomasz Grabiec
dcac0ac80c tests: sstables: Verify no index reads during scans which dont need it
Reproducer for https://github.com/scylladb/scylla/issues/3868

Message-Id: <1540459849-27612-2-git-send-email-tgrabiec@scylladb.com>
2018-10-25 16:14:45 +03:00
Tomasz Grabiec
46d0c157ae tests: sstables: Extract make_sstable_mutation_source()
Message-Id: <1540459849-27612-1-git-send-email-tgrabiec@scylladb.com>
2018-10-25 16:14:39 +03:00