* seastar 38aaa4a...6d80c6a (2):
> DPDK: Change the metrics registration with label support
> metric: Fix the error: could not convert {...} from <brace-enclosed initializer list> to struct metric_definition_impl
Add a counter field to RELEASE, just before the date, and fix it at zero.
This allows custom package builds to override it in a way that sorts before
the official packages.
Example:
Official release: 1.6.0-0.20160120.<githash>
Custom release 1: 1.6.0-1.avi.20160121.<githash>
Custom release 2: 1.6.0-2.avi.20160122.<githash>
The counter (0/1/2) ensures that the build number dominates over the date
when sorting.
Message-Id: <20170122102814.19649-1-avi@scylladb.com>
From Avi:
In many cases, batch statements are used to mutate a single partition, or
a number of partitions that is smaller than the number of statements within
the batch. We can detect this case and reduce the numbers of mutations
applied, and in some cases, convert a logged batch into an unlogged batch.
Ref #1689.
continuous_data_consumer::fast_forward_to() returns a future which was
later ignored by data_consume_context::fast_forward_to().
With the current implementation, the future in question is always ready
and that's why the problem didn't manifest itself in the form of crashes
or invalid results.
Message-Id: <20170120105746.7300-1-pdziepak@scylladb.com>
Batch statements are often used to insert multiple rows into the same
partition. Recognize this case and merge mutations to the same partition.
If the result is a single mutation, there is an additional win (already
present in the code), where a logged batch can be converted into an unlogged
batch.
Ref #1689.
Add a boolean to short circuit the read path on empty range
hoping for some speedup.
tested in read write with cs using:
cl=QUORUM duration=1m -mode native cql3 -rate threads=700 -node localhost
Will do some additional benchmark.
Fixes#1056
Signed-off-by: Benoît Canet <benoit@scylladb.com>
Message-Id: <20170118194451.16836-1-benoit@scylladb.com>
That's because a single shard is used to calculate generation for new
sstables in upload directory, and that will result in that single shard
sharing all the resources with other shards.
For refresh without upload dir, it currently works fine because we
reshuffle column family dir instead.
flush_upload_dir() is now a free function, takes a distributed database
object, and uses calculate_shard_from_sstable_generation() to decide
which shard will move sstable using its own generation namespace.
Fixes#2008.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <b0cccf7bbb61416ff8718bac92fdca90cc5fb9c9.1484253232.git.raphaelsc@scylladb.com>
This patch ensures that when adding a shared sstable, we select only
one cpu to update that column family's stats. This is important so we
don't overestimated the on-disk size of sstables when resharding
This fixes only a temporary miscount of the current load, since shared
sstables are eventually re-written, but a fixes a permanent miscount
of the total load.
Refs #1592
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <20170119144823.31041-1-duarte@scylladb.com>
Currently eviction is performed until occupancy of the whole region
drops below the 85% threshold. This may take a while if region had
high occupancy and is large. We could improve the situation by only
evicting until occupancy of the sparsest segment drops below the
threshold, as is done by this change.
I tested this using a c-s read workload in which the condition
triggers in the cache region, with 1G per shard:
lsa-timing - Reclamation cycle took 12.934 us.
lsa-timing - Reclamation cycle took 47.771 us.
lsa-timing - Reclamation cycle took 125.946 us.
lsa-timing - Reclamation cycle took 144356 us.
lsa-timing - Reclamation cycle took 655.765 us.
lsa-timing - Reclamation cycle took 693.418 us.
lsa-timing - Reclamation cycle took 509.869 us.
lsa-timing - Reclamation cycle took 1139.15 us.
The 144ms pause is when large eviction is necessary.
The change improves worst case latency. Reclamation time statistics
over 30 second period after cache fills up, in microseconds:
Before:
avg = 1524.283148
stdev = 11021.021118
min = 12.934000
max = 144356.000000
sum = 257603.852000
samples = 169
After:
avg = 1317.362414
stdev = 1913.542802
min = 263.935000
max = 19244.600000
sum = 175209.201000
samples = 133
Refs #1634.
Message-Id: <1484730859-11969-1-git-send-email-tgrabiec@scylladb.com>
* seastar 240b0bf...ff098c8 (15):
> metrics::impl::shard(): check if reactor is initialized before using it
> reactor: introduce engine_is_ready()
> fix metric name
> Merge "Add label support to the metric layer" from Amnon
> core: Avoid memory leak when submission to syscall_work_queue fails
> core: Avoid memory leak when submission to smp_message_queue fails
> core: append_challenged_posix_file_impl: Make exception-safe
> Merge "Log backtrace in report_failed_future" from Tomasz
> install-dependencies.sh: add systemtap-sdt-dev to Ubuntu/Debian dependencies
> core: add fsqual.cc/.hh to core
> dpdk: Fix compile error with rte_pci.h
> fstream_test: fix spurious failures due to BOOST_REQUIRE_EQUAL thread-unsafety
> reactor: unregister metrics of queue on shard 0
> build: track system header changes too
> Prometheus: do not rely on collectd for the hostname
close() operation is like a destructor, it cannot fail. It just
reports errors, but close itself succeeds. So we should proceed with
the closing even if it fails.
Message-Id: <1484245886-7269-1-git-send-email-tgrabiec@scylladb.com>
As Tomek pointed out, previous code, regardless of version mismatch, of generating
comparator description string was not correct (as in: in sync with origin).
This modifies it to look at
1.) Actual clustring size
2.) Compound-ness
3.) Dense-ness
to determine whether we should generate a compound desc, and whether it
should contain a trailing utf8-desc type.
v2: Simplify non-dense base column addition and ensure it handles
thrift non-utf8 (as per comments from tomek)
Message-Id: <1484670171-18362-1-git-send-email-calle@scylladb.com>
As the metrics migration progressed, some include to scollectd.hh left
behind.
Because of the nature of the scollecd implementation those include
brings alot of code with them to the header files and eventually to many
source file.
This patch remove those include and add a missing include to
storage_proxy.cc.
The reason the compiler didn't complain is an indication to the
problematic nature of those include in the first place.
Before this patch, change in metrics.hh would cause 169 files to
compile, after this change 17.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
Message-Id: <1484667536-2185-1-git-send-email-amnon@scylladb.com>
Fixes#2019
According to the Java driver and cassandra, all versions < 3
include the PK in the comparator descriptor string.
This broke for us when bumping the cassandra version 2.1 -> 2.2
Message-Id: <1484657580-14411-1-git-send-email-calle@scylladb.com>
This patch ensures that the host only announces and registers the
MATERIALIZED_VIEWS feature if it was started with the experimental
flag.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <20170116123412.21365-1-duarte@scylladb.com>
After we call unlink_leftmost_without_rebalance(), we must unlink all
elements before mutatation is destroyed. We did this properly from
~reader, but it would not be called if reader construction failed,
which it may.
Message-Id: <1484572581-6537-1-git-send-email-tgrabiec@scylladb.com>
Since ce083308a1
"random_mutation_generator: Generate RTs by default" random mutation
generator produces range tombstones. However, so far the tests were run
with all features disabled (because of incomplete initialization of all
services) which meant that RANGE_TOMBSTONE feature was not enabled and
the code couldn't handle range tombstones that weren't just prefixes.
This patch solves the problem by forcing all features to be enabled when
tests are run.
Message-Id: <20170116103324.22956-1-pdziepak@scylladb.com>
It still has problems:
- while resharding a very large leveled compaction strategy table, a huge
amount of tiny sstables are generated, overwhelming the file descriptor
limits
- there is a large impact on read latency while resharding is going on
(cherry picked from commit cf27d44412)
(forward-ported from branch-1.6)
Since scylla-housekeeping running as scylla user, it doesn't have a permission
to create a file on /etc/scylla.d.
So introduce /var/lib/scylla-housekeeping which owns by scylla user, place uuid
file on the directory.
Fixes#2009
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1484235946-12463-1-git-send-email-syuu@scylladb.com>
query_mutations_locally() takes one_or_two_partition_ranges by
reference and requires, indirectly, that it is kept alive until
operation resolves. However, we were passing expiring value to it, the
result of unwrap().
Fixes dtest failure in consistent_bootstrap_test.py:TestBootstrapConsistency.consistent_reads_after_bootstrap_test
Another potential problem was that we were dereferencing "s" in the same
expression which move-constructs an argument out of it.
Message-Id: <1484222759-4967-1-git-send-email-tgrabiec@scylladb.com>
On DPDK 16.11 dpdk_nic_bind.py is renamed to dpdk-devbind.py, so we are
getting "file not found" both on packaging and scripts, fixed that.
Also fixed inconsistent packaging.
Since Seastar copied dpdk_nic_bind.py to its scripts/ directory, there're two
different versions of the script, .rpm/.deb packaging different one:
dist/redhat: seastar/dpdk/tools/dpdk_nic_bind.py
dist/ubuntu: seastar/scripts/dpdk_nic_bind.py
That's won't work because we sharing setup scripts between two
distributions, so I changed dist/ubuntu package to use DPDK one.
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1484191955-28006-1-git-send-email-syuu@scylladb.com>
Explicitly generate tables' IDs of tables from the system_traces KS using
generate_legacy_id() in order to ensure all Nodes create these tables with
the same IDs.
This is going to prevent hitting issue #420.
Fixes#1976
Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
Message-Id: <1484153725-31030-1-git-send-email-vladz@scylladb.com>
This patch changes the random_mutation_generator so it generates range
tombstones by default. This fixes a bug where reversibly applying
range tombstones wasn't being tested.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <20170110164822.28747-1-duarte@scylladb.com>
