There is a user of fragment_and_freeze() (streaming) that will need
to be able to break the loop Right now, it does that between
streamed_mutation, but that won't be possible after we switch to flat
readers.
partition_entry::read() calls open_version() under standard allocator,
but it may allocate a new partition version if a snapshot already
exists which was created in an earlier phase. Versions are supposed to
be allocated using region's allocator, they will be freed using
region's allocator. LSA will delegate free() to the standard allocator
correctly in this case, but it will subtract from its
_non_lsa_occupancy, assuming the allocation was done through it. This
will corrupt occupancy() for cache region.
Fixes#2948.
Message-Id: <1510229584-14398-1-git-send-email-tgrabiec@scylladb.com>
"Ensure stop() waits for the accept loop to complete to avoid crashes
during shutdown."
* 'thrift-server-stop/v4' of https://github.com/duarten/scylla:
thrift/server: Restore code format
thrift/server: Stopping the server waits for connection shutdown
thrift/server: Abort listeners on stop()
thrift/server: Avoid manual memory management
thrift/server: Add move ctor for connection
thrift/server: Extract retry logic
thrift/server: Retry with backoff for some error types
thrift/server: Retry accept in case of error
This patch ensures the future returned from stop() resolves only when
all connections and listeners are no longer in use.
Fixes#2657Fixes#2942
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
In case of errors like ECONNABORTED, we want to retry accepting
connections. Right now we immediately retry the accept, but in
subsequent patches we introduce a backoff for other types of errors.
We also consider fatal errors like EBADFD, which should not trigger a
retry.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Currently, we use type type of the column as the accumulator when we
average it. This can easily overflow, e.g. (2^31-1)+(3) = overflow.
Fix by using __int128 for the accumulator. It's not standard, but
it's way more efficient and simpler than the alternatives.
Inspired by CASSANDRA-12417, but much simpler due to the availability
of __int128.
Message-Id: <20171112173529.30764-1-avi@scylladb.com>
"This patch adds support for varint and decimal to aggregate functions.
Some other types (like byte or smallint) weren't supported and they are
supported by C*. So their aggregate functions were added as well.
To allow aggregate functions for big_decimal, following methods were added
to big_decimal type:
* Division by int64_t that preservers number of decimal digits.
* Operator += .
* Comparison operators.
Fixes #2842."
* 'danfiala/scylla-2842-send-002' of https://github.com/hagrid-the-developer/scylla:
tests: Add tests for aggregate functions.
tests: Add tests for big_decimal type.
cql3/functions: Add aggregate functions for big_decimal.
utils/big_decimal: Added necessary operators and methods for aggregate functions.
cql3/functions: Add aggregate functions for types for which it is trivial.
Since we started accounting virtual dirty memory we no longer have a cap
on real dirty memory. In most situations that is not needed, since real
dirty will just be at most twice as much as virtual dirty (current
flushing memtable plus new memtable).
However, due to things like cache updates and component flushing we can
end up having a lot of memtables that are virtually freed but not yet
fully released, leading real dirty memory to explode using all the box'
memory.
This patch adds a cap on real dirty memory as well. Because of the
hierarchical nature of region_group, if the parent blocks due to memory
depletion, so will the child (virtual dirty region group).
After that is done, we need to make sure that dirty memory is not seen
as freed until the cache update is done. Until a particular partition is
moved to the cache it is not evictable. As a result we can OOM the
system if we have a lot of pending cache updates as the writes will not
be throttled and memory won't be made available.
This patch pins the memory used by the region as real dirty before the
cache update starts, and unpins it when it is over. In the mean time it
gradually releases memory of the partitions that are being moved to
cache.
I have verified in a couple of workloads that the amount of memory
accounted through this is the same amount of memory accounted through
the memtable flush procedure.
Fixes#1942
* git@github.com:glommer/scylla.git glommer/update-cache-v4:
row_cache: modernize use of seastar threads
mutation_partition: estimate size of partition
memtable: factor out calculation of memtable_entry memory size
memtable: add a method to export memtable's dirty memory manager
dirty_memory_manager: block if we hit the real dirty limit
dirty_memory_manager: add functions to manipulate real dirty
partition: add method to calculate memory size of a partition
row cache: pin real dirty during cache updates.
This patch fixes 'DROP INDEX' CQL statement to also drop the underlying
index view automatically so that we don't leave unused materialized
views behind.
Message-Id: <1510303421-15945-1-git-send-email-penberg@scylladb.com>
Right now, once a region is moved to the cache is no longer visible to
the dirty memory system. Not as real dirty nor virtual dirty.
The problem is that until a particular partition is moved to the cache
it is not evictable. As a result we can OOM the system if we have a lot
of pending cache updates as the writes will not be throttled and memory
won't be made available.
This patch pins the memory used by the region as real dirty before the
cache update starts, and unpins it when it is over. In the mean time it
gradually releases memory of the partitions that are being moved to
cache.
I have verified in a couple of workloads that the amount of memory
accounted through this is the same amount of memory accounted through
the memtable flush procedure.
Fixes#1942
Signed-off-by: Glauber Costa <glauber@scylladb.com>
Once that is added, also add a method to a memtable entry to calculate
the entire size of a memtable entry. Right now we only have one method
to calculate the size minus rows.
Signed-off-by: Glauber Costa <glauber@scylladb.com>
There are times in which we want to add and remove real dirty memory
without impacting virtual dirty. One such example is the cache update
process, where real dirty is the limiting factor.
Signed-off-by: Glauber Costa <glauber@scylladb.com>
Since we started accounting virtual dirty memory we no longer have a cap
on real dirty memory. In most situations that is not needed, since real
dirty will just be at most twice as much as virtual dirty (current
flushing memtable plus new memtable).
However, due to things like cache updates and component flushing we can
end up having a lot of memtables that are virtually freed but not yet
fully released, leading real dirty memory to explode using all the box'
memory.
This patch adds a cap on real dirty memory as well. Because of the
hierarchical nature of region_group, if the parent blocks due to memory
depletion, so will the child (virtual dirty region group).
A next step is to add a controller that will increase the priority of
the tasks involving in releasing real dirty memory if we get dangerously
close to the threshold.
Signed-off-by: Glauber Costa <glauber@scylladb.com>
The total size is the sum of two components. Add a method that
does that sum so this code gets easier to reuse.
Signed-off-by: Glauber Costa <glauber@scylladb.com>
In the memtable flusher, we account for the size of a partition as we
read them. However, there are other points in the architecture where we
would like to calculate the size of a partition in a point in which we
are not reading it. One such example is the cache update process.
This patch enhances the mutation_partition adding a method that returns
the total size for this partition.
Signed-off-by: Glauber Costa <glauber@scylladb.com>
For a while now we have an async() function, that simplifies the code by not
needing to issue an explicit join. This patch converts the row cache to use
async() as well, which most of our code already does. Doing so will make
it easier to make changes to update_cache.
Signed-off-by: Glauber Costa <glauber@scylladb.com>
"Implement flat_mutation_reader::consume and add tests for it.
For that implement flat_mutation_reader_from_mutations and
read_mutation_from_flat_mutation_reader."
* 'haaawk/flat_reader_consume_v3' of github.com:scylladb/seastar-dev:
Add tests for flat_mutation_reader::consume
Add tests for flat_mutation_reader utils
Introduce read_mutation_from_flat_mutation_reader
Make mutation_rebuilder streamed_mutation independent
flat_mutation_reader_from_mutation: support multiple mutations
Introduce flat_mutation_reader::consume
Move FlattenedConsumer concept to flat_mutation_reader.hh
"This patchset introduces memtable::make_flat_reader that returns
flat_mutation_reader and converts internal memtable readers into
flat_mutation_readers.
It also introduces some utility methods like make_forwardable and
make_partition_snapshot_flat_reader."
* 'haaawk/flat_reader_memtable_v4' of github.com:scylladb/seastar-dev:
Turn scanning_reader into flat_mutation_reader
Change memtable_entry::read to return flat_mutation_reader
Make iterator_reader independent from mutation_reader
Introduce make_partition_snapshot_flat_reader
Prepare partition_snapshot_flat_reader
Introduce flat_mutation_reader_from_mutation
Prepare flat_mutation_reader_from_mutation
Introduce make_forwardable
Prepare make_forwardable for flat_mutation_reader
Introduce empty_flat_reader
memtable: Introduce make_flat_reader
mutation_rebuilder will be used not only with streamed_mutations
but also with flat_mutation_readers so it's better for it to be
independent from streamed_mutation.
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
Rename flat_mutation_reader_from_mutation to
flat_mutation_reader_from_mutations.
Make it work with std::vector<mutation> instead of a single
mutation.
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
