mirrors/scylladb
1
0
Fork 0
mirror of https://github.com/scylladb/scylladb.git synced 2026-06-06 15:03:06 +00:00
Code Issues Packages Projects Releases Wiki Activity
11,552 Commits 257 Branches 558 Tags
7deeffc9536b71ac80cf789bd6219893ad3e0d50
Go to file
Clone
Open with VS Code Open with VSCodium Open with Intellij IDEA
Download ZIP Download TAR.GZ Download BUNDLE
Raphael S. Carvalho 7deeffc953 database: serialize sstable cleanup 
We're cleaning up sstables in parallel. That means cleanup may need
almost twice the disk space used by all sstables being cleaned up,
if almost all sstables need cleanup and every one will discard an
insignificant portion of its whole data.
Given that cleanup is frequently issued when node is running out of
disk space, we should serialize cleanups in every shard to decrease
the disk space requirement.

Fixes #192.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <20170317022911.10306-1-raphaelsc@scylladb.com>
2017-03-19 12:33:03 +02:00
.github
Make github issue template less shouty
2016-06-01 10:45:04 +03:00
api
Merge seastar upstream
2017-02-01 14:39:08 +02:00
auth
db: Create default auth and tracing keyspaces using lowest timestamp
2017-03-07 19:19:15 +01:00
conf
batchlog: introduce batch_size_fail_threshold_in_kb option
2017-03-09 12:20:17 +01:00
cql3
Merge "Ensure base and view schema versions match" from Duarte
2017-03-17 12:40:00 +01:00
db
db/schema_tables: Atomically publish base and view changes
2017-03-15 16:35:07 +01:00
debug
debug: add systemtap script to measure interesting latencies during cache updates.
2017-01-26 22:15:16 -05:00
dht
dht: add hashing and comparison helpers to dht::decorarted_key
2017-01-20 11:24:14 +02:00
dist
Update scylla-ami submodule
2017-03-17 11:13:07 +02:00
docs
docs: tracing.md: initial commit
2016-10-10 16:12:02 +03:00
exceptions
exceptions: mark function called before construction static
2016-10-03 16:29:02 +03:00
gms
gms::inet_address: Add lookup functionality.
2017-02-06 09:45:37 +00:00
idl
commitlog: avoid copying column_mapping
2017-02-27 17:05:58 +00:00
interface
Fix pre-ScyllaDB copyright statements
2016-04-08 08:12:47 +03:00
io
Fix pre-ScyllaDB copyright statements
2016-04-08 08:12:47 +03:00
licenses
Merge "CQL 3.3.1 support" from Pekka
2017-01-09 11:54:45 +02:00
locator
Convert to use dht::partition_range_vector and dht::token_range_vector
2016-12-19 14:08:50 +08:00
message
messaging_service: Move log printout to actual listen start
2017-02-22 17:08:21 +01:00
repair
repair: Fix midpoint is not contained in the split range assertion in split_and_add
2017-03-09 09:09:17 +01:00
scripts
scripts/scylla_install_pkg: support Debian
2016-12-06 12:06:30 +02:00
seastar @ 6b21197fd2
Update seastar submodule
2017-03-16 13:32:45 +01:00
service
migration_manager: Atomically migrate table and views
2017-03-15 16:03:56 +01:00
sstables
compaction_manager: stop manager after storage io error
2017-03-16 10:37:47 +02:00
streaming
streaming::stream_manager: move a collectd counters registration to the metrics registration layer
2017-01-10 16:24:54 -05:00
swagger-ui @ 1b212bbe71
Update swagger-ui for local fix (change URL to not to point to pet store)
2015-06-25 14:04:07 +03:00
tests
Merge "Ensure base and view schema versions match" from Duarte
2017-03-17 12:40:00 +01:00
thrift
migration_manager: Atomically migrate table and views
2017-03-15 16:03:56 +01:00
tools/scyllatop
scyllatop: dump all output to stdout instead of running a fancy console interface
2016-08-31 08:31:36 +03:00
tracing
db: Create default auth and tracing keyspaces using lowest timestamp
2017-03-07 19:19:15 +01:00
transport
transport: make process_request_one() an execution stage
2017-03-09 09:27:43 +00:00
utils
lsa: Fix debug-mode compilation error
2017-03-16 18:23:05 +01:00
.gitattributes
Add .gitattributes file to classify C++ source
2015-10-05 08:51:51 +02:00
.gitignore
ScyllaTop: top-like tool to see live scylla metrics
2016-02-23 12:32:47 +02:00
.gitmodules
dist: move ComboAMI related code to scylla-ami
2015-09-22 00:17:42 +03:00
.gitorderfile
gitorderfile: make changes into *.py files appear first
2015-05-12 10:13:25 +03:00
atomic_cell_hash.hh
mutation_hasher: handle counter cells properly
2017-02-02 10:35:14 +00:00
atomic_cell_or_collection.hh
atomic_cell: introduce atomic_cell_mutable_view
2017-03-02 09:05:11 +00:00
atomic_cell.hh
atomic_cell: add COUNTER_IN_PLACE_REVERT flag
2017-03-02 09:05:11 +00:00
bytes_ostream.hh
bytes_ostream: make max_chunk_size() an inline function
2016-10-17 11:49:33 +03:00
bytes.cc
Fix pre-ScyllaDB copyright statements
2016-04-08 08:12:47 +03:00
bytes.hh
bytes: add bytes_mutable_view
2017-03-02 09:05:11 +00:00
caching_options.hh
Fix pre-ScyllaDB copyright statements
2016-04-08 08:12:47 +03:00
canonical_mutation.cc
idl/mutation: add counter serialisation logic
2017-02-02 10:35:14 +00:00
canonical_mutation.hh
remove db/serializer.hh includes
2016-03-02 09:07:09 +00:00
cartesian_product.hh
Fix pre-ScyllaDB copyright statements
2016-04-08 08:12:47 +03:00
cell_locking.hh
cell_locker: add metrics for lock acquisition
2017-03-02 09:05:12 +00:00
checked-file-impl.hh
Merge seastar upstream
2017-03-14 13:38:38 +02:00
clustering_bounds_comparator.hh
bound_view: Extract converters for range start and end bounds
2017-02-13 16:12:16 +01:00
clustering_key_filter.hh
partition_snapshot_reader: Emit only relevant tombstones
2017-02-13 16:12:15 +01:00
clustering_ranges_walker.hh
sstables: Extract and use clustering_ranges_walker
2017-03-10 14:42:22 +01:00
combine.hh
Fix pre-ScyllaDB copyright statements
2016-04-08 08:12:47 +03:00
compaction_strategy.hh
lcs: fix broken token range distribution at higher levels
2016-09-30 14:09:16 -03:00
compatible_ring_position.hh
compatible_ring_position: add function to return token
2016-12-08 14:25:29 -02:00
compound_compat.hh
composite: Use operator[] instead of at()
2016-07-28 11:11:54 +02:00
compound.hh
Replace iostream include with iosfwd in headers
2017-01-17 14:52:44 +02:00
compress.hh
cql3: Disable compression on empty properties
2016-11-09 10:03:59 +02:00
configure.py
Merge "database: fix the 'nodetool clearsnapshot'" from Vlad
2017-03-06 16:45:31 +02:00
CONTRIBUTING.md
CONTRIBUTING.md: add sections for help and issues
2016-11-18 22:21:10 +02:00
converting_mutation_partition_applier.hh
schema: Dense schemas are correctly upgrades
2016-08-03 13:39:01 +02:00
counters.cc
counters: attempt to apply in place
2017-03-02 09:05:11 +00:00
counters.hh
counters: attempt to apply in place
2017-03-02 09:05:11 +00:00
cql_serialization_format.hh
Replace iostream include with iosfwd in headers
2017-01-17 14:52:44 +02:00
database_fwd.hh
database: keep a pointer to the memtable list in a memtable
2016-11-21 18:18:27 +02:00
database.cc
database: serialize sstable cleanup
2017-03-19 12:33:03 +02:00
database.hh
database: Upgrade mutation to current schema to push view updates
2017-03-15 18:15:27 +01:00
db_clock.hh
db: Use microsecond precision for server-side timestamps
2016-12-21 18:03:22 +00:00
debug.hh
Fix pre-ScyllaDB copyright statements
2016-04-08 08:12:47 +03:00
digest_algorithm.hh
storage_proxy: avoid calculating digest when only one replica is contacted
2016-11-17 13:04:30 +02:00
disk-error-handler.cc
db: make it possible to use custom error handler with io checker
2016-10-27 15:54:21 -02:00
disk-error-handler.hh
db: make it possible to use custom error handler with io checker
2016-10-27 15:54:21 -02:00
Doxyfile
docs: exclude dpdk
2015-06-24 13:09:51 +03:00
enum_set.hh
Fix pre-ScyllaDB copyright statements
2016-04-08 08:12:47 +03:00
fnv1a_hasher.hh
add fnv1a hasher
2017-02-02 10:35:14 +00:00
frozen_mutation.cc
idl/mutation: add counter serialisation logic
2017-02-02 10:35:14 +00:00
frozen_mutation.hh
frozen_mutation: avoid buffer linearization and copy
2016-08-22 09:31:33 +01:00
frozen_schema.cc
idl: allow writers to use any output stream
2016-12-22 13:35:04 +01:00
frozen_schema.hh
remove db/serializer.hh includes
2016-03-02 09:07:09 +00:00
gc_clock.cc
Fix pre-ScyllaDB copyright statements
2016-04-08 08:12:47 +03:00
gc_clock.hh
gc_clock: Remove orphaned comment
2017-03-02 12:56:09 +02:00
hashing_partition_visitor.hh
mutation_hasher: handle counter cells properly
2017-02-02 10:35:14 +00:00
hashing.hh
feed_hash: allow additional arguments
2017-02-02 10:35:14 +00:00
idl-compiler.py
idl: fix generated writers when member functions are used
2017-02-27 17:05:58 +00:00
IDL.md
Add an IDL definition file
2016-01-24 12:29:21 +02:00
init.cc
main/init: Lookup inet addresses from config by dns lookup
2017-02-06 09:45:37 +00:00
init.hh
scylla tls: Add option support for client auth and tls opts
2017-02-06 09:45:09 +00:00
intrusive_set_external_comparator.hh
intrusive_set_external_comparator: avoid using boost::intrusive::value_traits_pointers
2017-01-10 18:16:56 +02:00
json.hh
Fix pre-ScyllaDB copyright statements
2016-04-08 08:12:47 +03:00
keys.cc
partition_key_view: Implement operator<<
2016-09-30 10:54:54 +02:00
keys.hh
keys: add memory_usage()
2016-11-18 11:25:36 +00:00
LICENSE.AGPL
Add the AGPL license
2015-09-20 10:45:35 +03:00
lister.cc
loggers should not have external linkage
2017-03-09 12:27:20 +01:00
lister.hh
database: move lister class into separate files
2017-02-17 17:50:40 -05:00
log.hh
Merge seastar upstream
2016-06-01 18:28:42 +03:00
main.cc
main: Name the http servers
2017-03-09 12:30:49 +02:00
map_difference.hh
map_difference: Allow on unordered_map
2016-04-20 09:54:06 +02:00
md5_hasher.hh
md5_hasher: add finalize_array()
2016-03-11 18:27:13 +00:00
memtable.cc
memtable: do not yield while holding reclaim_lock
2017-03-06 17:24:22 +01:00
memtable.hh
db: Enable creating forwardable readers via mutation_source
2017-02-23 18:50:44 +01:00
mutation_compactor.hh
db: Fix overflow of gc_clock time point
2017-03-01 18:49:56 +02:00
mutation_partition_applier.hh
mutations: Row tombstones are now a set of ranges
2016-06-02 16:21:59 +02:00
mutation_partition_serializer.cc
do not use long_type for counter update
2017-03-01 16:33:37 +00:00
mutation_partition_serializer.hh
idl: allow writers to use any output stream
2016-12-22 13:35:04 +01:00
mutation_partition_view.cc
do not use long_type for counter update
2017-03-01 16:33:37 +00:00
mutation_partition_view.hh
idl: switch to utils::input_stream
2016-08-22 09:31:33 +01:00
mutation_partition_visitor.hh
mutations: Row tombstones are now a set of ranges
2016-06-02 16:21:59 +02:00
mutation_partition.cc
mutation_query: add an execution stage
2017-03-09 09:27:43 +00:00
mutation_partition.hh
introduce counter_write_query()
2017-03-01 16:33:36 +00:00
mutation_query.cc
mutation_query: to_data_query_result enforces row limit
2016-12-15 10:56:40 +00:00
mutation_query.hh
mutation_query: add an execution stage
2017-03-09 09:27:43 +00:00
mutation_reader.cc
db: Enable creating forwardable readers via mutation_source
2017-02-23 18:50:44 +01:00
mutation_reader.hh
mutation_reader: make mutation_source nothrow movable
2017-03-09 09:27:43 +00:00
mutation.cc
mutation: Remove dead get_cell() function
2017-03-17 11:18:23 +02:00
mutation.hh
mutation: Remove dead get_cell() function
2017-03-17 11:18:23 +02:00
noexcept_traits.hh
Introduce noexcept_traits
2015-12-07 09:50:27 +01:00
NOTICE.txt
Add NOTICE file as required by the Apache license.
2014-12-24 09:47:18 +02:00
nway_merger.hh
Fix pre-ScyllaDB copyright statements
2016-04-08 08:12:47 +03:00
ORIGIN
Update ORIGIN for gossip and storage_service
2015-12-01 19:45:04 +08:00
partition_builder.hh
mutations: Row tombstones are now a set of ranges
2016-06-02 16:21:59 +02:00
partition_range_compat.hh
Convert to use dht::partition_range_vector and dht::token_range_vector
2016-12-19 14:08:50 +08:00
partition_slice_builder.cc
partition_slice_builder: Add with_ranges()
2017-02-23 18:50:53 +01:00
partition_slice_builder.hh
partition_slice_builder: Add with_ranges()
2017-02-23 18:50:53 +01:00
partition_version.cc
partition_version: Fix corruption of partition_version list
2016-10-18 09:25:38 +01:00
partition_version.hh
db: Enable creating forwardable readers via mutation_source
2017-02-23 18:50:44 +01:00
query_result_merger.hh
query_result_merger: Limit rows
2016-12-15 11:00:36 +00:00
query-request.hh
query: Introduce full_clustering_range
2017-02-23 18:50:53 +01:00
query-result-reader.hh
storage_proxy: avoid calculating digest when only one replica is contacted
2016-11-17 13:04:30 +02:00
query-result-set.cc
query: use result_view::consume() where appropriate
2016-08-22 09:31:33 +01:00
query-result-set.hh
Merge "Fix query digest mismatch" from Tomasz
2016-04-08 12:13:29 +03:00
query-result-writer.hh
idl: allow writers to use any output stream
2016-12-22 13:35:04 +01:00
query-result.hh
query: Fix invalid initialization of _memory_tracker by moving-from-self
2017-03-01 11:38:28 +00:00
query.cc
query: Introduce full_clustering_range
2017-02-23 18:50:53 +01:00
range_tombstone_list.cc
range_tombstone_list: Introduce erase()
2017-02-13 16:12:15 +01:00
range_tombstone_list.hh
range_tombstone_list: Introduce erase()
2017-02-13 16:12:15 +01:00
range_tombstone_to_prefix_tombstone_converter.hh
range_tombstone: check for adjacent instead of equal bounds
2016-06-20 21:29:48 +01:00
range_tombstone.cc
range_tombstone: Introduce end_position()
2017-02-13 16:12:16 +01:00
range_tombstone.hh
range_tombstone: Introduce end_position()
2017-02-13 16:12:16 +01:00
range.hh
Replace iostream include with iosfwd in headers
2017-01-17 14:52:44 +02:00
README-DPDK.md
README: fix typos and paramter syntax
2015-06-28 10:24:48 +03:00
README.md
README: Guidelines for contributing
2016-11-16 12:50:02 +02:00
release.cc
Fix pre-ScyllaDB copyright statements
2016-04-08 08:12:47 +03:00
release.hh
Fix pre-ScyllaDB copyright statements
2016-04-08 08:12:47 +03:00
reversibly_mergeable.hh
Fix pre-ScyllaDB copyright statements
2016-04-08 08:12:47 +03:00
row_cache.cc
db: Enable creating forwardable readers via mutation_source
2017-02-23 18:50:44 +01:00
row_cache.hh
db: Enable creating forwardable readers via mutation_source
2017-02-23 18:50:44 +01:00
schema_builder.hh
schema: Move raw_view_info outside of raw_schema
2017-03-15 15:38:31 +01:00
schema_mutations.cc
frozen_schema: Support view schemas
2016-12-20 13:06:11 +00:00
schema_mutations.hh
frozen_schema: Support view schemas
2016-12-20 13:06:11 +00:00
schema_registry.cc
schema_registry: Don't leak schemas
2017-02-21 09:56:21 +01:00
schema_registry.hh
schema_registry: Don't leak schemas
2017-02-21 09:56:21 +01:00
schema.cc
materialized views: Replace db::view::view class
2017-03-15 15:50:05 +01:00
schema.hh
materialized views: Replace db::view::view class
2017-03-15 15:50:05 +01:00
scylla-blocktune
blocktune: fix syntax error in exception handling
2016-10-23 16:40:00 +03:00
scylla-gdb.py
gdb: Cast gdb.Values to int
2017-03-08 19:43:48 +02:00
scylla-housekeeping
scylla-housekeeping: Create a uuid file if one is missing
2017-01-08 14:11:03 +02:00
SCYLLA-VERSION-GEN
build: improve support for custom builds
2017-01-22 14:56:52 +02:00
serialization_visitors.hh
idl: add start_frame() overload for seastar::simple_output_stream
2017-02-27 17:05:58 +00:00
serializer_impl.hh
serializer_impl: add serializer for bool_class<Tag>
2016-12-14 14:10:01 +00:00
serializer.hh
Merge seastar upstream
2016-09-28 17:34:16 +03:00
sstable_mutation_readers.hh
db: Enable creating forwardable readers via mutation_source
2017-02-23 18:50:44 +01:00
stdx.hh
Introduce stdx.hh header file
2016-08-17 11:19:49 +01:00
streamed_mutation.cc
streamed_mutation: Add streamed_mutation_returning() helper
2017-03-10 14:42:22 +01:00
streamed_mutation.hh
streamed_mutation: Add streamed_mutation_returning() helper
2017-03-10 14:42:22 +01:00
supervisor.cc
init: move supervisor_notify() out of main.cc
2017-01-06 10:10:55 +00:00
supervisor.hh
init: move supervisor_notify() out of main.cc
2017-01-06 10:10:55 +00:00
test.py
tests: add test for counter cell locker
2017-02-15 15:09:40 +00:00
timestamp.hh
db: Use microsecond precision for server-side timestamps
2016-12-21 18:03:22 +00:00
to_string.hh
to_string: add operator<< overload for std::array<>
2017-03-01 16:33:36 +00:00
tombstone.hh
db: Avoiding checking bloom filters during compaction
2016-07-10 09:54:20 +02:00
types.cc
collection_type_impl: Use set difference for tombstones
2017-03-15 14:34:01 +01:00
types.hh
cql3: Wire up functions for floating-point types
2017-03-16 11:04:59 +00:00
unimplemented.cc
unimplemented: Add materialized views
2016-10-18 01:18:52 +00:00
unimplemented.hh
Replace iostream include with iosfwd in headers
2017-01-17 14:52:44 +02:00
validation.cc
validation: Add KS validation + convinence methods
2016-04-19 11:49:05 +00:00
validation.hh
validation: Add KS validation + convinence methods
2016-04-19 11:49:05 +00:00
version.hh
version: Bump Cassandra version to 2.2.8
2017-01-09 10:42:21 +02:00
view_info.hh
materialized views: Replace db::view::view class
2017-03-15 15:50:05 +01:00

README.md

Scylla

Building Scylla

In addition to required packages by Seastar, the following packages are required by Scylla.

Submodules

Scylla uses submodules, so make sure you pull the submodules first by doing:

git submodule init
git submodule update --init --recursive

Building and Running Scylla on Fedora

  • Installing required packages:
sudo dnf install yaml-cpp-devel lz4-devel zlib-devel snappy-devel jsoncpp-devel thrift-devel antlr3-tool antlr3-C++-devel libasan libubsan gcc-c++ gnutls-devel ninja-build ragel libaio-devel cryptopp-devel xfsprogs-devel numactl-devel hwloc-devel libpciaccess-devel libxml2-devel python3-pyparsing lksctp-tools-devel protobuf-devel protobuf-compiler systemd-devel libunwind-devel
  • Build Scylla
./configure.py --mode=release --with=scylla --disable-xen
ninja-build build/release/scylla -j2 # you can use more cpus if you have tons of RAM
  • Run Scylla
./build/release/scylla
  • run Scylla with one CPU and ./tmp as data directory
./build/release/scylla --datadir tmp --commitlog-directory tmp --smp 1
  • For more run options:
./build/release/scylla --help

Building Fedora RPM

As a pre-requisite, you need to install Mock on your machine:

# Install mock:
sudo yum install mock

# Add user to the "mock" group:
usermod -a -G mock $USER && newgrp mock

Then, to build an RPM, run:

./dist/redhat/build_rpm.sh

The built RPM is stored in /var/lib/mock/<configuration>/result directory. For example, on Fedora 21 mock reports the following:

INFO: Done(scylla-server-0.00-1.fc21.src.rpm) Config(default) 20 minutes 7 seconds
INFO: Results and/or logs in: /var/lib/mock/fedora-21-x86_64/result

Building Fedora-based Docker image

Build a Docker image with:

cd dist/docker
docker build -t <image-name> .

Run the image with:

docker run -p $(hostname -i):9042:9042 -i -t <image name>

Contributing to Scylla

Guidelines for contributing

Reference in New Issue View Git Blame Copy Permalink
Description
No description provided
Readme 505 MiB
Languages
C++ 72.1%
Python 26.7%
CMake 0.3%
GAP 0.3%
Shell 0.3%