Commit Graph

6597 Commits

Author SHA1 Message Date
Calle Wilund
7856d7fe02 config: Change "auto_snapshot" to "used" 2015-09-30 09:09:42 +02:00
Calle Wilund
56228fba24 column family: Add "snapshot" operation. 2015-09-30 09:09:42 +02:00
Calle Wilund
428557a66d sstables: add "create_links" method
Adds hard links in requested directory to all components of the sstable
Used for snapshotting
2015-09-30 09:09:42 +02:00
Calle Wilund
cdaafb0505 sstables: Expose directory, max age and all active files 2015-09-30 09:09:42 +02:00
Calle Wilund
c141e15a4a column family: Add "run_with_compaction_disabled" helper
A'la origin. Could as well been RAII.
2015-09-30 09:09:41 +02:00
Calle Wilund
b3c95ce42d system_keyspace: Change truncation record method to use context qp
Align with rest of file (for better or worse). This allows calls from
entity without query_processor handy (i.e. storage_proxy).

Added "minimal" setup method for the "global" state, to facilitate
tests. Doing a full setup either in cql_test_env or after it is created
breaks badly. (Not sure why). So quick workaround.

Updated the current two users (batchlog_manager and commitlog_replayer)
callsites to conform.
2015-09-30 09:09:41 +02:00
Calle Wilund
3abd8b38b6 query_context: Expose query_processor (local) 2015-09-30 09:09:41 +02:00
Calle Wilund
0444029a16 cql_test_env: expose distributed db and query processor 2015-09-30 09:09:41 +02:00
Calle Wilund
713860602b cql3/maps.cc : implement maps::marker::bind
Needed for system table (truncation pos)
2015-09-30 09:09:41 +02:00
Avi Kivity
4ae6c8c875 Merge seastar upstream
* seastar 5fe596a...283901a (10):
  > Add filesystem "link_file"
  > scripts: posix_net_conf.sh: take care of the case with more than 32 CPUs
  > posix: Add explanatory string to throw_system_error_on()
  > tests: fix memory leak in thread_test test_thread_1
  > tests: fix memory leak in timertest cancellation test
  > README: require xfsprogs-devel
  > file: query dma alignment from OS
  > file: separate disk and memory dma alignment
  > scripts: posix_net_conf.sh: Exclude CPU0 in RDS config for EN.
  > README: Add missing Ubuntu 14.04 dependencies to README.md
2015-09-29 19:06:37 +03:00
Avi Kivity
0ec0e32014 Merge "ommitlog: preallocate segments" from Calle
"Modified version of the initial patch (which was reverted), further
reducing the possible delay states in CL allocation and segment management."
2015-09-29 17:02:54 +03:00
Glauber Costa
91408d3cbc warn users on 100 % CPU usage
Although it is technically a seastar problem, most complains about that is
coming from the Scylla side.  I prefer to keep the message here so we can reference
a Scylla issue.

Signed-off-by: Glauber Costa <glommer@scylladb.com>
2015-09-29 16:40:24 +03:00
Avi Kivity
c30feb714c Merge "gossip heart_beat_version + time to wait for seed" from Asias 2015-09-29 15:49:24 +03:00
Avi Kivity
d5d271a45b Merge "improvement on code that handles temporary TOC" from Raphael 2015-09-29 11:47:35 +03:00
Tomasz Grabiec
4863d16fb6 Merge tag 'bloom-memory' from git@github.com:glommer/scylla.git
From Glauber:

We will export the total memory used by the filter as its "off heap"
size for the purposes of statistics.
2015-09-29 10:05:17 +02:00
Glauber Costa
22294dd6a0 do not re-read sstable components after write
When we write an SSTable, all its components are already in memory. load() is
to big of a hammer.

We still want to keep the write operation separated from the preparation to
read, but in the case of a newly written SSTable, all we need to do is to open
the index and data file.

Fixes #300

Signed-off-by: Glauber Costa <glommer@scylladb.com>
2015-09-29 10:00:26 +02:00
Calle Wilund
0ec50e8d36 create_index_statement: Bugfix. Inverted logic in full index check 2015-09-29 09:47:15 +02:00
Avi Kivity
c52d9f8da4 db: fix circular reference collection_type_impl <-> cql3_type
cql2_type is a simple wrapper around data_type.  But some data_types
(collection_type_impl) contain a cql3_type as a cache to avoid recomputing
it, resulting in a circular reference.  This leaks memory when as_cql3_type()
is called.

Fix by using a static hash table for the cache.
2015-09-29 08:38:15 +02:00
Raphael S. Carvalho
549a9e2ed4 sstable: rename file_existence to file_exists
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
2015-09-28 15:49:57 -03:00
Raphael S. Carvalho
59506eba24 sstable: close file returned by open_file_dma in file_existence
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
2015-09-28 15:49:47 -03:00
Raphael S. Carvalho
da316c982d sstable: fsync cf dir before removing temporary toc
That's important to guarantee that all other components were
deleted before deleting TemporaryTOC.

Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
2015-09-28 15:49:24 -03:00
Glauber Costa
5dd0953bb9 api: implement filter off heap memory calculation
For us, everything is "off heap", so this will just be the total amount of
memory used by the filters.

Fixes #339

Signed-off-by: Glauber Costa <glommer@scylladb.com>
2015-09-28 16:44:26 +02:00
Glauber Costa
8b3a6f19a1 sstables: export filter memory size
Signed-off-by: Glauber Costa <glommer@scylladb.com>
2015-09-28 16:43:20 +02:00
Glauber Costa
cbedd9ee41 Export Bloom Filter's memory size
Do it so we can estimate how much memory it is being used by the filters. This
estimate is not 100 % correct: the implementation of the bloom_filter class
uses a thread-local variable that is common to all filters. We won't include
that in the estimate. But aside from that, it should be quite accurate.

Signed-off-by: Glauber Costa <glommer@scylladb.com>
2015-09-28 16:43:06 +02:00
Pekka Enberg
f43f0d6f04 keys: Add compound_wrapper::from_singular()
Clean up code by adding a from_singular() helper function to compound
wrapper and use it in.
2015-09-28 16:29:44 +02:00
Asias He
3a36ec33db gossip: Wait longer for seed node during boot up
When start a cluster on AWS, the seed node might get ready after
non-seed nodes is ready to contact it. Wait for seed node longer to make
the boot up process more robust.
2015-09-28 11:11:11 +08:00
Asias He
e43b2c2c89 api: Add get_current_heart_beat_version
curl -X GET "http://127.0.0.1:10000/gossiper/heart_beat_version/127.0.0.2"

This is useful to check if the gossip code is still running when
debugging.

Now we can get both the generation version and heart beat version of a
node.

curl -X GET "http://127.0.0.1:10000/gossiper/generation_number/127.0.0.2"
2015-09-28 09:38:33 +08:00
Asias He
817c138034 gossip: Add get_current_heart_beat_version interface
HTTP API will use it.
2015-09-28 09:38:22 +08:00
Gleb Natapov
f0c3caa43b Do not ignore exceptions during compaction
As comment explains if both read and write fails write exception is
ignored. To fix that create one exception that contains both errors.
2015-09-27 14:16:35 +03:00
Gleb Natapov
d53be0a91e Move operator<< for std::exception_ptr to std namespace and make it get const
If the operator is not in std namespace it cannot be found in non global
contexts.
2015-09-27 14:16:35 +03:00
Takuya ASADA
b2630db514 dist: remove rpm dependency to libvirt
This is for testing virtio mode, since we don't officially recommend to use virtio mode we should drop it.

Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
2015-09-25 17:14:37 -07:00
Gleb Natapov
140641689b messaging: do not use rpc client in error state
Using rpc client in error state will result in a message loss. Try to
reconnect instead.
2015-09-24 17:50:51 +02:00
Raphael S. Carvalho
ce855577b6 add compaction stats to collectd
With this change, we can see the number and length of compaction
activity per shard from collectd.

Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
2015-09-24 16:51:11 +02:00
Asias He
e77cea382e rpm: Improve rpm build scripts
This makes we can build in a centos container.
2015-09-23 21:42:51 -07:00
Tomasz Grabiec
1b1cfd2cbf tests: Introduce tests/memory_footprint_test 2015-09-23 21:27:44 -07:00
Tomasz Grabiec
d033cdcefe db: Move "Populating Keyspace ..." message from WARN to INFO level
WARN level is for messages which should draw log reader's attention,
journalctl highlights them for example. Populating of keyspace is a
fairly normal thing, so it should be logged on lower level.
2015-09-23 15:28:44 +02:00
Avi Kivity
b3b6fc2f39 Merge branch 'branch-0.9' 2015-09-23 06:27:55 -07:00
Avi Kivity
2b8a3c3f81 Merge seastar upstream
* seastar 66569fd...5fe596a (1):
  > memory: tolerate mbind failures

Fixes #390.
2015-09-23 06:27:23 -07:00
Asias He
ea007485d8 ami: Copy the rpm we just built only
If there previous multiple build/rpms, the build will fail like

cp a.rpm b.rpm c.rpm

c.rpm is not a directory.
2015-09-22 21:06:43 -07:00
Takuya ASADA
9c3db5cfa3 dist: change /var/lib/scylla/*/ permission to 755
Make them readable from other users.

Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
2015-09-22 21:06:42 -07:00
Asias He
91b0019b50 rpm: Setup irq, network queue binding and cpuset
When scylla is deployed on AWS's c4.8xlarge or c4.8xlarge large
instances, we can apply irq and network queue binding to achieve better
performance.

Also make scylla skip using cpu0 which will be busy serving network
interrupts under high workload.
2015-09-22 21:00:59 -07:00
Avi Kivity
36c3439fae Merge branch 'branch-0.9' 2015-09-22 05:33:32 -07:00
Avi Kivity
cbc0aa4916 README: drop 'urchin' codename in favor of Scylla 2015-09-22 05:32:55 -07:00
Pekka Enberg
3d0106aa69 dist/docker: Limit Scylla to a single CPU for now
Limit to single CPU to work around abysmal performance...

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
2015-09-22 05:10:38 -07:00
Avi Kivity
d69cb91bea dist: add libasan and libubsan as dependencies
Fixes mock Fedora 22 build; it needed them for the -fsanitize=vptr
detection.
2015-09-22 04:46:33 -07:00
Calle Wilund
4941d91063 Commitlog: add some more verbosity 2015-09-22 12:57:33 +02:00
Takuya ASADA
df995d1815 dist: change license to AGPL
Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
2015-09-22 10:39:39 +02:00
Paweł Dziepak
34e66e60c1 main: disable thrift by default
Fixes #205.

Signed-off-by: Paweł Dziepak <pdziepak@cloudius-systems.com>
2015-09-22 09:48:44 +02:00
Asias He
9ab0c1e321 README: Remove ubuntu build instructions
Scylla needs thrift and antlr3 which are not provided by ubuntu. We need
to compile them from source in order to build scylla. For now, let's
only support build on Fedora.
2015-09-21 18:25:18 -07:00
Avi Kivity
99e19a9f73 Merge branch 'branch-0.9' 2015-09-21 17:03:47 -07:00