Commit Graph

47 Commits

Author SHA1 Message Date
Avi Kivity
f3eade2f62 treewide: relicense to ScyllaDB-Source-Available-1.0
Drop the AGPL license in favor of a source-available license.
See the blog post [1] for details.

[1] https://www.scylladb.com/2024/12/18/why-were-moving-to-a-source-available-license/
2024-12-18 17:45:13 +02:00
Gleb Natapov
41a57ed2e8 streaming: move streaming code to use host ids instead of host ips
The patch is rather large, but it is a straightforward conversion from
one type to another.
2024-12-15 11:31:11 +02:00
Kefu Chai
6ead5a4696 treewide: move log.hh into utils/log.hh
the log.hh under the root of the tree was created keep the backward
compatibility when seastar was extracted into a separate library.
so log.hh should belong to `utils` directory, as it is based solely
on seastar, and can be used all subsystems.

in this change, we move log.hh into utils/log.hh to that it is more
modularized. and this also improves the readability, when one see
`#include "utils/log.hh"`, it is obvious that this source file
needs the logging system, instead of its own log facility -- please
note, we do have two other `log.hh` in the tree.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2024-10-22 06:54:46 +03:00
Asias He
1ca779d287 streaming: Fix use after move in fire_stream_event
The event is used in a loop.

Found by clang-tidy:

```
streaming/stream_result_future.cc:80:49: warning: 'event' used after it was moved [bugprone-use-after-move]
        listener->handle_stream_event(std::move(event));
                                                ^
streaming/stream_result_future.cc:80:39: note: move occurred here
        listener->handle_stream_event(std::move(event));
                                      ^
streaming/stream_result_future.cc:80:49: note: the use happens in a later loop iteration than the move
        listener->handle_stream_event(std::move(event));
                                                ^
```

Fixes #18332

Closes scylladb/scylladb#18333
2024-04-25 16:48:54 +03:00
Kefu Chai
a439ebcfce treewide: include fmt/ranges.h and/or fmt/std.h
before this change, we rely on the default-generated fmt::formatter
created from operator<<, but fmt v10 dropped the default-generated
formatter.

in this change, we include `fmt/ranges.h` and/or `fmt/std.h`
for formatting the container types, like vector, map
optional and variant using {fmt} instead of the homebrew
formatter based on operator<<.
with this change, the changes adding fmt::formatter and
the changes using ostream formatter explicitly, we are
allowed to drop `FMT_DEPRECATED_OSTREAM` macro.

Refs scylladb#13245

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2024-04-19 22:56:16 +08:00
Kefu Chai
34259a03d0 treewide: use consteval string as format string when formatting log message
seastar::logger is using the compile-time format checking by default if
compiled using {fmt} 8.0 and up. and it requires the format string to be
consteval string, otherwise we have to use `fmt::runtime()` explicitly.

so adapt the change, let's use the consteval string when formatting
logging messages.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#16612
2024-01-02 19:08:47 +02:00
Kefu Chai
0cb842797a treewide: do not define/capture unused variables
these warnings are found by Clang-17 after removing
`-Wno-unused-lambda-capture` and '-Wno-unused-variable' from
the list of disabled warnings in `configure.py`.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2023-02-15 22:57:18 +02:00
Benny Halevy
314e45d957 streaming: define plan_id as a strong tagged_uuid type
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2022-08-22 19:45:30 +03:00
Avi Kivity
fcb8d040e8 treewide: use Software Package Data Exchange (SPDX) license identifiers
Instead of lengthy blurbs, switch to single-line, machine-readable
standardized (https://spdx.dev) license identifiers. The Linux kernel
switched long ago, so there is strong precedent.

Three cases are handled: AGPL-only, Apache-only, and dual licensed.
For the latter case, I chose (AGPL-3.0-or-later and Apache-2.0),
reasoning that our changes are extensive enough to apply our license.

The changes we applied mechanically with a script, except to
licenses/README.md.

Closes #9937
2022-01-18 12:15:18 +01:00
Pavel Emelyanov
307a2583ee stream_result_future: Use local manager reference
The reference is present in all the required places already.

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
2021-11-24 12:17:37 +03:00
Pavel Emelyanov
5b748a72de stream_result_future: Keep stream_manager reference
The stream_result_future needs manager to register on it and to
unregister from it. Also the result-future is referenced from
stream_session that also needs the manager (see next patches).

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
2021-11-24 12:17:37 +03:00
Avi Kivity
7abd105d79 streaming: replace seastar::sprint() with fmt::format()
sprint() is obsolete.
2021-10-27 17:02:00 +03:00
Avi Kivity
a55b434a2b treewide: extent copyright statements to present day 2021-06-06 19:18:49 +03:00
Rafael Ávila de Espíndola
c0072eab30 everywhere: Be more explicit that we don't want std::make_shared
If sstring is made an alias to std::string ADL causes std::make_shared
to be found. Explicitly ask for ::make_shared.

Signed-off-by: Rafael Ávila de Espíndola <espindola@scylladb.com>
2020-03-10 13:13:48 -07:00
Asias He
6b7344f6e5 streaming: Fix typo in stream_result_future::maybe_complete
s/progess/progress/

Refs: #5437
2019-12-16 11:12:03 +02:00
Botond Dénes
136fc856c5 treewide: silence discarded future warnings for questionable discards
This patches silences the remaining discarded future warnings, those
where it cannot be determined with reasonable confidence that this was
indeed the actual intent of the author, or that the discarding of the
future could lead to problems. For all those places a FIXME is added,
with the intent that these will be soon followed-up with an actual fix.
I deliberately haven't fixed any of these, even if the fix seems
trivial. It is too easy to overlook a bad fix mixed in with so many
mechanical changes.
2019-08-26 19:28:43 +03:00
Avi Kivity
fd513c42ad streaming: convert sprint() to format()
sprint() recently became more strict, throwing on sprint("%s", 5). Replace
with the more modern format().

Mechanically converted with https://github.com/avikivity/unsprint.
2018-11-01 13:16:17 +00:00
Asias He
caad7ced23 streaming: Less verbose in logging
Now, we will have large number of small streaming. Make the
not very important logging message debug level.
2017-07-18 11:17:09 +08:00
Asias He
d0dffd7346 streaming: Better stats
Log the number of bytes streamed and streaming bandwidth summary in the same line with session
complete message.
2017-07-18 11:17:09 +08:00
Asias He
49229964d0 streaming: Add streaming plan name when session is failed
Before:
[shard 0] stream_session - [Stream #fc1b66e0-b75b-11e6-b295-000000000000]
Stream failed, peers={127.0.0.1, 127.0.0.2}

After:
[shard 0] stream_session - [Stream #fc1b66e0-b75b-11e6-b295-000000000000]
Stream failed for streaming plan repair-in-29, peers={127.0.0.1, 127.0.0.2}
2016-12-05 08:20:18 +08:00
Asias He
1c47e26913 streaming: Add streaming plan name when all sessions are completed
Before:
[shard 0] stream_session - [Stream #e050b710-b758-11e6-9321-000000000000]
All sessions completed, peers={127.0.0.2}

After:
[shard 0] stream_session - [Stream #e050b710-b758-11e6-9321-000000000000]
All sessions completed for streaming plan repair-in-32, peers={127.0.0.2}
2016-12-05 08:20:18 +08:00
Asias He
984f427cb5 streaming: Log streaming bandwidth
It looks like:

[Stream #f3907fd0-a557-11e6-a583-000000000000] Session with 127.0.0.1 is complete, state=COMPLETE
[Stream #f3907fd0-a557-11e6-a583-000000000000] Session with 127.0.0.2 is complete, state=COMPLETE
[Stream #f3907fd0-a557-11e6-a583-000000000000] Session with 127.0.0.3 is complete, state=COMPLETE
[Stream #f3907fd0-a557-11e6-a583-000000000000] bytes_sent = 393284364, bytes_received = 0, tx_bandwidth = 17.048 MiB/s, rx_bandwidth = 0.000 MiB/s
[Stream #f3907fd0-a557-11e6-a583-000000000000] All sessions completed, peers={127.0.0.1, 127.0.0.2, 127.0.0.3}

Fixes #1826
2016-12-05 08:20:18 +08:00
Pekka Enberg
38a54df863 Fix pre-ScyllaDB copyright statements
People keep tripping over the old copyrights and copy-pasting them to
new files. Search and replace "Cloudius Systems" with "ScyllaDB".

Message-Id: <1460013664-25966-1-git-send-email-penberg@scylladb.com>
2016-04-08 08:12:47 +03:00
Asias He
8060b97d67 streaming: Log number of bytes sent and recevied when stream_plan completes
It is useful for test code to verify number of bytes sent/received.

It looks like below in the log.

/tmp/out1:INFO  [shard 0] stream_session - \
[Stream #1f3e23f0-db9e-11e5-9cfb-000000000000] bytes_sent = 0, bytes_received = 15760704

/tmp/out2:INFO  [shard 0] stream_session - \
[Stream #1f3e23f0-db9e-11e5-9cfb-000000000000] bytes_sent = 0, bytes_received = 18203964

/tmp/out3:INFO  [shard 0] stream_session - \
[Stream #1f3e23f0-db9e-11e5-9cfb-000000000000] bytes_sent = 33964668, bytes_received = 0
2016-02-26 17:38:37 +08:00
Asias He
cb92fe75e6 streaming: Introduce get_session helper
To simplify streaming verb handler.

- Use get_session instead of open coded logic to get get_coordinator and
  stream_session in all the verb handlers

- Use throw instead of assert for error handling

- init_receiving_side now returns a shared_ptr<stream_result_future>
2016-01-29 16:31:07 +08:00
Asias He
aa69d5ffb2 streaming: Drop update_progress in stream_coordinator
Since we have session_info inside stream_session now, we can call
update_progress directly in stream_session.
2016-01-29 16:31:07 +08:00
Asias He
46bec5980b streaming: Put session_info inside stream_session
It is 1:1 mapping between session_info and stream_session. Putting
session_info inside stream_session, we can get rid of the
stream_coordinator::host_streaming_data class.
2016-01-29 16:31:07 +08:00
Asias He
048965ea02 streaming: Do not print session_index in handle_session_prepared
session_index is always 0. It will be removed soon.
2016-01-25 16:24:16 +08:00
Asias He
bc4ac5004e streaming: Kill stream_result_future::create_and_register
The helper is used only once in init_sending_side and in
init_receiving_side we do not use create_and_register to create
stream_result_future. Kill the trivial helper to make the code more
consistent.

In addition, rename variables "future" and "f" to sr (streaming_result).
2016-01-25 11:38:13 +08:00
Asias He
face74a8f2 streaming: Rename stream_result_future::init to ::init_sending_side
So we have:

- init_sending_side
  called when the node initiates a stream_session

- init_receiving_side
  called when the node is a receiver of a stream_session initiated by a peer
2016-01-25 11:38:13 +08:00
Asias He
9a346d56b9 streaming: Drop unnecessary parameters in stream_init_message
- from
  We can get it form the rpc::client_info

- session_index
  There will always be one session in stream_coordinator::host_streaming_data with a peer.

- is_for_outgoing
  In cassandra, it initiates two tcp connections, one for incoming stream and one for outgoing stream.
  logger.debug("[Stream #{}] Sending stream init for incoming stream", session.planId());
  logger.debug("[Stream #{}] Sending stream init for outgoing stream", session.planId());
  In scylla, it only initiates one "connection" for sending, the peer initiates another "connection" for receiving.
  So, is_for_outgoing will also be true in scylla, we can drop it.

- keep_ss_table_level
  In scylla, again, we stream mutations instead of sstable file. It is
  not relevant to us.
2016-01-25 11:38:13 +08:00
Asias He
bdd6a69af7 streaming: Drop unused parameters
- int connections_per_host

Scylla does not create connections per stream_session, instead it uses
rpc, thus connections_per_host is not relevant to scylla.

- bool keep_ss_table_level
- int repaired_at

Scylla does not stream sstable files. They are not relevant to scylla.
2016-01-25 11:38:13 +08:00
Asias He
88e99e89d6 streaming: Add more debug info
- Add debug for the peer address info
- Add debug in stream_transfer_task and stream_receive_task
- Add debug when cancel the keep_alive timer
- Add debug for has_active_sessions in stream_result_future::maybe_complete
2016-01-22 07:43:16 +08:00
Asias He
13934140f6 streaming: Remove bogus file size info for prepare completed
Scylla does not streaming sstable files directly for streaming. The file
size info is incorrect, let's get rid of it.
2015-11-10 15:39:34 +08:00
Avi Kivity
d5cf0fb2b1 Add license notices 2015-09-20 10:43:39 +03:00
Asias He
804564fe76 streaming: Remove completed stream in stream_manager 2015-07-31 16:27:55 +08:00
Asias He
1749305165 streaming: Fix stream_result_future::create_and_register
It is used to create stream initiated by us.
2015-07-31 16:27:55 +08:00
Asias He
59cae82470 streaming: Make stream_plan::execute return a future
Returns a ready future if the stream_plan completes successfully, or a
failed future otherwise.
2015-07-31 16:27:55 +08:00
Asias He
3a5af0a7fb streaming: Fix stream_coordinator::is_receiving
Unlike Origin, We can not use _connections_per_host == 0 to indicate we
are a follower since we set _connections_per_host to 1 for follower too.
2015-07-31 16:27:55 +08:00
Asias He
93f17024a8 streaming: Fix a logger printout
Before:

[Stream #136c5310-2f7a-11e5-87df-000000000000 ID#0]
[Stream #136c5310-2f7a-11e5-87df-000000000000, ID#0]

After:

[Stream #136c5310-2f7a-11e5-87df-000000000000 ID#0]
[Stream #136c5310-2f7a-11e5-87df-000000000000 ID#0]
2015-07-21 16:12:54 +08:00
Asias He
0e5fa35bd2 streaming: Register to stream_manager in create_and_register 2015-07-21 16:12:54 +08:00
Asias He
1f5feee5f2 streaming: Implement stream_result_future::handle_progress 2015-07-21 16:12:54 +08:00
Asias He
d74e414d09 streaming: Implement stream_result_future::handle_session_complete 2015-07-21 16:12:54 +08:00
Asias He
5e6f84cba8 streaming: Implement stream_result_future::handle_session_prepared
Instead of playing the game of casting between stream_event and derived
class. We overload handle_stream_event with derived stream_event class.

virtual void handle_stream_event(session_complete_event event) {}
virtual void handle_stream_event(progress_event event) {}
virtual void handle_stream_event(session_prepared_event event) {}

Also, make the virtual function non pure virtual, so user can override
the interested event only without defining all of the three.
2015-07-21 16:12:54 +08:00
Asias He
d9b7cb4142 streaming: Enable logger in stream_result_future 2015-07-21 16:12:54 +08:00
Asias He
fd04337e80 streaming: Wire up register_receiving init_receiving_side
Now stream_result_future can create a stream_coordinator if not
provided.

So
 - On sending side, stream_coordinator is created by stream_plan
 - On receiving side, stream_coordinator is created by stream_result_future
2015-07-14 20:41:14 +08:00
Asias He
1828019d60 streaming: Convert StreamManager.java to C++ 2015-07-14 20:41:14 +08:00