scylladb/docs/dev/IDL.md

# IDL compiler

IDL compiler is a tool written in Python, that generates utility serialization/de-serialization code (specializations
for `ser::serializer<T>` class) for C++ classes and enums. It takes an IDL definition file as input and generates two
files from it: `<mod_name>.dist.hh`, which is declarations part, and `<mod_name>.dist.impl.hh`, which contains generated
code definitions (private part).

The syntax of IDL is similar to C++ to some degree (e.g. contains `struct/class` and `enum class` constructs), but is
also extended to support more complex things in the context of RPC serialization: RPC messages (also called
"RPC verbs").

By default, all generated code for Scylla is created under `ser` namespace.

## File syntax description

As noted above, the syntax of IDL definitions file resembles C++ by providing similar-looking code constructs for class
and enum serializers.

The IDL file contains a sequence of includes and entity declarations, most of which correspond directly to those from the actual
C++ code:

- Includes (`include`)
- Namespaces (`namespace`)
- Classes / structures (`class` / `struct`)
- Enums (`enum class`)
- RPC verbs (`verb`)

Classes can be templated, supporting the ordinary `template <typename T>`-style notation at the beginning of the class
declaration, similar to C++.

Some places (e.g. classes, class members and RPC verbs) also support modifying generation behavior by providing
attribute sequences (via C++ syntax of `[[attr]]` form).

### Including files nad modules

Data types that are defined elsewhere can be used in IDL definition files.

To include definitions from C++ headers, use the `#include "filename.hh"` statement.
The idl compiler generates a respective `#include "filename.hh"` directive in the generated `<mod_name>.dist.hh` file.

To import definitions from other IDL definition files, use the `#include "idl/<other_mod_name>.idl.hh"` statement.
The idl compiler generates a respective `#include "idl/<other_mod_name>.dist.hh"` directive in the generated `<mod_name>.dist.hh` file
and a `#include "idl/<other_mod_name>.dist.impl.hh"` directive in the generated `<mod_name>.dist.impl.hh` file

### Namespaces

Namespaces in IDL act the same way as in C++ or any other programming language, and can have arbitrary nesting depth.
Syntax:

```
namespace <ns-name> { <namespace-body> }
```

Where:

- `ns-name` — namespace identifier. Directly corresponds to the namespace from C++, defining a new namespace or
  extending an existing one.
- `namespace-body` — a sequence of 0+ nested entities of any kind: classes, enums, RPC verbs or other nested
  namespaces.

Example:

```
namespace ns {

class my_class {
    int a;
    bool b;
    std::vector<int> c;
};

enum class e {
    E1,
    E2
};

// ...

} // namespace ns
```

### Classes

Class declaration creates a `ser::serializer<T>` specialization для for a class with a given name, along with an
implementation of `write`, `read` and `skip` methods. Syntax:

```
template <template-parameter-list> (optional)
class-key <class-name> final(optional) attributes_seq(optional) stub(optional) { <nested-class> | <member-specification> } ;(optional)
```

Where:

- `template-parameter-list` - the list of template arguments in case a class is a template.
- `class-key` - either `class`, or `struct`.
- `class-name` - class identifier, for which `ser::serializer<T>` specialization is to be created.
- `final` specifier — an optimization option, which denotes that a class is final, i.e. cannot be extended in the
  future, in which case the size is not serialized. Should be used with care.
- `stub` specifier — skip generating serialization code for a class. Initially designed for documentation within IDL
  (there is one exception to that with `[[writable]]` attribute, though. More on that below).
- `attributes_seq` - optional sequence of C++-style attributes. Only `[[writable]]` attribute is supported at the
  moment, which means that: if specified, the writers and serialization  views will also be generated for a class
  (inside the private `<mod_name>.dist.impl.hh` part). Other attributes are ignored.
- `nested-class` - nested class definition following the same syntax.
- `member-specification` — data members and getter functions declarations (more details on the syntax below). Getter
  functions should be used in cases where access to private class fields is needed. Both kinds of field accessors can 
  also be marked with `[[version id]]` version attribute, which denotes, that a field is accessible starting from
  version `id`.

A class declaration can optionally include a semicolon at the end to more closely resemble C++ syntax.

If a class contains both `stub` specifier and `[[writable]]` attribute at the same time, the `ser::serializer` code is
not generated but the serialization views (classes with `_view` suffix in the name, that support reading and writing
data in the stream according to some fixed data layout) are created, nonetheless.

Class members are declared the following way:

```
<type> name <getter-marker>(optional) [[version version-id]](optional) <default-value>(optional);
```

Where:

- `type` - any valid C++ type, following the regular C++ syntax. Naturally, a serializer specialization for this
  type should exist in order to serialize/deserialize it.
- `name` - accessor name. For ordinary data fields it is just a C++ name of a class field. In case it's a getter 
  function, it also should contain a "getter marker", which is denoted as empty `()` braces sequence right after the
  name. As noted above, getter functions should be used if a field is not accessible (i.e. is private), otherwise a
  regular data field can be used. Note, that getter functions can (and probably should) be const methods.
- version attribute — specify that a field is accessible starting from version `version-id` and above.
- `default-value` — an optional clause to specify default value for a field accessor. The syntax is: `= <value>`.

### Enums

Analogous to classes, `ser::serializer<T>` specializations can also be generated for enums. Declaration syntax:

```
enum class identifier enum-base { enumerator-list } ;(optional)
```

Where:

- `identifier` - the name of C++ enum class.
- `enum-base` - mandatory specification of C++ underlying type for the enum, following the regular C++ syntax: `: integer-type`
- `enumerator-list` - a list of enum cases or initializers of the following form: `name = integer-value`, where
  `integer-value` is a plain integer literal value.

Note that although C++ allows `constexpr` as an initializer value, it makes the documentation less readable, hence is
not permitted.

### RPC Verbs

IDL can also contain declarations of RPC messages with a given signature (also called: "RPC verbs"). It allows to
automatically generate boilerplate code for message handlers registration and message passing code via an instance of
`netw::messaging_service` class. Declaration syntax:

```
verb id [[attributes...]](optional) (parameters...) (-> return-type...)(optional) ;(optional)
```

There should be a corresponding upper-cased enumerator `ID` inside the `netw::messaging_verb` enum for a verb with name
`id`. For example, for a `my_verb` declaration there should be a corresponding `netw::messaging_verb::MY_VERB` constant
to specify an id for the RPC client.

The parameters of the verb declarations will also act as parameters for the handler functions and corresponding `send`
functions. In case `[[with_timeout]]` attribute is set, the argument list is extended with a `time_point` argument at the
beginning of the parameter list to specify an RPC timeout when sending or handling the message.

The return value type is calculated as `future<return_type>` if only one type is present and
`future<rpc::tuple<type1, type2, ...>>` if there are more than one. It is used as return type for message handler and `send`
function. If the `-> return type` clause is omitted, the return type is assumed to be `future<>`. If `[[one_way]]`
attribute is specified, handler function return type is fixed to `future<rpc::no_wait_type>`, and the return type
clause should not be used, otherwise an exception will be thrown during  IDL generation process.

If `[[with_client_info]]` attribute is used then the handler will contain a const reference to  an `rpc::client_info` as
its first argument.

`[[with_client_info]]`, `[[with_timeout]]` and `[[one_way]]` attributes can be combined.

For an RPC verb with the definition of `verb x (arg1_t, arg2_t) -> ret_t;` , which is defined in some `my_mod.idl.hh`
module, the following `my_mod_rpc_verbs` class will be generated (approximately):

```cpp
// my_mod.dist.hh

namespace ser {

struct my_mod_rpc_verbs {
	static void register_x(netw::messaging_service* ms, std::function<future<ret_t> (const rpc::client_info&, arg1_t, arg2_t>&&);
	static future<> unregister_x(netw::messaging_service* ms);
	static future<ret_t> send_x(netw::messaging_service* ms, netw::msg_addr id, arg1_t, arg2_t);

	// calls unregister_x, and the same for other verbs, if there are any,
  // and waits for all of them to resolve.
	static future<> unregister(netw::messaging_service* ms);
};

} // namespace ser
```

Each parameter can optionally have a name, otherwise a placeholder name of form `_N` will be used, where `N` is the
index within the RPC verb parameters list. Also, each argument and return type can be annotated with `[[version id]]`
attribute, which will cause it to be accepted as a `rpc::optional<>` in the handler function signature or returned by send()
function respectively,

If a parameter is annotated with [[ref]] attribute send function will receive it as a const reference, but a handler
will still receive it by value.

Return type value can be also annotated with `[[unique_ptr]]` or `[[lw_shared_ptr]]` attributes. If the attribute is present
handler function's return value will have the type as `foreign_ptr<unique_ptr<>>` and `foreign_ptr<lw_shared_ptr<>>` respectively.

## IDL example
Forward slashes comments are ignored until the end of the line.
```
namespace utils {
// An example of a stub class
class UUID stub {
    int64_t most_sig_bits;
    int64_t least_sig_bits;
}
}

namespace gms {
//an enum example
enum class application_state:int {STATUS = 0,
        LOAD,
        SCHEMA,
        DC};

// example of final class
class versioned_value final {
// getter and setter as public member
    int version;
    sstring value;
}

class heart_beat_state {
//getter as function
    int32_t get_generation();
//default value example
    int32_t get_heart_beat_version() = 1;
}

class endpoint_state {
    heart_beat_state get_heart_beat_state();
    std::map<application_state, versioned_value> get_application_state_map();
}

class gossip_digest {
    inet_address get_endpoint();
    int32_t get_generation();
//mark that a field was added on a specific version
    int32_t get_max_version() [ [version 0.14.2] ];
}

class gossip_digest_ack {
    std::vector<gossip_digest> digests();
    std::map<inet_address, gms::endpoint_state> get_endpoint_state_map();
}
}
```