mirrors/scoutfs
1
0
Fork 0
mirror of https://github.com/versity/scoutfs.git synced 2026-09-07 08:37:03 +00:00
Code Issues Packages Projects Releases Wiki Activity

scoutfs: refine quorum voting

Browse Source 
The current quorum voting implementatoin had some rough edges that
increased the complexity of the system and introduced undesirable
failure modes.  We can keep the same basic pattern but move
functionality around a few places, and rethink the quorum voting, to end
up with a meaningfully simpler system.

The motivation for this work was to remove the need to provide a
uniq_name option for every mount instance.

The first big change is to remove the idea of static configuration slots
for mounts.  This removes the use of uniq_name.  Mounts now simply have
a server_addr mount option instead of using their uniq_name to find
their address in the configuration.

The server can't check the configuration to see if a given connected
client's name is found in the quorum config.  Clients can set a flag in
their sent greeting which indicates that they're a voter.  This removes
the uniq_name from the greeting and mounted client records.

Without a static configuration mounts no longer have dedicated block
locations to write to.  We increase the size of the region of quorum
blocks and have voters simply write to a random block.  Overwriting vote
blocks is OK because we move from heartbeating design patterns to a
protocol strongly based on raft's election.  We're using quorum blocks
to communicate votes instead of network messages and overwriting blocks
is analagous to lossy networks droping vote messages in the raft
election protocol.

We were using the dedicated per-mount quorum blocks to track mounts that
had been elected and needed to be fenced.  We no longer have that
storage so instead we add the idea of an election log that is stored in
every voting block.  Readers merge the logs from all the blocks they
read and write the resulting merged log in their block.

With no static quorum configuration we no longer have to worry about the
complexity of changing the slot configurations while they're in use.
The only persistent configuration is the number of votes a candidate
needs to be elected by a quorum.

It was a mistake to use quorum voting blocks to communicate state
between the server and the quorum voters.  We can easily move the
unmount_barrier, server address, and fencing state from the quorum
blocks into the super block.  The server no longer needs the quorum
election info struct to be able to later write its quorum block.  It
instead writes a few fields in the super.  There's only one place where
clients need to look to find out who they should connect to or if they
can finish unmount.

Signed-off-by: Zach Brown <zab@versity.com>
This commit is contained in:
Zach Brown
2019-08-20 15:52:13 -07:00
committed by Zach Brown
parent 5929a36747
commit 5b258cee3b
9 changed files with 874 additions and 997 deletions
Download Patch File Download Diff File Expand all files Collapse all files
kmod/src/client.c
+74 -84
View File
@@ -53,10 +53,7 @@ struct client_info {
atomic_t shutting_down;
struct workqueue_struct *workq;
struct work_struct connect_work;
struct scoutfs_quorum_elected_info qei;
u64 old_elected_nr;
struct delayed_work connect_dwork;
u64 server_term;
u64 greeting_umb;
@@ -373,117 +370,108 @@ out:
}
/*
* If the previous election told us to start the server then stop it and
* clear the indication that we were elected. We get the current
* version of the election info from the server because they might have
* modified it while they were running. the old election info.
* This work is responsible for maintaining a connection from the client
* to the server. It's queued on mount and disconnect and we requeue
* the work if the work fails and we're not shutting down.
*
* If we're not fast enough to clear the election from the quorum block
* then the next server might fence us. Should be very unlikely as
* election requires multiple RMW cycles.
*/
static void stop_our_server(struct super_block *sb,
struct scoutfs_quorum_elected_info *qei)
{
if (qei->run_server) {
scoutfs_server_stop(sb, qei);
scoutfs_quorum_clear_elected(sb, qei);
memset(qei, 0, sizeof(*qei));
}
}
/*
* This work is responsible for managing leader elections, running the
* server, and connecting clients to the server.
*
* In the typical case a mount reads the quorum blocks and finds the
* In the typical case a mount reads the super blocks and finds the
* address of the currently running server and connects to it.
* Non-voting clients who can't connect will keep trying alternating
* reading the address and getting connect timeouts.
*
* More rarely clients who aren't connected and are configured to
* participate in quorum need to elect the new leader. The elected info
* filled by quorum tells us if we were elected to run the server.
* Voting mounts will try to elect a leader if they can't connect to the
* server. When a quorum can't connect and are able to elect a leader
* then a new server is started. The new server will write its address
* in the super and everyone will be able to connect.
*
* This leads to the possibility that the mount who is running the
* server had its mount disconnect. This is only weirdly different from
* other clients disconnecting and trying to reconnect because of the
* way quorum slots are reconfigured and reclaimed. If we connect to a
* server with the new quorum config then we can't have any old servers
* running in the stale old quorum slot. The simplest way to do this is
* to *always* stop the server if we're running it and we got
* disconnected. It's a big hammer, but it's reliable, and arguably if
* *we* couldn't' use *our* server then something bad is happening and
* someone else should be the server.
*
* This only executes on mount, error, or as a connection disconnects
* and there's only ever one executing.
* There's a tricky bit of coordination required to safely unmount.
* Clients need to tell the server that they won't be coming back with a
* farewell request. Once a client receives its farewell response it
* can exit. But a majority of clients need to stick around to elect a
* server to process all their farewell requests. This is coordinated
* by having the greeting tell the server that a client is a voter. The
* server then holds on to farewell requests from voters until only
* requests from the final quorum remain. These farewell responses are
* only sent after updating an unmount barrier in the super to indicate
* to the final quorum that they can safely exit without having received
* a farewell response over the network.
*/
static void scoutfs_client_connect_worker(struct work_struct *work)
{
struct client_info *client = container_of(work, struct client_info,
connect_work);
connect_dwork.work);
struct super_block *sb = client->sb;
struct scoutfs_quorum_elected_info *qei = &client->qei;
struct scoutfs_sb_info *sbi = SCOUTFS_SB(sb);
struct scoutfs_super_block *super = &sbi->super;
struct scoutfs_super_block *super = NULL;
struct mount_options *opts = &sbi->opts;
const bool am_voter = opts->server_addr.sin_addr.s_addr != 0;
struct scoutfs_net_greeting greet;
struct sockaddr_in sin;
ktime_t timeout_abs;
u64 elected_term;
int ret;
/* don't try quorum and connecting while our mount runs a server */
stop_our_server(sb, qei);
super = kmalloc(sizeof(struct scoutfs_super_block), GFP_NOFS);
if (!super) {
ret = -ENOMEM;
goto out;
}
timeout_abs = ktime_add_ms(ktime_get(), CLIENT_QUORUM_TIMEOUT_MS);
ret = scoutfs_quorum_election(sb, opts->uniq_name,
client->old_elected_nr,
timeout_abs, client->sending_farewell,
client->greeting_umb, qei);
ret = scoutfs_read_super(sb, super);
if (ret)
goto out;
/* we saw that the server wrote a new unmount barrier */
if (client->sending_farewell && qei->elected_nr == 0 &&
qei->unmount_barrier > client->greeting_umb) {
/* can safely unmount if we see that server processed our farewell */
if (am_voter && client->sending_farewell &&
(le64_to_cpu(super->unmount_barrier) > client->greeting_umb)) {
client->farewell_error = 0;
complete(&client->farewell_comp);
ret = 0;
goto out;
}
if (qei->run_server) {
ret = scoutfs_server_start(sb, &qei->sin, qei->elected_nr, qei);
if (ret) {
/* forget that we tried to start the server */
memset(qei, 0, sizeof(*qei));
/* try to connect to the super's server address */
scoutfs_addr_to_sin(&sin, &super->server_addr);
if (sin.sin_addr.s_addr != 0 && sin.sin_port != 0)
ret = scoutfs_net_connect(sb, client->conn, &sin,
CLIENT_CONNECT_TIMEOUT_MS);
else
ret = -ENOTCONN;
/* voters try to elect a leader if they couldn't connect */
if (ret < 0) {
/* non-voters will keep retrying */
if (!am_voter)
goto out;
}
}
/* always give the server some time before connecting */
msleep(CLIENT_CONNECT_DELAY_MS);
/* make sure local server isn't writing super during votes */
scoutfs_server_stop(sb);
ret = scoutfs_net_connect(sb, client->conn, &qei->sin,
CLIENT_CONNECT_TIMEOUT_MS);
if (ret) {
/* we couldn't connect, try electing a new server */
client->old_elected_nr = qei->elected_nr;
timeout_abs = ktime_add_ms(ktime_get(),
CLIENT_QUORUM_TIMEOUT_MS);
ret = scoutfs_quorum_election(sb, timeout_abs,
le64_to_cpu(super->quorum_server_term),
&elected_term);
/* start the server if we were asked to */
if (elected_term > 0)
ret = scoutfs_server_start(sb, &opts->server_addr,
elected_term);
ret = -ENOTCONN;
goto out;
}
/* trust this server again if it's still around after we disconnect */
client->old_elected_nr = 0;
/* send a greeting to verify endpoints of each connection */
memcpy(greet.name, opts->uniq_name, sizeof(greet.name));
greet.fsid = super->hdr.fsid;
greet.format_hash = super->format_hash;
greet.server_term = cpu_to_le64(client->server_term);
greet.unmount_barrier = 0;
greet.unmount_barrier = cpu_to_le64(client->greeting_umb);
greet.node_id = cpu_to_le64(sbi->node_id);
greet.flags = 0;
if (client->sending_farewell)
greet.flags |= cpu_to_le64(SCOUTFS_NET_GREETING_FLAG_FAREWELL);
if (am_voter)
greet.flags |= cpu_to_le64(SCOUTFS_NET_GREETING_FLAG_VOTER);
ret = scoutfs_net_submit_request(sb, client->conn,
SCOUTFS_NET_CMD_GREETING,
@@ -492,8 +480,12 @@ static void scoutfs_client_connect_worker(struct work_struct *work)
if (ret)
scoutfs_net_shutdown(sb, client->conn);
out:
kfree(super);
/* always have a small delay before retrying to avoid storms */
if (ret && !atomic_read(&client->shutting_down))
queue_work(client->workq, &client->connect_work);
queue_delayed_work(client->workq, &client->connect_dwork,
msecs_to_jiffies(CLIENT_CONNECT_DELAY_MS));
}
/*
@@ -566,7 +558,7 @@ static void client_notify_down(struct super_block *sb,
struct client_info *client = SCOUTFS_SB(sb)->client_info;
if (!atomic_read(&client->shutting_down))
queue_work(client->workq, &client->connect_work);
queue_delayed_work(client->workq, &client->connect_dwork, 0);
}
/*
@@ -597,7 +589,8 @@ int scoutfs_client_setup(struct super_block *sb)
client->sb = sb;
init_completion(&client->node_id_comp);
atomic_set(&client->shutting_down, 0);
INIT_WORK(&client->connect_work, scoutfs_client_connect_worker);
INIT_DELAYED_WORK(&client->connect_dwork,
scoutfs_client_connect_worker);
init_completion(&client->farewell_comp);
client->conn = scoutfs_net_alloc_conn(sb, NULL, client_notify_down, 0,
@@ -613,7 +606,7 @@ int scoutfs_client_setup(struct super_block *sb)
goto out;
}
queue_work(client->workq, &client->connect_work);
queue_delayed_work(client->workq, &client->connect_dwork, 0);
ret = 0;
out:
@@ -693,16 +686,13 @@ void scoutfs_client_destroy(struct super_block *sb)
atomic_set(&client->shutting_down, 1);
/* make sure worker isn't using the conn */
cancel_work_sync(&client->connect_work);
cancel_delayed_work_sync(&client->connect_dwork);
/* make racing conn use explode */
conn = client->conn;
client->conn = NULL;
scoutfs_net_free_conn(sb, conn);
/* stop running the server if we were, harmless otherwise */
stop_our_server(sb, &client->qei);
if (client->workq)
destroy_workqueue(client->workq);
kfree(client);
kmod/src/counters.h
+8 -7
View File
@@ -117,18 +117,19 @@
EXPAND_COUNTER(net_recv_invalid_message) \
EXPAND_COUNTER(net_recv_messages) \
EXPAND_COUNTER(net_unknown_request) \
EXPAND_COUNTER(quorum_elected) \
EXPAND_COUNTER(quorum_election_error) \
EXPAND_COUNTER(quorum_fenced) \
EXPAND_COUNTER(quorum_found_leader) \
EXPAND_COUNTER(quorum_no_leader) \
EXPAND_COUNTER(quorum_cycle) \
EXPAND_COUNTER(quorum_elected_leader) \
EXPAND_COUNTER(quorum_election_timeout) \
EXPAND_COUNTER(quorum_failure) \
EXPAND_COUNTER(quorum_new_leader) \
EXPAND_COUNTER(quorum_read_block) \
EXPAND_COUNTER(quorum_read_block_error) \
EXPAND_COUNTER(quorum_read_invalid_block) \
EXPAND_COUNTER(quorum_read_invalid_config) \
EXPAND_COUNTER(quorum_waited) \
EXPAND_COUNTER(quorum_saw_super_leader) \
EXPAND_COUNTER(quorum_timedout) \
EXPAND_COUNTER(quorum_write_block) \
EXPAND_COUNTER(quorum_write_block_error) \
EXPAND_COUNTER(quorum_fenced) \
EXPAND_COUNTER(seg_alloc) \
EXPAND_COUNTER(seg_csum_error) \
EXPAND_COUNTER(seg_free) \
kmod/src/format.h
+48 -65
View File
@@ -42,11 +42,14 @@
/*
* A reasonably large region of aligned quorum blocks follow the super
* block.
* block. Each voting cycle reads the entire region so we don't want it
* to be too enormous. 256K seems like a reasonably chunky single IO.
* The number of blocks in the region also determines the number of
* mounts that have a reasonable probability of not overwriting each
* other's random block locations.
*/
#define SCOUTFS_QUORUM_BLKNO ((128ULL * 1024) >> SCOUTFS_BLOCK_SHIFT)
#define SCOUTFS_QUORUM_BLOCKS ((128ULL * 1024) >> SCOUTFS_BLOCK_SHIFT)
#define SCOUTFS_QUORUM_MAX_SLOTS SCOUTFS_QUORUM_BLOCKS
#define SCOUTFS_QUORUM_BLKNO ((256ULL * 1024) >> SCOUTFS_BLOCK_SHIFT)
#define SCOUTFS_QUORUM_BLOCKS ((256ULL * 1024) >> SCOUTFS_BLOCK_SHIFT)
#define SCOUTFS_UNIQUE_NAME_MAX_BYTES 64 /* includes null */
@@ -304,9 +307,10 @@ struct scoutfs_mounted_client_btree_key {
} __packed;
struct scoutfs_mounted_client_btree_val {
__u8 name[SCOUTFS_UNIQUE_NAME_MAX_BYTES];
__u8 flags;
} __packed;
#define SCOUTFS_MOUNTED_CLIENT_VOTER (1 << 0)
/*
* The max number of links defines the max number of entries that we can
@@ -421,70 +425,47 @@ struct scoutfs_xattr {
#define SCOUTFS_UUID_BYTES 16
/*
* During each quorum voting interval the fabric has to process 2 reads
* and a write for each voting mount. The only reason we limit the
* number of active quorum mounts is to limit the number of IOs per
* interval. We use a pretty conservative interval given that IOs will
* generally be faster than our constant and we'll have fewer active
* than the max.
* Mounts read all the quorum blocks and write to one random quorum
* block during a cycle. The min cycle time limits the per-mount iop
* load during elections. The random cycle delay makes it less likely
* that mounts will read and write at the same time and miss each
* other's writes. An election only completes if a quorum of mounts
* vote for a leader before any of their elections timeout. This is
* made less likely by the probability that mounts will overwrite each
* others random block locations. The max quorum count limits that
* probability. 9 mounts only have a 55% chance of writing to unique 4k
* blocks in a 256k region. The election timeout is set to include
* enough cycles to usually complete the election. Once a leader is
* elected it spends a number of cycles writing out blocks with itself
* logged as a leader. This reduces the possibility that servers
* will have their log entries overwritten and not be fenced.
*/
#define SCOUTFS_QUORUM_MAX_ACTIVE 7
#define SCOUTFS_QUORUM_IO_LATENCY_MS 10
#define SCOUTFS_QUORUM_INTERVAL_MS \
(SCOUTFS_QUORUM_MAX_ACTIVE * 3 * SCOUTFS_QUORUM_IO_LATENCY_MS)
#define SCOUTFS_QUORUM_MAX_COUNT 9
#define SCOUTFS_QUORUM_CYCLE_LO_MS 10
#define SCOUTFS_QUORUM_CYCLE_HI_MS 20
#define SCOUTFS_QUORUM_TERM_LO_MS 250
#define SCOUTFS_QUORUM_TERM_HI_MS 500
#define SCOUTFS_QUORUM_ELECTED_LOG_CYCLES 10
/*
* Each mount that is found in the quorum config in the super block can
* write to quorum blocks indicating which mount they vote for as
* the leader.
*
* @config_gen: references the config gen in the super block
* @write_nr: incremented for every write, only 0 when never written
* @elected_nr: incremented when elected, 0 otherwise
* @unmount_barrier: incremented by servers when all members have unmounted
* @vote_slot: the active config slot that the writer is voting for
*/
struct scoutfs_quorum_block {
__le64 fsid;
__le64 blkno;
__le64 config_gen;
__le64 term;
__le64 write_nr;
__le64 elected_nr;
__le64 unmount_barrier;
__le64 voter_rid;
__le64 vote_for_rid;
__le32 crc;
__u8 vote_slot;
__u8 flags;
} __packed;
#define SCOUTFS_QUORUM_BLOCK_FLAG_ELECTED (1 << 0)
#define SCOUTFS_QUORUM_BLOCK_FLAG_LISTENING (1 << 1)
#define SCOUTFS_QUORUM_BLOCK_FLAGS_UNKNOWN (U8_MAX << 2)
#define SCOUTFS_QUORUM_MAX_SLOTS SCOUTFS_QUORUM_BLOCKS
/*
* Each quorum voter is described by a slot which corresponds to the
* block that the voter will write to.
*
* The stale flag is used to support config migration. A new
* configuration is written in free slots and the old configuration is
* marked stale. Stale slots can only be reclaimed once we have
* evidence that the named mount won't try and write to it by seeing it
* write to other slots or connect with the new gen.
*/
struct scoutfs_quorum_config {
__le64 gen;
struct scoutfs_quorum_slot {
__u8 name[SCOUTFS_UNIQUE_NAME_MAX_BYTES];
__u8 log_nr;
struct scoutfs_quorum_log {
__le64 term;
__le64 rid;
struct scoutfs_inet_addr addr;
__u8 vote_priority;
__u8 flags;
} __packed slots[SCOUTFS_QUORUM_MAX_SLOTS];
} __packed log[0];
} __packed;
#define SCOUTFS_QUORUM_SLOT_ACTIVE (1 << 0)
#define SCOUTFS_QUORUM_SLOT_STALE (1 << 1)
#define SCOUTFS_QUORUM_SLOT_FLAGS_UNKNOWN (U8_MAX << 2)
#define SCOUTFS_QUORUM_LOG_MAX \
((SCOUTFS_BLOCK_SIZE - sizeof(struct scoutfs_quorum_block)) / \
sizeof(struct scoutfs_quorum_log))
struct scoutfs_super_block {
struct scoutfs_block_header hdr;
@@ -500,9 +481,13 @@ struct scoutfs_super_block {
__le64 next_seg_seq;
__le64 next_node_id;
__le64 next_compact_id;
__le64 quorum_fenced_term;
__le64 quorum_server_term;
__le64 unmount_barrier;
__u8 quorum_count;
struct scoutfs_inet_addr server_addr;
struct scoutfs_btree_root alloc_root;
struct scoutfs_manifest manifest;
struct scoutfs_quorum_config quorum_config;
struct scoutfs_btree_root lock_clients;
struct scoutfs_btree_root trans_seqs;
struct scoutfs_btree_root mounted_clients;
@@ -624,8 +609,6 @@ enum {
* Greetings verify identity of communicating nodes. The sender sends
* their credentials and the receiver verifies them.
*
* @name: The client sends its unique name to the server.
*
* @server_term: The raft term that elected the server. Initially 0
* from the client, sent by the server, then sent by the client as it
* tries to reconnect. Used to identify a client reconnecting to a
@@ -634,7 +617,7 @@ enum {
* @unmount_barrier: Incremented every time the remaining majority of
* quorum members all agree to leave. The server tells a quorum member
* the value that it's connecting under so that if the client sees the
* value increase in a quorum block it knows that the server has
* value increase in the super block then it knows that the server has
* processed its farewell and can safely unmount.
*
* @node_id: The id of the client. Initially 0 from the client,
@@ -643,7 +626,6 @@ enum {
* state must be dealt with.
*/
struct scoutfs_net_greeting {
__u8 name[SCOUTFS_UNIQUE_NAME_MAX_BYTES];
__le64 fsid;
__le64 format_hash;
__le64 server_term;
@@ -653,7 +635,8 @@ struct scoutfs_net_greeting {
} __packed;
#define SCOUTFS_NET_GREETING_FLAG_FAREWELL (1 << 0)
#define SCOUTFS_NET_GREETING_FLAG_INVALID (~(__u64)0 << 1)
#define SCOUTFS_NET_GREETING_FLAG_VOTER (1 << 1)
#define SCOUTFS_NET_GREETING_FLAG_INVALID (~(__u64)0 << 2)
/*
* This header precedes and describes all network messages sent over
kmod/src/quorum.c
+549 -720
View File
File diff suppressed because it is too large Load Diff
kmod/src/quorum.h
+3 -27
View File
@@ -1,33 +1,9 @@
#ifndef _SCOUTFS_QUORUM_H_
#define _SCOUTFS_QUORUM_H_
struct scoutfs_quorum_elected_info {
struct sockaddr_in sin;
__le64 config_gen;
__le64 write_nr;
u64 elected_nr;
u64 unmount_barrier;
unsigned int config_slot;
bool run_server;
u8 flags;
};
int scoutfs_quorum_election(struct super_block *sb, char *our_name,
u64 old_elected_nr, ktime_t timeout_abs,
bool unmounting, u64 our_umb,
struct scoutfs_quorum_elected_info *qei);
int scoutfs_quorum_set_listening(struct super_block *sb,
struct scoutfs_quorum_elected_info *qei);
int scoutfs_quorum_clear_elected(struct super_block *sb,
struct scoutfs_quorum_elected_info *qei);
int scoutfs_quorum_update_barrier(struct super_block *sb,
struct scoutfs_quorum_elected_info *qei,
u64 unmount_barrier);
int scoutfs_quorum_majority(struct super_block *sb,
struct scoutfs_quorum_config *conf);
bool scoutfs_quorum_voting_member(struct super_block *sb,
struct scoutfs_quorum_config *conf,
char *name);
int scoutfs_quorum_election(struct super_block *sb, ktime_t timeout_abs,
u64 prev_term, u64 *elected_term);
void scoutfs_quorum_clear_leader(struct super_block *sb);
int scoutfs_quorum_setup(struct super_block *sb);
void scoutfs_quorum_destroy(struct super_block *sb);
kmod/src/scoutfs_trace.h
+98 -33
View File
@@ -2544,53 +2544,118 @@ TRACE_EVENT(scoutfs_lock_message,
__entry->old_mode, __entry->new_mode)
);
DECLARE_EVENT_CLASS(scoutfs_quorum_block_class,
TP_PROTO(struct super_block *sb, u64 io_blkno,
struct scoutfs_quorum_block *blk),
TP_ARGS(sb, io_blkno, blk),
TRACE_EVENT(scoutfs_quorum_election,
TP_PROTO(struct super_block *sb, u64 prev_term),
TP_ARGS(sb, prev_term),
TP_STRUCT__entry(
SCSB_TRACE_FIELDS
__field(__u64, io_blkno)
__field(__u64, hdr_blkno)
__field(__u64, config_gen)
__field(__u64, write_nr)
__field(__u64, elected_nr)
__field(__u64, unmount_barrier)
__field(__u32, crc)
__field(__u8, vote_slot)
__field(__u8, flags)
__field(__u64, prev_term)
),
TP_fast_assign(
SCSB_TRACE_ASSIGN(sb);
__entry->io_blkno = io_blkno;
__entry->hdr_blkno = le64_to_cpu(blk->blkno);
__entry->config_gen = le64_to_cpu(blk->config_gen);
__entry->write_nr = le64_to_cpu(blk->write_nr);
__entry->elected_nr = le64_to_cpu(blk->elected_nr);
__entry->unmount_barrier = le64_to_cpu(blk->unmount_barrier);
__entry->crc = le32_to_cpu(blk->crc);
__entry->vote_slot = blk->vote_slot;
__entry->flags = blk->flags;
__entry->prev_term = prev_term;
),
TP_printk(SCSBF" io_blkno %llu hdr_blkno %llu config_gen %llu write_nr %llu elected_nr %llu umb %llu crc 0x%08x vote_slot %u flags %02x",
SCSB_TRACE_ARGS, __entry->io_blkno, __entry->hdr_blkno,
__entry->config_gen, __entry->write_nr, __entry->elected_nr,
__entry->unmount_barrier, __entry->crc, __entry->vote_slot,
__entry->flags)
TP_printk(SCSBF" prev_term %llu",
SCSB_TRACE_ARGS, __entry->prev_term)
);
TRACE_EVENT(scoutfs_quorum_election_ret,
TP_PROTO(struct super_block *sb, int ret, u64 elected_term),
TP_ARGS(sb, ret, elected_term),
TP_STRUCT__entry(
SCSB_TRACE_FIELDS
__field(int, ret)
__field(__u64, elected_term)
),
TP_fast_assign(
SCSB_TRACE_ASSIGN(sb);
__entry->ret = ret;
__entry->elected_term = elected_term;
),
TP_printk(SCSBF" ret %d elected_term %llu",
SCSB_TRACE_ARGS, __entry->ret, __entry->elected_term)
);
TRACE_EVENT(scoutfs_quorum_election_vote,
TP_PROTO(struct super_block *sb, int role, u64 term, u64 vote_for_rid,
int votes, int log_cycles, int quorum_count),
TP_ARGS(sb, role, term, vote_for_rid, votes, log_cycles, quorum_count),
TP_STRUCT__entry(
SCSB_TRACE_FIELDS
__field(int, role)
__field(__u64, term)
__field(__u64, vote_for_rid)
__field(int, votes)
__field(int, log_cycles)
__field(int, quorum_count)
),
TP_fast_assign(
SCSB_TRACE_ASSIGN(sb);
__entry->role = role;
__entry->term = term;
__entry->vote_for_rid = vote_for_rid;
__entry->votes = votes;
__entry->log_cycles = log_cycles;
__entry->quorum_count = quorum_count;
),
TP_printk(SCSBF" role %d term %llu vote_for_rid %016llx votes %d log_cycles %d quorum_count %d",
SCSB_TRACE_ARGS, __entry->role, __entry->term,
__entry->vote_for_rid, __entry->votes, __entry->log_cycles,
__entry->quorum_count)
);
DECLARE_EVENT_CLASS(scoutfs_quorum_block_class,
TP_PROTO(struct super_block *sb, struct scoutfs_quorum_block *blk),
TP_ARGS(sb, blk),
TP_STRUCT__entry(
SCSB_TRACE_FIELDS
__field(__u64, blkno)
__field(__u64, term)
__field(__u64, write_nr)
__field(__u64, voter_rid)
__field(__u64, vote_for_rid)
__field(__u32, crc)
__field(__u8, log_nr)
),
TP_fast_assign(
SCSB_TRACE_ASSIGN(sb);
__entry->blkno = le64_to_cpu(blk->blkno);
__entry->term = le64_to_cpu(blk->term);
__entry->write_nr = le64_to_cpu(blk->write_nr);
__entry->voter_rid = le64_to_cpu(blk->voter_rid);
__entry->vote_for_rid = le64_to_cpu(blk->vote_for_rid);
__entry->crc = le32_to_cpu(blk->crc);
__entry->log_nr = blk->log_nr;
),
TP_printk(SCSBF" blkno %llu term %llu write_nr %llu voter_rid %016llx vote_for_rid %016llx crc 0x%08x log_nr %u",
SCSB_TRACE_ARGS, __entry->blkno, __entry->term,
__entry->write_nr, __entry->voter_rid, __entry->vote_for_rid,
__entry->crc, __entry->log_nr)
);
DEFINE_EVENT(scoutfs_quorum_block_class, scoutfs_quorum_read_block,
TP_PROTO(struct super_block *sb, u64 io_blkno,
struct scoutfs_quorum_block *blk),
TP_ARGS(sb, io_blkno, blk)
TP_PROTO(struct super_block *sb, struct scoutfs_quorum_block *blk),
TP_ARGS(sb, blk)
);
DEFINE_EVENT(scoutfs_quorum_block_class, scoutfs_quorum_write_block,
TP_PROTO(struct super_block *sb, u64 io_blkno,
struct scoutfs_quorum_block *blk),
TP_ARGS(sb, io_blkno, blk)
TP_PROTO(struct super_block *sb, struct scoutfs_quorum_block *blk),
TP_ARGS(sb, blk)
);
/*
kmod/src/server.c
+79 -57
View File
@@ -42,8 +42,8 @@
* connection and accepts connections from all the other mounts acting
* as clients.
*
* The server is started when raft elects the mount as the leader. If
* it sees errors it shuts down the server in the hopes that another
* The server is started by the mount that is elected leader by quorum.
* If it sees errors it shuts down the server in the hopes that another
* mount will become the leader and have less trouble.
*/
@@ -61,8 +61,6 @@ struct server_info {
u64 term;
struct scoutfs_net_connection *conn;
struct scoutfs_quorum_elected_info qei;
/* request processing coordinates committing manifest and alloc */
struct rw_semaphore commit_rwsem;
struct llist_head commit_waiters;
@@ -1185,14 +1183,16 @@ int scoutfs_server_lock_recover_request(struct super_block *sb, u64 node_id,
}
static int insert_mounted_client(struct super_block *sb, u64 node_id,
char *name)
u64 gr_flags)
{
struct scoutfs_super_block *super = &SCOUTFS_SB(sb)->super;
struct scoutfs_mounted_client_btree_key mck;
struct scoutfs_mounted_client_btree_val mcv;
mck.node_id = cpu_to_be64(node_id);
strncpy(mcv.name, name, sizeof(mcv.name));
mcv.flags = 0;
if (gr_flags & SCOUTFS_NET_GREETING_FLAG_VOTER)
mcv.flags |= SCOUTFS_MOUNTED_CLIENT_VOTER;
return scoutfs_btree_insert(sb, &super->mounted_clients,
&mck, sizeof(mck), &mcv, sizeof(mcv));
@@ -1260,6 +1260,7 @@ static int server_greeting(struct super_block *sb,
DECLARE_SERVER_INFO(sb, server);
struct commit_waiter cw;
__le64 node_id = 0;
__le64 umb = 0;
bool sent_node_id;
bool first_contact;
bool farewell;
@@ -1293,10 +1294,12 @@ static int server_greeting(struct super_block *sb,
spin_lock(&server->lock);
node_id = super->next_node_id;
le64_add_cpu(&super->next_node_id, 1);
umb = super->unmount_barrier;
spin_unlock(&server->lock);
mutex_lock(&server->farewell_mutex);
ret = insert_mounted_client(sb, le64_to_cpu(node_id), gr->name);
ret = insert_mounted_client(sb, le64_to_cpu(node_id),
le64_to_cpu(gr->flags));
mutex_unlock(&server->farewell_mutex);
if (ret == 0)
@@ -1308,6 +1311,7 @@ static int server_greeting(struct super_block *sb,
}
} else {
node_id = gr->node_id;
umb = gr->unmount_barrier;
}
send_err:
@@ -1315,11 +1319,10 @@ send_err:
if (err)
node_id = 0;
memset(greet.name, 0, sizeof(greet.name));
greet.fsid = super->hdr.fsid;
greet.format_hash = super->format_hash;
greet.server_term = cpu_to_le64(server->term);
greet.unmount_barrier = cpu_to_le64(server->qei.unmount_barrier);
greet.unmount_barrier = umb;
greet.node_id = node_id;
greet.flags = 0;
@@ -1379,31 +1382,20 @@ static bool invalid_mounted_client_item(struct scoutfs_btree_item_ref *iref)
/*
* This work processes farewell requests asynchronously. Requests from
* voting quorum members can be held until they're no longer needed to
* vote for quorum and elect a server to process farewell requests.
*
* This will hold farewell requests from voting clients until either it
* isn't needed for quorum because a majority remains without it, or it
* won't be needed for quorum because all the remaining mounted clients
* are voting and waiting for farewell.
* voting clients can be held until only the final quorum remains and
* they've all sent farewell requests.
*
* When we remove the last mounted client record for the last voting
* client then we increase the unmount_barrier and write it to the
* server's quorum block. If voting clients don't get their farewell
* response they'll attempt to form quorum again to start the server for
* their farewell response but will find the increased umount_barrier.
* The'll know that their farewell has been processed and they can exit
* without forming quorum.
* client then we increase the unmount_barrier and write it to the super
* block. If voting clients don't get their farewell response they'll
* see the greater umount_barrier in the super and will know that their
* farewell has been processed and that they can exit.
*
* Responses that are waiting for clients who aren't voting are
* immediately sent. Clients that don't have a mounted client record
* have already had their farewell processed by another server and can
* proceed.
*
* This can trust the quorum config found in the super that was read
* when the server started. Only the current server can rewrite the
* working config.
*
* Farewell responses are unique in that sending them causes the server
* to shutdown the connection to the client next time the socket
* disconnects. If the socket is destroyed before the client gets the
@@ -1420,7 +1412,6 @@ static void farewell_worker(struct work_struct *work)
farewell_work);
struct super_block *sb = server->sb;
struct scoutfs_super_block *super = &SCOUTFS_SB(sb)->super;
struct scoutfs_quorum_config *conf = &super->quorum_config;
struct scoutfs_mounted_client_btree_key mck;
struct scoutfs_mounted_client_btree_val *mcv;
struct farewell_request *tmp;
@@ -1429,7 +1420,6 @@ static void farewell_worker(struct work_struct *work)
struct commit_waiter cw;
unsigned int nr_unmounting = 0;
unsigned int nr_mounted = 0;
unsigned int majority;
LIST_HEAD(reqs);
LIST_HEAD(send);
bool deleted = false;
@@ -1437,8 +1427,6 @@ static void farewell_worker(struct work_struct *work)
bool more_reqs;
int ret;
majority = scoutfs_quorum_majority(sb, conf);
/* grab all the requests that are waiting */
mutex_lock(&server->farewell_mutex);
list_splice_init(&server->farewell_requests, &reqs);
@@ -1463,7 +1451,7 @@ static void farewell_worker(struct work_struct *work)
}
mcv = iref.val;
voting = scoutfs_quorum_voting_member(sb, conf, mcv->name);
voting = (mcv->flags & SCOUTFS_MOUNTED_CLIENT_VOTER) != 0;
scoutfs_btree_put_iref(&iref);
if (!voting) {
@@ -1492,7 +1480,7 @@ static void farewell_worker(struct work_struct *work)
memcpy(&mck, iref.key, sizeof(mck));
mcv = iref.val;
if (scoutfs_quorum_voting_member(sb, conf, mcv->name))
if (mcv->flags & SCOUTFS_MOUNTED_CLIENT_VOTER)
nr_mounted++;
scoutfs_btree_put_iref(&iref);
@@ -1503,7 +1491,8 @@ static void farewell_worker(struct work_struct *work)
/* send as many responses as we can to maintain quorum */
while ((fw = list_first_entry_or_null(&reqs, struct farewell_request,
entry)) &&
(nr_mounted > majority || nr_unmounting >= nr_mounted)) {
(nr_mounted > super->quorum_count ||
nr_unmounting >= nr_mounted)) {
list_move_tail(&fw->entry, &send);
nr_mounted--;
@@ -1529,10 +1518,13 @@ static void farewell_worker(struct work_struct *work)
goto out;
}
/* update the unmount barrier the first time we delete all mounted */
/* update the unmount barrier if we deleted all voting clients */
if (deleted && nr_mounted == 0) {
ret = scoutfs_quorum_update_barrier(sb, &server->qei,
server->qei.unmount_barrier + 1);
down_read(&server->commit_rwsem);
le64_add_cpu(&super->unmount_barrier, 1);
queue_commit_work(server, &cw);
up_read(&server->commit_rwsem);
ret = wait_for_commit(&cw);
if (ret)
goto out;
}
@@ -2290,9 +2282,14 @@ static void scoutfs_server_worker(struct work_struct *work)
struct sockaddr_in sin;
LIST_HEAD(conn_list);
int ret;
int err;
trace_scoutfs_server_work_enter(sb, 0, 0);
sin = server->listen_sin;
scoutfs_info(sb, "server setting up at "SIN_FMT, SIN_ARG(&sin));
conn = scoutfs_net_alloc_conn(sb, server_notify_up, server_notify_down,
sizeof(struct server_client_info),
server_req_funcs, "server");
@@ -2301,8 +2298,6 @@ static void scoutfs_server_worker(struct work_struct *work)
goto out;
}
sin = server->listen_sin;
ret = scoutfs_net_bind(sb, conn, &sin);
if (ret) {
scoutfs_err(sb, "server failed to bind to "SIN_FMT", err %d%s",
@@ -2312,37 +2307,44 @@ static void scoutfs_server_worker(struct work_struct *work)
goto out;
}
ret = scoutfs_read_super(sb, super);
if (ret)
goto out;
/* start up the server subsystems before accepting */
ret = scoutfs_btree_setup(sb) ?:
ret = scoutfs_read_super(sb, super) ?:
scoutfs_btree_setup(sb) ?:
scoutfs_manifest_setup(sb) ?:
scoutfs_lock_server_setup(sb);
if (ret)
goto shutdown;
complete(&server->start_comp);
/*
* Write our address in the super before it's possible for net
* processing to start writing the super as part of
* transactions. In theory clients could be trying to connect
* to our address without having seen it in the super (maybe
* they saw it a long time ago).
*/
scoutfs_addr_from_sin(&super->server_addr, &sin);
super->quorum_server_term = cpu_to_le64(server->term);
ret = scoutfs_write_super(sb, super);
if (ret < 0)
goto shutdown;
server->stable_manifest_root = super->manifest.root;
scoutfs_info(sb, "server started on "SIN_FMT, SIN_ARG(&sin));
/* start accepting connections and processing work */
server->conn = conn;
scoutfs_net_listen(sb, conn);
ret = scoutfs_quorum_set_listening(sb, &server->qei);
scoutfs_info(sb, "server ready at "SIN_FMT, SIN_ARG(&sin));
complete(&server->start_comp);
if (ret == 0) {
/* wait_event/wake_up provide barriers */
wait_event_interruptible(server->waitq, server->shutting_down);
}
scoutfs_info(sb, "server shutting down on "SIN_FMT, SIN_ARG(&sin));
/* wait_event/wake_up provide barriers */
wait_event_interruptible(server->waitq, server->shutting_down);
shutdown:
scoutfs_info(sb, "server shutting down at "SIN_FMT, SIN_ARG(&sin));
/* wait for request processing */
scoutfs_net_shutdown(sb, conn);
/* drain compact work queued by responses */
@@ -2357,17 +2359,41 @@ shutdown:
scoutfs_lock_server_destroy(sb);
out:
scoutfs_quorum_clear_leader(sb);
scoutfs_net_free_conn(sb, conn);
scoutfs_info(sb, "server stopped at "SIN_FMT, SIN_ARG(&sin));
trace_scoutfs_server_work_exit(sb, 0, ret);
/*
* Always try to clear our presence in the super so that we're
* not fenced. We do this last because other mounts will try to
* reach quorum the moment they see zero here. The later we do
* this the longer we have to finish shutdown while clients
* timeout.
*/
err = scoutfs_read_super(sb, super);
if (err == 0) {
super->quorum_fenced_term = cpu_to_le64(server->term);
memset(&super->server_addr, 0, sizeof(super->server_addr));
err = scoutfs_write_super(sb, super);
}
if (err < 0) {
scoutfs_err(sb, "failed to clear election term %llu at "SIN_FMT", this mount could be fenced",
server->term, SIN_ARG(&sin));
}
server->err = ret;
complete(&server->start_comp);
}
/* XXX can we call start multiple times? */
/*
* Wait for the server to successfully start. If this returns error then
* the super block's fence_term has been set to the new server's term so
* that it won't be fenced.
*/
int scoutfs_server_start(struct super_block *sb, struct sockaddr_in *sin,
u64 term, struct scoutfs_quorum_elected_info *qei)
u64 term)
{
DECLARE_SERVER_INFO(sb, server);
@@ -2375,7 +2401,6 @@ int scoutfs_server_start(struct super_block *sb, struct sockaddr_in *sin,
server->shutting_down = false;
server->listen_sin = *sin;
server->term = term;
server->qei = *qei;
init_completion(&server->start_comp);
queue_work(server->wq, &server->work);
@@ -2398,8 +2423,7 @@ void scoutfs_server_abort(struct super_block *sb)
* Once the server is stopped we give the caller our election info
* which might have been modified while we were running.
*/
void scoutfs_server_stop(struct super_block *sb,
struct scoutfs_quorum_elected_info *qei)
void scoutfs_server_stop(struct super_block *sb)
{
DECLARE_SERVER_INFO(sb, server);
@@ -2407,8 +2431,6 @@ void scoutfs_server_stop(struct super_block *sb,
/* XXX not sure both are needed */
cancel_work_sync(&server->work);
cancel_work_sync(&server->commit_work);
*qei = server->qei;
}
int scoutfs_server_setup(struct super_block *sb)
kmod/src/server.h
+2 -3
View File
@@ -74,10 +74,9 @@ int scoutfs_server_lock_recover_request(struct super_block *sb, u64 node_id,
struct sockaddr_in;
struct scoutfs_quorum_elected_info;
int scoutfs_server_start(struct super_block *sb, struct sockaddr_in *sin,
u64 term, struct scoutfs_quorum_elected_info *qei);
u64 term);
void scoutfs_server_abort(struct super_block *sb);
void scoutfs_server_stop(struct super_block *sb,
struct scoutfs_quorum_elected_info *qei);
void scoutfs_server_stop(struct super_block *sb);
int scoutfs_server_setup(struct super_block *sb);
void scoutfs_server_destroy(struct super_block *sb);
kmod/src/super.c
+13 -1
View File
@@ -200,7 +200,6 @@ static void scoutfs_put_super(struct super_block *sb)
scoutfs_shutdown_trans(sb);
scoutfs_client_destroy(sb);
scoutfs_quorum_destroy(sb);
scoutfs_inode_destroy(sb);
/* the server locks the listen address and compacts */
@@ -210,6 +209,9 @@ static void scoutfs_put_super(struct super_block *sb)
scoutfs_seg_destroy(sb);
scoutfs_lock_destroy(sb);
/* server clears quorum leader flag during shutdown */
scoutfs_quorum_destroy(sb);
scoutfs_item_destroy(sb);
scoutfs_destroy_triggers(sb);
scoutfs_options_destroy(sb);
@@ -319,6 +321,16 @@ int scoutfs_read_super(struct super_block *sb,
goto out;
}
/* XXX do we want more rigorous invalid super checking? */
if (super->quorum_count == 0 ||
super->quorum_count > SCOUTFS_QUORUM_MAX_COUNT) {
scoutfs_err(sb, "super block has invalid quorum count %u, must be > 0 and <= %u",
super->quorum_count, SCOUTFS_QUORUM_MAX_COUNT);
ret = -EINVAL;
goto out;
}
*super_res = *super;
ret = 0;
out: