Add support for our format version

We had previously started on a relatively simple notion of an
interoperability version which wasn't quite right.  This fleshes out
support for a more functional format version.   The super blocks have a
single version that defines behaviour of the running system.   The code
supports a range of versions and we add some initial interfaces for
updating the version while the system is offline.   All of this together
should let us safely change the underlying format over time.

Signed-off-by: Zach Brown <zab@versity.com>

utils/man/scoutfs.5

@@ -198,7 +198,86 @@ with the
 .IB READ_XATTR_TOTALS
 ioctl.
 .RE
- 
+
+.SH FORMAT VERSION
+The format version defines the layout and use of structures stored on
+devices and passed over the network.  The version is incremented for
+every change in structures that is not backwards compatible with
+previous versions.  A single version implies all changes, individual
+changes can't be selectively adopted.
+.sp
+As a new file system is created the format version is stored in both of
+the super blocks written to the metadata and data devices.  By default
+the greatest supported version is written while an older supported
+version may be specified.
+.sp
+During mount the kernel module verifies that the format versions stored
+in both of the super blocks match and are supported.   That version
+defines the set of features and behavior of all the mounts using the
+file system, including the network protocol that is communicated over
+the wire.
+.sp
+Any combination of software release versions that support the current
+format version of the file system can safely be used concurrently.  This
+allows for rolling software updates of multiple mounts using a shared
+file system.
+.sp
+To use new incompatible features added in newer format versions the super blocks must
+be updated.   This can currently only be safely performed on a
+completely and cleanly unmounted file system.  The
+.BR scoutfs (8)
+.I change-format-version
+command can be used with the 
+.I --offline
+option to write a newer supported version into the super blocks.  It
+will fail if it sees any indication of unresolved mounts that may be
+using the devices: either active quorum members working with their
+quorum blocks or persistent records of mounted clients that haven't been
+resolved.  Like creating a new file system, there is no protection
+against multiple invocations of the change command corrupting the
+system.  Once the version is updated older software can no longer use
+the file system so this change should be performed with care.  Once the
+newer format version is successfully written it can be mounted and newer
+features can be used.
+.sp
+Each layer of the system can show its supported format versions:
+.RS
+.TP
+.B Userspace utilities
+.B scoutfs --help
+includes the range of supported format versions for a given release
+of the userspace utilities.
+.TP
+.B Kernel module
+.I modinfo MODULE
+shows the range of supproted versions for a kernel module file in the
+.I scoutfs_format_version_min
+and
+.I scoutfs_format_version_min
+fields.
+.TP
+.B Inserted module
+The supported version range of an inserted module can be found in
+.I .note.scoutfs_format_version_min
+and
+.I .note.scoutfs_format_version_max
+notes files in the sysfs notes directory for the inserted module,
+typically
+.I /sys/module/scoutfs/notes/
+.TP
+.B Metadata and data devices
+.I scoutfs print DEVICE
+shows the
+.I fmt_vers
+field in the initial output of the super block on the device.
+.TP
+.B Mounted filesystem
+The version that a mount is using is shown in the
+.I format_version
+file in the mount's sysfs directory, typically
+.I /sys/fs/scoutfs/f.FSID.r.RID/
+.RE
+
 .SH CORRUPTION DETECTION
 A
 .B scoutfs

utils/man/scoutfs.8

@@ -14,6 +14,34 @@ option will, when the option is omitted, fall back to using the value of the
 environment variable. If that variable is also absent the current working
 directory will be used.
 
+.TP
+.BI "change-format-version [-V, --format-version VERS] [-F|--offline META-DEVICE DATA-DEVICE]"
+.sp
+Change the format version of an existing file system.  The maxmimum
+supported version is used by default.   A specific version in the range
+can be specified.   The range of supported versions in shown in the
+output of --help.
+.RS 1.0i
+.PD 0
+.TP
+.sp
+.B "-F, --offline META-DEVICE DATA-DEVICE"
+Change the format version by writing directly to the metadata and data
+devices.   Like mkfs, this writes directly to the devices without
+protection and must only be used on completely unmounted devices.   The
+command will fail if it sees evidence of active quorum use of the device
+or of previously connected clients which haven't been reclaimed.  The
+only way to avoid these checks is to fully mount and cleanly unmount the
+file system. 
+.sp
+This is not an atomic operation because it writes to blocks on two
+devices.   Write failure can result in the versions becoming out of sync
+which will prevent the system from mouting.  To recover the error must
+be resolved so the command can be repeated and successfully write to
+the super blocks on both devices.
+.RE
+.PD
+
 .TP
 .BI "df [-h|--human-readable] [-p|--path PATH]"
 .sp
@@ -32,7 +60,7 @@ A path within a ScoutFS filesystem.
 .PD
 
 .TP
-.BI "mkfs META-DEVICE DATA-DEVICE {-Q|--quorum-slot} NR,ADDR,PORT [-m|--max-meta-size SIZE] [-d|--max-data-size SIZE] [-z|--data-alloc-zone-blocks BLOCKS] [-f|--force] [-A|--allow-small-size]"
+.BI "mkfs META-DEVICE DATA-DEVICE {-Q|--quorum-slot} NR,ADDR,PORT [-m|--max-meta-size SIZE] [-d|--max-data-size SIZE] [-z|--data-alloc-zone-blocks BLOCKS] [-f|--force] [-A|--allow-small-size] [-V|--format-version VERS]"
 .sp
 Initialize a new ScoutFS filesystem on the target devices. Since ScoutFS uses
 separate block devices for its metadata and data storage, two are required.
@@ -99,6 +127,13 @@ Set the data_alloc_zone_blocks volume option, as described in
 .TP
 .B "-f, --force"
 Ignore presence of existing data on the data and metadata devices.
+.TP
+.B "-V, --format-verson"
+Specify the format version to use in the newly created file system.
+The range of supported versions is visible in the output of 
++.BR scoutfs (8)
++.I --help
+.
 .RE
 .PD
 

utils/src/change_format_version.c

@@ -0,0 +1,287 @@
+#define _GNU_SOURCE /* O_DIRECT */
+#include <unistd.h>
+#include <stdbool.h>
+#include <stdlib.h>
+#include <stdio.h>
+#include <string.h>
+#include <errno.h>
+#include <sys/time.h>
+#include <uuid/uuid.h>
+#include <fcntl.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <unistd.h>
+#include <assert.h>
+#include <sys/socket.h>
+#include <netinet/in.h>
+#include <arpa/inet.h>
+#include <ctype.h>
+#include <inttypes.h>
+#include <argp.h>
+
+#include "sparse.h"
+#include "cmd.h"
+#include "util.h"
+#include "format.h"
+#include "parse.h"
+#include "crc.h"
+#include "rand.h"
+#include "dev.h"
+#include "key.h"
+#include "bitops.h"
+#include "btree.h"
+#include "leaf_item_hash.h"
+#include "blkid.h"
+#include "quorum.h"
+
+struct change_fmt_vers_args {
+	char *meta_device;
+	char *data_device;
+	u64 fmt_vers;
+	bool offline;
+};
+
+static int do_change_fmt_vers(struct change_fmt_vers_args *args)
+{
+	struct scoutfs_super_block *meta_super = NULL;
+	struct scoutfs_super_block *data_super = NULL;
+	struct scoutfs_quorum_block *qblk = NULL;
+	struct scoutfs_quorum_block_event *beg;
+	struct scoutfs_quorum_block_event *end;
+	bool wrote_meta = false;
+	bool in_use = false;
+	char uuid_str[37];
+	int meta_fd = -1;
+	int data_fd = -1;
+	int ret;
+	int i;
+
+	meta_fd = open(args->meta_device, O_DIRECT | O_SYNC | O_RDWR | O_EXCL);
+	if (meta_fd < 0) {
+		ret = -errno;
+		fprintf(stderr, "failed to open meta device '%s': %s (%d)\n",
+			args->meta_device, strerror(errno), errno);
+		goto out;
+	}
+
+	data_fd = open(args->data_device, O_DIRECT | O_SYNC | O_RDWR | O_EXCL);
+	if (data_fd < 0) {
+		ret = -errno;
+		fprintf(stderr, "failed to open data device '%s': %s (%d)\n",
+			args->data_device, strerror(errno), errno);
+		goto out;
+	}
+
+	ret = read_block_verify(meta_fd, SCOUTFS_BLOCK_MAGIC_SUPER, 0, SCOUTFS_SUPER_BLKNO,
+				SCOUTFS_BLOCK_SM_SHIFT, (void **)&meta_super);
+	if (ret) {
+		ret = -errno;
+		fprintf(stderr, "failed to read meta super block: %s (%d)\n",
+			strerror(errno), errno);
+		goto out;
+	}
+
+	ret = read_block_verify(data_fd, SCOUTFS_BLOCK_MAGIC_SUPER,
+				le64_to_cpu(meta_super->hdr.fsid), SCOUTFS_SUPER_BLKNO,
+				SCOUTFS_BLOCK_SM_SHIFT, (void **)&data_super);
+	if (ret) {
+		ret = -errno;
+		fprintf(stderr, "failed to read data super block: %s (%d)\n",
+			strerror(errno), errno);
+		goto out;
+	}
+
+	if (le64_to_cpu(meta_super->fmt_vers) == args->fmt_vers &&
+	    meta_super->fmt_vers == data_super->fmt_vers) {
+		printf("both metadata and data device format version are already %llu, nothing to do.\n",
+			args->fmt_vers);
+		ret = 0;
+		goto out;
+	}
+
+	if (le64_to_cpu(meta_super->fmt_vers) < SCOUTFS_FORMAT_VERSION_MIN ||
+	    le64_to_cpu(meta_super->fmt_vers) > SCOUTFS_FORMAT_VERSION_MAX) {
+		fprintf(stderr, "meta super block has format version %llu outside of supported version range %u-%u",
+			    le64_to_cpu(meta_super->fmt_vers), SCOUTFS_FORMAT_VERSION_MIN,
+			    SCOUTFS_FORMAT_VERSION_MAX);
+		ret = -EINVAL;
+		goto out;
+	}
+
+	if (le64_to_cpu(data_super->fmt_vers) < SCOUTFS_FORMAT_VERSION_MIN ||
+	    le64_to_cpu(data_super->fmt_vers) > SCOUTFS_FORMAT_VERSION_MAX) {
+		fprintf(stderr, "data super block has format version %llu outside of supported version range %u-%u",
+			    le64_to_cpu(data_super->fmt_vers), SCOUTFS_FORMAT_VERSION_MIN,
+			    SCOUTFS_FORMAT_VERSION_MAX);
+		ret = -EINVAL;
+		goto out;
+	}
+
+	if (meta_super->mounted_clients.ref.blkno != 0) {
+		fprintf(stderr, "meta superblock mounted clients btree is not empty.\n");
+		ret = -EBUSY;
+		in_use = true;
+		goto out;
+	}
+
+	/* check for active quorum slots */
+	for (i = 0; i < SCOUTFS_QUORUM_BLOCKS; i++) {
+		if (!quorum_slot_present(meta_super, i))
+			continue;
+		ret = read_block(meta_fd, SCOUTFS_QUORUM_BLKNO + i, SCOUTFS_BLOCK_SM_SHIFT,
+				 (void **)&qblk);
+		if (ret < 0) {
+			fprintf(stderr, "error reading quorum block for slot %u\n", i);
+			goto out;
+		}
+
+		beg = &qblk->events[SCOUTFS_QUORUM_EVENT_BEGIN];
+		end = &qblk->events[SCOUTFS_QUORUM_EVENT_END];
+
+		if (le64_to_cpu(beg->write_nr) > le64_to_cpu(end->write_nr)) {
+			fprintf(stderr, "mount in quorum slot %u could still be running.\n"
+					"  begin event: write_nr %llu timestamp %llu.%08u\n"
+					"    end event: write_nr %llu timestamp %llu.%08u\n",
+				i, le64_to_cpu(beg->write_nr), le64_to_cpu(beg->ts.sec),
+				le32_to_cpu(beg->ts.nsec),
+				le64_to_cpu(end->write_nr), le64_to_cpu(end->ts.sec),
+				le32_to_cpu(end->ts.nsec));
+			ret = -EBUSY;
+			in_use = true;
+			goto out;
+		}
+
+		free(qblk);
+		qblk = NULL;
+	}
+
+	if (le64_to_cpu(meta_super->fmt_vers) != args->fmt_vers) {
+		meta_super->fmt_vers = cpu_to_le64(args->fmt_vers);
+
+		ret = write_block(meta_fd, SCOUTFS_BLOCK_MAGIC_SUPER, meta_super->hdr.fsid, 1,
+				  SCOUTFS_SUPER_BLKNO, SCOUTFS_BLOCK_SM_SHIFT, &meta_super->hdr);
+		if (ret)
+			goto out;
+
+		wrote_meta = true;
+	}
+
+	if (le64_to_cpu(data_super->fmt_vers) != args->fmt_vers) {
+		data_super->fmt_vers = cpu_to_le64(args->fmt_vers);
+
+		ret = write_block(data_fd, SCOUTFS_BLOCK_MAGIC_SUPER, data_super->hdr.fsid, 1,
+				  SCOUTFS_SUPER_BLKNO, SCOUTFS_BLOCK_SM_SHIFT, &data_super->hdr);
+		if (ret < 0 && wrote_meta) {
+			fprintf(stderr, "Error writing data super block after writing the meta\n"
+					"super block.  The two super blocks may now be out of sync which\n"
+					"would prevent mounting.   Correct the source of the write error\n"
+					"and retry changing the version to write both super blocks.\n");
+			goto out;
+		}
+	}
+
+	uuid_unparse(meta_super->uuid, uuid_str);
+
+	printf("Successfully updated format version for scoutfs filesystem:\n"
+	       "  meta device path:     %s\n"
+	       "  data device path:     %s\n"
+	       "  fsid:                 %llx\n"
+	       "  uuid:                 %s\n"
+	       "  format version:       %llu\n",
+		args->meta_device,
+	        args->data_device,
+		le64_to_cpu(meta_super->hdr.fsid),
+		uuid_str,
+		le64_to_cpu(meta_super->fmt_vers));
+
+out:
+	if (in_use)
+		fprintf(stderr, "The filesystem must be fully recovered and cleanly unmounted to change the format version\n");
+
+	if (qblk)
+		free(qblk);
+	if (meta_super)
+		free(meta_super);
+	if (data_super)
+		free(data_super);
+	if (meta_fd != -1)
+		close(meta_fd);
+	if (data_fd != -1)
+		close(data_fd);
+	return ret;
+}
+
+static int parse_opt(int key, char *arg, struct argp_state *state)
+{
+	struct change_fmt_vers_args *args = state->input;
+	int ret;
+
+	switch (key) {
+	case 'F':
+		args->offline = true;
+		break;
+	case 'V':
+		ret = parse_u64(arg, &args->fmt_vers);
+		if (ret)
+			return ret;
+		if (args->fmt_vers < SCOUTFS_FORMAT_VERSION_MIN ||
+		    args->fmt_vers > SCOUTFS_FORMAT_VERSION_MAX)
+			argp_error(state, "format-version %llu is outside supported range of %u-%u",
+				   args->fmt_vers, SCOUTFS_FORMAT_VERSION_MIN,
+				   SCOUTFS_FORMAT_VERSION_MAX);
+		break;
+	case ARGP_KEY_ARG:
+		if (!args->meta_device)
+			args->meta_device = strdup_or_error(state, arg);
+		else if (!args->data_device)
+			args->data_device = strdup_or_error(state, arg);
+		else
+			argp_error(state, "more than two device arguments given");
+		break;
+	case ARGP_KEY_FINI:
+		if (!args->offline)
+			argp_error(state, "must specify --offline");
+		if (!args->meta_device)
+			argp_error(state, "no metadata device argument given");
+		if (!args->data_device)
+			argp_error(state, "no data device argument given");
+		break;
+	default:
+		break;
+	}
+
+	return 0;
+}
+
+static struct argp_option options[] = {
+	{ "offline", 'F', NULL, 0, "Write format version in offline device super blocks"},
+	{ "format-version", 'V', "VERS", 0, "Specify a format version within supported range ("SCOUTFS_FORMAT_VERSION_MIN_STR"-"SCOUTFS_FORMAT_VERSION_MAX_STR", default "SCOUTFS_FORMAT_VERSION_MAX_STR")"},
+	{ NULL }
+};
+
+static struct argp argp = {
+	options,
+	parse_opt,
+	"",
+	"Change format version of an existing ScoutFS filesystem"
+};
+
+static int change_fmt_vers_cmd(int argc, char *argv[])
+{
+	struct change_fmt_vers_args change_fmt_vers_args = {
+		.offline = false,
+		.fmt_vers = SCOUTFS_FORMAT_VERSION_MAX,
+	};
+	int ret;
+
+	ret = argp_parse(&argp, argc, argv, 0, NULL, &change_fmt_vers_args);
+	if (ret)
+		return ret;
+
+	return do_change_fmt_vers(&change_fmt_vers_args);
+}
+
+static void __attribute__((constructor)) change_fmt_vers_ctor(void)
+{
+	cmd_register_argp("change-format-version", &argp, GROUP_CORE, change_fmt_vers_cmd);
+}

utils/src/cmd.c

@@ -8,6 +8,7 @@
 
 #include "cmd.h"
 #include "util.h"
+#include "format.h"
 
 static struct argp_command {
 	char *name;
@@ -69,6 +70,9 @@ static void usage(void)
 	fprintf(stderr, "Selected fs defaults to current working directory.\n");
 	fprintf(stderr, "See <command> --help for more details.\n");
 
+	fprintf(stderr, "\nSupported format version: %u-%u\n",
+			SCOUTFS_FORMAT_VERSION_MIN, SCOUTFS_FORMAT_VERSION_MAX);
+
 	fprintf(stderr, "\nCore admin:\n");
 	print_cmds_for_group(GROUP_CORE);
 	fprintf(stderr, "\nAdditional Information:\n");

utils/src/mkfs.c

@@ -110,6 +110,7 @@ struct mkfs_args {
 	unsigned long long max_meta_size;
 	unsigned long long max_data_size;
 	u64 data_alloc_zone_blocks;
+	u64 fmt_vers;
 	bool force;
 	bool allow_small_size;
 	int nr_slots;
@@ -212,7 +213,7 @@ static int do_mkfs(struct mkfs_args *args)
 
 	/* partially initialize the super so we can use it to init others */
 	memset(super, 0, SCOUTFS_BLOCK_SM_SIZE);
-	super->version = cpu_to_le64(SCOUTFS_INTEROP_VERSION);
+	super->fmt_vers = cpu_to_le64(args->fmt_vers);
 	uuid_generate(super->uuid);
 	super->next_ino = cpu_to_le64(round_up(SCOUTFS_ROOT_INO + 1, SCOUTFS_LOCK_INODE_GROUP_NR));
 	super->seq = cpu_to_le64(1);
@@ -351,16 +352,16 @@ static int do_mkfs(struct mkfs_args *args)
 	       "  meta device path:     %s\n"
 	       "  data device path:     %s\n"
 	       "  fsid:                 %llx\n"
-	       "  version:              %llx\n"
 	       "  uuid:                 %s\n"
+	       "  format version:       %llu\n"
 	       "  64KB metadata blocks: "SIZE_FMT"\n"
 	       "  4KB data blocks:      "SIZE_FMT"\n"
 	       "  quorum slots:         ",
 		args->meta_device,
 	        args->data_device,
 		le64_to_cpu(super->hdr.fsid),
-		le64_to_cpu(super->version),
 		uuid_str,
+		le64_to_cpu(super->fmt_vers),
 		SIZE_ARGS(le64_to_cpu(super->total_meta_blocks),
 			  SCOUTFS_BLOCK_LG_SIZE),
 		SIZE_ARGS(le64_to_cpu(super->total_data_blocks),
@@ -484,6 +485,16 @@ static int parse_opt(int key, char *arg, struct argp_state *state)
 	case 'A':
 		args->allow_small_size = true;
 		break;
+	case 'V':
+		ret = parse_u64(arg, &args->fmt_vers);
+		if (ret)
+			return ret;
+		if (args->fmt_vers < SCOUTFS_FORMAT_VERSION_MIN ||
+		    args->fmt_vers > SCOUTFS_FORMAT_VERSION_MAX)
+			argp_error(state, "format-version %llu is outside supported range of %u-%u",
+				   args->fmt_vers, SCOUTFS_FORMAT_VERSION_MIN,
+				   SCOUTFS_FORMAT_VERSION_MAX);
+		break;
 	case 'z': /* data-alloc-zone-blocks */
 	{
 		ret = parse_u64(arg, &args->data_alloc_zone_blocks);
@@ -527,6 +538,7 @@ static struct argp_option options[] = {
 	{ "max-meta-size", 'm', "SIZE", 0, "Use a size less than the base metadata device size (bytes or KMGTP units)"},
 	{ "max-data-size", 'd', "SIZE", 0, "Use a size less than the base data device size (bytes or KMGTP units)"},
 	{ "data-alloc-zone-blocks", 'z', "BLOCKS", 0, "Divide data device into block zones so each mounts writes to a zone (4KB blocks)"},
+	{ "format-version", 'V', "version", 0, "Specify a format version within supported range, ("SCOUTFS_FORMAT_VERSION_MIN_STR"-"SCOUTFS_FORMAT_VERSION_MAX_STR", default "SCOUTFS_FORMAT_VERSION_MAX_STR")"},
 	{ NULL }
 };
 
@@ -539,7 +551,9 @@ static struct argp argp = {
 
 static int mkfs_cmd(int argc, char *argv[])
 {
-	struct mkfs_args mkfs_args = {NULL,};
+	struct mkfs_args mkfs_args = {
+		.fmt_vers = SCOUTFS_FORMAT_VERSION_MAX,
+	};
 	int ret;
 
 	ret = argp_parse(&argp, argc, argv, 0, NULL, &mkfs_args);

utils/src/print.c

@@ -935,8 +935,8 @@ static void print_super_block(struct scoutfs_super_block *super, u64 blkno)
 
 	printf("super blkno %llu\n", blkno);
 	print_block_header(&super->hdr, SCOUTFS_BLOCK_SM_SIZE);
-	printf("  version %llx uuid %s\n",
-	       le64_to_cpu(super->version), uuid_str);
+	printf("  fmt_vers %llu uuid %s\n",
+	       le64_to_cpu(super->fmt_vers), uuid_str);
 	printf("  flags: 0x%016llx\n", le64_to_cpu(super->flags));
 
 	/* XXX these are all in a crazy order */

utils/src/util.c

@@ -79,11 +79,16 @@ int read_block(int fd, u64 blkno, int shift, void **ret_val)
 	void *buf;
 	int ret;
 
+	buf = NULL;
 	*ret_val = NULL;
 
-	buf = malloc(size);
-	if (!buf)
-		return -ENOMEM;
+	ret = posix_memalign(&buf, size, size);
+	if (ret != 0) {
+		ret = -errno;
+		fprintf(stderr, "%zu byte aligned buffer allocation failed: %s (%d)\n",
+			size, strerror(errno), errno);
+		return ret;
+	}
 
 	ret = pread(fd, buf, size, blkno << shift);
 	if (ret == -1) {
@@ -136,7 +141,7 @@ int read_block_verify(int fd, u32 magic, u64 fsid, u64 blkno, int shift, void **
 		if (le32_to_cpu(hdr->magic) != magic)
 			fprintf(stderr, "read blkno %llu has bad magic %08x != expected %08x\n",
 				blkno, le32_to_cpu(hdr->magic), magic);
-		else if (fsid != 0 && le32_to_cpu(hdr->fsid) != fsid)
+		else if (fsid != 0 && le64_to_cpu(hdr->fsid) != fsid)
 			fprintf(stderr, "read blkno %llu has bad fsid %016llx != expected %016llx\n",
 				blkno, le64_to_cpu(hdr->fsid), fsid);
 		else if (le32_to_cpu(hdr->blkno) != blkno)