diff --git a/fcst/README b/fcst/README index 5e0e50a5c..1004eef34 100644 --- a/fcst/README +++ b/fcst/README @@ -39,7 +39,6 @@ To build for linux-2.6.34, do: because FCST supports only it. You need to do cd trunk/scst - make disable_proc # first time only make make install diff --git a/iscsi-scst/README b/iscsi-scst/README index 94b887ae1..6397a4038 100644 --- a/iscsi-scst/README +++ b/iscsi-scst/README @@ -12,7 +12,7 @@ ISCSI-SCST is a deeply reworked fork of iSCSI Enterprise Target (IET) - To fix all the problems, corner cases issues and iSCSI standard violations which IET has. -See more info at http://iscsi-scst.sourceforge.net. +See for more info http://iscsi-scst.sourceforge.net. This version is compatible with SCST version 2.0.0 and higher. @@ -38,9 +38,10 @@ compiler complains about redefinition of some symbol, you should either switch to vanilla kernel, or add or change as necessary the corresponding to that symbol "#if LINUX_VERSION_CODE" statement. -The sysfs build supports only kernels 2.6.26 and higher, because in -2.6.26 internal kernel's sysfs interface had a major change, which made -it heavily incompatible with pre-2.6.26 version. +Default sysfs interface supports only kernels 2.6.26 and higher, because +in 2.6.26 internal kernel's sysfs interface had a major change, which +made it heavily incompatible with pre-2.6.26 version. But with the +obsolete procfs interface kernels 2.6.16+ are supported. If during compilation you see message like "*** No rule to make target `xxx.h', needed by `yyy.o'. Stop.", then your autogenerated @@ -124,6 +125,30 @@ CAUTION: Working of target and initiator on the same host isn't fully ======= supported. See SCST README file for details. +Migration from the obsolete proc interface +------------------------------------------ + +Sysfs enabled scstadmin supports old procfs config file format, so with +it you should do the following steps to migrate your proc-based +configuration to the sysfs interface: + +1. Load SCST modules + +2. Run "scstadmin -config old_config_file" + +3. Start iSCSI-SCST + +4. Run "scstadmin -write_config new_config_file" + +5. Check new_config_file and make sure it has everything written +properly. + +6. Delete /etc/iscsi-scst.conf and forget about it. + +7. Start using "scstadmin -config new_config_file" to configure both +SCST and iSCSI-SCST. + + Migration from IET ------------------ @@ -151,13 +176,12 @@ ietd.conf files for us to play with). Sysfs interface --------------- -Starting from 2.0.0 iSCSI-SCST has sysfs interface. You can switch to it -by running "make disable_proc". To switch back to the procfs interface -you should run "make enable_proc". The procfs interface starting from -version 2.0.0 is obsolete and will be removed in one of the next -versions. The sysfs build supports only kernels 2.6.26 and higher, -because in 2.6.26 internal kernel's sysfs interface had a major change, -which made it heavily incompatible with pre-2.6.26 version. +Starting from 2.0.0 iSCSI-SCST uses sysfs interface. The procfs +interface is obsolete and will be removed in one of the next versions. +The sysfs build supports only kernels 2.6.26 and higher, because in +2.6.26 internal kernel's sysfs interface had a major change, which made +it heavily incompatible with pre-2.6.26 version. But with the obsolete +procfs interface kernels 2.6.16+ are supported. Root of SCST sysfs interface is /sys/kernel/scst_tgt. Root of iSCSI-SCST is /sys/kernel/scst_tgt/targets/iscsi. It has the following entries: @@ -880,3 +904,4 @@ Thanks to: * Bart Van Assche for a lot of help Vladislav Bolkhovitin , http://scst.sourceforge.net + diff --git a/iscsi-scst/README_in-tree b/iscsi-scst/README_in-tree index e50da6738..8d7d3ef52 100644 --- a/iscsi-scst/README_in-tree +++ b/iscsi-scst/README_in-tree @@ -9,7 +9,7 @@ ISCSI-SCST is a deeply reworked fork of iSCSI Enterprise Target (IET) - To fix all the problems, corner cases issues and iSCSI standard violations which IET has. -See more info at http://iscsi-scst.sourceforge.net. +See for more info http://iscsi-scst.sourceforge.net. Usage ----- @@ -731,3 +731,4 @@ Thanks to: * Bart Van Assche for a lot of help Vladislav Bolkhovitin , http://scst.sourceforge.net + diff --git a/iscsi-scst/doc/SCST_Gentoo_HOWTO.txt b/iscsi-scst/doc/SCST_Gentoo_HOWTO.txt index ed1f9b928..cc570e2c9 100644 --- a/iscsi-scst/doc/SCST_Gentoo_HOWTO.txt +++ b/iscsi-scst/doc/SCST_Gentoo_HOWTO.txt @@ -1,3 +1,5 @@ +This HOWTO is for iSCSI-SCST v1.x. Updates for v2.x are welcome! + 1) If you haven't done so already install a base gentoo installation: download: http://www.gentoo.org/main/en/where.xml documentation: http://www.gentoo.org/doc/en/handbook/index.xml diff --git a/iscsi-scst/doc/iscsi-scst-howto.txt b/iscsi-scst/doc/iscsi-scst-howto.txt index 971e17370..d36668620 100644 --- a/iscsi-scst/doc/iscsi-scst-howto.txt +++ b/iscsi-scst/doc/iscsi-scst-howto.txt @@ -6,7 +6,7 @@ Installing and using iSCSI-SCST with scstadmin Start with downloading the iSCSI-SCST source code. You can either download the released version from the following URL: - http://sourceforge.net/project/showfiles.php?group_id=110471 + http://scst.sourceforge.net/downloads.html or you can download the latest development version by running the following command in a shell: @@ -46,8 +46,9 @@ iSCSI-SCST includes the following components: * /etc/init.d/iscsi-scst - service script * iscsi-scstd - daemon * man pages - * Configuration files (located under iscsi-scst/etc/, need to be copied to - /etc if you want to use them): + * Obsolete configuration files located under iscsi-scst/etc/obsolete, need to be copied to + /etc if you want to use them. In iSCSI-SCST 2.0 it is recommended to use scstadmin + and SCST sysfs interface for that: * initiators.allow - used for assigning specific initiators to targets * initiators.deny - used for assigning specific initiators to targets * iscsi-scstd.conf - list of targets and their properties @@ -64,39 +65,60 @@ shown below. Verify the contents of the generated file. /etc/init.d/open-iscsi start cat /etc/iscsi/initiatorname.iscsi -3. Set up /etc/iscsi-scstd.conf +3. Set up /etc/scst.conf -You can do this by e.g. running the following shell commands: +The scst.conf configuration file is loaded at system boot time by the +shell script /etc/init.d/scst. All targets, devices and LUNs +configuration defined in this file. - echo "Target $(sed -n 's/InitiatorName=//p' /etc/iscsi/initiatorname.iscsi):storage" >/etc/iscsi-scstd.conf - cat /etc/iscsi-scstd.conf +Here is the sample config file. Copy it in /etc/scst.conf -4. Set up /etc/scst.conf +HANDLER dev_disk { +} -The scst.conf configuration file is loaded at system boot time by the shell -script /etc/init.d/scst. All devices known by SCST are defined in this file. -The shell statements shown below set up a configuration file for two vdisks. -Replace the device names by names that apply to your system: +HANDLER dev_cdrom { +} + +HANDLER vdisk_fileio { + DEVICE disk01 { + filename /dev/ram0 + nv_cache 1 + } + DEVICE disk02 { + filename /dev/ram1 + nv_cache 1 + } +} + +TARGET_DRIVER iscsi { + IncomingUser "joe 12charsecret" + + TARGET iqn.2006-10.net.vlnb:tgt { + IncomingUser "joe 12charsecret" + + LUN 0 disk01 + LUN 1 disk02 + + enabled 1 + } +} + +Then run the following commands: modprobe scst modprobe scst_vdisk - scstadmin -ClearConfig /etc/scst.conf - scstadmin -adddev disk01 -path /dev/ram0 -handler vdisk -options NV_CACHE - scstadmin -adddev disk02 -path /dev/ram1 -handler vdisk -options NV_CACHE - scstadmin -assigndev disk01 -group Default -lun 0 - scstadmin -assigndev disk02 -group Default -lun 1 - scstadmin -WriteConfig /etc/scst.conf - cat /etc/scst.conf + service iscsi-scst start + scstadmin -config /etc/scst.conf Note that for each SCSI device LUN 0 must exist. This is required by the SCSI protocol and is also documented in SCST's README file. -5. Edit /etc/init.d/scst +4. Edit /etc/init.d/scst Remove the modules you do not need from the SCST_MODULES variable, e.g. qla2x00tgt. -6. Create soft links in /etc/init.d +5. Create soft links in /etc/init.d This will make SCST to start automatically upon system boot. Use either chkconfig or update-rc.d, depending on the Linux distribution you are using. @@ -110,21 +132,21 @@ On openSUSE you can let SCST start automatically through the following commands: insserv scst insserv iscsi-scst -7. Restart SCST and iSCSI-SCST +6. Restart SCST and iSCSI-SCST If you don't want to reboot now, you can make the above settings effective via the following commands: - /etc/init.d/iscsi-scst stop /etc/init.d/scst stop - /etc/init.d/scst start + /etc/init.d/iscsi-scst stop /etc/init.d/iscsi-scst start + /etc/init.d/scst start Installing and using iSCSI-SCST without using scstadmin via /sys interface ========================================================================== -First repeat steps 1, 2 and 3 from the first section but leave out +First repeat steps 1 and 2 from the first section but leave out "scstadm scstadm_install" from the make command in step 1. Next load the scst_disk and scst_vdisk kernel modules as follows: @@ -266,25 +288,29 @@ echo 1 >/sys/kernel/scst_tgt/targets/iscsi/enabled Assigning targets to specific initiators ---------------------------------------- -In order to assign targets to specific initiators, you need to have -/etc/initiators.allow and /etc/initiators.deny. You can find -example files in etc/initiators.allow and etc/initiators.deny. +In order to assign targets to specific initiators, you should use LUN +masking defined above. If an initiator assigned to an empty security +group of a target, the initiator will not see this target in the +SendTargets discovery and will not be able to connect to it. -Note that all targets are allowed to all initiators by default, so if -you want to use /etc/initiators.allow, you will need to have -/etc/initiators.deny that looks like this: - -ALL ALL - -This will deny all initiators expect for those defined in /etc/initiators.allow. +Additionally, you can make the target be visible only via selected +portals using allowed_portal sysfs attribute. Read iSCSI-SCST's README +file for more details about this facility. Installing and using iSCSI-SCST without using scstadmin via /proc interface =========================================================================== -First repeat steps 1, 2 and 3 from the previous section but leave out +First repeat steps 1 and 2 from the previous section but leave out "scstadm scstadm_install" from the make command in step 1. +Next set up /etc/iscsi-scstd.conf + +You can do this by e.g. running the following shell commands: + + echo "Target $(sed -n 's/InitiatorName=//p' /etc/iscsi/initiatorname.iscsi):storage" >/etc/iscsi-scstd.conf + cat /etc/iscsi-scstd.conf + Next load the scst_disk kernel module as follows: modprobe scst_disk diff --git a/iscsi-scst/etc/initiators.allow b/iscsi-scst/etc/obsolete/initiators.allow similarity index 100% rename from iscsi-scst/etc/initiators.allow rename to iscsi-scst/etc/obsolete/initiators.allow diff --git a/iscsi-scst/etc/initiators.deny b/iscsi-scst/etc/obsolete/initiators.deny similarity index 100% rename from iscsi-scst/etc/initiators.deny rename to iscsi-scst/etc/obsolete/initiators.deny diff --git a/iscsi-scst/etc/iscsi-scstd.conf b/iscsi-scst/etc/obsolete/iscsi-scstd.conf similarity index 100% rename from iscsi-scst/etc/iscsi-scstd.conf rename to iscsi-scst/etc/obsolete/iscsi-scstd.conf diff --git a/iscsi-scst/etc/scst.conf b/iscsi-scst/etc/obsolete/scst.conf similarity index 100% rename from iscsi-scst/etc/scst.conf rename to iscsi-scst/etc/obsolete/scst.conf diff --git a/qla2x00t/qla2x00-target/README b/qla2x00t/qla2x00-target/README index 48d8daf45..1ea1fb2a2 100644 --- a/qla2x00t/qla2x00-target/README +++ b/qla2x00t/qla2x00-target/README @@ -86,17 +86,17 @@ core. It is highly recommended to use scstadmin utility for that purpose. Then target mode should be enabled via a sysfs interface on a per card -basis. Under the appropriate scsi_host there is an entry -target_mode_enabled, where you should write "1", like: - -echo "1" >/sys/class/scsi_host/host0/target_mode_enabled - -For the sysfs build you should instead +basis, like: echo "1" >/sys/kernel/scst_tgt/targets/qla2x00t/target/enabled See below for full description of the driver's sysfs interface. +With the obsolete proc interface you should instead use +target_mode_enabled under the appropriate scsi_host entry, like: + +echo "1" >/sys/class/scsi_host/host0/target_mode_enabled + You can find some installation and configuration HOWTOs in http://scst.sourceforge.net/qla2x00t-howto.html and https://forums.openfiler.com/viewtopic.php?id=3422. @@ -142,16 +142,14 @@ Explicit conformation --------------------- This option should (actually, almost always must) be enabled by echoing -"1" in /sys/class/scsi_host/hostX/explicit_conform_enabled, if a target -card exports at least one stateful SCSI device, like tape, and class 2 -isn't used, otherwise link-level errors could lead to loss of the -target/initiator state synchronization. Also check if initiator supports -this feature, it is reported in the kernel logs ("confirmed completion -supported" or not). No major performance degradation was noticed, if it -is enabled. Supported only for 23xx+. Disabled by default. - -In the sysfs build it is moved in /sys/kernel/scst_tgt/targets/qla2x00t/target/ -subdirectory, see the sysfs interface description below. +"1" in /sys/kernel/scst_tgt/targets/qla2x00t/target/host/explicit_conform_enabled, +if a target card exports at least one stateful SCSI device, like tape, +and class 2 isn't used, otherwise link-level errors could lead to loss +of the target/initiator state synchronization. Also check if initiator +supports this feature, it is reported in the kernel logs ("confirmed +completion supported" or not). No major performance degradation was +noticed, if it is enabled. Supported only for 23xx+. Disabled by +default. Class 2 @@ -159,9 +157,10 @@ Class 2 Class 2 is the close equivalent of the TCP in the IP world. If you enable it, all the Fibre Channel packets will be acknowledged. By -default, class 3 is used, which is UDP-like. Enable it by echoing -"1" in /sys/class/scsi_host/hostX/class2_enabled. This option needs -a special firmware with class 2 support. Disabled by default. +default, class 3 is used, which is UDP-like. Enable it by echoing "1" in +/sys/kernel/scst_tgt/targets/qla2x00t/target/host/class2_enabled. This +option needs a special firmware with class 2 support. Disabled by +default. Compilation options @@ -191,11 +190,9 @@ in/out in Makefile: Sysfs interface --------------- -Starting from 2.0.0 this driver has sysfs interface. You can switch to -it by running "make disable_proc". To switch back to the procfs -interface you should run "make enable_proc". The procfs interface from -version 2.0.0 is obsolete and will be removed in one of the next -versions. +Starting from 2.0.0 this driver has sysfs interface. The procfs +interface from version 2.0.0 is obsolete and will be removed in one of +the next versions. Root of SCST sysfs interface is /sys/kernel/scst_tgt. Root of this driver is /sys/kernel/scst_tgt/targets/qla2x00t. It has the following diff --git a/qla2x00t/qla2x00-target/qla2x00t-howto.html b/qla2x00t/qla2x00-target/qla2x00t-howto.html index c1ebdc998..495e6c4c0 100644 --- a/qla2x00t/qla2x00-target/qla2x00t-howto.html +++ b/qla2x00t/qla2x00-target/qla2x00t-howto.html @@ -44,12 +44,18 @@ How to configure QLogic target driver for 22xx/23xx/24xx/25xx adapters [root@proj src ] ln -s /usr/src/linux-2.6.26 linux
[root@proj src ] ln -s /usr/src/linux-2.6.26 kernel


-5: Get the SCST code from their SVN code base.
+5: Get the SCST code. It is possibile either from the SCST downloads page + or from the SVN code base.

+ + Below we will consider the case of the downloaded from the SVN code. For the release code everything is the same. + You only need unpack all downloaded SCST archives in one directory. It then will contain similar content as + ~scst/trunk/ below.

+ [root@proj ] svn co https://scst.svn.sourceforge.net/svnroot/scst
If the command returns successfully then you'll have 'scst'
directory under your current directory ( here it is /root )

- Please check if svn (open source version control system) is installed on + You need subversion (open source version control system) installed on your Linux box.


6: Change directory to ~scst/trunk/
@@ -187,7 +193,7 @@ How to configure QLogic target driver for 22xx/23xx/24xx/25xx adapters < snip from dmesg >

- scst: SCST version 1.0.0-rc1 loaded successfully (max mem for commands 16251MB, per device 6500MB)
+ scst: SCST version 2.0.0 loaded successfully (max mem for commands 16251MB, per device 6500MB)
scst: Enabled features: TRACING
scst: Target template qla2x00tgt registered successfully
scst: Virtual device handler vdisk for type 0 registered successfully
@@ -196,8 +202,7 @@ How to configure QLogic target driver for 22xx/23xx/24xx/25xx adapters --------
< snip from dmesg >


-15:To see the devices remotely, we have to add them to atleast "Default" security group.
- Where each security group will have LUN 0(LUs numeration must not start from, e.g., 1)

+15: Now let's create our virtual device:

[root@proj ] dd if=/dev/zero of=/mnt/disk1 bs=1024k count=512
@@ -210,33 +215,54 @@ How to configure QLogic target driver for 22xx/23xx/24xx/25xx adapters [root@proj ] file /mnt/disk1
/mnt/disk1: data

- - [root@proj ] echo "open vm_disk /mnt/disk1" > /proc/scsi_tgt/vdisk/vdisk
- - [root@proj ] echo "add vm_disk 0" >/proc/scsi_tgt/groups/Default/devices

- < snip from dmesg >
- dev_vdisk: Attached SCSI target virtual disk vm_disk (file="/mnt/disk1", fs=512MB, bs=512, nblocks=1048576, cyln=512)
- scst: Attached SCSI target mid-level to virtual device vm_disk (id 1)
- scst: Added device vm_disk to group Default (LUN 0, rd_only 0)
- < snip from dmesg >

- - [root@proj ] cat /proc/scsi_tgt/groups/Default/devices
- Device (host:ch:id:lun or name) Virtual lun Options
- vm_disk 0

+16:To see the device remotely, we have to add it to at least in the LUNs set for our target.
+ We must have LUN 0 (LUs numeration must not start from, e.g., 1).

- [root@proj ] cat /proc/scsi_tgt/vdisk/vdisk
- Name Size(MB) Block size Options File name
- vm_disk 512 512 /mnt/disk1


- - -16:After the drivers are loaded, the target mode should be enabled via a
+ The simplest way to do that is to write an scstadmin's config file:

- sysfs interface on a per card basis. Under the appropriate scsi_host - there is an entry target_mode_enabled, where you should write "1", like:
+HANDLER vdisk_fileio {
+  DEVICE disk1 {
+   filename /mnt/disk1
+  }
+}

- [root@proj ] echo "1" >/sys/class/scsi_host/host11/target_mode_enabled

+TARGET_DRIVER qla2x00t {
+  TARGET 25:00:00:f0:98:87:92:f3 {
+   LUN 0 disk1

+   enabled 1
+  }
+}

+ +To use it, we need to install scstadmin:

+ +[root@proj ]# make
+[root@proj ]# make install

+ +Let's save our config file as /etc/scst.conf

+ +Now run scstadmin:

+ +[root@proj ]# scstadmin -config /etc/scst.conf

+ +Collecting current configuration: done.

+ +-> Checking configuration file 'qq' for errors.
+ -> Done, 0 warnings found.

+ + -> Opening device 'disk1' using handler 'vdisk_fileio': done.
+ -> Adding device 'disk1' at LUN 0 to driver/target 'qla2x00t/25:00:00:f0:98:87:92:f3': done.
+ -> Enabling driver/target 'qla2x00t/25:00:00:f0:98:87:92:f3': done.

+ + All done.

+ + +The same can be done directly via SCST's sysfs interface:

+ +[root@proj ]# echo "add_device disk1 filename=/mnt/disk1" >/sys/kernel/scst_tgt/handlers/vdisk_fileio/mgmt
+[root@proj ]# echo "add disk1 0" >/sys/kernel/scst_tgt/targets/qla2x00t/25:00:00:f0:98:87:92:f3/luns/mgmt
+[root@proj ]# echo "1" >/sys/kernel/scst_tgt/targets/qla2x00t/25:00:00:f0:98:87:92:f3/enabled

ON INITIATOR

diff --git a/scripts/run-regression-tests b/scripts/run-regression-tests index 5bcbfbf61..56a219409 100755 --- a/scripts/run-regression-tests +++ b/scripts/run-regression-tests @@ -552,7 +552,7 @@ test_scst_tree_patches || exit $? compile_scst_unpatched || exit $? compile_scst_patched debug2release || exit $? compile_scst_patched debug2perf || exit $? -compile_scst_patched disable_proc || exit $? +compile_scst_patched enable_proc || exit $? first_iteration="true" for kv in ${kernel_versions} diff --git a/scst/README b/scst/README index da937adfa..857263c88 100644 --- a/scst/README +++ b/scst/README @@ -7,8 +7,7 @@ Version 2.0.0, XX XXXXX 2010 SCST is designed to provide unified, consistent interface between SCSI target drivers and Linux kernel and simplify target drivers development as much as possible. Detail description of SCST's features and internals -could be found in "Generic SCSI Target Middle Level for Linux" document -SCST's Internet page http://scst.sourceforge.net. +could be found on its Internet page http://scst.sourceforge.net. SCST supports the following I/O modes: @@ -16,30 +15,34 @@ SCST supports the following I/O modes: initiators can connect to the exported pass-through devices, for the following SCSI devices types: disks (type 0), tapes (type 1), processors (type 3), CDROMs (type 5), MO disks (type 7), medium - changers (type 8) and RAID controllers (type 0xC) + changers (type 8) and RAID controllers (type 0xC). * FILEIO mode, which allows to use files on file systems or block devices as virtual remotely available SCSI disks or CDROMs with - benefits of the Linux page cache + benefits of the Linux page cache. * BLOCKIO mode, which performs direct block IO with a block device, bypassing page-cache for all operations. This mode works ideally with high-end storage HBAs and for applications that either do not need caching between application and disk or need the large block - throughput + throughput. * User space mode using scst_user device handler, which allows to - implement in the user space virtual SCSI devices in the SCST - environment + implement in the user space high performance virtual SCSI + devices. Comparing with fully in-kernel dev handlers this mode has + very low overhead (few %%) * "Performance" device handlers, which provide in pseudo pass-through mode a way for direct performance measurements without overhead of - actual data transferring from/to underlying SCSI device + actual data transferring from/to underlying SCSI device. In addition, SCST supports advanced per-initiator access and devices visibility management, so different initiators could see different set of devices with different access permissions. See below for details. +Full list of SCST features and comparison with other Linux targets you +can find on http://scst.sourceforge.net/comparison.html. + Installation ------------ @@ -54,9 +57,10 @@ your compiler complains about redefinition of some symbol, you should either switch to vanilla kernel, or add or change as necessary the corresponding to that symbol "#if LINUX_VERSION_CODE" statement. -The sysfs build supports only kernels 2.6.26 and higher, because in -2.6.26 internal kernel's sysfs interface had a major change, which made -it heavily incompatible with pre-2.6.26 version. +Default sysfs interface supports only kernels 2.6.26 and higher, because +in 2.6.26 internal kernel's sysfs interface had a major change, which +made it heavily incompatible with pre-2.6.26 version. But with the +obsolete procfs interface kernels 2.6.16+ are supported. At first, make sure that the link "/lib/modules/`you_kernel_version`/build" points to the source code for your currently running kernel. @@ -114,10 +118,10 @@ are: - scst_vdisk - device handler for virtual disks (file, device or ISO CD image). - scst_user - user space device handler -Then, to see your devices remotely, you need to add them to at least -"Default" security group (see below how). By default, no local devices -are seen remotely. There must be LUN 0 in each security group, i.e. LUs -numeration must not start from, e.g., 1. Otherwise you will see no +Then, to see your devices remotely, you need to add a corresponding LUN +for them (see below how). By default, no local devices are seen +remotely. There must be LUN 0 in each LUNs set (security group), i.e. +LUs numeration must not start from, e.g., 1. Otherwise you will see no devices on remote initiators and SCST core will write into the kernel log message: "tgt_dev for LUN 0 not found, command to unexisting LU?" @@ -156,12 +160,31 @@ IMPORTANT: In the current version simultaneous access to local SCSI devices the state of devices and their parameters, because that could lead to data corruption. If any such command is done, at least related device handler(s) must be restarted. For block - devices READ/WRITE commands using direct disk handler look to - be safe. + devices READ/WRITE commands using direct disk handler are + generally safe. To uninstall, type 'make scst_uninstall'. +Migration from the obsolete proc interface +------------------------------------------ + +Sysfs enabled scstadmin supports old procfs config file format, so with +it you should do the following steps to migrate your proc-based +configuration to the sysfs interface: + +1. Load SCST modules + +2. Run "scstadmin -config old_config_file" + +3. Run "scstadmin -write_config new_config_file" + +4. Check new_config_file and make sure it has everything written +properly. + +5. Start using "scstadmin -config new_config_file" to configure SCST. + + Usage in failover mode ---------------------- @@ -231,22 +254,22 @@ in/out in Makefile: the various places. - CONFIG_SCST_USE_EXPECTED_VALUES - if not defined (default), initiator - supplied expected data transfer length and direction will be used only for - verification purposes to return error or warn in case if one of them - is invalid. Instead, locally decoded from SCSI command values will be - used. This is necessary for security reasons, because otherwise a - faulty initiator can crash target by supplying invalid value in one - of those parameters. This is especially important in case of - pass-through mode. If CONFIG_SCST_USE_EXPECTED_VALUES is defined, initiator - supplied expected data transfer length and direction will override - the locally decoded values. This might be necessary if internal SCST - commands translation table doesn't contain SCSI command, which is - used in your environment. You can know that if you have messages like - "Unknown opcode XX for YY. Should you update scst_scsi_op_table?" in - your kernel log and your initiator returns an error. Also report - those messages in the SCST mailing list - scst-devel@lists.sourceforge.net. Note, that not all SCSI transports - support supplying expected values. + supplied expected data transfer length and direction will be used + only for verification purposes to return error or warn in case if one + of them is invalid. Instead, locally decoded from SCSI command values + will be used. This is necessary for security reasons, because + otherwise a faulty initiator can crash target by supplying invalid + value in one of those parameters. This is especially important in + case of pass-through mode. If CONFIG_SCST_USE_EXPECTED_VALUES is + defined, initiator supplied expected data transfer length and + direction will override the locally decoded values. This might be + necessary if internal SCST commands translation table doesn't contain + SCSI command, which is used in your environment. You can know that if + you enable "minor" trace level and have messages like "Unknown + opcode XX for YY. Should you update scst_scsi_op_table?" in your + kernel log and your initiator returns an error. Also report those + messages in the SCST mailing list scst-devel@lists.sourceforge.net. + Note, that not all SCSI transports support supplying expected values. - CONFIG_SCST_DEBUG_TM - if defined, turns on task management functions debugging, when on LUN 6 some of the commands will be delayed for @@ -283,8 +306,8 @@ in/out in Makefile: CONFIG_QLA_TGT_DEBUG_WORK_IN_THREAD disabled - does. This option is designed to be used with vdisk NULLIO backend. - WARNING! Using with other backend than vdisk NULLIO is unsafe and can - lead you to a kernel crash! + WARNING! Using this option enabled with other backend than vdisk + NULLIO is unsafe and can lead you to a kernel crash! - CONFIG_SCST_STRICT_SECURITY - if defined, makes SCST zero allocated data buffers. Undefining it (default) considerably improves performance @@ -302,17 +325,16 @@ in/out in Makefile: Then sometimes get crazy itself. So, this option is disabled by default. - - CONFIG_SCST_MEASURE_LATENCY - if defined, provides in /proc/scsi_tgt/latency - file average commands processing latency. You can clear already - measured results by writing 0 in this file. For the sysfs build you - can find those results in /sys/kernel/scst_tgt and below. Note, you need a - non-preemptible kernel to have correct results. + - CONFIG_SCST_MEASURE_LATENCY - if defined, provides in "latency" files + global and per-LUN average commands processing latency statistic. You + can clear already measured results by writing 0 in each file. Note, + you need a non-preemptible kernel to have correct results. HIGHMEM kernel configurations are fully supported, but not recommended for performance reasons, except for scst_user, where they are not supported, because this module deals with user supplied memory on a -zero-copy manner. If you need to use it, consider change VMSPLIT option -or use 64-bit system configuration instead. +zero-copy manner. If you need to use HIGHMEM enabled, consider change +VMSPLIT option or use 64-bit system configuration instead. For changing VMSPLIT option (CONFIG_VMSPLIT to be precise) you should in "make menuconfig" command set the following variables: @@ -340,15 +362,1243 @@ Module scst supports the following parameters: consumed by the SCST commands for data buffers at any given time. By default it is approximately TotalMem/4. + - scst_max_dev_cmd_mem - sets maximum amount of memory in MB allowed + to be consumed by all SCSI commands of a device at any given time. By + default, it is approximately 2/5 of scst_max_cmd_mem. -SCST /proc interface + +SCST sysfs interface -------------------- -For communications with user space programs SCST provides proc-based -interface in /proc/scsi_tgt directory. This interface is available in -the procfs build only. Starting from version 2.0.0 it is obsolete and -will be removed in one of the next versions. It contains the following -entries. +Starting from 2.0.0 SCST has sysfs interface. It supports only kernels +2.6.26 and higher, because in 2.6.26 internal kernel's sysfs interface +had a major change, which made it heavily incompatible with pre-2.6.26 +version. If you need pre-2.6.26 kernel, you need to use obsolete procfs +interface (see below). + +Root of SCST sysfs interface is /sys/kernel/scst_tgt. It has the +following entries: + + - devices - this is a root subdirectory for all SCST devices + + - handlers - this is a root subdirectory for all SCST dev handlers + + - sgv - this is a root subdirectory for all SCST SGV caches + + - targets - this is a root subdirectory for all SCST targets + + - setup_id - allows to read and write SCST setup ID. This ID can be + used in cases, when the same SCST configuration should be installed + on several targets, but exported from those targets devices should + have different IDs and SNs. For instance, VDISK dev handler uses this + ID to generate T10 vendor specific identifier and SN of the devices. + + - threads - allows to read and set number of global SCST I/O threads. + Those threads used with async. dev handlers, for instance, vdisk + BLOCKIO or NULLIO. + + - trace_level - allows to enable and disable various tracing + facilities. See content of this file for help how to use it. + + - version - read-only attribute, which allows to see version of + SCST and enabled optional features. + + - last_sysfs_mgmt_res - read-only attribute returning completion status + of the last management command. In the sysfs implementation there are + some problems between internal sysfs and internal SCST locking. To + avoid them in some cases sysfs calls can return error with errno + EAGAIN. This doesn't mean the operation failed. It only means that + the operation queued and not yet completed. To wait for it to + complete, an management tool should poll this file. If the operation + hasn't yet completed, it will also return EAGAIN. But after it's + completed, it will return the result of this operation (0 for success + or -errno for error). + +Each SCST sysfs file (attribute) can contain in the last line mark +"[key]". It is automatically added mark used to allow scstadmin to see +which attributes it should save in the config file. You can ignore it. + +"Devices" subdirectory contains subdirectories for each SCST devices. + +Content of each device's subdirectory is dev handler specific. See +documentation for your dev handlers for more info about it as well as +SysfsRules file for more info about common to all dev handlers rules. +SCST dev handlers can have the following common entries: + + - exported - subdirectory containing links to all LUNs where this + device was exported. + + - handler - if dev handler determined for this device, this link points + to it. The handler can be not set for pass-through devices. + + - threads_num - shows and allows to set number of threads in this device's + threads pool. If 0 - no threads will be created, and global SCST + threads pool will be used. If <0 - creation of the threads pool is + prohibited. + + - threads_pool_type - shows and allows to sets threads pool type. + Possible values: "per_initiator" and "shared". When the value is + "per_initiator" (default), each session from each initiator will use + separate dedicated pool of threads. When the value is "shared", all + sessions from all initiators will share the same per-device pool of + threads. Valid only if threads_num attribute >0. + + - dump_prs - allows to dump persistent reservations information in the + kernel log. + + - type - SCSI type of this device + +See below for more information about other entries of this subdirectory +of the standard SCST dev handlers. + +"Handlers" subdirectory contains subdirectories for each SCST dev +handler. + +Content of each handler's subdirectory is dev handler specific. See +documentation for your dev handlers for more info about it as well as +SysfsRules file for more info about common to all dev handlers rules. +SCST dev handlers can have the following common entries: + + - mgmt - this entry allows to create virtual devices and their + attributes (for virtual devices dev handlers) or assign/unassign real + SCSI devices to/from this dev handler (for pass-through dev + handlers). + + - trace_level - allows to enable and disable various tracing + facilities. See content of this file for help how to use it. + + - type - SCSI type of devices served by this dev handler. + +See below for more information about other entries of this subdirectory +of the standard SCST dev handlers. + +"Sgv" subdirectory contains statistic information of SCST SGV caches. It +has the following entries: + + - None, one or more subdirectories for each existing SGV cache. + + - global_stats - file containing global SGV caches statistics. + +Each SGV cache's subdirectory has the following item: + + - stats - file containing statistics for this SGV caches. + +"Targets" subdirectory contains subdirectories for each SCST target. + +Content of each target's subdirectory is target specific. See +documentation for your target for more info about it as well as +SysfsRules file for more info about common to all targets rules. +Every target should have at least the following entries: + + - ini_groups - subdirectory, which contains and allows to define + initiator-oriented access control information, see below. + + - luns - subdirectory, which contains list of available LUNs in the + target-oriented access control and allows to define it, see below. + + - sessions - subdirectory containing connected to this target sessions. + + - enabled - using this attribute you can enable or disable this target/ + It allows to finish configuring it before it starts accepting new + connections. 0 by default. + + - addr_method - used LUNs addressing method. Possible values: + "Peripheral" and "Flat". Most initiators work well with Peripheral + addressing method (default), but some (HP-UX, for instance) may + require Flat method. This attribute is also available in the + initiators security groups, so you can assign the addressing method + on per-initiator basis. + + - io_grouping_type - defines how I/O from sessions to this target are + grouped together. This I/O grouping is very important for + performance. By setting this attribute in a right value, you can + considerably increase performance of your setup. This grouping is + performed only if you use CFQ I/O scheduler on the target and for + devices with threads_num >= 0 and, if threads_num > 0, with + threads_pool_type "per_initiator". Possible values: + "this_group_only", "never", "auto", or I/O group number >0. When the + value is "this_group_only" all I/O from all sessions in this target + will be grouped together. When the value is "never", I/O from + different sessions will not be grouped together, i.e. all sessions in + this target will have separate dedicated I/O groups. When the value + is "auto" (default), all I/O from initiators with the same name + (iSCSI initiator name, for instance) in all targets will be grouped + together with a separate dedicated I/O group for each initiator name. + For iSCSI this mode works well, but other transports usually use + different initiator names for different sessions, so using such + transports in MPIO configurations you should either use value + "this_group_only", or an explicit I/O group number. This attribute is + also available in the initiators security groups, so you can assign + the I/O grouping on per-initiator basis. See below for more info how + to use this attribute. + + - rel_tgt_id - allows to read or write SCSI Relative Target Port + Identifier attribute. This identifier is used to identify SCSI Target + Ports by some SCSI commands, mainly by Persistent Reservations + commands. This identifier must be unique among all SCST targets, but + for convenience SCST allows disabled targets to have not unique + rel_tgt_id. In this case SCST will not allow to enable this target + until rel_tgt_id becomes unique. This attribute initialized unique by + SCST by default. + +A target driver may have also the following entries: + + - "hw_target" - if the target driver supports both hardware and virtual + targets (for instance, an FC adapter supporting NPIV, which has + hardware targets for its physical ports as well as virtual NPIV + targets), this read only attribute for all hardware targets will + exist and contain value 1. + +Subdirectory "sessions" contains one subdirectory for each connected +session with name equal to name of the connected initiator. + +Each session subdirectory contains the following entries: + + - initiator_name - contains initiator name + + - force_close - optional write-only attribute, which allows to force + close this session. + + - active_commands - contains number of active, i.e. not yet or being + executed, SCSI commands in this session. + + - commands - contains overall number of SCSI commands in this session. + + - latency - if CONFIG_SCST_MEASURE_LATENCY enabled, contains latency + statistics for this session. + + - luns - a link pointing out to the corresponding LUNs set (security + group) where this session was attached to. + + - One or more "lunX" subdirectories, where 'X' is a number, for each LUN + this session has (see below). + + - other target driver specific attributes and subdirectories. + +See below description of the VDISK's sysfs interface for samples. + + +Access and devices visibility management (LUN masking) +------------------------------------------------------ + +Access and devices visibility management allows for an initiator or +group of initiators to see different devices with different LUNs +with necessary access permissions. + +SCST supports two modes of access control: + +1. Target-oriented. In this mode you define for each target a default +set of LUNs, which are accessible to all initiators, connected to that +target. This is a regular access control mode, which people usually mean +thinking about access control in general. For instance, in IET this is +the only supported mode. + +2. Initiator-oriented. In this mode you define which LUNs are accessible +for each initiator. In this mode you should create for each set of one +or more initiators, which should access to the same set of devices with +the same LUNs, a separate security group, then add to it devices and +names of allowed initiator(s). + +Both modes can be used simultaneously. In this case the +initiator-oriented mode has higher priority, than the target-oriented, +i.e. initiators are at first searched in all defined security groups for +this target and, if none matches, the default target's set of LUNs is +used. This set of LUNs might be empty, then the initiator will not see +any LUNs from the target. + +You can at any time find out which set of LUNs each session is assigned +to by looking where link +/sys/kernel/scst_tgt/targets/target_driver/target_name/sessions/initiator_name/luns +points to. + +To configure the target-oriented access control SCST provides the +following interface. Each target's sysfs subdirectory +(/sys/kernel/scst_tgt/targets/target_driver/target_name) has "luns" +subdirectory. This subdirectory contains the list of already defined +target-oriented access control LUNs for this target as well as file +"mgmt". This file has the following commands, which you can send to it, +for instance, using "echo" shell command. You can always get a small +help about supported commands by looking inside this file. "Parameters" +are one or more param_name=value pairs separated by ';'. + + - "add H:C:I:L lun [parameters]" - adds a pass-through device with + host:channel:id:lun with LUN "lun". Optionally, the device could be + marked as read only by using parameter "read_only". The recommended + way to find out H:C:I:L numbers is use of lsscsi utility. + + - "replace H:C:I:L lun [parameters]" - replaces by pass-through device + with host:channel:id:lun existing with LUN "lun" device with + generation of INQUIRY DATA HAS CHANGED Unit Attention. If the old + device doesn't exist, this command acts as the "add" command. + Optionally, the device could be marked as read only by using + parameter "read_only". The recommended way to find out H:C:I:L + numbers is use of lsscsi utility. + + - "add VNAME lun [parameters]" - adds a virtual device with name VNAME + with LUN "lun". Optionally, the device could be marked as read only + by using parameter "read_only". + + - "replace VNAME lun [parameters]" - replaces by virtual device + with name VNAME existing with LUN "lun" device with generation of + INQUIRY DATA HAS CHANGED Unit Attention. If the old device doesn't + exist, this command acts as the "add" command. Optionally, the device + could be marked as read only by using parameter "read_only". + + - "del lun" - deletes LUN lun + + - "clear" - clears the list of devices + +To configure the initiator-oriented access control SCST provides the +following interface. Each target's sysfs subdirectory +(/sys/kernel/scst_tgt/targets/target_driver/target_name) has "ini_groups" +subdirectory. This subdirectory contains the list of already defined +security groups for this target as well as file "mgmt". This file has +the following commands, which you can send to it, for instance, using +"echo" shell command. You can always get a small help about supported +commands by looking inside this file. + + - "create GROUP_NAME" - creates a new security group. + + - "del GROUP_NAME" - deletes a new security group. + +Each security group's subdirectory contains 2 subdirectories: initiators +and luns. + +Each "initiators" subdirectory contains list of added to this groups +initiator as well as as well as file "mgmt". This file has the following +commands, which you can send to it, for instance, using "echo" shell +command. You can always get a small help about supported commands by +looking inside this file. + + - "add INITIATOR_NAME" - adds initiator with name INITIATOR_NAME to the + group. + + - "del INITIATOR_NAME" - deletes initiator with name INITIATOR_NAME + from the group. + + - "move INITIATOR_NAME DEST_GROUP_NAME" moves initiator with name + INITIATOR_NAME from the current group to group with name + DEST_GROUP_NAME. + + - "clear" - deletes all initiators from this group. + +For "add" and "del" commands INITIATOR_NAME can be a simple DOS-type +patterns, containing '*' and '?' symbols. '*' means match all any +symbols, '?' means match only any single symbol. For instance, +"blah.xxx" will match "bl?h.*". Additionally, you can use negative sign +'!' to revert the value of the pattern. For instance, "ah.xxx" will +match "!bl?h.*". + +Each "luns" subdirectory contains the list of already defined LUNs for +this group as well as file "mgmt". Content of this file as well as list +of available in it commands is fully identical to the "luns" +subdirectory of the target-oriented access control. + +Examples: + + - echo "create INI" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ini_groups/mgmt - + creates security group INI for target iqn.2006-10.net.vlnb:tgt1. + + - echo "add 2:0:1:0 11" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ini_groups/INI/luns/mgmt - + adds a pass-through device sitting on host 2, channel 0, ID 1, LUN 0 + to group with name INI as LUN 11. + + - echo "add disk1 0" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ini_groups/INI/luns/mgmt - + adds a virtual disk with name disk1 to group with name INI as LUN 0. + + - echo "add 21:*:e0:?b:83:*" >/sys/kernel/scst_tgt/targets/21:00:00:a0:8c:54:52:12/ini_groups/INI/initiators/mgmt - + adds a pattern to group with name INI to Fibre Channel target with + WWN 21:00:00:a0:8c:54:52:12, which matches WWNs of Fibre Channel + initiator ports. + +Consider you need to have an iSCSI target with name +"iqn.2007-05.com.example:storage.disk1.sys1.xyz", which should export +virtual device "dev1" with LUN 0 and virtual device "dev2" with LUN 1, +but initiator with name +"iqn.2007-05.com.example:storage.disk1.spec_ini.xyz" should see only +virtual device "dev2" read only with LUN 0. To achieve that you should +do the following commands: + +# echo "iqn.2007-05.com.example:storage.disk1.sys1.xyz" >/sys/kernel/scst_tgt/targets/iscsi/mgmt +# echo "add dev1 0" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2007-05.com.example:storage.disk1.sys1.xyz/luns/mgmt +# echo "add dev2 1" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2007-05.com.example:storage.disk1.sys1.xyz/luns/mgmt +# echo "create SPEC_INI" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2007-05.com.example:storage.disk1.sys1.xyz/ini_groups/mgmt +# echo "add dev2 0 read_only=1" \ + >/sys/kernel/scst_tgt/targets/iscsi/iqn.2007-05.com.example:storage.disk1.sys1.xyz/ini_groups/SPEC_INI/luns/mgmt +# echo "iqn.2007-05.com.example:storage.disk1.spec_ini.xyz" \ + >/sys/kernel/scst_tgt/targets/iscsi/iqn.2007-05.com.example:storage.disk1.sys1.xyz/ini_groups/SPEC_INI/initiators/mgmt + +For Fibre Channel or SAS in the above example you should use target's +and initiator ports WWNs instead of iSCSI names. + +It is highly recommended to use scstadmin utility instead of described +in this section low level interface. + +IMPORTANT +========= + +There must be LUN 0 in each set of LUNs, i.e. LUs numeration must not +start from, e.g., 1. Otherwise you will see no devices on remote +initiators and SCST core will write into the kernel log message: "tgt_dev +for LUN 0 not found, command to unexisting LU?" + +IMPORTANT +========= + +All the access control must be fully configured BEFORE the corresponding +target is enabled. When you enable a target, it will immediately start +accepting new connections, hence creating new sessions, and those new +sessions will be assigned to security groups according to the +*currently* configured access control settings. For instance, to +the default target's set of LUNs, instead of "HOST004" group as you may +need, because "HOST004" doesn't exist yet. So, you must configure all +the security groups before new connections from the initiators are +created, i.e. before the target enabled. + + +VDISK device handler +-------------------- + +Starting from 2.0.0 VDISK device handler uses sysfs interface. The +procfs interface is obsolete and will be removed in one of the next +versions. + +VDISK has 4 built-in dev handlers: vdisk_fileio, vdisk_blockio, +vdisk_nullio and vcdrom. Roots of their sysfs interface are +/sys/kernel/scst_tgt/handlers/handler_name, e.g. for vdisk_fileio: +/sys/kernel/scst_tgt/handlers/vdisk_fileio. Each root has the following +entries: + + - None, one or more links to devices with name equal to names + of the corresponding devices. + + - trace_level - allows to enable and disable various tracing + facilities. See content of this file for help how to use it. + + - mgmt - main management entry, which allows to add/delete VDISK + devices with the corresponding type. + +The "mgmt" file has the following commands, which you can send to it, +for instance, using "echo" shell command. You can always get a small +help about supported commands by looking inside this file. "Parameters" +are one or more param_name=value pairs separated by ';'. + + - echo "add_device device_name [parameters]" - adds a virtual device + with name device_name and specified parameters (see below) + + - echo "del_device device_name" - deletes a virtual device with name + device_name. + +Handler vdisk_fileio provides FILEIO mode to create virtual devices. +This mode uses as backend files and accesses to them using regular +read()/write() file calls. This allows to use full power of Linux page +cache. The following parameters possible for vdisk_fileio: + + - filename - specifies path and file name of the backend file. The path + must be absolute. + + - blocksize - specifies block size used by this virtual device. The + block size must be power of 2 and >= 512 bytes. Default is 512. + + - write_through - disables write back caching. Note, this option + has sense only if you also *manually* disable write-back cache in + *all* your backstorage devices and make sure it's actually disabled, + since many devices are known to lie about this mode to get better + benchmark results. Default is 0. + + - read_only - read only. Default is 0. + + - o_direct - disables both read and write caching. This mode isn't + currently fully implemented, you should use user space fileio_tgt + program in O_DIRECT mode instead (see below). + + - nv_cache - enables "non-volatile cache" mode. In this mode it is + assumed that the target has a GOOD UPS with ability to cleanly + shutdown target in case of power failure and it is software/hardware + bugs free, i.e. all data from the target's cache are guaranteed + sooner or later to go to the media. Hence all data synchronization + with media operations, like SYNCHRONIZE_CACHE, are ignored in order + to bring more performance. Also in this mode target reports to + initiators that the corresponding device has write-through cache to + disable all write-back cache workarounds used by initiators. Use with + extreme caution, since in this mode after a crash of the target + journaled file systems don't guarantee the consistency after journal + recovery, therefore manual fsck MUST be ran. Note, that since usually + the journal barrier protection (see "IMPORTANT" note below) turned + off, enabling NV_CACHE could change nothing from data protection + point of view, since no data synchronization with media operations + will go from the initiator. This option overrides "write_through" + option. Disabled by default. + + - removable - with this flag set the device is reported to remote + initiators as removable. + +Handler vdisk_blockio provides BLOCKIO mode to create virtual devices. +This mode performs direct block I/O with a block device, bypassing the +page cache for all operations. This mode works ideally with high-end +storage HBAs and for applications that either do not need caching +between application and disk or need the large block throughput. See +below for more info. + +The following parameters possible for vdisk_blockio: filename, +blocksize, nv_cache, read_only, removable. See vdisk_fileio above for +description of those parameters. + +Handler vdisk_nullio provides NULLIO mode to create virtual devices. In +this mode no real I/O is done, but success returned to initiators. +Intended to be used for performance measurements at the same way as +"*_perf" handlers. The following parameters possible for vdisk_nullio: +blocksize, read_only, removable. See vdisk_fileio above for description +of those parameters. + +Handler vcdrom allows emulation of a virtual CDROM device using an ISO +file as backend. It doesn't have any parameters. + +For example: + +echo "add_device disk1 filename=/disk1; blocksize=4096; nv_cache=1" >/sys/kernel/scst_tgt/handlers/vdisk_fileio/mgmt + +will create a FILEIO virtual device disk1 with backend file /disk1 +with block size 4K and NV_CACHE enabled. + +Each vdisk_fileio's device has the following attributes in +/sys/kernel/scst_tgt/devices/device_name: + + - filename - contains path and file name of the backend file. + + - blocksize - contains block size used by this virtual device. + + - write_through - contains status of write back caching of this virtual + device. + + - read_only - contains read only status of this virtual device. + + - o_direct - contains O_DIRECT status of this virtual device. + + - nv_cache - contains NV_CACHE status of this virtual device. + + - removable - contains removable status of this virtual device. + + - size_mb - contains size of this virtual device in MB. + + - t10_dev_id - contains and allows to set T10 vendor specific + identifier for Device Identification VPD page (0x83) of INQUIRY data. + By default VDISK handler always generates t10_dev_id for every new + created device at creation time based on the device name and + scst_vdisk_ID scst_vdisk.ko module parameter (see below). + + - usn - contains the virtual device's serial number of INQUIRY data. It + is created at the device creation time based on the device name and + scst_vdisk_ID scst_vdisk.ko module parameter (see below). + + - type - contains SCSI type of this virtual device. + + - resync_size - write only attribute, which makes vdisk_fileio to + rescan size of the backend file. It is useful if you changed it, for + instance, if you resized it. + +For example: + +/sys/kernel/scst_tgt/devices/disk1 +|-- blocksize +|-- exported +| |-- export0 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt/luns/0 +| |-- export1 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt/ini_groups/INI/luns/0 +| |-- export2 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt1/luns/0 +| |-- export3 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ini_groups/INI1/luns/0 +| |-- export4 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ini_groups/INI2/luns/0 +|-- filename +|-- handler -> ../../handlers/vdisk_fileio +|-- nv_cache +|-- o_direct +|-- read_only +|-- removable +|-- resync_size +|-- size_mb +|-- t10_dev_id +|-- threads_num +|-- threads_pool_type +|-- type +|-- usn +`-- write_through + +Each vdisk_blockio's device has the following attributes in +/sys/kernel/scst_tgt/devices/device_name: blocksize, filename, nv_cache, +read_only, removable, resync_size, size_mb, t10_dev_id, threads_num, +threads_pool_type, type, usn. See above description of those parameters. + +Each vdisk_nullio's device has the following attributes in +/sys/kernel/scst_tgt/devices/device_name: blocksize, read_only, +removable, size_mb, t10_dev_id, threads_num, threads_pool_type, type, +usn. See above description of those parameters. + +Each vcdrom's device has the following attributes in +/sys/kernel/scst_tgt/devices/device_name: filename, size_mb, +t10_dev_id, threads_num, threads_pool_type, type, usn. See above +description of those parameters. Exception is filename attribute. For +vcdrom it is writable. Writing to it allows to virtually insert or +change virtual CD media in the virtual CDROM device. For example: + + - echo "/image.iso" >/sys/kernel/scst_tgt/devices/cdrom/filename - will + insert file /image.iso as virtual media to the virtual CDROM cdrom. + + - echo "" >/sys/kernel/scst_tgt/devices/cdrom/filename - will remove + "media" from the virtual CDROM cdrom. + +Additionally VDISK handler has module parameter "num_threads", which +specifies count of I/O threads for each FILEIO VDISK's or VCDROM device. +If you have a workload, which tends to produce rather random accesses +(e.g. DB-like), you should increase this count to a bigger value, like +32. If you have a rather sequential workload, you should decrease it to +a lower value, like number of CPUs on the target or even 1. Due to some +limitations of Linux I/O subsystem, increasing number of I/O threads too +much leads to sequential performance drop, especially with deadline +scheduler, so decreasing it can improve sequential performance. The +default provides a good compromise between random and sequential +accesses. + +You shouldn't be afraid to have too many VDISK I/O threads if you have +many VDISK devices. Kernel threads consume very little amount of +resources (several KBs) and only necessary threads will be used by SCST, +so the threads will not trash your system. + +CAUTION: If you partitioned/formatted your device with block size X, *NEVER* +======== ever try to export and then mount it (even accidentally) with another + block size. Otherwise you can *instantly* damage it pretty + badly as well as all your data on it. Messages on initiator + like: "attempt to access beyond end of device" is the sign of + such damage. + + Moreover, if you want to compare how well different block sizes + work for you, you **MUST** EVERY TIME AFTER CHANGING BLOCK SIZE + **COMPLETELY** **WIPE OFF** ALL THE DATA FROM THE DEVICE. In + other words, THE **WHOLE** DEVICE **MUST** HAVE ONLY **ZEROS** + AS THE DATA AFTER YOU SWITCH TO NEW BLOCK SIZE. Switching block + sizes isn't like switching between FILEIO and BLOCKIO, after + changing block size all previously written with another block + size data MUST BE ERASED. Otherwise you will have a full set of + very weird behaviors, because blocks addressing will be + changed, but initiators in most cases will not have a + possibility to detect that old addresses written on the device + in, e.g., partition table, don't refer anymore to what they are + intended to refer. + +IMPORTANT: Some disk and partition table management utilities don't support +========= block sizes >512 bytes, therefore make sure that your favorite one + supports it. Currently only cfdisk is known to work only with + 512 bytes blocks, other utilities like fdisk on Linux or + standard disk manager on Windows are proved to work well with + non-512 bytes blocks. Note, if you export a disk file or + device with some block size, different from one, with which + it was already partitioned, you could get various weird + things like utilities hang up or other unexpected behavior. + Hence, to be sure, zero the exported file or device before + the first access to it from the remote initiator with another + block size. On Window initiator make sure you "Set Signature" + in the disk manager on the imported from the target drive + before doing any other partitioning on it. After you + successfully mounted a file system over non-512 bytes block + size device, the block size stops matter, any program will + work with files on such file system. + + +Persistent Reservations +----------------------- + +SCST implements Persistent Reservations with full set of capabilities, +including "Persistence Through Power Loss". + +The "Persistence Through Power Loss" data are saved in /var/lib/scst/pr +with files with names the same as the names of the corresponding +devices. Also this directory contains backup versions of those files +with suffix ".1". Those backup files are used in case of power or other +failure to prevent Persistent Reservation information from corruption +during update. + +The "Persistence Through Power Loss" feature is not available in the +procfs build, because the SCST proc interface doesn't allow to keep +persistent Relative Target IDs of each target between reboots/reloads +(they are load and initialization order dependent). + +The Persistent Reservations available on all transports implementing +get_initiator_port_transport_id() callback. Transports not implementing +this callback will act in one of 2 possible scenarios ("all or +nothing"): + +1. If a device has such transport connected and doesn't have persistent +reservations, it will refuse Persistent Reservations commands as if it +doesn't support them. + +2. If a device has persistent reservations, all initiators newly +connecting via such transports will not see this device. After all +persistent reservations from this device are released, upon reconnect +the initiators will see it. + + +Caching +------- + +By default for performance reasons VDISK FILEIO devices use write back +caching policy. + +Generally, write back caching is safe for use and danger of it is +greatly overestimated, because most modern (especially, Enterprise +level) applications are well prepared to work with write back cached +storage. Particularly, such are all transactions-based applications. +Those applications flush cache to completely avoid ANY data loss on a +crash or power failure. For instance, journaled file systems flush cache +on each meta data update, so they survive power/hardware/software +failures pretty well. + +Since locally on initiators write back caching is always on, if an +application cares about its data consistency, it does flush the cache +when necessary or on any write, if open files with O_SYNC. If it doesn't +care, it doesn't flush the cache. As soon as the cache flushes +propagated to the storage, write back caching on it doesn't make any +difference. If application doesn't flush the cache, it's doomed to loose +data in case of a crash or power failure doesn't matter where this cache +located, locally or on the storage. + +To illustrate that consider, for example, a user who wants to copy /src +directory to /dst directory reliably, i.e. after the copy finished no +power failure or software/hardware crash could lead to a loss of the +data in /dst. There are 2 ways to achieve this. Let's suppose for +simplicity cp opens files for writing with O_SYNC flag, hence bypassing +the local cache. + +1. Slow. Make the device behind /dst working in write through caching +mode and then run "cp -a /src /dst". + +2. Fast. Let the device behind /dst working in write back caching mode +and then run "cp -a /src /dst; sync". The reliability of the result is +the same, but it's much faster than (1). Nobody would care if a crash +happens during the copy, because after recovery simply leftovers from +the not completed attempt would be deleted and the operation would be +restarted from the very beginning. + +So, you can see in (2) there is no danger of ANY data loss from the +write back caching. Moreover, since on practice cp doesn't open files +for writing with O_SYNC flag, to get the copy done reliably, sync +command must be called after cp anyway, so enabling write back caching +wouldn't make any difference for reliability. + +Also you can consider it from another side. Modern HDDs have at least +16MB of cache working in write back mode by default, so for a 10 drives +RAID it is 160MB of a write back cache. How many people are happy with +it and how many disabled write back cache of their HDDs? Almost all and +almost nobody correspondingly? Moreover, many HDDs lie about state of +their cache and report write through while working in write back mode. +They are also successfully used. + +Note, Linux I/O subsystem guarantees to propagated cache flushes to the +storage only using data protection barriers, which usually turned off by +default (see http://lwn.net/Articles/283161). Without barriers enabled +Linux doesn't provide a guarantee that after sync()/fsync() all written +data really hit permanent storage. They can be stored in the cache of +your backstorage devices and, hence, lost on a power failure event. +Thus, ever with write-through cache mode, you still either need to +enable barriers on your backend file system on the target (for direct +/dev/sdX devices this is, indeed, impossible), or need a good UPS to +protect yourself from not committed data loss. Some info about barriers +from the XFS point of view could be found at +http://oss.sgi.com/projects/xfs/faq.html#wcache. On Linux initiators for +Ext3 and ReiserFS file systems the barrier protection could be turned on +using "barrier=1" and "barrier=flush" mount options correspondingly. You +can check if the barriers turn on or off by looking in /proc/mounts. +Windows and, AFAIK, other UNIX'es don't need any special explicit +options and do necessary barrier actions on write-back caching devices +by default. + +To limit this data loss with write back caching you can use files in +/proc/sys/vm to limit amount of unflushed data in the system cache. + +If you for some reason have to use VDISK FILEIO devices in write through +caching mode, don't forget to disable internal caching on their backend +devices or make sure they have additional battery or supercapacitors +power supply on board. Otherwise, you still on a power failure would +loose all the unsaved yet data in the devices internal cache. + +Note, on some real-life workloads write through caching might perform +better, than write back one with the barrier protection turned on. + + +BLOCKIO VDISK mode +------------------ + +This module works best for these types of scenarios: + +1) Data that are not aligned to 4K sector boundaries and <4K block sizes +are used, which is normally found in virtualization environments where +operating systems start partitions on odd sectors (Windows and it's +sector 63). + +2) Large block data transfers normally found in database loads/dumps and +streaming media. + +3) Advanced relational database systems that perform their own caching +which prefer or demand direct IO access and, because of the nature of +their data access, can actually see worse performance with +non-discriminate caching. + +4) Multiple layers of targets were the secondary and above layers need +to have a consistent view of the primary targets in order to preserve +data integrity which a page cache backed IO type might not provide +reliably. + +Also it has an advantage over FILEIO that it doesn't copy data between +the system cache and the commands data buffers, so it saves a +considerable amount of CPU power and memory bandwidth. + +IMPORTANT: Since data in BLOCKIO and FILEIO modes are not consistent between +========= each other, if you try to use a device in both those modes + simultaneously, you will almost instantly corrupt your data + on that device. + +IMPORTANT: Some kernels starting from 2.6.32 have a problem, which prevents +========= prevents BLOCKIO from working correctly with RAID5/DM. See + http://lkml.org/lkml/2010/7/28/315. That problem was fixed in + 2.6.32.19, 2.6.34.4, 2.6.35.2 and 2.6.36-rc1. It is strongly + recommended to not use affected kernels with BLOCKIO. + +IMPORTANT: In SCST 1.x BLOCKIO worked by default in NV_CACHE mode, when +========= each device reported to remote initiators as having write through + caching. But if your backend block device has internal write + back caching it might create a possibility for data loss of + the cached in the internal cache data in case of a power + failure. Starting from SCST 2.0 BLOCKIO works by default in + non-NV_CACHE mode, when each device reported to remote + initiators as having write back caching, and synchronizes the + internal device's cache on each SYNCHRONIZE_CACHE command + from the initiators. It might lead to some *PERFORMANCE LOSS*, + so if you are are sure in your power supply and want to + restore the 1.x behavior, your should recreate your BLOCKIO + devices in NV_CACHE mode. + + +Pass-through mode +----------------- + +In the pass-through mode (i.e. using the pass-through device handlers +scst_disk, scst_tape, etc) SCSI commands, coming from remote initiators, +are passed to local SCSI devices on target as is, without any +modifications. + +SCST supports 1 to many pass-through, when several initiators can safely +connect a single pass-through device (a tape, for instance). For such +cases SCST emulates all the necessary functionality. + +In the sysfs interface all real SCSI devices are listed in +/sys/kernel/scst_tgt/devices in form host:channel:id:lun numbers, for +instance 1:0:0:0. The recommended way to match those numbers to your +devices is use of lsscsi utility. + +Each pass-through dev handler has in its root subdirectory +/sys/kernel/scst_tgt/handlers/handler_name, e.g. +/sys/kernel/scst_tgt/handlers/dev_disk, "mgmt" file. It allows the +following commands. They can be sent to it using, e.g., echo command. + + - "add_device" - this command assigns SCSI device with +host:channel:id:lun numbers to this dev handler. + +echo "add_device 1:0:0:0" >/sys/kernel/scst_tgt/handlers/dev_disk/mgmt + +will assign SCSI device 1:0:0:0 to this dev handler. + + - "del_device" - this command unassigns SCSI device with +host:channel:id:lun numbers from this dev handler. + +As usually, on read the "mgmt" file returns small help about available +commands. + +You need to manually assign each your real SCSI device to the +corresponding pass-through dev handler using the "add_device" command, +otherwise the real SCSI devices will not be visible remotely. The +assignment isn't done automatically, because it could lead to the +pass-through dev handlers load and initialization problems if any of the +local real SCSI devices are malfunctioning. + +As any other hardware, the local SCSI hardware can not handle commands +with amount of data and/or segments count in scatter-gather array bigger +some values. Therefore, when using the pass-through mode you should note +that values for maximum number of segments and maximum amount of +transferred data for each SCSI command on devices on initiators can not +be bigger, than corresponding values of the corresponding SCSI devices +on the target. Otherwise you will see symptoms like small transfers work +well, but large ones stall and messages like: "Unable to complete +command due to SG IO count limitation" are printed in the kernel logs. + +You can't control from the user space limit of the scatter-gather +segments, but for block devices usually it is sufficient if you set on +the initiators /sys/block/DEVICE_NAME/queue/max_sectors_kb in the same +or lower value as in /sys/block/DEVICE_NAME/queue/max_hw_sectors_kb for +the corresponding devices on the target. + +For not-block devices SCSI commands are usually generated directly by +applications, so, if you experience large transfers stalls, you should +check documentation for your application how to limit the transfer +sizes. + +Another way to solve this issue is to build SG entries with more than 1 +page each. See the following patch as an example: +http://scst.sourceforge.net/sgv_big_order_alloc.diff + + +User space mode using scst_user dev handler +------------------------------------------- + +User space program fileio_tgt uses interface of scst_user dev handler +and allows to see how it works in various modes. Fileio_tgt provides +mostly the same functionality as scst_vdisk handler with the most +noticeable difference that it supports O_DIRECT mode. O_DIRECT mode is +basically the same as BLOCKIO, but also supports files, so for some +loads it could be significantly faster, than the regular FILEIO access. +All the words about BLOCKIO from above apply to O_DIRECT as well. See +fileio_tgt's README file for more details. + + +Performance +----------- + +SCST from the very beginning has been designed and implemented to +provide the best possible performance. Since there is no "one fit all" +the best performance configuration for different setups and loads, SCST +provides extensive set of settings to allow to tune it for the best +performance in each particular case. You don't have to necessary use +those settings. If you don't, SCST will do very good job to autotune for +you, so the resulting performance will, in average, be better +(sometimes, much better) than with other SCSI targets. But in some cases +you can by manual tuning improve it even more. + +If you want to get maximum performance from your target, RHEL/CentOS 5.x +kernels are not recommended on both target and initiators, if you are +using Linux initiators, because those kernels are based on very outdated +2.6.18 kernel, hence, missed >3 years of important improvements in the +kernel's storage area. You should use at least long maintained vanilla +2.6.27.x kernel, although 2.6.29+ would be even better. + +Before doing any performance measurements note that performance results +are very much dependent from your type of load, so it is crucial that +you choose access mode (FILEIO, BLOCKIO, O_DIRECT, pass-through), which +suits your needs the best. + +In order to get the maximum performance you should: + +1. For SCST: + + - Disable in Makefile CONFIG_SCST_STRICT_SERIALIZING, CONFIG_SCST_EXTRACHECKS, + CONFIG_SCST_TRACING, CONFIG_SCST_DEBUG*, CONFIG_SCST_STRICT_SECURITY, + CONFIG_SCST_MEASURE_LATENCY + +2. For target drivers: + + - Disable in Makefiles CONFIG_SCST_EXTRACHECKS, CONFIG_SCST_TRACING, + CONFIG_SCST_DEBUG* + +3. For device handlers, including VDISK: + + - Disable in Makefile CONFIG_SCST_TRACING and CONFIG_SCST_DEBUG. + +IMPORTANT: The development version of SCST in the SVN is optimized for +========= development and bug hunting, not for performance. To reconfigure + it for performance you should run "make debug2perf" in the + root of your source code (e.g. trunk/). It will set the above + options as needed. The only option it doesn't set is + CONFIG_SCST_TEST_IO_IN_SIRQ, so, if needed, you should change + it manually. + +IMPORTANT: You can't use debug SCST drivers with non-debug SCST core. +========= So, after disabling both CONFIG_SCST_TRACING and CONFIG_SCST_DEBUG + for SCST core you have to disable them for all SCST drivers + you are using as well. + +4. Make sure you have io_grouping_type option set correctly, especially +in the following cases: + + - Several initiators share your target's backstorage. It can be a + shared LU using some cluster FS, like VMFS, as well as can be + different LUs located on the same backstorage (RAID array). For + instance, if you have 3 initiators and each of them using its own + dedicated FILEIO device file from the same RAID-6 array on the + target. + + In this case for the best performance you should have + io_grouping_type option set in value "never" in all the LUNs' targets + and security groups. + + - Your initiator connected to your target in MPIO mode. In this case for + the best performance you should: + + * Either connect all the sessions from the initiator to a single + target or security group and have io_grouping_type option set in + value "this_group_only" in the target or security group, + + * Or, if it isn't possible to connect all the sessions from the + initiator to a single target or security group, assign the same + numeric io_grouping_type value for each target/security group this + initiator connected to. The exact value itself doesn't matter, + important only that all the targets/security groups use the same + value. + +Don't forget, io_grouping_type makes sense only if you use CFQ I/O +scheduler on the target and for devices with threads_num >= 0 and, if +threads_num > 0, with threads_pool_type "per_initiator". + +You can check if in your setup io_grouping_type set correctly as well as +if the "auto" io_grouping_type value works for you by tests like the +following: + + - For not MPIO case you can run single thread sequential reading, e.g. + using buffered dd, from one initiator, then run the same single + thread sequential reading from the second initiator in parallel. If + io_grouping_type is set correctly the aggregate throughput measured + on the target should only slightly decrease as well as all initiators + should have nearly equal share of it. If io_grouping_type is not set + correctly, the aggregate throughput and/or throughput on any + initiator will decrease significantly, in 2 times or even more. For + instance, you have 80MB/s single thread sequential reading from the + target on any initiator. When then both initiators are reading in + parallel you should see on the target aggregate throughput something + like 70-75MB/s with correct io_grouping_type and something like + 35-40MB/s or 8-10MB/s on any initiator with incorrect. + + - For the MPIO case it's quite easier. With incorrect io_grouping_type + you simply won't see performance increase from adding the second + session (assuming your hardware is capable to transfer data through + both sessions in parallel), or can even see a performance decrease. + +5. If you are going to use your target in an VM environment, for +instance as a shared storage with VMware, make sure all your VMs +connected to the target via *separate* sessions. For instance, for iSCSI +it means that each VM has own connection to the target, not all VMs +connected using a single connection. You can check it using SCST sysfs +interface. For other transports you should use available facilities, +like NPIV for Fibre Channel, to make separate sessions for each VM. If +you miss it, you can greatly loose performance of parallel access to +your target from different VMs. This isn't related to the case if your +VMs are using the same shared storage, like with VMFS, for instance. In +this case all your VM hosts will be connected to the target via separate +sessions, which is enough. + +6. For other target and initiator software parts: + + - Make sure you applied on your kernel all available SCST patches. + If for your kernel version this patch doesn't exist, it is strongly + recommended to upgrade your kernel to version, for which this patch + exists. + + - Don't enable debug/hacking features in the kernel, i.e. use them as + they are by default. + + - The default kernel read-ahead and queuing settings are optimized + for locally attached disks, therefore they are not optimal if they + attached remotely (SCSI target case), which sometimes could lead to + unexpectedly low throughput. You should increase read-ahead size to at + least 512KB or even more on all initiators and the target. + + You should also limit on all initiators maximum amount of sectors per + SCSI command. This tuning is also recommended on targets with large + read-ahead values. To do it on Linux, run: + + echo “64” > /sys/block/sdX/queue/max_sectors_kb + + where specify instead of X your imported from target device letter, + like 'b', i.e. sdb. + + To increase read-ahead size on Linux, run: + + blockdev --setra N /dev/sdX + + where N is a read-ahead number in 512-byte sectors and X is a device + letter like above. + + Note: you need to set read-ahead setting for device sdX again after + you changed the maximum amount of sectors per SCSI command for that + device. + + Note2: you need to restart SCST after you changed read-ahead settings + on the target. + + - You may need to increase amount of requests that OS on initiator + sends to the target device. To do it on Linux initiators, run + + echo “64” > /sys/block/sdX/queue/nr_requests + + where X is a device letter like above. + + You may also experiment with other parameters in /sys/block/sdX + directory, they also affect performance. If you find the best values, + please share them with us. + + - On the target use CFQ IO scheduler. In most cases it has performance + advantage over other IO schedulers, sometimes huge (2+ times + aggregate throughput increase). + + - It is recommended to turn the kernel preemption off, i.e. set + the kernel preemption model to "No Forced Preemption (Server)". + + - Looks like XFS is the best filesystem on the target to store device + files, because it allows considerably better linear write throughput, + than ext3. + +7. For hardware on target. + + - Make sure that your target hardware (e.g. target FC or network card) + and underlaying IO hardware (e.g. IO card, like SATA, SCSI or RAID to + which your disks connected) don't share the same PCI bus. You can + check it using lspci utility. They have to work in parallel, so it + will be better if they don't compete for the bus. The problem is not + only in the bandwidth, which they have to share, but also in the + interaction between cards during that competition. This is very + important, because in some cases if target and backend storage + controllers share the same PCI bus, it could lead up to 5-10 times + less performance, than expected. Moreover, some motherboard (by + Supermicro, particularly) have serious stability issues if there are + several high speed devices on the same bus working in parallel. If + you have no choice, but PCI bus sharing, set in the BIOS PCI latency + as low as possible. + +8. If you use VDISK IO module in FILEIO mode, NV_CACHE option will +provide you the best performance. But using it make sure you use a good +UPS with ability to shutdown the target on the power failure. + +Baseline performance numbers you can find in those measurements: +http://lkml.org/lkml/2009/3/30/283. + +IMPORTANT: If you use on initiator some versions of Windows (at least W2K) +========= you can't get good write performance for VDISK FILEIO devices with + default 512 bytes block sizes. You could get about 10% of the + expected one. This is because of the partition alignment, which + is (simplifying) incompatible with how Linux page cache + works, so for each write the corresponding block must be read + first. Use 4096 bytes block sizes for VDISK devices and you + will have the expected write performance. Actually, any OS on + initiators, not only Windows, will benefit from block size + max(PAGE_SIZE, BLOCK_SIZE_ON_UNDERLYING_FS), where PAGE_SIZE + is the page size, BLOCK_SIZE_ON_UNDERLYING_FS is block size + on the underlying FS, on which the device file located, or 0, + if a device node is used. Both values are from the target. + See also important notes about setting block sizes >512 bytes + for VDISK FILEIO devices above. + + +9. In some cases, for instance working with SSD devices, which consume 100% +of a single CPU load for data transfers in their internal threads, to +maximize IOPS it can be needed to assign for those threads dedicated +CPUs using Linux CPU affinity facilities. No IRQ processing should be +done on those CPUs. Check that using /proc/interrupts. See taskset +command and Documentation/IRQ-affinity.txt in your kernel's source tree +for how to assign IRQ affinity to tasks and IRQs. + +The reason for that is that processing of coming commands in SIRQ +context might be done on the same CPUs as SSD devices' threads doing data +transfers. As the result, those threads won't receive all the processing +power of those CPUs and perform worse. + + +Work if target's backstorage or link is too slow +------------------------------------------------ + +Under high I/O load, when your target's backstorage gets overloaded, or +working over a slow link between initiator and target, when the link +can't serve all the queued commands on time, you can experience I/O +stalls or see in the kernel log abort or reset messages. + +At first, consider the case of too slow target's backstorage. On some +seek intensive workloads even fast disks or RAIDs, which able to serve +continuous data stream on 500+ MB/s speed, can be as slow as 0.3 MB/s. +Another possible cause for that can be MD/LVM/RAID on your target as in +http://lkml.org/lkml/2008/2/27/96 (check the whole thread as well). + +Thus, in such situations simply processing of one or more commands takes +too long time, hence initiator decides that they are stuck on the target +and tries to recover. Particularly, it is known that the default amount +of simultaneously queued commands (48) is sometimes too high if you do +intensive writes from VMware on a target disk, which uses LVM in the +snapshot mode. In this case value like 16 or even 8-10 depending of your +backstorage speed could be more appropriate. + +Unfortunately, currently SCST lacks dynamic I/O flow control, when the +queue depth on the target is dynamically decreased/increased based on +how slow/fast the backstorage speed comparing to the target link. So, +there are 6 possible actions, which you can do to workaround or fix this +issue in this case: + +1. Ignore incoming task management (TM) commands. It's fine if there are +not too many of them, so average performance isn't hurt and the +corresponding device isn't getting put offline, i.e. if the backstorage +isn't too slow. + +2. Decrease /sys/block/sdX/device/queue_depth on the initiator in case +if it's Linux (see below how) or/and SCST_MAX_TGT_DEV_COMMANDS constant +in scst_priv.h file until you stop seeing incoming TM commands. +ISCSI-SCST driver also has its own iSCSI specific parameter for that, +see its README file. + +To decrease device queue depth on Linux initiators you can run command: + +# echo Y >/sys/block/sdX/device/queue_depth + +where Y is the new number of simultaneously queued commands, X - your +imported device letter, like 'a' for sda device. There are no special +limitations for Y value, it can be any value from 1 to possible maximum +(usually, 32), so start from dividing the current value on 2, i.e. set +16, if /sys/block/sdX/device/queue_depth contains 32. + +3. Increase the corresponding timeout on the initiator. For Linux it is +located in +/sys/devices/platform/host*/session*/target*:0:0/*:0:0:1/timeout. It can +be done automatically by an udev rule. For instance, the following +rule will increase it to 300 seconds: + +SUBSYSTEM=="scsi", KERNEL=="[0-9]*:[0-9]*", ACTION=="add", ATTR{type}=="0|7|14", ATTR{timeout}="300" + +By default, this timeout is 30 or 60 seconds, depending on your distribution. + +4. Try to avoid such seek intensive workloads. + +5. Increase speed of the target's backstorage. + +6. Implement in SCST dynamic I/O flow control. This will be an ultimate +solution. See "Dynamic I/O flow control" section on +http://scst.sourceforge.net/contributing.html page for possible +implementation idea. + +Next, consider the case of too slow link between initiator and target, +when the initiator tries to simultaneously push N commands to the target +over it. In this case time to serve those commands, i.e. send or receive +data for them over the link, can be more, than timeout for any single +command, hence one or more commands in the tail of the queue can not be +served on time less than the timeout, so the initiator will decide that +they are stuck on the target and will try to recover. + +To workaround/fix this issue in this case you can use ways 1, 2, 3, 6 +above or (7): increase speed of the link between target and initiator. +But for some initiators implementations for WRITE commands there might +be cases when target has no way to detect the issue, so dynamic I/O flow +control will not be able to help. In those cases you could also need on +the initiator(s) to either decrease the queue depth (way 2), or increase +the corresponding timeout (way 3). + +Note, that logged messages about QUEUE_FULL status are quite different +by nature. This is a normal work, just SCSI flow control in action. +Simply don't enable "mgmt_minor" logging level, or, alternatively, if +you are confident in the worst case performance of your back-end storage +or initiator-target link, you can increase SCST_MAX_TGT_DEV_COMMANDS in +scst_priv.h to 64. Usually initiators don't try to push more commands on +the target. + + +Obsolete /proc interface +------------------------ + +Also for communications with user space programs SCST provides +proc-based interface in /proc/scsi_tgt directory. This interface is +available in the procfs build only. Starting from version 2.0.0 it is +obsolete and will be removed in one of the next versions. To switch in +the procfs build you need to run "make enable_proc" command before +building anything else. + +It contains the following entries. - "help" file, which provides online help for SCST commands @@ -580,388 +1830,12 @@ initiators or add new devices to old groups, but not altering existing LUNs in them. -SCST sysfs interface --------------------- - -Starting from 2.0.0 SCST has sysfs interface. You can switch to it by -running "make disable_proc". To switch back to the procfs interface you -should run "make enable_proc". The sysfs build supports only kernels -2.6.26 and higher, because in 2.6.26 internal kernel's sysfs interface -had a major change, which made it heavily incompatible with pre-2.6.26 -version. - -Root of SCST sysfs interface is /sys/kernel/scst_tgt. It has the -following entries: - - - devices - this is a root subdirectory for all SCST devices - - - handlers - this is a root subdirectory for all SCST dev handlers - - - sgv - this is a root subdirectory for all SCST SGV caches - - - targets - this is a root subdirectory for all SCST targets - - - setup_id - allows to read and write SCST setup ID. This ID can be - used in cases, when the same SCST configuration should be installed - on several targets, but exported from those targets devices should - have different IDs and SNs. For instance, VDISK dev handler uses this - ID to generate T10 vendor specific identifier and SN of the devices. - - - threads - allows to read and set number of global SCST I/O threads. - Those threads used with async. dev handlers, for instance, vdisk - BLOCKIO or NULLIO. - - - trace_level - allows to enable and disable various tracing - facilities. See content of this file for help how to use it. - - - version - read-only attribute, which allows to see version of - SCST and enabled optional features. - -Each SCST sysfs file (attribute) can contain in the last line mark -"[key]". It is automatically added mark used to allow scstadmin to see -which attributes it should save in the config file. You can ignore it. - -"Devices" subdirectory contains subdirectories for each SCST devices. - -Content of each device's subdirectory is dev handler specific. See -documentation for your dev handlers for more info about it as well as -SysfsRules file for more info about common to all dev handlers rules. -Standard SCST dev handlers have at least the following common entries: - - - exported - subdirectory containing links to all LUNs where this - device was exported. - - - handler - if dev handler determined for this device, this link points - to it. The handler can be not set for pass-through devices. - - - threads_num - shows and allows to set number of threads in this device's - threads pool. If 0 - no threads will be created, and global SCST - threads pool will be used. If <0 - creation of the threads pool is - prohibited. - - - threads_pool_type - shows and allows to sets threads pool type. - Possible values: "per_initiator" and "shared". When the value is - "per_initiator" (default), each session from each initiator will use - separate dedicated pool of threads. When the value is "shared", all - sessions from all initiators will share the same per-device pool of - threads. Valid only if threads_num attribute >0. - - - dump_prs - allows to dump persistent reservations information in the - kernel log. - - - type - SCSI type of this device - -See below for more information about other entries of this subdirectory -of the standard SCST dev handlers. - -"Handlers" subdirectory contains subdirectories for each SCST dev -handler. - -Content of each handler's subdirectory is dev handler specific. See -documentation for your dev handlers for more info about it as well as -SysfsRules file for more info about common to all dev handlers rules. -Standard SCST dev handlers have at least the following common entries: - - - mgmt - this entry allows to create virtual devices and their - attributes (for virtual devices dev handlers) or assign/unassign real - SCSI devices to/from this dev handler (for pass-through dev - handlers). - - - trace_level - allows to enable and disable various tracing - facilities. See content of this file for help how to use it. - - - type - SCSI type of devices served by this dev handler. - -See below for more information about other entries of this subdirectory -of the standard SCST dev handlers. - -"Sgv" subdirectory contains statistic information of SCST SGV caches. It -has the following entries: - - - None, one or more subdirectories for each existing SGV cache. - - - global_stats - file containing global SGV caches statistics. - -Each SGV cache's subdirectory has the following item: - - - stats - file containing statistics for this SGV caches. - -"Targets" subdirectory contains subdirectories for each SCST target. - -Content of each target's subdirectory is target specific. See -documentation for your target for more info about it as well as -SysfsRules file for more info about common to all targets rules. -Every target should have at least the following entries: - - - ini_groups - subdirectory, which contains and allows to define - initiator-oriented access control information, see below. - - - luns - subdirectory, which contains list of available LUNs in the - target-oriented access control and allows to define it, see below. - - - sessions - subdirectory containing connected to this target sessions. - - - enabled - using this attribute you can enable or disable this target/ - It allows to finish configuring it before it starts accepting new - connections. 0 by default. - - - addr_method - used LUNs addressing method. Possible values: - "Peripheral" and "Flat". Most initiators work well with Peripheral - addressing method (default), but some (HP-UX, for instance) may - require Flat method. This attribute is also available in the - initiators security groups, so you can assign the addressing method - on per-initiator basis. - - - io_grouping_type - defines how I/O from sessions to this target are - grouped together. This I/O grouping is very important for - performance. By setting this attribute in a right value, you can - considerably increase performance of your setup. This grouping is - performed only if you use CFQ I/O scheduler on the target and for - devices with threads_num >= 0 and, if threads_num > 0, with - threads_pool_type "per_initiator". Possible values: - "this_group_only", "never", "auto", or I/O group number >0. When the - value is "this_group_only" all I/O from all sessions in this target - will be grouped together. When the value is "never", I/O from - different sessions will not be grouped together, i.e. all sessions in - this target will have separate dedicated I/O groups. When the value - is "auto" (default), all I/O from initiators with the same name - (iSCSI initiator name, for instance) in all targets will be grouped - together with a separate dedicated I/O group for each initiator name. - For iSCSI this mode works well, but other transports usually use - different initiator names for different sessions, so using such - transports in MPIO configurations you should either use value - "this_group_only", or an explicit I/O group number. This attribute is - also available in the initiators security groups, so you can assign - the I/O grouping on per-initiator basis. See below for more info how - to use this attribute. - - - rel_tgt_id - allows to read or write SCSI Relative Target Port - Identifier attribute. This identifier is used to identify SCSI Target - Ports by some SCSI commands, mainly by Persistent Reservations - commands. This identifier must be unique among all SCST targets, but - for convenience SCST allows disabled targets to have not unique - rel_tgt_id. In this case SCST will not allow to enable this target - until rel_tgt_id becomes unique. This attribute initialized unique by - SCST by default. - -A target driver may have also the following entries: - - - "hw_target" - if the target driver supports both hardware and virtual - targets (for instance, an FC adapter supporting NPIV, which has - hardware targets for its physical ports as well as virtual NPIV - targets), this read only attribute for all hardware targets will - exist and contain value 1. - -Subdirectory "sessions" contains one subdirectory for each connected -session with name equal to name of the connected initiator. - -Each session subdirectory contains the following entries: - - - initiator_name - contains initiator name - - - force_close - optional write-only attribute, which allows to force - close this session. - - - active_commands - contains number of active, i.e. not yet or being - executed, SCSI commands in this session. - - - commands - contains overall number of SCSI commands in this session. - - - other target driver specific attributes and subdirectories. - -See below description of the VDISK's sysfs interface for samples. - - -Access and devices visibility management (LUN masking) - sysfs interface ------------------------------------------------------------------------- - -Access and devices visibility management allows for an initiator or -group of initiators to see different devices with different LUNs -with necessary access permissions. - -SCST supports two modes of access control: - -1. Target-oriented. In this mode you define for each target a default -set of LUNs, which are accessible to all initiators, connected to that -target. This is a regular access control mode, which people usually mean -thinking about access control in general. For instance, in IET this is -the only supported mode. - -2. Initiator-oriented. In this mode you define which LUNs are accessible -for each initiator. In this mode you should create for each set of one -or more initiators, which should access to the same set of devices with -the same LUNs, a separate security group, then add to it devices and -names of allowed initiator(s). - -Both modes can be used simultaneously. In this case the -initiator-oriented mode has higher priority, than the target-oriented, -i.e. initiators are at first searched in all defined security groups for -this target and, if none matches, the default target's set of LUNs is -used. This set of LUNs might be empty, then the initiator will not see -any LUNs from the target. - -You can at any time find out which set of LUNs each session is assigned -to by looking where link -/sys/kernel/scst_tgt/targets/target_driver/target_name/sessions/initiator_name/luns -points to. - -To configure the target-oriented access control SCST provides the -following interface. Each target's sysfs subdirectory -(/sys/kernel/scst_tgt/targets/target_driver/target_name) has "luns" -subdirectory. This subdirectory contains the list of already defined -target-oriented access control LUNs for this target as well as file -"mgmt". This file has the following commands, which you can send to it, -for instance, using "echo" shell command. You can always get a small -help about supported commands by looking inside this file. "Parameters" -are one or more param_name=value pairs separated by ';'. - - - "add H:C:I:L lun [parameters]" - adds a pass-through device with - host:channel:id:lun with LUN "lun". Optionally, the device could be - marked as read only by using parameter "read_only". The recommended - way to find out H:C:I:L numbers is use of lsscsi utility. - - - "replace H:C:I:L lun [parameters]" - replaces by pass-through device - with host:channel:id:lun existing with LUN "lun" device with - generation of INQUIRY DATA HAS CHANGED Unit Attention. If the old - device doesn't exist, this command acts as the "add" command. - Optionally, the device could be marked as read only by using - parameter "read_only". The recommended way to find out H:C:I:L - numbers is use of lsscsi utility. - - - "del H:C:I:L" - deletes a pass-through device with host:channel:id:lun - The recommended way to find out H:C:I:L numbers is use of lsscsi - utility. - - - "add VNAME lun [parameters]" - adds a virtual device with name VNAME - with LUN "lun". Optionally, the device could be marked as read only - by using parameter "read_only". - - - "replace VNAME lun [parameters]" - replaces by virtual device - with name VNAME existing with LUN "lun" device with generation of - INQUIRY DATA HAS CHANGED Unit Attention. If the old device doesn't - exist, this command acts as the "add" command. Optionally, the device - could be marked as read only by using parameter "read_only". - - - "del VNAME" - deletes a virtual device with name VNAME. - - - "clear" - clears the list of devices - -To configure the initiator-oriented access control SCST provides the -following interface. Each target's sysfs subdirectory -(/sys/kernel/scst_tgt/targets/target_driver/target_name) has "ini_groups" -subdirectory. This subdirectory contains the list of already defined -security groups for this target as well as file "mgmt". This file has -the following commands, which you can send to it, for instance, using -"echo" shell command. You can always get a small help about supported -commands by looking inside this file. - - - "create GROUP_NAME" - creates a new security group. - - - "del GROUP_NAME" - deletes a new security group. - -Each security group's subdirectory contains 2 subdirectories: initiators -and luns. - -Each "initiators" subdirectory contains list of added to this groups -initiator as well as as well as file "mgmt". This file has the following -commands, which you can send to it, for instance, using "echo" shell -command. You can always get a small help about supported commands by -looking inside this file. - - - "add INITIATOR_NAME" - adds initiator with name INITIATOR_NAME to the - group. - - - "del INITIATOR_NAME" - deletes initiator with name INITIATOR_NAME - from the group. - - - "move INITIATOR_NAME DEST_GROUP_NAME" moves initiator with name - INITIATOR_NAME from the current group to group with name - DEST_GROUP_NAME. - - - "clear" - deletes all initiators from this group. - -For "add" and "del" commands INITIATOR_NAME can be a simple DOS-type -patterns, containing '*' and '?' symbols. '*' means match all any -symbols, '?' means match only any single symbol. For instance, -"blah.xxx" will match "bl?h.*". Additionally, you can use negative sign -'!' to revert the value of the pattern. For instance, "ah.xxx" will -match "!bl?h.*". - -Each "luns" subdirectory contains the list of already defined LUNs for -this group as well as file "mgmt". Content of this file as well as list -of available in it commands is fully identical to the "luns" -subdirectory of the target-oriented access control. - -Examples: - - - echo "create INI" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ini_groups/mgmt - - creates security group INI for target iqn.2006-10.net.vlnb:tgt1. - - - echo "add 2:0:1:0 11" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ini_groups/INI/luns/mgmt - - adds a pass-through device sitting on host 2, channel 0, ID 1, LUN 0 - to group with name INI as LUN 11. - - - echo "add disk1 0" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ini_groups/INI/luns/mgmt - - adds a virtual disk with name disk1 to group with name INI as LUN 0. - - - echo "add 21:*:e0:?b:83:*" >/sys/kernel/scst_tgt/targets/21:00:00:a0:8c:54:52:12/ini_groups/INI/initiators/mgmt - - adds a pattern to group with name INI to Fibre Channel target with - WWN 21:00:00:a0:8c:54:52:12, which matches WWNs of Fibre Channel - initiator ports. - -Consider you need to have an iSCSI target with name -"iqn.2007-05.com.example:storage.disk1.sys1.xyz", which should export -virtual device "dev1" with LUN 0 and virtual device "dev2" with LUN 1, -but initiator with name -"iqn.2007-05.com.example:storage.disk1.spec_ini.xyz" should see only -virtual device "dev2" read only with LUN 0. To achieve that you should -do the following commands: - -# echo "iqn.2007-05.com.example:storage.disk1.sys1.xyz" >/sys/kernel/scst_tgt/targets/iscsi/mgmt -# echo "add dev1 0" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2007-05.com.example:storage.disk1.sys1.xyz/luns/mgmt -# echo "add dev2 1" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2007-05.com.example:storage.disk1.sys1.xyz/luns/mgmt -# echo "create SPEC_INI" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2007-05.com.example:storage.disk1.sys1.xyz/ini_groups/mgmt -# echo "add dev2 0 read_only=1" \ - >/sys/kernel/scst_tgt/targets/iscsi/iqn.2007-05.com.example:storage.disk1.sys1.xyz/ini_groups/SPEC_INI/luns/mgmt -# echo "iqn.2007-05.com.example:storage.disk1.spec_ini.xyz" \ - >/sys/kernel/scst_tgt/targets/iscsi/iqn.2007-05.com.example:storage.disk1.sys1.xyz/ini_groups/SPEC_INI/initiators/mgmt - -For Fibre Channel or SAS in the above example you should use target's -and initiator ports WWNs instead of iSCSI names. - -It is highly recommended to use scstadmin utility instead of described -in this section low level interface. - -IMPORTANT -========= - -There must be LUN 0 in each set of LUNs, i.e. LUs numeration must not -start from, e.g., 1. Otherwise you will see no devices on remote -initiators and SCST core will write into the kernel log message: "tgt_dev -for LUN 0 not found, command to unexisting LU?" - -IMPORTANT -========= - -All the access control must be fully configured BEFORE the corresponding -target is enabled! When you enable a target, it will immediately start -accepting new connections, hence creating new sessions, and those new -sessions will be assigned to security groups according to the -*currently* configured access control settings. For instance, to -the default target's set of LUNs, instead of "HOST004" group as you may -need, because "HOST004" doesn't exist yet. So, one must configure all -the security groups before new connections from the initiators are -created, i.e. before the target enabled. - - -VDISK device handler --------------------- - -/proc interface -~~~~~~~~~~~~~~~ +VDISK /proc interface +--------------------- This interface starting from version 2.0.0 is obsolete and will be -removed in one of the next versions. +removed in one of the next versions. To switch to it you should run +"make enable_proc". After loading VDISK device handler creates in /proc/scsi_tgt/ subdirectories "vdisk" and "vcdrom". They have the following layout: @@ -1044,833 +1918,6 @@ mode is used. For example, "echo "open disk1 /vdisks/disk1" >/proc/scsi_tgt/vdisk/vdisk" will open file /vdisks/disk1 as virtual FILEIO disk with name "disk1". -/sys interface -~~~~~~~~~~~~~~ - -Starting from 2.0.0 VDISK device handler has sysfs interface. You can -switch to it by running "make disable_proc". To switch back to the -procfs interface you should run "make enable_proc". The procfs interface -starting from version 2.0.0 is obsolete and will be removed in one of -the next versions. - -VDISK has 4 built-in dev handlers: vdisk_fileio, vdisk_blockio, -vdisk_nullio and vcdrom. Roots of their sysfs interface are -/sys/kernel/scst_tgt/handlers/handler_name, e.g. for vdisk_fileio: -/sys/kernel/scst_tgt/handlers/vdisk_fileio. Each root has the following -entries: - - - None, one or more links to devices with name equal to names - of the corresponding devices. - - - trace_level - allows to enable and disable various tracing - facilities. See content of this file for help how to use it. - - - mgmt - main management entry, which allows to add/delete VDISK - devices with the corresponding type. - -The "mgmt" file has the following commands, which you can send to it, -for instance, using "echo" shell command. You can always get a small -help about supported commands by looking inside this file. "Parameters" -are one or more param_name=value pairs separated by ';'. - - - echo "add_device device_name [parameters]" - adds a virtual device - with name device_name and specified parameters (see below) - - - echo "del_device device_name" - deletes a virtual device with name - device_name. - -Handler vdisk_fileio provides FILEIO mode to create virtual devices. -This mode uses as backend files and accesses to them using regular -read()/write() file calls. This allows to use full power of Linux page -cache. The following parameters possible for vdisk_fileio: - - - filename - specifies path and file name of the backend file. The path - must be absolute. - - - blocksize - specifies block size used by this virtual device. The - block size must be power of 2 and >= 512 bytes. Default is 512. - - - write_through - disables write back caching. Note, this option - has sense only if you also *manually* disable write-back cache in - *all* your backstorage devices and make sure it's actually disabled, - since many devices are known to lie about this mode to get better - benchmark results. Default is 0. - - - read_only - read only. Default is 0. - - - o_direct - disables both read and write caching. This mode isn't - currently fully implemented, you should use user space fileio_tgt - program in O_DIRECT mode instead (see below). - - - nv_cache - enables "non-volatile cache" mode. In this mode it is - assumed that the target has a GOOD UPS with ability to cleanly - shutdown target in case of power failure and it is software/hardware - bugs free, i.e. all data from the target's cache are guaranteed - sooner or later to go to the media. Hence all data synchronization - with media operations, like SYNCHRONIZE_CACHE, are ignored in order - to bring more performance. Also in this mode target reports to - initiators that the corresponding device has write-through cache to - disable all write-back cache workarounds used by initiators. Use with - extreme caution, since in this mode after a crash of the target - journaled file systems don't guarantee the consistency after journal - recovery, therefore manual fsck MUST be ran. Note, that since usually - the journal barrier protection (see "IMPORTANT" note below) turned - off, enabling NV_CACHE could change nothing from data protection - point of view, since no data synchronization with media operations - will go from the initiator. This option overrides "write_through" - option. Disabled by default. - - - removable - with this flag set the device is reported to remote - initiators as removable. - -Handler vdisk_blockio provides BLOCKIO mode to create virtual devices. -This mode performs direct block I/O with a block device, bypassing the -page cache for all operations. This mode works ideally with high-end -storage HBAs and for applications that either do not need caching -between application and disk or need the large block throughput. See -below for more info. - -The following parameters possible for vdisk_blockio: filename, -blocksize, nv_cache, read_only, removable. See vdisk_fileio above for -description of those parameters. - -Handler vdisk_nullio provides NULLIO mode to create virtual devices. In -this mode no real I/O is done, but success returned to initiators. -Intended to be used for performance measurements at the same way as -"*_perf" handlers. The following parameters possible for vdisk_nullio: -blocksize, read_only, removable. See vdisk_fileio above for description -of those parameters. - -Handler vcdrom allows emulation of a virtual CDROM device using an ISO -file as backend. It doesn't have any parameters. - -For example: - -echo "add_device disk1 filename=/disk1; blocksize=4096; nv_cache=1" >/sys/kernel/scst_tgt/handlers/vdisk_fileio/mgmt - -will create a FILEIO virtual device disk1 with backend file /disk1 -with block size 4K and NV_CACHE enabled. - -Each vdisk_fileio's device has the following attributes in -/sys/kernel/scst_tgt/devices/device_name: - - - filename - contains path and file name of the backend file. - - - blocksize - contains block size used by this virtual device. - - - write_through - contains status of write back caching of this virtual - device. - - - read_only - contains read only status of this virtual device. - - - o_direct - contains O_DIRECT status of this virtual device. - - - nv_cache - contains NV_CACHE status of this virtual device. - - - removable - contains removable status of this virtual device. - - - size_mb - contains size of this virtual device in MB. - - - t10_dev_id - contains and allows to set T10 vendor specific - identifier for Device Identification VPD page (0x83) of INQUIRY data. - By default VDISK handler always generates t10_dev_id for every new - created device at creation time based on the device name and - scst_vdisk_ID scst_vdisk.ko module parameter (see below). - - - usn - contains the virtual device's serial number of INQUIRY data. It - is created at the device creation time based on the device name and - scst_vdisk_ID scst_vdisk.ko module parameter (see below). - - - type - contains SCSI type of this virtual device. - - - resync_size - write only attribute, which makes vdisk_fileio to - rescan size of the backend file. It is useful if you changed it, for - instance, if you resized it. - -For example: - -/sys/kernel/scst_tgt/devices/disk1 -|-- blocksize -|-- exported -| |-- export0 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt/luns/0 -| |-- export1 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt/ini_groups/INI/luns/0 -| |-- export2 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt1/luns/0 -| |-- export3 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ini_groups/INI1/luns/0 -| |-- export4 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ini_groups/INI2/luns/0 -|-- filename -|-- handler -> ../../handlers/vdisk_fileio -|-- nv_cache -|-- o_direct -|-- read_only -|-- removable -|-- resync_size -|-- size_mb -|-- t10_dev_id -|-- threads_num -|-- threads_pool_type -|-- type -|-- usn -`-- write_through - -Each vdisk_blockio's device has the following attributes in -/sys/kernel/scst_tgt/devices/device_name: blocksize, filename, nv_cache, -read_only, removable, resync_size, size_mb, t10_dev_id, threads_num, -threads_pool_type, type, usn. See above description of those parameters. - -Each vdisk_nullio's device has the following attributes in -/sys/kernel/scst_tgt/devices/device_name: blocksize, read_only, -removable, size_mb, t10_dev_id, threads_num, threads_pool_type, type, -usn. See above description of those parameters. - -Each vcdrom's device has the following attributes in -/sys/kernel/scst_tgt/devices/device_name: filename, size_mb, -t10_dev_id, threads_num, threads_pool_type, type, usn. See above -description of those parameters. Exception is filename attribute. For -vcdrom it is writable. Writing to it allows to virtually insert or -change virtual CD media in the virtual CDROM device. For example: - - - echo "/image.iso" >/sys/kernel/scst_tgt/devices/cdrom/filename - will - insert file /image.iso as virtual media to the virtual CDROM cdrom. - - - echo "" >/sys/kernel/scst_tgt/devices/cdrom/filename - will remove - "media" from the virtual CDROM cdrom. - -Additionally to the sysfs/procfs interface VDISK handler has module -parameter "num_threads", which specifies count of I/O threads for each -VDISK's device. If you have a workload, which tends to produce rather -random accesses (e.g. DB-like), you should increase this count to a -bigger value, like 32. If you have a rather sequential workload, you -should decrease it to a lower value, like number of CPUs on the target -or even 1. Due to some limitations of Linux I/O subsystem, increasing -number of I/O threads too much leads to sequential performance drop, -especially with deadline scheduler, so decreasing it can improve -sequential performance. The default provides a good compromise between -random and sequential accesses. - -You shouldn't be afraid to have too many VDISK I/O threads if you have -many VDISK devices. Kernel threads consume very little amount of -resources (several KBs) and only necessary threads will be used by SCST, -so the threads will not trash your system. - -CAUTION: If you partitioned/formatted your device with block size X, *NEVER* -======== ever try to export and then mount it (even accidentally) with another - block size. Otherwise you can *instantly* damage it pretty - badly as well as all your data on it. Messages on initiator - like: "attempt to access beyond end of device" is the sign of - such damage. - - Moreover, if you want to compare how well different block sizes - work for you, you **MUST** EVERY TIME AFTER CHANGING BLOCK SIZE - **COMPLETELY** **WIPE OFF** ALL THE DATA FROM THE DEVICE. In - other words, THE **WHOLE** DEVICE **MUST** HAVE ONLY **ZEROS** - AS THE DATA AFTER YOU SWITCH TO NEW BLOCK SIZE. Switching block - sizes isn't like switching between FILEIO and BLOCKIO, after - changing block size all previously written with another block - size data MUST BE ERASED. Otherwise you will have a full set of - very weird behaviors, because blocks addressing will be - changed, but initiators in most cases will not have a - possibility to detect that old addresses written on the device - in, e.g., partition table, don't refer anymore to what they are - intended to refer. - -IMPORTANT: Some disk and partition table management utilities don't support -========= block sizes >512 bytes, therefore make sure that your favorite one - supports it. Currently only cfdisk is known to work only with - 512 bytes blocks, other utilities like fdisk on Linux or - standard disk manager on Windows are proved to work well with - non-512 bytes blocks. Note, if you export a disk file or - device with some block size, different from one, with which - it was already partitioned, you could get various weird - things like utilities hang up or other unexpected behavior. - Hence, to be sure, zero the exported file or device before - the first access to it from the remote initiator with another - block size. On Window initiator make sure you "Set Signature" - in the disk manager on the imported from the target drive - before doing any other partitioning on it. After you - successfully mounted a file system over non-512 bytes block - size device, the block size stops matter, any program will - work with files on such file system. - - -Persistent Reservations ------------------------ - -SCST implements Persistent Reservations with full set of capabilities, -including "Persistence Through Power Loss". - -The "Persistence Through Power Loss" data are saved in /var/lib/scst/pr -with files with names the same as the names of the corresponding -devices. Also this directory contains backup versions of those files -with suffix ".1". Those backup files are used in case of power or other -failure to prevent Persistent Reservation information from corruption -during update. - -The "Persistence Through Power Loss" feature is not available in the -procfs build, because the SCST proc interface doesn't allow to keep -persistent Relative Target IDs of each target between reboots/reloads -(they are load and initialization order dependent). - -The Persistent Reservations available on all transports implementing -get_initiator_port_transport_id() callback. Transports not implementing -this callback will act in one of 2 possible scenarios ("all or -nothing"): - -1. If a device has such transport connected and doesn't have persistent -reservations, it will refuse Persistent Reservations commands as if it -doesn't support them. - -2. If a device has persistent reservations, all initiators newly -connecting via such transports will not see this device. After all -persistent reservations from this device are released, upon reconnect -the initiators will see it. - - -Caching -------- - -By default for performance reasons VDISK FILEIO devices use write back -caching policy. - -Generally, write back caching is safe for use and danger of it is -greatly overestimated, because most modern (especially, enterprise -level) applications are well prepared to work with write back cached -storage. Particularly, such are all transactions-based applications. -Those applications flush cache to completely AVOID ANY data loss on a -crash or power failure. For instance, journaled file systems flush cache -on each meta data update, so they survive power/hardware/software -failures pretty well. - -Since locally on initiators write back caching is always on, if an -application cares about its data consistency, it does flush the cache -when necessary or on any write, if open files with O_SYNC. If it doesn't -care, it doesn't flush the cache. As soon as the cache flushes -propagated to the storage, write back caching on it doesn't make any -difference. If application doesn't flush the cache, it's doomed to loose -data in case of a crash or power failure doesn't matter where this cache -located, locally or on the storage. - -To illustrate how data loss can be avoided with write back caching, -consider, for example, a user who wants to copy /src directory to /dst -directory reliably, i.e. after the copy finished no power failure or -software/hardware crash could lead to a loss of the data in /dst. There -are 2 ways to achieve this. Let's suppose for simplicity cp opens files -for writing with O_SYNC flag, hence bypassing the local cache. - -1. Slow. Make the device behind /dst working in write through caching -mode and then run "cp -a /src /dst". - -2. Fast. Let the device behind /dst working in write back caching mode -and then run "cp -a /src /dst; sync". The reliability of the result is -the same, but it's much faster than (1). Nobody would care if a crash -happens during the copy, because after recovery simply leftovers from -the not completed attempt would be deleted and the operation would be -restarted from the very beginning. - -So, you can see in (2) there is no danger of ANY data loss from the -write back caching. Moreover, since on practice cp doesn't open files -for writing with O_SYNC flag, to get the copy done reliably, sync -command must be called after cp anyway, so enabling write back caching -wouldn't make any difference for reliability. - -Also you can consider it from another side. Modern HDDs have at least -16MB of cache working in write back mode by default, so for a 10 drives -RAID it is 160MB of a write back cache. How many people are happy with -it and how many disabled write back cache of their HDDs? Almost all and -almost nobody correspondingly? Moreover, many HDDs lie about state of -their cache and report write through while working in write back mode. -They are also successfully used. - -Note, Linux I/O subsystem guarantees to propagated cache flushes to the -storage only using data protection barriers, which usually turned off by -default (see http://lwn.net/Articles/283161). Without barriers enabled -Linux doesn't provide a guarantee that after sync()/fsync() all written -data really hit permanent storage. They can be stored in the cache of -your backstorage devices and, hence, lost on a power failure event. -Thus, ever with write-through cache mode, you still either need to -enable barriers on your backend file system on the target (for direct -/dev/sdX devices this is, indeed, impossible), or need a good UPS to -protect yourself from not committed data loss. Some info about barriers -from the XFS point of view could be found at -http://oss.sgi.com/projects/xfs/faq.html#wcache. On Linux initiators for -Ext3 and ReiserFS file systems the barrier protection could be turned on -using "barrier=1" and "barrier=flush" mount options correspondingly. You -can check if the barriers turn on or off by looking in /proc/mounts. -Windows and, AFAIK, other UNIX'es don't need any special explicit -options and do necessary barrier actions on write-back caching devices -by default. - -To limit this data loss with write back caching you can use files in -/proc/sys/vm to limit amount of unflushed data in the system cache. - -If you for some reason have to use VDISK FILEIO devices in write through -caching mode, don't forget to disable internal caching on their backend -devices or make sure they have additional battery or supercapacitors -power supply on board. Otherwise, you still on a power failure would -loose all the unsaved yet data in the devices internal cache. - -Note, on some real-life workloads write through caching might perform -better, than write back one with the barrier protection turned on. - - -BLOCKIO VDISK mode ------------------- - -This module works best for these types of scenarios: - -1) Data that are not aligned to 4K sector boundaries and <4K block sizes -are used, which is normally found in virtualization environments where -operating systems start partitions on odd sectors (Windows and it's -sector 63). - -2) Large block data transfers normally found in database loads/dumps and -streaming media. - -3) Advanced relational database systems that perform their own caching -which prefer or demand direct IO access and, because of the nature of -their data access, can actually see worse performance with -non-discriminate caching. - -4) Multiple layers of targets were the secondary and above layers need -to have a consistent view of the primary targets in order to preserve -data integrity which a page cache backed IO type might not provide -reliably. - -Also it has an advantage over FILEIO that it doesn't copy data between -the system cache and the commands data buffers, so it saves a -considerable amount of CPU power and memory bandwidth. - -IMPORTANT: Since data in BLOCKIO and FILEIO modes are not consistent between -========= each other, if you try to use a device in both those modes - simultaneously, you will almost instantly corrupt your data - on that device. - -IMPORTANT: Some kernels starting from 2.6.32 have a problem, which prevents -========= prevents BLOCKIO from working correctly with RAID5/DM. See - http://lkml.org/lkml/2010/7/28/315. That problem was fixed in - 2.6.32.19, 2.6.34.4, 2.6.35.2 and 2.6.36-rc1. It is strongly - recommended to not use affected kernels with BLOCKIO. - -IMPORTANT: In SCST 1.x BLOCKIO worked by default in NV_CACHE mode, when -========= each device reported to remote initiators as having write through - caching. But if your backend block device has internal write - back caching it might create a possibility for data loss of - the cached in the internal cache data in case of a power - failure. Starting from SCST 2.0 BLOCKIO works by default in - non-NV_CACHE mode, when each device reported to remote - initiators as having write back caching, and synchronizes the - internal device's cache on each SYNCHRONIZE_CACHE command - from the initiators. It might lead to some *PERFORMANCE LOSS*, - so if you are are sure in your power supply and want to - restore the 1.x behavior, your should recreate your BLOCKIO - devices in NV_CACHE mode. - -Pass-through mode ------------------ - -In the pass-through mode (i.e. using the pass-through device handlers -scst_disk, scst_tape, etc) SCSI commands, coming from remote initiators, -are passed to local SCSI devices on target as is, without any -modifications. - -In the SYSFS interface all real SCSI devices are listed in -/sys/kernel/scst_tgt/devices in form host:channel:id:lun numbers, for -instance 1:0:0:0. The recommended way to match those numbers to your -devices is use of lsscsi utility. - -Each pass-through dev handler has in its root subdirectory -/sys/kernel/scst_tgt/handlers/handler_name, e.g. -/sys/kernel/scst_tgt/handlers/dev_disk, "mgmt" file. It allows the -following commands. They can be sent to it using, e.g., echo command. - - - "add_device" - this command assigns SCSI device with -host:channel:id:lun numbers to this dev handler. - -echo "add_device 1:0:0:0" >/sys/kernel/scst_tgt/handlers/dev_disk/mgmt - -will assign SCSI device 1:0:0:0 to this dev handler. - - - "del_device" - this command unassigns SCSI device with -host:channel:id:lun numbers from this dev handler. - -As usually, on read the "mgmt" file returns small help about available -commands. - -You need to manually assign each your real SCSI device to the -corresponding pass-through dev handler using the "add_device" command, -otherwise the real SCSI devices will not be visible remotely. The -assignment isn't done automatically, because it could lead to the -pass-through dev handlers load and initialization problems if any of the -local real SCSI devices are malfunctioning. - -As any other hardware, the local SCSI hardware can not handle commands -with amount of data and/or segments count in scatter-gather array bigger -some values. Therefore, when using the pass-through mode you should note -that values for maximum number of segments and maximum amount of -transferred data for each SCSI command on devices on initiators can not -be bigger, than corresponding values of the corresponding SCSI devices -on the target. Otherwise you will see symptoms like small transfers work -well, but large ones stall and messages like: "Unable to complete -command due to SG IO count limitation" are printed in the kernel logs. - -You can't control from the user space limit of the scatter-gather -segments, but for block devices usually it is sufficient if you set on -the initiators /sys/block/DEVICE_NAME/queue/max_sectors_kb in the same -or lower value as in /sys/block/DEVICE_NAME/queue/max_hw_sectors_kb for -the corresponding devices on the target. - -For not-block devices SCSI commands are usually generated directly by -applications, so, if you experience large transfers stalls, you should -check documentation for your application how to limit the transfer -sizes. - -Another way to solve this issue is to build SG entries with more than 1 -page each. See the following patch as an example: -http://scst.sourceforge.net/sgv_big_order_alloc.diff - - -User space mode using scst_user dev handler -------------------------------------------- - -User space program fileio_tgt uses interface of scst_user dev handler -and allows to see how it works in various modes. Fileio_tgt provides -mostly the same functionality as scst_vdisk handler with the most -noticeable difference that it supports O_DIRECT mode. O_DIRECT mode is -basically the same as BLOCKIO, but also supports files, so for some -loads it could be significantly faster, than the regular FILEIO access. -All the words about BLOCKIO from above apply to O_DIRECT as well. See -fileio_tgt's README file for more details. - - -Performance ------------ - -SCST from the very beginning has been designed and implemented to -provide the best possible performance. Since there is no "one fit all" -the best performance configuration for different setups and loads, SCST -provides extensive set of settings to allow to tune it for the best -performance in each particular case. You don't have to necessary use -those settings. If you don't, SCST will do very good job to autotune for -you, so the resulting performance will, in average, be better -(sometimes, much better) than with other SCSI targets. But in some cases -you can by manual tuning improve it even more. - -If you want to get maximum performance from your target, RHEL/CentOS 5.x -kernels are not recommended on both target and initiators, if you are -using Linux initiators, because those kernels are based on very outdated -2.6.18 kernel, hence, missed >3 years of important improvements in the -kernel's storage area. You should use at least long maintained vanilla -2.6.27.x kernel, although 2.6.29+ would be even better. - -Before doing any performance measurements note that performance results -are very much dependent from your type of load, so it is crucial that -you choose access mode (FILEIO, BLOCKIO, O_DIRECT, pass-through), which -suits your needs the best. - -In order to get the maximum performance you should: - -1. For SCST: - - - Disable in Makefile CONFIG_SCST_STRICT_SERIALIZING, CONFIG_SCST_EXTRACHECKS, - CONFIG_SCST_TRACING, CONFIG_SCST_DEBUG*, CONFIG_SCST_STRICT_SECURITY - -2. For target drivers: - - - Disable in Makefiles CONFIG_SCST_EXTRACHECKS, CONFIG_SCST_TRACING, - CONFIG_SCST_DEBUG* - -3. For device handlers, including VDISK: - - - Disable in Makefile CONFIG_SCST_TRACING and CONFIG_SCST_DEBUG. - -IMPORTANT: The development version of SCST in the SVN is optimized for -========= development and bug hunting, not for performance. To reconfigure - it for performance you should run "make debug2perf" in the - root of your source code (e.g. trunk/). It will set the above - options as needed. The only option it doesn't set is - CONFIG_SCST_TEST_IO_IN_SIRQ, so, if needed, you should change - it manually. - -IMPORTANT: You can't use debug SCST drivers with non-debug SCST core. -========= So, after disabling both CONFIG_SCST_TRACING and CONFIG_SCST_DEBUG - for SCST core you have to disable them for all SCST drivers - you are using as well. - -4. Make sure you have io_grouping_type option set correctly, especially -in the following cases: - - - Several initiators share your target's backstorage. It can be a - shared LU using some cluster FS, like VMFS, as well as can be - different LUs located on the same backstorage (RAID array). For - instance, if you have 3 initiators and each of them using its own - dedicated FILEIO device file from the same RAID-6 array on the - target. - - In this case for the best performance you should have - io_grouping_type option set in value "never" in all the LUNs' targets - and security groups. - - - Your initiator connected to your target in MPIO mode. In this case for - the best performance you should: - - * Either connect all the sessions from the initiator to a single - target or security group and have io_grouping_type option set in - value "this_group_only" in the target or security group, - - * Or, if it isn't possible to connect all the sessions from the - initiator to a single target or security group, assign the same - numeric io_grouping_type value for each target/security group this - initiator connected to. The exact value itself doesn't matter, - important only that all the targets/security groups use the same - value. - -Don't forget, io_grouping_type makes sense only if you use CFQ I/O -scheduler on the target and for devices with threads_num >= 0 and, if -threads_num > 0, with threads_pool_type "per_initiator". - -You can check if in your setup io_grouping_type set correctly as well as -if the "auto" io_grouping_type value works for you by tests like the -following: - - - For not MPIO case you can run single thread sequential reading, e.g. - using buffered dd, from one initiator, then run the same single - thread sequential reading from the second initiator in parallel. If - io_grouping_type is set correctly the aggregate throughput measured - on the target should only slightly decrease as well as all initiators - should have nearly equal share of it. If io_grouping_type is not set - correctly, the aggregate throughput and/or throughput on any - initiator will decrease significantly, in 2 times or even more. For - instance, you have 80MB/s single thread sequential reading from the - target on any initiator. When then both initiators are reading in - parallel you should see on the target aggregate throughput something - like 70-75MB/s with correct io_grouping_type and something like - 35-40MB/s or 8-10MB/s on any initiator with incorrect. - - - For the MPIO case it's quite easier. With incorrect io_grouping_type - you simply won't see performance increase from adding the second - session (assuming your hardware is capable to transfer data through - both sessions in parallel), or can even see a performance decrease. - -5. If you are going to use your target in an VM environment, for -instance as a shared storage with VMware, make sure all your VMs -connected to the target via *separate* sessions. For instance, for iSCSI -it means that each VM has own connection to the target, not all VMs -connected using a single connection. You can check it using SCST proc or -sysfs interface. For other transports you should use available -facilities, like NPIV for Fibre Channel, to make separate sessions for -each VM. If you miss it, you can greatly loose performance of parallel -access to your target from different VMs. This isn't related to the case -if your VMs are using the same shared storage, like with VMFS, for -instance. In this case all your VM hosts will be connected to the target -via separate sessions, which is enough. - -6. For other target and initiator software parts: - - - Make sure you applied on your kernel all available SCST patches. - If for your kernel version this patch doesn't exist, it is strongly - recommended to upgrade your kernel to version, for which this patch - exists. - - - Don't enable debug/hacking features in the kernel, i.e. use them as - they are by default. - - - The default kernel read-ahead and queuing settings are optimized - for locally attached disks, therefore they are not optimal if they - attached remotely (SCSI target case), which sometimes could lead to - unexpectedly low throughput. You should increase read-ahead size to at - least 512KB or even more on all initiators and the target. - - You should also limit on all initiators maximum amount of sectors per - SCSI command. This tuning is also recommended on targets with large - read-ahead values. To do it on Linux, run: - - echo “64” > /sys/block/sdX/queue/max_sectors_kb - - where specify instead of X your imported from target device letter, - like 'b', i.e. sdb. - - To increase read-ahead size on Linux, run: - - blockdev --setra N /dev/sdX - - where N is a read-ahead number in 512-byte sectors and X is a device - letter like above. - - Note: you need to set read-ahead setting for device sdX again after - you changed the maximum amount of sectors per SCSI command for that - device. - - Note2: you need to restart SCST after you changed read-ahead settings - on the target. - - - You may need to increase amount of requests that OS on initiator - sends to the target device. To do it on Linux initiators, run - - echo “64” > /sys/block/sdX/queue/nr_requests - - where X is a device letter like above. - - You may also experiment with other parameters in /sys/block/sdX - directory, they also affect performance. If you find the best values, - please share them with us. - - - On the target use CFQ IO scheduler. In most cases it has performance - advantage over other IO schedulers, sometimes huge (2+ times - aggregate throughput increase). - - - It is recommended to turn the kernel preemption off, i.e. set - the kernel preemption model to "No Forced Preemption (Server)". - - - Looks like XFS is the best filesystem on the target to store device - files, because it allows considerably better linear write throughput, - than ext3. - -7. For hardware on target. - - - Make sure that your target hardware (e.g. target FC or network card) - and underlaying IO hardware (e.g. IO card, like SATA, SCSI or RAID to - which your disks connected) don't share the same PCI bus. You can - check it using lspci utility. They have to work in parallel, so it - will be better if they don't compete for the bus. The problem is not - only in the bandwidth, which they have to share, but also in the - interaction between cards during that competition. This is very - important, because in some cases if target and backend storage - controllers share the same PCI bus, it could lead up to 5-10 times - less performance, than expected. Moreover, some motherboard (by - Supermicro, particularly) have serious stability issues if there are - several high speed devices on the same bus working in parallel. If - you have no choice, but PCI bus sharing, set in the BIOS PCI latency - as low as possible. - -8. If you use VDISK IO module in FILEIO mode, NV_CACHE option will -provide you the best performance. But using it make sure you use a good -UPS with ability to shutdown the target on the power failure. - -Baseline performance numbers you can find in those measurements: -http://lkml.org/lkml/2009/3/30/283. - -IMPORTANT: If you use on initiator some versions of Windows (at least W2K) -========= you can't get good write performance for VDISK FILEIO devices with - default 512 bytes block sizes. You could get about 10% of the - expected one. This is because of the partition alignment, which - is (simplifying) incompatible with how Linux page cache - works, so for each write the corresponding block must be read - first. Use 4096 bytes block sizes for VDISK devices and you - will have the expected write performance. Actually, any OS on - initiators, not only Windows, will benefit from block size - max(PAGE_SIZE, BLOCK_SIZE_ON_UNDERLYING_FS), where PAGE_SIZE - is the page size, BLOCK_SIZE_ON_UNDERLYING_FS is block size - on the underlying FS, on which the device file located, or 0, - if a device node is used. Both values are from the target. - See also important notes about setting block sizes >512 bytes - for VDISK FILEIO devices above. - - -9. In some cases, for instance working with SSD devices, which consume 100% -of a single CPU load for data transfers in their internal threads, to -maximize IOPS it can be needed to assign for those threads dedicated -CPUs using Linux CPU affinity facilities. No IRQ processing should be -done on those CPUs. Check that using /proc/interrupts. See taskset -command and Documentation/IRQ-affinity.txt in your kernel's source tree -for how to assign IRQ affinity to tasks and IRQs. - -The reason for that is that processing of coming commands in SIRQ -context might be done on the same CPUs as SSD devices' threads doing data -transfers. As the result, those threads won't receive all the processing -power of those CPUs and perform worse. - - -Work if target's backstorage or link is too slow ------------------------------------------------- - -Under high I/O load, when your target's backstorage gets overloaded, or -working over a slow link between initiator and target, when the link -can't serve all the queued commands on time, you can experience I/O -stalls or see in the kernel log abort or reset messages. - -At first, consider the case of too slow target's backstorage. On some -seek intensive workloads even fast disks or RAIDs, which able to serve -continuous data stream on 500+ MB/s speed, can be as slow as 0.3 MB/s. -Another possible cause for that can be MD/LVM/RAID on your target as in -http://lkml.org/lkml/2008/2/27/96 (check the whole thread as well). - -Thus, in such situations simply processing of one or more commands takes -too long time, hence initiator decides that they are stuck on the target -and tries to recover. Particularly, it is known that the default amount -of simultaneously queued commands (48) is sometimes too high if you do -intensive writes from VMware on a target disk, which uses LVM in the -snapshot mode. In this case value like 16 or even 8-10 depending of your -backstorage speed could be more appropriate. - -Unfortunately, currently SCST lacks dynamic I/O flow control, when the -queue depth on the target is dynamically decreased/increased based on -how slow/fast the backstorage speed comparing to the target link. So, -there are 6 possible actions, which you can do to workaround or fix this -issue in this case: - -1. Ignore incoming task management (TM) commands. It's fine if there are -not too many of them, so average performance isn't hurt and the -corresponding device isn't getting put offline, i.e. if the backstorage -isn't too slow. - -2. Decrease /sys/block/sdX/device/queue_depth on the initiator in case -if it's Linux (see below how) or/and SCST_MAX_TGT_DEV_COMMANDS constant -in scst_priv.h file until you stop seeing incoming TM commands. -ISCSI-SCST driver also has its own iSCSI specific parameter for that, -see its README file. - -To decrease device queue depth on Linux initiators you can run command: - -# echo Y >/sys/block/sdX/device/queue_depth - -where Y is the new number of simultaneously queued commands, X - your -imported device letter, like 'a' for sda device. There are no special -limitations for Y value, it can be any value from 1 to possible maximum -(usually, 32), so start from dividing the current value on 2, i.e. set -16, if /sys/block/sdX/device/queue_depth contains 32. - -3. Increase the corresponding timeout on the initiator. For Linux it is -located in -/sys/devices/platform/host*/session*/target*:0:0/*:0:0:1/timeout. It can -be done automatically by an udev rule. For instance, the following -rule will increase it to 300 seconds: - -SUBSYSTEM=="scsi", KERNEL=="[0-9]*:[0-9]*", ACTION=="add", ATTR{type}=="0|7|14", ATTR{timeout}="300" - -By default, this timeout is 30 or 60 seconds, depending on your distribution. - -4. Try to avoid such seek intensive workloads. - -5. Increase speed of the target's backstorage. - -6. Implement in SCST dynamic I/O flow control. This will be an ultimate -solution. See "Dynamic I/O flow control" section on -http://scst.sourceforge.net/contributing.html page for possible -implementation idea. - -Next, consider the case of too slow link between initiator and target, -when the initiator tries to simultaneously push N commands to the target -over it. In this case time to serve those commands, i.e. send or receive -data for them over the link, can be more, than timeout for any single -command, hence one or more commands in the tail of the queue can not be -served on time less than the timeout, so the initiator will decide that -they are stuck on the target and will try to recover. - -To workaround/fix this issue in this case you can use ways 1, 2, 3, 6 -above or (7): increase speed of the link between target and initiator. -But for some initiators implementations for WRITE commands there might -be cases when target has no way to detect the issue, so dynamic I/O flow -control will not be able to help. In those cases you could also need on -the initiator(s) to either decrease the queue depth (way 2), or increase -the corresponding timeout (way 3). - -Note, that logged messages about QUEUE_FULL status are quite different -by nature. This is a normal work, just SCSI flow control in action. -Simply don't enable "mgmt_minor" logging level, or, alternatively, if -you are confident in the worst case performance of your back-end storage -or initiator-target link, you can increase SCST_MAX_TGT_DEV_COMMANDS in -scst_priv.h to 64. Usually initiators don't try to push more commands on -the target. - Credits ------- @@ -1916,8 +1963,8 @@ Thanks to: for UNH-iSCSI project (http://www.iol.unh.edu/consortiums/iscsi/index.html) on which interface between SCST core and target drivers was based. - * Daniel Debonzi for a big part of SCST sysfs tree - implementation + * Daniel Debonzi for a big part of the + initial SCST sysfs tree implementation Vladislav Bolkhovitin , http://scst.sourceforge.net diff --git a/scst/README_in-tree b/scst/README_in-tree index 87409ca89..43dae4d5d 100644 --- a/scst/README_in-tree +++ b/scst/README_in-tree @@ -4,8 +4,7 @@ Generic SCSI target mid-level for Linux (SCST) SCST is designed to provide unified, consistent interface between SCSI target drivers and Linux kernel and simplify target drivers development as much as possible. Detail description of SCST's features and internals -could be found in "Generic SCSI Target Middle Level for Linux" document -SCST's Internet page http://scst.sourceforge.net. +could be found on its Internet page http://scst.sourceforge.net. SCST supports the following I/O modes: @@ -13,37 +12,36 @@ SCST supports the following I/O modes: initiators can connect to the exported pass-through devices, for the following SCSI devices types: disks (type 0), tapes (type 1), processors (type 3), CDROMs (type 5), MO disks (type 7), medium - changers (type 8) and RAID controllers (type 0xC) + changers (type 8) and RAID controllers (type 0xC). * FILEIO mode, which allows to use files on file systems or block devices as virtual remotely available SCSI disks or CDROMs with - benefits of the Linux page cache + benefits of the Linux page cache. * BLOCKIO mode, which performs direct block IO with a block device, bypassing page-cache for all operations. This mode works ideally with high-end storage HBAs and for applications that either do not need caching between application and disk or need the large block - throughput - - * User space mode using scst_user device handler, which allows to - implement in the user space virtual SCSI devices in the SCST - environment + throughput. * "Performance" device handlers, which provide in pseudo pass-through mode a way for direct performance measurements without overhead of - actual data transferring from/to underlying SCSI device + actual data transferring from/to underlying SCSI device. In addition, SCST supports advanced per-initiator access and devices visibility management, so different initiators could see different set of devices with different access permissions. See below for details. +Full list of SCST features and comparison with other Linux targets you +can find on http://scst.sourceforge.net/comparison.html. + Installation ------------ -To see your devices remotely, you need to add them to at least "Default" -security group (see below how). By default, no local devices are seen -remotely. There must be LUN 0 in each security group, i.e. LUs +To see your devices remotely, you need to add a corresponding LUN for +them (see below how). By default, no local devices are seen remotely. +There must be LUN 0 in each LUNs set (security group), i.e. LUs numeration must not start from, e.g., 1. Otherwise you will see no devices on remote initiators and SCST core will write into the kernel log message: "tgt_dev for LUN 0 not found, command to unexisting LU?" @@ -83,8 +81,8 @@ IMPORTANT: In the current version simultaneous access to local SCSI devices the state of devices and their parameters, because that could lead to data corruption. If any such command is done, at least related device handler(s) must be restarted. For block - devices READ/WRITE commands using direct disk handler look to - be safe. + devices READ/WRITE commands using direct disk handler are + generally safe. Usage in failover mode @@ -121,13 +119,6 @@ addressed in the next release. Virtual CDROM's are useful for remote installation. See below for details how to setup and use VDISK device handler. -SCST user space device handler provides an interface between SCST and -the user space, which allows to create pure user space devices. The -simplest example, where one would want it is if he/she wants to write a -VTL. With scst_user he/she can write it purely in the user space. Or one -would want it if he/she needs some sophisticated for kernel space -processing of the passed data, like encrypting them or making snapshots. - "Performance" device handlers for disks, MO disks and tapes in their exec() method skip (pretend to execute) all READ and WRITE operations and thus provide a way for direct link performance measurements without @@ -156,22 +147,22 @@ your favorite kernel configuration Makefile target, e.g. "make xconfig": the various places. - CONFIG_SCST_USE_EXPECTED_VALUES - if not defined (default), initiator - supplied expected data transfer length and direction will be used only for - verification purposes to return error or warn in case if one of them - is invalid. Instead, locally decoded from SCSI command values will be - used. This is necessary for security reasons, because otherwise a - faulty initiator can crash target by supplying invalid value in one - of those parameters. This is especially important in case of - pass-through mode. If CONFIG_SCST_USE_EXPECTED_VALUES is defined, initiator - supplied expected data transfer length and direction will override - the locally decoded values. This might be necessary if internal SCST - commands translation table doesn't contain SCSI command, which is - used in your environment. You can know that if you have messages like - "Unknown opcode XX for YY. Should you update scst_scsi_op_table?" in - your kernel log and your initiator returns an error. Also report - those messages in the SCST mailing list - scst-devel@lists.sourceforge.net. Note, that not all SCSI transports - support supplying expected values. + supplied expected data transfer length and direction will be used + only for verification purposes to return error or warn in case if one + of them is invalid. Instead, locally decoded from SCSI command values + will be used. This is necessary for security reasons, because + otherwise a faulty initiator can crash target by supplying invalid + value in one of those parameters. This is especially important in + case of pass-through mode. If CONFIG_SCST_USE_EXPECTED_VALUES is + defined, initiator supplied expected data transfer length and + direction will override the locally decoded values. This might be + necessary if internal SCST commands translation table doesn't contain + SCSI command, which is used in your environment. You can know that if + you enable "minor" trace level and have messages like "Unknown + opcode XX for YY. Should you update scst_scsi_op_table?" in your + kernel log and your initiator returns an error. Also report those + messages in the SCST mailing list scst-devel@lists.sourceforge.net. + Note, that not all SCSI transports support supplying expected values. - CONFIG_SCST_DEBUG_TM - if defined, turns on task management functions debugging, when on LUN 6 some of the commands will be delayed for @@ -208,8 +199,8 @@ your favorite kernel configuration Makefile target, e.g. "make xconfig": CONFIG_QLA_TGT_DEBUG_WORK_IN_THREAD disabled - does. This option is designed to be used with vdisk NULLIO backend. - WARNING! Using with other backend than vdisk NULLIO is unsafe and can - lead you to a kernel crash! + WARNING! Using this option enabled with other backend than vdisk + NULLIO is unsafe and can lead you to a kernel crash! - CONFIG_SCST_STRICT_SECURITY - if defined, makes SCST zero allocated data buffers. Undefining it (default) considerably improves performance @@ -227,30 +218,13 @@ your favorite kernel configuration Makefile target, e.g. "make xconfig": Then sometimes get crazy itself. So, this option is disabled by default. - - CONFIG_SCST_MEASURE_LATENCY - if defined, provides in /sys/kernel/scst_tgt - and below statistics about average commands processing latency. You - can clear already measured results by writing 0 in the corresponding - file. Note, you need a non-preemptible kernel to have correct - results. + - CONFIG_SCST_MEASURE_LATENCY - if defined, provides in "latency" files + global and per-LUN average commands processing latency statistic. You + can clear already measured results by writing 0 in each file. Note, + you need a non-preemptible kernel to have correct results. HIGHMEM kernel configurations are fully supported, but not recommended -for performance reasons, except for scst_user, where they are not -supported, because this module deals with user supplied memory on a -zero-copy manner. If you need to use it, consider change VMSPLIT option -or use 64-bit system configuration instead. - -For changing VMSPLIT option (CONFIG_VMSPLIT to be precise) you should in -"make menuconfig" command set the following variables: - - - General setup->Configure standard kernel features (for small systems): ON - - - General setup->Prompt for development and/or incomplete code/drivers: ON - - - Processor type and features->High Memory Support: OFF - - - Processor type and features->Memory split: according to amount of - memory you have. If it is less than 800MB, you may not touch this - option at all. +for performance reasons. Module parameters @@ -296,6 +270,17 @@ following entries: - version - read-only attribute, which allows to see version of SCST and enabled optional features. + - last_sysfs_mgmt_res - read-only attribute returning completion status + of the last management command. In the sysfs implementation there are + some problems between internal sysfs and internal SCST locking. To + avoid them in some cases sysfs calls can return error with errno + EAGAIN. This doesn't mean the operation failed. It only means that + the operation queued and not yet completed. To wait for it to + complete, an management tool should poll this file. If the operation + hasn't yet completed, it will also return EAGAIN. But after it's + completed, it will return the result of this operation (0 for success + or -errno for error). + Each SCST sysfs file (attribute) can contain in the last line mark "[key]". It is automatically added mark used to allow scstadmin to see which attributes it should save in the config file. You can ignore it. @@ -305,7 +290,7 @@ which attributes it should save in the config file. You can ignore it. Content of each device's subdirectory is dev handler specific. See documentation for your dev handlers for more info about it as well as SysfsRules file for more info about common to all dev handlers rules. -Standard SCST dev handlers have at least the following common entries: +SCST dev handlers can have the following common entries: - exported - subdirectory containing links to all LUNs where this device was exported. @@ -339,7 +324,7 @@ handler. Content of each handler's subdirectory is dev handler specific. See documentation for your dev handlers for more info about it as well as SysfsRules file for more info about common to all dev handlers rules. -Standard SCST dev handlers have at least the following common entries: +SCST dev handlers can have the following common entries: - mgmt - this entry allows to create virtual devices and their attributes (for virtual devices dev handlers) or assign/unassign real @@ -446,13 +431,22 @@ Each session subdirectory contains the following entries: - commands - contains overall number of SCSI commands in this session. + - latency - if CONFIG_SCST_MEASURE_LATENCY enabled, contains latency + statistics for this session. + + - luns - a link pointing out to the corresponding LUNs set (security + group) where this session was attached to. + + - One or more "lunX" subdirectories, where 'X' is a number, for each LUN + this session has (see below). + - other target driver specific attributes and subdirectories. See below description of the VDISK's sysfs interface for samples. -Access and devices visibility management (LUN masking) - sysfs interface ------------------------------------------------------------------------- +Access and devices visibility management (LUN masking) +------------------------------------------------------ Access and devices visibility management allows for an initiator or group of initiators to see different devices with different LUNs @@ -507,10 +501,6 @@ are one or more param_name=value pairs separated by ';'. parameter "read_only". The recommended way to find out H:C:I:L numbers is use of lsscsi utility. - - "del H:C:I:L" - deletes a pass-through device with host:channel:id:lun - The recommended way to find out H:C:I:L numbers is use of lsscsi - utility. - - "add VNAME lun [parameters]" - adds a virtual device with name VNAME with LUN "lun". Optionally, the device could be marked as read only by using parameter "read_only". @@ -521,7 +511,7 @@ are one or more param_name=value pairs separated by ';'. exist, this command acts as the "add" command. Optionally, the device could be marked as read only by using parameter "read_only". - - "del VNAME" - deletes a virtual device with name VNAME. + - "del lun" - deletes LUN lun - "clear" - clears the list of devices @@ -623,12 +613,12 @@ IMPORTANT ========= All the access control must be fully configured BEFORE the corresponding -target is enabled! When you enable a target, it will immediately start +target is enabled. When you enable a target, it will immediately start accepting new connections, hence creating new sessions, and those new sessions will be assigned to security groups according to the *currently* configured access control settings. For instance, to the default target's set of LUNs, instead of "HOST004" group as you may -need, because "HOST004" doesn't exist yet. So, one must configure all +need, because "HOST004" doesn't exist yet. So, you must configure all the security groups before new connections from the initiators are created, i.e. before the target enabled. @@ -806,7 +796,7 @@ removable, size_mb, t10_dev_id, threads_num, threads_pool_type, type, usn. See above description of those parameters. Each vcdrom's device has the following attributes in -/sys/kernel/scst_tgt/devices/device_name: filename, size_mb, +/sys/kernel/scst_tgt/devices/device_name: filename, size_mb, t10_dev_id, threads_num, threads_pool_type, type, usn. See above description of those parameters. Exception is filename attribute. For vcdrom it is writable. Writing to it allows to virtually insert or @@ -818,17 +808,17 @@ change virtual CD media in the virtual CDROM device. For example: - echo "" >/sys/kernel/scst_tgt/devices/cdrom/filename - will remove "media" from the virtual CDROM cdrom. -Additionally to the sysfs interface VDISK handler has module parameter -"num_threads", which specifies count of I/O threads for each VDISK's -device. If you have a workload, which tends to produce rather random -accesses (e.g. DB-like), you should increase this count to a bigger -value, like 32. If you have a rather sequential workload, you should -decrease it to a lower value, like number of CPUs on the target or even -1. Due to some limitations of Linux I/O subsystem, increasing number of -I/O threads too much leads to sequential performance drop, especially -with deadline scheduler, so decreasing it can improve sequential -performance. The default provides a good compromise between random and -sequential accesses. +Additionally VDISK handler has module parameter "num_threads", which +specifies count of I/O threads for each FILEIO VDISK's or VCDROM device. +If you have a workload, which tends to produce rather random accesses +(e.g. DB-like), you should increase this count to a bigger value, like +32. If you have a rather sequential workload, you should decrease it to +a lower value, like number of CPUs on the target or even 1. Due to some +limitations of Linux I/O subsystem, increasing number of I/O threads too +much leads to sequential performance drop, especially with deadline +scheduler, so decreasing it can improve sequential performance. The +default provides a good compromise between random and sequential +accesses. You shouldn't be afraid to have too many VDISK I/O threads if you have many VDISK devices. Kernel threads consume very little amount of @@ -888,11 +878,6 @@ with suffix ".1". Those backup files are used in case of power or other failure to prevent Persistent Reservation information from corruption during update. -The "Persistence Through Power Loss" feature is not available in the -procfs build, because the SCST proc interface doesn't allow to keep -persistent Relative Target IDs of each target between reboots/reloads -(they are load and initialization order dependent). - The Persistent Reservations available on all transports implementing get_initiator_port_transport_id() callback. Transports not implementing this callback will act in one of 2 possible scenarios ("all or @@ -915,10 +900,10 @@ By default for performance reasons VDISK FILEIO devices use write back caching policy. Generally, write back caching is safe for use and danger of it is -greatly overestimated, because most modern (especially, enterprise +greatly overestimated, because most modern (especially, Enterprise level) applications are well prepared to work with write back cached storage. Particularly, such are all transactions-based applications. -Those applications flush cache to completely AVOID ANY data loss on a +Those applications flush cache to completely avoid ANY data loss on a crash or power failure. For instance, journaled file systems flush cache on each meta data update, so they survive power/hardware/software failures pretty well. @@ -932,12 +917,12 @@ difference. If application doesn't flush the cache, it's doomed to loose data in case of a crash or power failure doesn't matter where this cache located, locally or on the storage. -To illustrate how data loss can be avoided with write back caching, -consider, for example, a user who wants to copy /src directory to /dst -directory reliably, i.e. after the copy finished no power failure or -software/hardware crash could lead to a loss of the data in /dst. There -are 2 ways to achieve this. Let's suppose for simplicity cp opens files -for writing with O_SYNC flag, hence bypassing the local cache. +To illustrate that consider, for example, a user who wants to copy /src +directory to /dst directory reliably, i.e. after the copy finished no +power failure or software/hardware crash could lead to a loss of the +data in /dst. There are 2 ways to achieve this. Let's suppose for +simplicity cp opens files for writing with O_SYNC flag, hence bypassing +the local cache. 1. Slow. Make the device behind /dst working in write through caching mode and then run "cp -a /src /dst". @@ -1041,6 +1026,7 @@ IMPORTANT: If SCST 1.x BLOCKIO worked by default in NV_CACHE mode, when restore 1.x behavior, your should recreate your BLOCKIO devices in NV_CACHE mode. + Pass-through mode ----------------- @@ -1049,7 +1035,11 @@ scst_disk, scst_tape, etc) SCSI commands, coming from remote initiators, are passed to local SCSI devices on target as is, without any modifications. -In the SYSFS interface all real SCSI devices are listed in +SCST supports 1 to many pass-through, when several initiators can safely +connect a single pass-through device (a tape, for instance). For such +cases SCST emulates all the necessary functionality. + +In the sysfs interface all real SCSI devices are listed in /sys/kernel/scst_tgt/devices in form host:channel:id:lun numbers, for instance 1:0:0:0. The recommended way to match those numbers to your devices is use of lsscsi utility. @@ -1105,19 +1095,6 @@ page each. See the following patch as an example: http://scst.sourceforge.net/sgv_big_order_alloc.diff -User space mode using scst_user dev handler -------------------------------------------- - -User space program fileio_tgt uses interface of scst_user dev handler -and allows to see how it works in various modes. Fileio_tgt provides -mostly the same functionality as scst_vdisk handler with the most -noticeable difference that it supports O_DIRECT mode. O_DIRECT mode is -basically the same as BLOCKIO, but also supports files, so for some -loads it could be significantly faster, than the regular FILEIO access. -All the words about BLOCKIO from above apply to O_DIRECT as well. See -fileio_tgt's README file for more details. - - Performance ----------- @@ -1141,7 +1118,8 @@ In order to get the maximum performance you should: 1. For SCST: - Disable in Makefile CONFIG_SCST_STRICT_SERIALIZING, CONFIG_SCST_EXTRACHECKS, - CONFIG_SCST_TRACING, CONFIG_SCST_DEBUG*, CONFIG_SCST_STRICT_SECURITY + CONFIG_SCST_TRACING, CONFIG_SCST_DEBUG*, CONFIG_SCST_STRICT_SECURITY, + CONFIG_SCST_MEASURE_LATENCY 2. For target drivers: @@ -1468,8 +1446,8 @@ Thanks to: * Bart Van Assche for a lot of help - * Daniel Debonzi for a big part of SCST sysfs tree - implementation + * Daniel Debonzi for a big part of the + initial SCST sysfs tree implementation Vladislav Bolkhovitin , http://scst.sourceforge.net diff --git a/scstadmin/README b/scstadmin/README index f440fe88a..2d84deb26 100644 --- a/scstadmin/README +++ b/scstadmin/README @@ -51,9 +51,9 @@ NOTE: The init.d startup & shutdown scripts now depend on lsb-core. Ensure you h lsb-core installed or 'make install' will fail. Make sure /usr/lib/lsb/install_initd exists. -To enable the newer sysfs version: +To enable the obsolete procfs version: - #> make disable_proc + #> make enable_proc #> make #> make install @@ -61,11 +61,7 @@ To enable the newer sysfs version: Getting Started: ================ -The following pertains to the older procfs based SCST. The new sysfs version is still -a work in progress, but should be mostly complete at this point. You may still use your -old scst.conf, however, I very much recommend you perform a -write_config and review -the resulting configuration file. - +The following pertains to the older procfs based SCST. The scstadmin script is much more functional than scst_db at this point but uses a standard text-based config file. The original thought behind scst_db was to write diff --git a/scstadmin/README.sysfs b/scstadmin/README.sysfs index bc76c5e54..710200ca9 100644 --- a/scstadmin/README.sysfs +++ b/scstadmin/README.sysfs @@ -1,11 +1,3 @@ -ENABLING SYSFS VERSION ----------------------- - -To enable the sysfs version of scstadmin (you must have the sysfs version of -SCST already installed): - -make disable_proc - INSTALLING ----------