iSCSI SCST target driver ======================== Version 2.0.0, XX XXXXX 2010 ---------------------------- ISCSI-SCST is a deeply reworked fork of iSCSI Enterprise Target (IET) (http://iscsitarget.sourceforge.net). Reasons of the fork were: - To be able to use full power of SCST core. - To fix all the problems, corner cases issues and iSCSI standard violations which IET has. See more info at http://iscsi-scst.sourceforge.net. This version is compatible with SCST version 2.0.0 and higher. Installation if your Linux kernel already has iSCSI-SCST built-in ----------------------------------------------------------------- Simply run "make all", then "make install". Installation out of Linux kernel tree ------------------------------------- See HOWTOs in the doc/ subdirectory. Only vanilla kernels from kernel.org and RHEL/CentOS 5.2 kernels are supported, but it should work on other (vendors') kernels, if you manage to successfully compile on them. The main problem with vendor's kernels is that they often contain patches, which appear only in the next version of the vanilla kernel, therefore it's quite hard to track such changes. Thus, if during compilation for some vendor's kernel your compiler complains about redefinition of some symbol, you should either switch to vanilla kernel, or add or change as necessary the corresponding to that symbol "#if LINUX_VERSION_CODE" statement. If during compilation you see message like "*** No rule to make target `xxx.h', needed by `yyy.o'. Stop.", then your autogenerated dependencies don't match your compiler configuration anymore. You should run "make extraclean" to remove them. On the next compilation they will be regenerated. If you experience problems during kernel module load or running, check your system and/or kernel logs (or run dmesg command for the few most recent kernel messages). To use full power of TCP zero-copy transmit functions, especially dealing with user space supplied via scst_user module memory, iSCSI-SCST needs to be notified when Linux networking finished data transmission. For that you should enable CONFIG_TCP_ZERO_COPY_TRANSFER_COMPLETION_NOTIFICATION kernel config option. This is highly recommended, but not required. Basically, iSCSI-SCST works fine with an unpatched Linux kernel with the same or better speed as other open source iSCSI targets, including IET, but if you want even better performance you have to patch and rebuild the kernel. Without CONFIG_TCP_ZERO_COPY_TRANSFER_COMPLETION_NOTIFICATION enabled you will just revert to the original behavior of other open source iSCSI targets, when for data transmission: - For in-kernel allocated memory (scst_vdisk and pass-through handlers) usage of SGV cache on transmit path (READ-type commands) will be disabled, but data will still be sent in zero-copy manner. - For user space allocated memory (scst_user handler) all transmitted data will be additionally copied into temporary TCP buffers. The performance hit will be quite noticeable. Note, that if your network hardware does not support TX offload functions or has them disabled, then TCP zero-copy transmit functions on your system will not be used by Linux networking in any case, so put_page_callback patch will not be able to improve performance for you. You can check your network hardware offload capabilities by command "ethtool -k ethX", where X is the network device number. At least "tx-checksumming" and "scatter-gather" should be enabled. If you have in your kernel log error messages like: iscsi-scst: ***ERROR*** net_priv isn't NULL and != ref_cmd with the corresponding kernel BUG dump, then put_page_callback patch you use isn't sufficient for your kernel. This might be because the kernel you use has some additional patches applied, which affect the functionality, which put_page_callback patch provides. For example, Fedora or Gentoo use kernels, which, although have version number like 2.6.18, are greatly differ from the "vanilla" kernel 2.6.18, maintained by Linus Torvalds for that the put_page_callback patch was created. In this case it is recommended either: - Search net/ in your kernel source for "put_page" and "get_page" functions. If you find any in some place, except in net/sunrpc/svc.c and net/core/pktgen.c, then, most likely, you found the reason of your problem. Replace them by "net_put_page" and "net_get_page" correspondingly and try again. If the problem is solved, then please prepare a new put_page_callback patch and send it to the SCST mailing list scst-devel@lists.sourceforge.net. or - Unapply this patch and use iSCSI-SCST without it. Also report this problem to the SCST mailing list scst-devel@lists.sourceforge.net. Usage ----- See HOWTOs in the doc/ subdirectory. In 2.0.0 usage of iscsi-scstd.conf as well as iscsi-scst-adm utility is obsolete. Use the sysfs interface facilities instead. It is recommended to use TEST UNIT READY ("tur") command to check if iSCSI-SCST target is alive in MPIO configurations. Also see SCST README file how to tune for the best performance. CAUTION: Working of target and initiator on the same host isn't fully ======= supported. See SCST README file for details. Sysfs interface --------------- Starting from 2.0.0 iSCSI-SCST has sysfs interface. You can switch to it by running "make disable_proc". To switch back to the procfs interface you should run "make enable_proc". The procfs interface starting from version 2.0.0 is obsolete and will be removed in one of the next versions. Root of SCST sysfs interface is /sys/kernel/scst_tgt. Root of iSCSI-SCST is /sys/kernel/scst_tgt/targets/iscsi. It has the following entries: - None, one or more subdirectories for targets with name equal to names of the corresponding targets. - IncomingUser[num] - optional one or more attributes containing user name and password for incoming discovery user name. Not exist by default and can be added through "mgmt" entry, see below. - OutgoingUser - optional attribute containing user name and password for outgoing discovery user name. Not exist by default and can be added through "mgmt" entry, see below. - iSNSServer - contains name or IP address of iSNS server with optional "AccessControl" attribute, which allows to enable iSNS access control. Empty by default. - enabled - using this attribute you can enable or disable iSCSI-SCST accept new connections. It allows to finish configuring global iSCSI-SCST attributes before it starts accepting new connections. 0 by default. - open_state - read-only attribute, which allows to see if the user space part of iSCSI-SCST connected to the kernel part. - trace_level - allows to enable and disable various tracing facilities. See content of this file for help how to use it. - version - read-only attribute, which allows to see version of iSCSI-SCST and enabled optional features. - mgmt - main management entry, which allows to configure iSCSI-SCST. Namely, add/delete targets as well as add/delete optional global and per-target attributes. See content of this file for help how to use it. Each iSCSI-SCST sysfs file (attribute) can contain in the last line mark "[key]". It is automatically added mark used to allow scstadmin to see which attributes it should save in the config file. You can ignore it. Each target subdirectory contains the following entries: - ini_groups - subdirectory defining initiator groups for this target, used to define per-initiator access control. See SCST core README for more details. - luns - subdirectory defining LUNs of this target. See SCST core README for more details. - sessions - subdirectory containing connected to this target sessions. - IncomingUser[num] - optional one or more attributes containing user name and password for incoming user name. Not exist by default and can be added through the "mgmt" entry, see above. - OutgoingUser - optional attribute containing user name and password for outgoing user name. Not exist by default and can be added through the "mgmt" entry, see above. - Entries defining default iSCSI parameters values used during iSCSI parameters negotiation. Only entries which can be changed or make sense are listed there. - QueuedCommands - defines maximum number of commands queued to any session of this target. Default is 32 commands. - RspTimeout - defines the maximum time in seconds a command can wait for response from initiator, otherwise the corresponding connection will be closed. For performance reasons it is implemented as a timer, which once in RspTimeout time checks the oldest command waiting for response and, if it's older than RspTimeout, then it closes the connection. Hence, a stalled connection will be closed in time between RspTimeout and 2*RspTimeout. Default is 30 seconds. - NopInInterval - defines interval between NOP-In requests, which the target will send on idle connections to check if the initiator is still alive. If there is no NOP-Out reply from the initiator in RspTimeout time, the corresponding connection will be closed. Default is 30 seconds. If it's set to 0, then NOP-In requests are disabled. - enabled - using this attribute you can enable or disable iSCSI-SCST accept new connections to this target. It allows to finish configuring it before it starts accepting new connections. 0 by default. - rel_tgt_id - allows to read or write SCSI Relative Target Port Identifier attribute. This identifier is used to identify SCSI Target Ports by some SCSI commands, mainly by Persistent Reservations commands. This identifier must be unique among all SCST targets, but for convenience SCST allows disabled targets to have not unique rel_tgt_id. In this case SCST will not allow to enable this target until rel_tgt_id becomes unique. This attribute initialized unique by SCST by default. - tid - TID of this target. Subdirectory "sessions" contains one subdirectory for each connected session with name equal to name of the connected initiator. Each session subdirectory contains the following entries: - One subdirectory for each TCP connection in this session. ISCSI-SCST supports 1 connection per session, but the session subdirectory can contain several connections: one active and other being closed. - Entries defining negotiated iSCSI parameters. Only parameters which can be changed or make sense are listed there. - initiator_name - contains initiator name - sid - contains SID of this session - reinstating - contains reinstatement state of this session - force_close - write-only attribute, which allows to force close this session. This is the only writable session attribute. - active_commands - contains number of active, i.e. not yet or being executed, SCSI commands in this session. - commands - contains overall number of SCSI commands in this session. Each connection subdirectory contains the following entries: - cid - contains CID of this connection. - ip - contains IP address of the connected initiator. - state - contains processing state of this connection. Below is a sample script, which configures 1 virtual disk "disk1" using /disk1 image and one target iqn.2006-10.net.vlnb:tgt with all default parameters: #!/bin/bash modprobe scst modprobe scst_vdisk echo "add_device disk1 filename=/disk1; nv_cache=1" >/sys/kernel/scst_tgt/handlers/vdisk_fileio/mgmt service iscsi-scst start echo "add_target iqn.2006-10.net.vlnb:tgt" >/sys/kernel/scst_tgt/targets/iscsi/mgmt echo "add disk1 0" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/luns/mgmt echo 1 >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/enabled echo 1 >/sys/kernel/scst_tgt/targets/iscsi/enabled Below is more advanced sample script, which configures more virtual devices of various types, including virtual CDROM and 2 targets, one with all default parameters, another one with some not default parameters, incoming and outgoing user names for CHAP authentification, and special permissions for initiator iqn.2005-03.org.open-iscsi:cacdcd2520, which will see another set of devices. Also this sample configures CHAP authentication for discovery sessions and iSNS server with access control. #!/bin/bash modprobe scst modprobe scst_vdisk echo "add_device disk1 filename=/disk1; nv_cache=1" >/sys/kernel/scst_tgt/handlers/vdisk_fileio/mgmt echo "add_device disk2 filename=/disk2; blocksize=4096; nv_cache=1" >/sys/kernel/scst_tgt/handlers/vdisk_fileio/mgmt echo "add_device blockio filename=/dev/sda5" >/sys/kernel/scst_tgt/handlers/vdisk_blockio/mgmt echo "add_device nullio" >/sys/kernel/scst_tgt/handlers/vdisk_nullio/mgmt echo "add_device cdrom" >/sys/kernel/scst_tgt/handlers/vcdrom/mgmt service iscsi-scst start echo "192.168.1.16 AccessControl" >/sys/kernel/scst_tgt/targets/iscsi/iSNSServer echo "add_attribute IncomingUser joeD 12charsecret" >/sys/kernel/scst_tgt/targets/iscsi/mgmt echo "add_attribute OutgoingUser jackD 12charsecret1" >/sys/kernel/scst_tgt/targets/iscsi/mgmt echo "add_target iqn.2006-10.net.vlnb:tgt" >/sys/kernel/scst_tgt/targets/iscsi/mgmt echo "add disk1 0" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/luns/mgmt echo "add cdrom 1" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/luns/mgmt echo "add_target iqn.2006-10.net.vlnb:tgt1" >/sys/kernel/scst_tgt/targets/iscsi/mgmt echo "add_target_attribute iqn.2006-10.net.vlnb:tgt1 IncomingUser1 joe2 12charsecret2" >/sys/kernel/scst_tgt/targets/iscsi/mgmt echo "add_target_attribute iqn.2006-10.net.vlnb:tgt1 IncomingUser joe 12charsecret" >/sys/kernel/scst_tgt/targets/iscsi/mgmt echo "add_target_attribute iqn.2006-10.net.vlnb:tgt1 OutgoingUser jim1 12charpasswd" >/sys/kernel/scst_tgt/targets/iscsi/mgmt echo "No" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/InitialR2T echo "Yes" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ImmediateData echo "8192" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/MaxRecvDataSegmentLength echo "8192" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/MaxXmitDataSegmentLength echo "131072" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/MaxBurstLength echo "32768" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/FirstBurstLength echo "1" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/MaxOutstandingR2T echo "CRC32C,None" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/HeaderDigest echo "CRC32C,None" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/DataDigest echo "32" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/QueuedCommands echo "add disk2 0" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/luns/mgmt echo "add nullio 26" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/luns/mgmt echo "create special_ini" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ini_groups/mgmt echo "add blockio 0 read_only=1" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ini_groups/special_ini/luns/mgmt echo "add iqn.2005-03.org.open-iscsi:cacdcd2520" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ini_groups/special_ini/initiators/mgmt echo 1 >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/enabled echo 1 >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/enabled echo 1 >/sys/kernel/scst_tgt/targets/iscsi/enabled The resulting overall SCST sysfs hierarchy with an initiator connected to both iSCSI-SCST targets will look like: /sys/kernel/scst_tgt |-- devices | |-- blockio | | |-- blocksize | | |-- exported | | | `-- export0 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ini_groups/special_ini/luns/0 | | |-- filename | | |-- handler -> ../../handlers/vdisk_blockio | | |-- read_only | | |-- removable | | |-- resync_size | | |-- size_mb | | |-- t10_dev_id | | |-- type | | `-- usn | |-- cdrom | | |-- exported | | | `-- export0 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt/luns/1 | | |-- filename | | |-- handler -> ../../handlers/vcdrom | | |-- size_mb | | |-- t10_dev_id | | |-- type | | `-- usn | |-- disk1 | | |-- blocksize | | |-- exported | | | `-- export0 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt/luns/0 | | |-- filename | | |-- handler -> ../../handlers/vdisk_fileio | | |-- nv_cache | | |-- o_direct | | |-- read_only | | |-- removable | | |-- resync_size | | |-- size_mb | | |-- t10_dev_id | | |-- type | | |-- usn | | `-- write_through | |-- disk2 | | |-- blocksize | | |-- exported | | | `-- export0 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt1/luns/0 | | |-- filename | | |-- handler -> ../../handlers/vdisk_fileio | | |-- nv_cache | | |-- o_direct | | |-- read_only | | |-- removable | | |-- resync_size | | |-- size_mb | | |-- t10_dev_id | | |-- type | | |-- usn | | `-- write_through | `-- nullio | |-- blocksize | |-- exported | | `-- export0 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt1/luns/26 | |-- handler -> ../../handlers/vdisk_nullio | |-- read_only | |-- removable | |-- size_mb | |-- t10_dev_id | |-- type | `-- usn |-- handlers | |-- vcdrom | | |-- cdrom -> ../../devices/cdrom | | |-- mgmt | | |-- trace_level | | `-- type | |-- vdisk_blockio | | |-- blockio -> ../../devices/blockio | | |-- mgmt | | |-- trace_level | | `-- type | |-- vdisk_fileio | | |-- disk1 -> ../../devices/disk1 | | |-- disk2 -> ../../devices/disk2 | | |-- mgmt | | |-- trace_level | | `-- type | `-- vdisk_nullio | |-- mgmt | |-- nullio -> ../../devices/nullio | |-- trace_level | `-- type |-- sgv | |-- global_stats | |-- sgv | | `-- stats | |-- sgv-clust | | `-- stats | `-- sgv-dma | `-- stats |-- targets | `-- iscsi | |-- IncomingUser | |-- OutgoingUser | |-- enabled | |-- iSNSServer | |-- iqn.2006-10.net.vlnb:tgt | | |-- DataDigest | | |-- FirstBurstLength | | |-- HeaderDigest | | |-- ImmediateData | | |-- InitialR2T | | |-- MaxBurstLength | | |-- MaxOutstandingR2T | | |-- MaxRecvDataSegmentLength | | |-- MaxXmitDataSegmentLength | | |-- NopInInterval | | |-- QueuedCommands | | |-- RspTimeout | | |-- enabled | | |-- ini_groups | | | `-- mgmt | | |-- luns | | | |-- 0 | | | | |-- device -> ../../../../../devices/disk1 | | | | `-- read_only | | | |-- 1 | | | | |-- device -> ../../../../../devices/cdrom | | | | `-- read_only | | | `-- mgmt | | |-- rel_tgt_id | | |-- sessions | | | `-- iqn.2005-03.org.open-iscsi:cacdcd2520 | | | |-- 10.170.75.2 | | | | |-- cid | | | | |-- ip | | | | `-- state | | | |-- DataDigest | | | |-- FirstBurstLength | | | |-- HeaderDigest | | | |-- ImmediateData | | | |-- InitialR2T | | | |-- MaxBurstLength | | | |-- MaxOutstandingR2T | | | |-- MaxRecvDataSegmentLength | | | |-- MaxXmitDataSegmentLength | | | |-- active_commands | | | |-- commands | | | |-- force_close | | | |-- initiator_name | | | |-- luns -> ../../luns | | | |-- reinstating | | | `-- sid | | `-- tid | |-- iqn.2006-10.net.vlnb:tgt1 | | |-- DataDigest | | |-- FirstBurstLength | | |-- HeaderDigest | | |-- ImmediateData | | |-- IncomingUser | | |-- IncomingUser1 | | |-- InitialR2T | | |-- MaxBurstLength | | |-- MaxOutstandingR2T | | |-- MaxRecvDataSegmentLength | | |-- MaxXmitDataSegmentLength | | |-- OutgoingUser | | |-- NopInInterval | | |-- QueuedCommands | | |-- RspTimeout | | |-- enabled | | |-- ini_groups | | | |-- mgmt | | | `-- special_ini | | | |-- initiators | | | | |-- iqn.2005-03.org.open-iscsi:cacdcd2520 | | | | `-- mgmt | | | `-- luns | | | |-- 0 | | | | |-- device -> ../../../../../../../devices/blockio | | | | `-- read_only | | | `-- mgmt | | |-- luns | | | |-- 0 | | | | |-- device -> ../../../../../devices/disk2 | | | | `-- read_only | | | |-- 26 | | | | |-- device -> ../../../../../devices/nullio | | | | `-- read_only | | | `-- mgmt | | |-- rel_tgt_id | | |-- sessions | | | `-- iqn.2005-03.org.open-iscsi:cacdcd2520 | | | |-- 10.170.75.2 | | | | |-- cid | | | | |-- ip | | | | `-- state | | | |-- DataDigest | | | |-- FirstBurstLength | | | |-- HeaderDigest | | | |-- ImmediateData | | | |-- InitialR2T | | | |-- MaxBurstLength | | | |-- MaxOutstandingR2T | | | |-- MaxRecvDataSegmentLength | | | |-- MaxXmitDataSegmentLength | | | |-- active_commands | | | |-- commands | | | |-- force_close | | | |-- initiator_name | | | |-- luns -> ../../ini_groups/special_ini/luns | | | |-- reinstating | | | `-- sid | | `-- tid | |-- mgmt | |-- open_state | |-- trace_level | `-- version |-- threads |-- trace_level `-- version Troubleshooting --------------- If you have any problems, start troubleshooting from looking at the kernel and system logs. In the kernel log iSCSI-SCST and SCST core send their messages, in the system log iscsi-scstd sends its messages. In most Linux distributions both those logs are put to /var/log/messages file. Then, it might be helpful to increase level of logging. For kernel modules you should make the debug build, by either running "make release2debug" if you work with SCST SVN tree, or by enabling the corresponding debug symbols (see below). If after looking on the logs the reason of your problem is still unclear for you, report to SCST mailing list scst-devel@lists.sourceforge.net. Work if target's backstorage or link is too slow ------------------------------------------------ In some cases you can experience I/O stalls or see in the kernel log abort or reset messages. It can happen under high I/O load, when your target's backstorage gets overloaded, or working over a slow link, when the link can't serve all the queued commands on time, To workaround it you can reduce QueuedCommands parameter for the corresponding target to some lower value, like 8 (default is 32). Also see SCST README file for more details about that issue and ways to prevent it. Performance advices ------------------- 1. If you use Windows XP or Windows 2003+ as initiators, you should consider to decrease TcpAckFrequency parameter to 1. See http://support.microsoft.com/kb/328890/ or google for "TcpAckFrequency" for more details. 2. See how to get the maximum throughput from iSCSI, for instance, at http://virtualgeek.typepad.com/virtual_geek/2009/01/a-multivendor-post-to-help-our-mutual-iscsi-customers-using-vmware.html. It's about VMware, but its recommendations apply to other environments as well. 3. ISCSI initiators from pre-CentOS/RHEL 5 reported to have some performance problems. If you use it, it is strongly advised to upgrade. Compilation options ------------------- There are the following compilation options, that could be commented in/out in the kernel's module Makefile: - CONFIG_SCST_DEBUG - turns on some debugging code, including some logging. Makes the driver considerably bigger and slower, producing large amount of log data. - CONFIG_SCST_TRACING - turns on ability to log events. Makes the driver considerably bigger and leads to some performance loss. - CONFIG_SCST_EXTRACHECKS - adds extra validity checks in the various places. - CONFIG_SCST_ISCSI_DEBUG_DIGEST_FAILURES - simulates digest failures in random places. Creating version of put_page_callback patch for your kernel ----------------------------------------------------------- If you need your own version of put_page_callback patch for your custom kernel, for which there is no prepared version, you can create it yourself. This is pretty mechanical work, you don't need to understand how it works, you only need to do the following two steps: 1. Apply the closest version of put_page_callback-.patch on your kernel. Resolve only failed hunks from include/ and net/core/utils.c, ignore other failures. 2. Search net/ in your kernel source for "put_page" and "get_page" functions. Replace them by "net_put_page" and "net_get_page" correspondingly. That's all. Then please send your new put_page_callback-.patch to the SCST mailing list scst-devel@lists.sourceforge.net. Credits ------- Thanks to: * Ming Zhang for fixes * Krzysztof Blaszkowski for many fixes * Alexey Kuznetsov for comments and help in debugging * Tomasz Chmielewski for testing and suggestions * Bart Van Assche for a lot of help Vladislav Bolkhovitin , http://scst.sourceforge.net