scst/iscsi-scst

iSCSI SCST target driver
========================

Version 1.0.2, XX XXXXX 2009
----------------------------

This driver is a forked with all respects version of iSCSI Enterprise
Target (IET) (http://iscsitarget.sourceforge.net/) with updates to work
over SCST as well as with many improvements and bugfixes (see ChangeLog
file). The reason of fork is that the necessary changes are intrusive
and with the current IET merge policy, where only simple bugfix-like
patches, which doesn't touch the core code, could be merged, it is very
unlikely that they will be merged in the main IET trunk.

To let it be installed and work at the same host together with IET
simultaneously all the driver's modules and files were renamed:

 * ietd.conf -> iscsi-scstd.conf
 * ietadm -> iscsi-scst-adm
 * ietd -> iscsi-scstd
 * iscsi-target -> iscsi-scst
 * iscsi-target.ko -> iscsi-scst.ko

This version is compatible with SCST version 1.0.2 and higher.


Installation if your Linux kernel already has iSCSI-SCST built-in
-----------------------------------------------------------------

Simply run "make all", then "make install".


Installation out of Linux kernel tree
-------------------------------------

Basically as in README-IET, where file names are changed as specified
above.

Only vanilla kernels from kernel.org and RHEL/CentOS 5.2 kernels are
supported, but it should work on other (vendors') kernels, if you manage
to successfully compile on them. The main problem with vendor's kernels
is that they often contain patches, which will appear only in the next
version of the vanilla kernel, therefore it's quite hard to track such
changes. Thus, if during compilation for some vendor's kernel your
compiler complains about redefinition of some symbol, you should either
switch to vanilla kernel, or add or change as necessary the
corresponding to that symbol "#if LINUX_VERSION_CODE" statement.

If during compilation you see message like "*** No rule to make target
`xxx.h', needed by `yyy.o'.  Stop.", then your autogenerated
dependencies don't match your compiler configuration anymore. You should
run "make extraclean" to remove them. On the next compilation they will
be regenerated.

If you experience problems during kernel module load or running, check
your system and/or kernel logs (or run dmesg command for the few most
recent kernel messages).

To use full power of TCP zero-copy transmit functions, especially
dealing with user space supplied via scst_user module memory, iSCSI-SCST
needs to be notified when Linux networking finished data transmission.
Patch put_page_callback-<kernel-version>.patch provides such
functionality. The corresponding version of it should be applied on your
kernel. Then you should enable CONFIG_TCP_ZERO_COPY_TRANSFER_COMPLETION_NOTIFICATION
kernel config option. This is highly recommended, but not required. Basically,
you should consider using of this patch as some optimization, which IET
doesn't have, so if you don't use it, you will just revert to the
original IET behavior, when for data transmission:

 - For in-kernel allocated memory (scst_vdisk and pass-through
   handlers) usage of SGV cache on transmit path (READ-type commands)
   will be disabled, but data will still be sent in zero-copy manner.
   The performance hit will be not big, but performance will still
   remain better, than for IET, because SGV cache will remain used on
   receive path while IET doesn't have such feature.

 - For user space allocated memory (scst_user handler) all transmitted
   data will be additionally copied into temporary TCP buffers. The
   performance hit will be quite noticeable.

Note, that if your network hardware does not support TX offload
functions or has them disabled, then TCP zero-copy transmit functions on
your system will not be used by Linux networking in any case, so
put_page_callback patch will not be able to improve performance for you.
You can check your network hardware offload capabilities by command
"ethtool -k ethX", where X is the network device number. At least
"tx-checksumming" and "scatter-gather" should be enabled.

If you have in your kernel log error messages like:

iscsi-scst: ***ERROR*** net_priv isn't NULL and != ref_cmd

with the corresponding kernel BUG dump, then put_page_callback patch you
use isn't sufficient for your kernel. This might be because the kernel
you use has some additional patches applied, which affect the
functionality, which put_page_callback patch provides. For example,
Fedora or Gentoo use kernels, which, although have version number like
2.6.18, are greatly differ from the "vanilla" kernel 2.6.18,
maintained by Linus Torvalds for that the put_page_callback patch was
created. In this case I would recommend you either:

 - Search net/ in your kernel source for "put_page" and "get_page" functions.
   If you find any in some place, except in net/sunrpc/svc.c and
   net/core/pktgen.c, then, most likely, you found the reason of your
   problem. Replace them by "net_put_page" and "net_get_page"
   correspondingly and try again. If the problem is solved, then please
   prepare a new put_page_callback patch and send it to the SCST mailing
   list scst-devel@lists.sourceforge.net.

or

 - Unapply this patch and use iSCSI-SCST without it. Also report this
   problem to the SCST mailing list scst-devel@lists.sourceforge.net.


Usage
-----

See in doc/iscsi-scst-howto.txt examples how to configure iSCSI-SCST.

ISCSI parameters like iSNS, CHAP and target parameters are configured in
iscsi-scstd.conf. All LUN information is configured using the
corresponding SCST interface. See in SCST README file section "Access
and devices visibility management (LUN masking)" to find out how to do
it. It is highly recommended to use scstadmin utility for that purpose.
The LUN information in iscsi-scstd.conf will be ignored. This is because
now responsibilities are divided between the target driver (iSCSI-SCST)
and the SCST core as it logically should be: the target driver is
responsible for handling targets and their parameters, SCST core is
responsible for handling backstorage.

It is recommended to use TEST UNIT READY ("tur") command to check if
iSCSI-SCST target is alive.

IMPORTANT: All LUN information (access control) MUST be configured
=========  BEFORE iscsi-scstd started!

Also see SCST README file how to tune for the best performance.

CAUTION: Working of target and initiator on the same host isn't fully
=======  supported. See SCST README file for details.


Troubleshooting
---------------

If you have any problems, start troubleshooting from looking at the
kernel and system logs. In the kernel log iSCSI-SCST and SCST core send
their messages, in the system log iscsi-scstd sends its messages. In
most Linux distributions both those logs are put to /var/log/messages
file.

Then, it might be helpful to increase level of logging. For kernel
modules you should make the debug build, by either running "make
release2debug" if you work with SCST SVN tree, or by enabling the
corresponding debug symbols (see below). 

If after looking on the logs the reason of your problem is still unclear
for you, report to SCST mailing list scst-devel@lists.sourceforge.net.


Work if target's backstorage or link is too slow
------------------------------------------------

In some cases you can experience I/O stalls or see in the kernel log
abort or reset messages. It can happen under high I/O load, when your
target's backstorage gets overloaded, or working over a slow link, when
the link can't serve all the queued commands on time, 

To workaround it you can reduce QueuedCommands parameter in
iscsi-scstd.conf file for the corresponding target to some lower value,
like 8 (default is 32).

Also see SCST README file for more details about that issue and ways to
prevent it.


Performance advices
-------------------

1. If you use Windows XP or Windows 2003+ as initiators, you should
consider to decrease TcpAckFrequency parameter to 1. See
http://support.microsoft.com/kb/328890/ or google for "TcpAckFrequency"
for more details.

2. See how to get the maximum throughput from iSCSI, for instance, at
http://virtualgeek.typepad.com/virtual_geek/2009/01/a-multivendor-post-to-help-our-mutual-iscsi-customers-using-vmware.html.
It's about VMware, but its recommendations apply to other environments
as well.

3. ISCSI initiators built in pre-CentOS/RHEL 5 reported to have some
performance problems. If you use it, it is strongly advised to upgrade.


Compilation options
-------------------

There are the following compilation options, that could be commented
in/out in the kernel's module Makefile:

 - CONFIG_SCST_DEBUG - turns on some debugging code, including some logging.
   Makes the driver considerably bigger and slower, producing large amount of
   log data.

 - CONFIG_SCST_TRACING - turns on ability to log events. Makes the driver
   considerably bigger and leads to some performance loss.

 - CONFIG_SCST_EXTRACHECKS - adds extra validity checks in the various places.

 - CONFIG_SCST_ISCSI_DEBUG_DIGEST_FAILURES - simulates digest failures in
   random places.


Creating version of put_page_callback patch for your kernel
-----------------------------------------------------------

If you need your own version of put_page_callback patch for your custom
kernel, for which there is no prepared version, you can create it
yourself. This is pretty mechanical work, you don't need to understand
how it works, you only need to do the following two steps:

1. Apply the closest version of put_page_callback-<kernel-version>.patch
on your kernel. Resolve only failed hunks from include/ and
net/core/utils.c, ignore other failures.

2. Search net/ in your kernel source for "put_page" and "get_page"
functions. Replace them by "net_put_page" and "net_get_page"
correspondingly. 

That's all. Then please send your new
put_page_callback-<kernel-version>.patch to the SCST mailing list
scst-devel@lists.sourceforge.net.

Credits
-------

Thanks to:

 * IET developers for IET

 * Ming Zhang <blackmagic02881@gmail.com> for fixes

 * Krzysztof Blaszkowski <kb@sysmikro.com.pl> for many fixes

 * Alexey Kuznetsov <kuznet@ms2.inr.ac.ru> for comments and help in
   debugging

 * Tomasz Chmielewski <mangoo@wpkg.org> for testing and suggestions

 * Bart Van Assche <bart.vanassche@gmail.com> for a lot of help

Vladislav Bolkhovitin <vst@vlnb.net>, http://scst.sourceforge.net