mirrors/scst
1
0
Fork 0
mirror of https://github.com/SCST-project/scst.git synced 2026-08-19 05:36:22 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
75c83bb8932e6979fffaaf081dbd10c926714c8a
scst/iscsi-scst/README_in-tree
T
Vladislav Bolkhovitin 32082dc44a Small docs update 
git-svn-id: http://svn.code.sf.net/p/scst/svn/trunk@1607 d57e44dd-8a1f-0410-8b47-8ef2f437770f
2010-04-12 17:38:08 +00:00

592 lines
22 KiB
Plaintext
Raw Blame History

iSCSI SCST target driver
========================
ISCSI-SCST is a deeply reworked fork of iSCSI Enterprise Target (IET)
(http://iscsitarget.sourceforge.net). Reasons of the fork were:
- To be able to use full power of SCST core.
- To fix all the problems, corner cases issues and iSCSI standard
violations which IET has.
See more info at http://iscsi-scst.sourceforge.net.
Usage
-----
See in http://iscsi-scst.sourceforge.net/iscsi-scst-howto.txt how to
configure iSCSI-SCST.
In 2.0.0 usage of iscsi-scstd.conf as well as iscsi-scst-adm utility is
obsolete. Use the sysfs interface facilities instead.
It is recommended to use TEST UNIT READY ("tur") command to check if
iSCSI-SCST target is alive in MPIO configurations.
Also see SCST README file how to tune for the best performance.
CAUTION: Working of target and initiator on the same host isn't fully
======= supported. See SCST README file for details.
Sysfs interface
---------------
Root of SCST sysfs interface is /sys/kernel/scst_tgt. Root of iSCSI-SCST
is /sys/kernel/scst_tgt/targets/iscsi. It has the following entries:
- None, one or more subdirectories for targets with name equal to names
of the corresponding targets.
- IncomingUser[num] - optional one or more attributes containing user
name and password for incoming discovery user name. Not exist by
default and can be added through "mgmt" entry, see below.
- OutgoingUser - optional attribute containing user name and password
for outgoing discovery user name. Not exist by default and can be
added through "mgmt" entry, see below.
- iSNSServer - contains name or IP address of iSNS server with optional
"AccessControl" attribute, which allows to enable iSNS access
control. Empty by default.
- enabled - using this attribute you can enable or disable iSCSI-SCST
accept new connections. It allows to finish configuring global
iSCSI-SCST attributes before it starts accepting new connections. 0
by default.
- open_state - read-only attribute, which allows to see if the user
space part of iSCSI-SCST connected to the kernel part.
- trace_level - allows to enable and disable various tracing
facilities. See content of this file for help how to use it.
- version - read-only attribute, which allows to see version of
iSCSI-SCST and enabled optional features.
- mgmt - main management entry, which allows to configure iSCSI-SCST.
Namely, add/delete targets as well as add/delete optional global and
per-target attributes. See content of this file for help how to use
it.
Each iSCSI-SCST sysfs file (attribute) can contain in the last line mark
"[key]". It is automatically added mark used to allow scstadmin to see
which attributes it should save in the config file. You can ignore it.
Each target subdirectory contains the following entries:
- ini_groups - subdirectory defining initiator groups for this target,
used to define per-initiator access control. See SCST core README for
more details.
- luns - subdirectory defining LUNs of this target. See SCST core
README for more details.
- sessions - subdirectory containing connected to this target sessions.
- IncomingUser[num] - optional one or more attributes containing user
name and password for incoming user name. Not exist by default and can
be added through the "mgmt" entry, see above.
- OutgoingUser - optional attribute containing user name and password
for outgoing user name. Not exist by default and can be added through
the "mgmt" entry, see above.
- Entries defining default iSCSI parameters values used during iSCSI
parameters negotiation. Only entries which can be changed or make
sense are listed there.
- QueuedCommands - defines maximum number of commands queued to any
session of this target. Default is 32 commands.
- RspTimeout - defines the maximum time in seconds a command can wait for
response from initiator, otherwise the corresponding connection will
be closed. For performance reasons it is implemented as a timer,
which once in RspTimeout time checks the oldest command waiting for
response and, if it's older than RspTimeout, then it closes the
connection. Hence, a stalled connection will be closed in time
between RspTimeout and 2*RspTimeout. Default is 30 seconds.
- NopInInterval - defines interval between NOP-In requests, which the
target will send on idle connections to check if the initiator is
still alive. If there is no NOP-Out reply from the initiator in
RspTimeout time, the corresponding connection will be closed. Default
is 30 seconds. If it's set to 0, then NOP-In requests are disabled.
- enabled - using this attribute you can enable or disable iSCSI-SCST
accept new connections to this target. It allows to finish
configuring it before it starts accepting new connections. 0 by
default.
- rel_tgt_id - allows to read or write SCSI Relative Target Port
Identifier attribute. This identifier is used to identify SCSI Target
Ports by some SCSI commands, mainly by Persistent Reservations
commands. This identifier must be unique among all SCST targets, but
for convenience SCST allows disabled targets to have not unique
rel_tgt_id. In this case SCST will not allow to enable this target
until rel_tgt_id becomes unique. This attribute initialized unique by
SCST by default.
- tid - TID of this target.
Subdirectory "sessions" contains one subdirectory for each connected
session with name equal to name of the connected initiator.
Each session subdirectory contains the following entries:
- One subdirectory for each TCP connection in this session. ISCSI-SCST
supports 1 connection per session, but the session subdirectory can
contain several connections: one active and other being closed.
- Entries defining negotiated iSCSI parameters. Only parameters which
can be changed or make sense are listed there.
- initiator_name - contains initiator name
- sid - contains SID of this session
- reinstating - contains reinstatement state of this session
- force_close - write-only attribute, which allows to force close this
session. This is the only writable session attribute.
- active_commands - contains number of active, i.e. not yet or being
executed, SCSI commands in this session.
- commands - contains overall number of SCSI commands in this session.
Each connection subdirectory contains the following entries:
- cid - contains CID of this connection.
- ip - contains IP address of the connected initiator.
- state - contains processing state of this connection.
Below is a sample script, which configures 1 virtual disk "disk1" using
/disk1 image and one target iqn.2006-10.net.vlnb:tgt with all default
parameters:
#!/bin/bash
modprobe scst
modprobe scst_vdisk
echo "add_device disk1 filename=/disk1; nv_cache=1" >/sys/kernel/scst_tgt/handlers/vdisk_fileio/mgmt
service iscsi-scst start
echo "add_target iqn.2006-10.net.vlnb:tgt" >/sys/kernel/scst_tgt/targets/iscsi/mgmt
echo "add disk1 0" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/luns/mgmt
echo 1 >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/enabled
echo 1 >/sys/kernel/scst_tgt/targets/iscsi/enabled
Below is more advanced sample script, which configures more virtual
devices of various types, including virtual CDROM and 2 targets, one
with all default parameters, another one with some not default
parameters, incoming and outgoing user names for CHAP authentification,
and special permissions for initiator iqn.2005-03.org.open-iscsi:cacdcd2520,
which will see another set of devices. Also this sample configures CHAP
authentication for discovery sessions and iSNS server with access control.
#!/bin/bash
modprobe scst
modprobe scst_vdisk
echo "add_device disk1 filename=/disk1; nv_cache=1" >/sys/kernel/scst_tgt/handlers/vdisk_fileio/mgmt
echo "add_device disk2 filename=/disk2; blocksize=4096; nv_cache=1" >/sys/kernel/scst_tgt/handlers/vdisk_fileio/mgmt
echo "add_device blockio filename=/dev/sda5" >/sys/kernel/scst_tgt/handlers/vdisk_blockio/mgmt
echo "add_device nullio" >/sys/kernel/scst_tgt/handlers/vdisk_nullio/mgmt
echo "add_device cdrom" >/sys/kernel/scst_tgt/handlers/vcdrom/mgmt
service iscsi-scst start
echo "192.168.1.16 AccessControl" >/sys/kernel/scst_tgt/targets/iscsi/iSNSServer
echo "add_attribute IncomingUser joeD 12charsecret" >/sys/kernel/scst_tgt/targets/iscsi/mgmt
echo "add_attribute OutgoingUser jackD 12charsecret1" >/sys/kernel/scst_tgt/targets/iscsi/mgmt
echo "add_target iqn.2006-10.net.vlnb:tgt" >/sys/kernel/scst_tgt/targets/iscsi/mgmt
echo "add disk1 0" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/luns/mgmt
echo "add cdrom 1" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/luns/mgmt
echo "add_target iqn.2006-10.net.vlnb:tgt1" >/sys/kernel/scst_tgt/targets/iscsi/mgmt
echo "add_target_attribute iqn.2006-10.net.vlnb:tgt1 IncomingUser1 joe2 12charsecret2" >/sys/kernel/scst_tgt/targets/iscsi/mgmt
echo "add_target_attribute iqn.2006-10.net.vlnb:tgt1 IncomingUser joe 12charsecret" >/sys/kernel/scst_tgt/targets/iscsi/mgmt
echo "add_target_attribute iqn.2006-10.net.vlnb:tgt1 OutgoingUser jim1 12charpasswd" >/sys/kernel/scst_tgt/targets/iscsi/mgmt
echo "No" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/InitialR2T
echo "Yes" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ImmediateData
echo "8192" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/MaxRecvDataSegmentLength
echo "8192" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/MaxXmitDataSegmentLength
echo "131072" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/MaxBurstLength
echo "32768" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/FirstBurstLength
echo "1" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/MaxOutstandingR2T
echo "CRC32C,None" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/HeaderDigest
echo "CRC32C,None" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/DataDigest
echo "32" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/QueuedCommands
echo "add disk2 0" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/luns/mgmt
echo "add nullio 26" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/luns/mgmt
echo "create special_ini" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ini_groups/mgmt
echo "add blockio 0 read_only=1" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ini_groups/special_ini/luns/mgmt
echo "add iqn.2005-03.org.open-iscsi:cacdcd2520" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ini_groups/special_ini/initiators/mgmt
echo 1 >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/enabled
echo 1 >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/enabled
echo 1 >/sys/kernel/scst_tgt/targets/iscsi/enabled
The resulting overall SCST sysfs hierarchy with an initiator connected to
both iSCSI-SCST targets will look like:
/sys/kernel/scst_tgt
|-- devices
| |-- blockio
| | |-- blocksize
| | |-- exported
| | | `-- export0 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ini_groups/special_ini/luns/0
| | |-- filename
| | |-- handler -> ../../handlers/vdisk_blockio
| | |-- read_only
| | |-- removable
| | |-- resync_size
| | |-- size_mb
| | |-- t10_dev_id
| | |-- threads_num
| | |-- threads_pool_type
| | |-- type
| | `-- usn
| |-- cdrom
| | |-- exported
| | | `-- export0 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt/luns/1
| | |-- filename
| | |-- handler -> ../../handlers/vcdrom
| | |-- size_mb
| | |-- t10_dev_id
| | |-- threads_num
| | |-- threads_pool_type
| | |-- type
| | `-- usn
| |-- disk1
| | |-- blocksize
| | |-- exported
| | | `-- export0 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt/luns/0
| | |-- filename
| | |-- handler -> ../../handlers/vdisk_fileio
| | |-- nv_cache
| | |-- o_direct
| | |-- read_only
| | |-- removable
| | |-- resync_size
| | |-- size_mb
| | |-- t10_dev_id
| | |-- type
| | |-- usn
| | `-- write_through
| |-- disk2
| | |-- blocksize
| | |-- exported
| | | `-- export0 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt1/luns/0
| | |-- filename
| | |-- handler -> ../../handlers/vdisk_fileio
| | |-- nv_cache
| | |-- o_direct
| | |-- read_only
| | |-- removable
| | |-- resync_size
| | |-- size_mb
| | |-- t10_dev_id
| | |-- threads_num
| | |-- threads_pool_type
| | |-- threads_num
| | |-- threads_pool_type
| | |-- type
| | |-- usn
| | `-- write_through
| `-- nullio
| |-- blocksize
| |-- exported
| | `-- export0 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt1/luns/26
| |-- handler -> ../../handlers/vdisk_nullio
| |-- read_only
| |-- removable
| |-- size_mb
| |-- t10_dev_id
| |-- threads_num
| |-- threads_pool_type
| |-- type
| `-- usn
|-- handlers
| |-- vcdrom
| | |-- cdrom -> ../../devices/cdrom
| | |-- mgmt
| | |-- trace_level
| | `-- type
| |-- vdisk_blockio
| | |-- blockio -> ../../devices/blockio
| | |-- mgmt
| | |-- trace_level
| | `-- type
| |-- vdisk_fileio
| | |-- disk1 -> ../../devices/disk1
| | |-- disk2 -> ../../devices/disk2
| | |-- mgmt
| | |-- trace_level
| | `-- type
| `-- vdisk_nullio
| |-- mgmt
| |-- nullio -> ../../devices/nullio
| |-- trace_level
| `-- type
|-- sgv
| |-- global_stats
| |-- sgv
| | `-- stats
| |-- sgv-clust
| | `-- stats
| `-- sgv-dma
| `-- stats
|-- targets
| `-- iscsi
| |-- IncomingUser
| |-- OutgoingUser
| |-- enabled
| |-- iSNSServer
| |-- iqn.2006-10.net.vlnb:tgt
| | |-- DataDigest
| | |-- FirstBurstLength
| | |-- HeaderDigest
| | |-- ImmediateData
| | |-- InitialR2T
| | |-- MaxBurstLength
| | |-- MaxOutstandingR2T
| | |-- MaxRecvDataSegmentLength
| | |-- MaxXmitDataSegmentLength
| | |-- NopInInterval
| | |-- QueuedCommands
| | |-- RspTimeout
| | |-- enabled
| | |-- ini_groups
| | | `-- mgmt
| | |-- luns
| | | |-- 0
| | | | |-- device -> ../../../../../devices/disk1
| | | | `-- read_only
| | | |-- 1
| | | | |-- device -> ../../../../../devices/cdrom
| | | | `-- read_only
| | | `-- mgmt
| | |-- rel_tgt_id
| | |-- sessions
| | | `-- iqn.2005-03.org.open-iscsi:cacdcd2520
| | | |-- 10.170.75.2
| | | | |-- cid
| | | | |-- ip
| | | | `-- state
| | | |-- DataDigest
| | | |-- FirstBurstLength
| | | |-- HeaderDigest
| | | |-- ImmediateData
| | | |-- InitialR2T
| | | |-- MaxBurstLength
| | | |-- MaxOutstandingR2T
| | | |-- MaxRecvDataSegmentLength
| | | |-- MaxXmitDataSegmentLength
| | | |-- active_commands
| | | |-- commands
| | | |-- force_close
| | | |-- initiator_name
| | | |-- luns -> ../../luns
| | | |-- reinstating
| | | `-- sid
| | `-- tid
| |-- iqn.2006-10.net.vlnb:tgt1
| | |-- DataDigest
| | |-- FirstBurstLength
| | |-- HeaderDigest
| | |-- ImmediateData
| | |-- IncomingUser
| | |-- IncomingUser1
| | |-- InitialR2T
| | |-- MaxBurstLength
| | |-- MaxOutstandingR2T
| | |-- MaxRecvDataSegmentLength
| | |-- MaxXmitDataSegmentLength
| | |-- OutgoingUser
| | |-- NopInInterval
| | |-- QueuedCommands
| | |-- RspTimeout
| | |-- enabled
| | |-- ini_groups
| | | |-- mgmt
| | | `-- special_ini
| | | |-- initiators
| | | | |-- iqn.2005-03.org.open-iscsi:cacdcd2520
| | | | `-- mgmt
| | | `-- luns
| | | |-- 0
| | | | |-- device -> ../../../../../../../devices/blockio
| | | | `-- read_only
| | | `-- mgmt
| | |-- luns
| | | |-- 0
| | | | |-- device -> ../../../../../devices/disk2
| | | | `-- read_only
| | | |-- 26
| | | | |-- device -> ../../../../../devices/nullio
| | | | `-- read_only
| | | `-- mgmt
| | |-- rel_tgt_id
| | |-- sessions
| | | `-- iqn.2005-03.org.open-iscsi:cacdcd2520
| | | |-- 10.170.75.2
| | | | |-- cid
| | | | |-- ip
| | | | `-- state
| | | |-- DataDigest
| | | |-- FirstBurstLength
| | | |-- HeaderDigest
| | | |-- ImmediateData
| | | |-- InitialR2T
| | | |-- MaxBurstLength
| | | |-- MaxOutstandingR2T
| | | |-- MaxRecvDataSegmentLength
| | | |-- MaxXmitDataSegmentLength
| | | |-- active_commands
| | | |-- commands
| | | |-- force_close
| | | |-- initiator_name
| | | |-- luns -> ../../ini_groups/special_ini/luns
| | | |-- reinstating
| | | `-- sid
| | `-- tid
| |-- mgmt
| |-- open_state
| |-- trace_level
| `-- version
|-- threads
|-- trace_level
`-- version
Troubleshooting
---------------
If you have any problems, start troubleshooting from looking at the
kernel and system logs. In the kernel log iSCSI-SCST and SCST core send
their messages, in the system log iscsi-scstd sends its messages. In
most Linux distributions both those logs are put to /var/log/messages
file.
Then, it might be helpful to increase level of logging. For kernel
modules you should make the debug build by enabling CONFIG_SCST_DEBUG.
If after looking on the logs the reason of your problem is still unclear
for you, report to SCST mailing list scst-devel@lists.sourceforge.net.
Work if target's backstorage or link is too slow
------------------------------------------------
In some cases you can experience I/O stalls or see in the kernel log
abort or reset messages. It can happen under high I/O load, when your
target's backstorage gets overloaded, or working over a slow link, when
the link can't serve all the queued commands on time,
To workaround it you can reduce QueuedCommands parameter for the
corresponding target to some lower value, like 8 (default is 32).
Also see SCST README file for more details about that issue and ways to
prevent it.
Performance advices
-------------------
1. If you use Windows XP or Windows 2003+ as initiators, you should
consider to decrease TcpAckFrequency parameter to 1. See
http://support.microsoft.com/kb/328890/ or google for "TcpAckFrequency"
for more details.
2. See how to get the maximum throughput from iSCSI, for instance, at
http://virtualgeek.typepad.com/virtual_geek/2009/01/a-multivendor-post-to-help-our-mutual-iscsi-customers-using-vmware.html.
It's about VMware, but its recommendations apply to other environments
as well.
3. ISCSI initiators built in pre-CentOS/RHEL 5 reported to have some
performance problems. If you use it, it is strongly advised to upgrade.
4. If you are going to use your target in an VM environment, for
instance as a shared storage with VMware, make sure all your VMs
connected to the target via *separate* sessions, i.e. each VM has own
connection to the target, not all VMs connected using a single
connection. You can check it using SCST sysfs interface. If you
miss it, you can greatly loose performance of parallel access to your
target from different VMs. This isn't related to the case if your VMs
are using the same shared storage, like with VMFS, for instance. In this
case all your VM hosts will be connected to the target via separate
sessions, which is enough.
5. Many dual port network adapters are not able to transfer data
simultaneously on both ports, i.e. they transfer data via both ports on
the same speed as via any single port. Thus, using such adapters in MPIO
configuration can't improve performance. To allow MPIO to have double
performance you should either use separate network adapters, or find a
dual-port adapter capable to to transfer data simultaneously on both
ports. You can check it by running 2 iperf's through both ports in
parallel.
6. Since network offload works much better in the write direction, than
for reading (simplifying, in the read direction often there's additional
data copy) in many cases with 10GbE in a single initiator-target pair
the initiator's CPU is a bottleneck, so you can see the initiator can
read data on much slower rate, than write. You can check it by watching
*each particular* CPU load to find out if any of them is close to 100%
load, including IRQ processing load. Note, many tools like vmstat give
aggregate load on all CPUs, so with 4 cores 25% corresponds to 100% load
of any single CPU.
7. See SCST core's README for more advices. Especially pay attention to
have io_grouping_type option set correctly.
Compilation options
-------------------
There are the following compilation options, that could be commented
in/out in the kernel's module Makefile:
- CONFIG_SCST_DEBUG - turns on some debugging code, including some logging.
Makes the driver considerably bigger and slower, producing large amount of
log data.
- CONFIG_SCST_TRACING - turns on ability to log events. Makes the driver
considerably bigger and leads to some performance loss.
- CONFIG_SCST_EXTRACHECKS - adds extra validity checks in the various places.
- CONFIG_SCST_ISCSI_DEBUG_DIGEST_FAILURES - simulates digest failures in
random places.
Credits
-------
Thanks to:
* Ming Zhang <blackmagic02881@gmail.com> for fixes
* Krzysztof Blaszkowski <kb@sysmikro.com.pl> for many fixes
* Alexey Kuznetsov <kuznet@ms2.inr.ac.ru> for comments and help in
debugging
* Tomasz Chmielewski <mangoo@wpkg.org> for testing and suggestions
* Bart Van Assche <bart.vanassche@gmail.com> for a lot of help
Vladislav Bolkhovitin <vst@vlnb.net>, http://scst.sourceforge.net
Reference in New Issue View Git Blame Copy Permalink