mirror of
https://github.com/SCST-project/scst.git
synced 2026-09-26 09:54:17 +00:00
Gleb Chesnokov
b836a17e35
The SRPT guides mix current RDMA/CM code with Linux 2.6, procfs and OFED 1.5 procedures. Their parameter defaults, session names and tuning advice no longer describe the driver built by this tree. Commit020ec68ddc("ib_srpt: Change default behavior from using SRQ to not using SRQ") changed receive queues, and commit434207ab53("ib_srpt: Restore immediate data support") changed request sizing. Commitfa58faf95b("ib_srpt: Add RDMA/CM support") added the IP-based session naming mode. Commit08e532715d("Remove procfs support") invalidated the in-tree setup, while commitd6fd03107f("Split Makefiles: move kernel build rules into Kbuild files") moved the fault-injection flag. Commitc2bd2e8bf5("ib_srpt: Remove "thread" kernel module parameter and keep thread=1 behavior - the other two modes could cause soft lockup complaints") made threaded completion unconditional. Use ib_srpt.c, ib_srpt.h and conftests for current behavior. Scope SCSI multiqueue advice after Linux commit f664a3cc17b7 ("scsi: kill off the legacy IO path"), and mark OFED, HA, benchmark and test recipes as historical or destructive.
119 lines
5.0 KiB
Plaintext
119 lines
5.0 KiB
Plaintext
SCSI RDMA Protocol (SRP) Target driver for Linux
|
|
=================================================
|
|
|
|
Historical procfs procedure
|
|
---------------------------
|
|
|
|
The procedure below predates removal of the SCST procfs control interface.
|
|
Its /proc/scsi_tgt commands cannot configure current SCST and its examples
|
|
would expose or access live storage. Do not follow them. Use ../INSTALL.md
|
|
and README for current build and SRPT guidance, and use the scstadmin
|
|
documentation for current sysfs-based target configuration.
|
|
|
|
The SRP Target driver is designed to work directly on top of the
|
|
OpenFabrics OFED-1.x software stack (http://www.openfabrics.org) or
|
|
the Infiniband drivers in the Linux kernel tree
|
|
(http://www.kernel.org). The SRP target driver also interfaces with
|
|
the generic SCSI target mid-level driver called SCST
|
|
(http://scst.sourceforge.net).
|
|
|
|
How-to run
|
|
-----------
|
|
|
|
A. On srp target machine
|
|
1. Please refer to SCST's README for loading scst driver and its
|
|
dev_handlers drivers (scst_disk, scst_vdisk block or file IO mode, nullio, ...)
|
|
|
|
Example 1: working with real back-end scsi disks
|
|
a. modprobe scst
|
|
b. modprobe scst_disk
|
|
c. cat /proc/scsi_tgt/scsi_tgt
|
|
|
|
ibstor00:~ # cat /proc/scsi_tgt/scsi_tgt
|
|
Device (host:ch:id:lun or name) Device handler
|
|
0:0:0:0 dev_disk
|
|
4:0:0:0 dev_disk
|
|
5:0:0:0 dev_disk
|
|
6:0:0:0 dev_disk
|
|
7:0:0:0 dev_disk
|
|
|
|
Now you want to exclude the first scsi disk and expose the last 4 scsi disks as
|
|
IB/SRP luns for I/O
|
|
echo "add 4:0:0:0 0" >/proc/scsi_tgt/groups/Default/devices
|
|
echo "add 5:0:0:0 1" >/proc/scsi_tgt/groups/Default/devices
|
|
echo "add 6:0:0:0 2" >/proc/scsi_tgt/groups/Default/devices
|
|
echo "add 7:0:0:0 3" >/proc/scsi_tgt/groups/Default/devices
|
|
|
|
Example 2: working with VDISK FILEIO mode (using md0 device and file 10G-file)
|
|
a. modprobe scst
|
|
b. modprobe scst_vdisk
|
|
c. echo "open vdisk0 /dev/md0" > /proc/scsi_tgt/vdisk/vdisk
|
|
d. echo "open vdisk1 /10G-file" > /proc/scsi_tgt/vdisk/vdisk
|
|
e. echo "add vdisk0 0" >/proc/scsi_tgt/groups/Default/devices
|
|
f. echo "add vdisk1 1" >/proc/scsi_tgt/groups/Default/devices
|
|
|
|
Example 3: working with VDISK BLOCKIO mode (using md0 device, sda, and cciss/c1d0)
|
|
a. modprobe scst
|
|
b. modprobe scst_vdisk
|
|
c. echo "open vdisk0 /dev/md0 BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk
|
|
d. echo "open vdisk1 /dev/sda BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk
|
|
e. echo "open vdisk2 /dev/cciss/c1d0 BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk
|
|
f. echo "add vdisk0 0" >/proc/scsi_tgt/groups/Default/devices
|
|
g. echo "add vdisk1 1" >/proc/scsi_tgt/groups/Default/devices
|
|
h. echo "add vdisk2 2" >/proc/scsi_tgt/groups/Default/devices
|
|
|
|
2. modprobe ib_srpt
|
|
|
|
|
|
B. On initiator machines you can manually do the following steps:
|
|
1. modprobe ib_srp
|
|
2. ibsrpdm -c (to discover new SRP target)
|
|
3. echo <new target info> > /sys/class/infiniband_srp/srp-mthca0-1/add_target
|
|
4. fdisk -l (will show new discovered scsi disks)
|
|
|
|
Example:
|
|
Assume that you use port 1 of first HCA in the system ie. mthca0
|
|
|
|
[root@lab104 ~]# ibsrpdm -c -d /dev/infiniband/umad0
|
|
id_ext=0002c90200226cf4,ioc_guid=0002c90200226cf4,
|
|
dgid=fe800000000000000002c90200226cf5,pkey=ffff,service_id=0002c90200226cf4
|
|
[root@lab104 ~]# echo id_ext=0002c90200226cf4,ioc_guid=0002c90200226cf4,
|
|
dgid=fe800000000000000002c90200226cf5,pkey=ffff,service_id=0002c90200226cf4 >
|
|
/sys/class/infiniband_srp/srp-mthca0-1/add_target
|
|
|
|
OR
|
|
|
|
+ You can edit /etc/infiniband/openib.conf to load srp driver and srp HA daemon
|
|
automatically ie. set SRP_LOAD=yes, and SRPHA_ENABLE=yes
|
|
+ To set up and use high availability feature you need dm-multipath driver
|
|
and multipath tool
|
|
+ Please refer to OFED-1.x SRP's user manual for more in-details instructions
|
|
on how-to enable/use HA feature
|
|
|
|
To minimize QUEUE_FULL conditions, you can apply scst_increase_max_tgt_cmds
|
|
patch from SRPT package from http://sourceforge.net/project/showfiles.php?group_id=110471
|
|
|
|
|
|
Performance notes
|
|
-----------------
|
|
|
|
In some cases, for instance working with SSD devices, which consume 100%
|
|
of a single CPU load for data transfers in their internal threads, to
|
|
maximize IOPS it can be needed to assign for those threads dedicated
|
|
CPUs using Linux CPU affinity facilities. No IRQ processing should be
|
|
done on those CPUs. Check that using /proc/interrupts. See taskset
|
|
command and Documentation/core-api/irq/irq-affinity.rst in your kernel's
|
|
source tree for how to assign CPU affinity to tasks and IRQs.
|
|
|
|
The reason for that is that processing of coming commands in SIRQ context
|
|
can be done on the same CPUs as SSD devices' threads doing data
|
|
transfers. As the result, those threads won't receive all the CPU power
|
|
and perform worse.
|
|
|
|
The ib_srpt thread parameter was removed when its former thread=1 mode
|
|
became unconditional. No module option is needed to select that mode.
|
|
|
|
|
|
Send questions about this driver to scst-devel@lists.sourceforge.net, CC:
|
|
Vu Pham <vuhuong@mellanox.com> and Bart Van Assche <bvanassche@acm.org>.
|