diff --git a/iscsi-scst/README.iser b/iscsi-scst/README.iser index 014bef310..4236d5c70 100644 --- a/iscsi-scst/README.iser +++ b/iscsi-scst/README.iser @@ -1,122 +1,57 @@ iSCSI extensions for RDMA driver ================================ -Installation & Configuration: ---------------------------- -For installation and configuration, see iscsi README. -There are no specific configuration options for iSER. -See below for performance optimizations as well as troubleshooting. -There is also a HOWTO on http://community.mellanox.com/docs/DOC-1479 +Overview +-------- -Performance considerations: ---------------------------- +isert-scst provides the iSER transport for the iSCSI-SCST target. Target, +authentication, LUN, and access-control configuration uses the same interfaces +described in README. The iscsi-scst Makefile builds isert-scst when the selected +kernel build tree enables InfiniBand support. -In order to achieve better performance, it is recommended to specify -"QueuedCommands 128" parameter per iSER target, since the transport -is very fast and you usually want to connect it to fast backstorage. +For the repository build and installation workflow, see ../INSTALL.md. For the +external RDMA header selection implemented by the iscsi-scst Makefile, see +README.iser_ofed. -For performance tuning of initiator and target machines, see -http://community.mellanox.com/docs/DOC-1483 - -Note that if you have an SSD controller that is close to a particular -NUMA node, you want the HCA to be close to the same node. - -Limitations: +Configuration ------------- -* Bidirectional commands are not supported -* Maximum number of concurent login requests that can be handled is 127 by default. - Note that there may be more connections, but only up to 127 login requests - can be handled at the same time. If you wish to increase this, load isert_scst with - module parameter isert_nr_devs set to the number of login requests you need to handle. +There is no separate iSER target type. Use allowed_portal attributes when a +target must be restricted to particular RDMA-capable addresses; see README for +the current syntax and matching rules. -Troubleshooting: ------------------ -* Initiator fails to connect to target. The following message is seen in dmesg: - Failed to accept conn request, err: -22 - The cause of this is often compilation issues if you have OFED or MLNX_OFED installed: - If you are compiling for OFED/MLNX_OFED, make sure OFED is installed for - the kernel you are running. Also, make sure you followed ALL steps described - in README.iser_ofed. - If you are compiling for non-OFED kernel, make sure you don't have - OFED/MLNX_OFED installed. +The read-only isert_nr_devs module parameter controls the pool of concurrent +connection requests handled during login. The source default is 128 and values +greater than 999 are rejected. This limit does not restrict the number of +established connections. +Performance +----------- -* Discovery of iSER targets takes a long time or login to all discovered targets fails. - iSCSI discovery does not have a way to determine between iSCSI and iSER - enabled portals. Thus, initiator tries to connect to all interfaces it - discovered (by default discovery is done over iSCSI TCP). - In order to prevent this behaviour, you should specify - "allowed_portal " parameter for each target you want - to export through specific RDMA capable adapters. +QueuedCommands defaults to 32 and its effective maximum is limited by the +registered kernel component. Do not assume that 128 is accepted or optimal; +measure the intended workload before changing it. +NUMA locality between backing storage, memory, CPUs, and the RDMA adapter can +affect performance. Use current kernel and adapter-vendor guidance for +system-wide RDMA, IRQ, IOMMU, and network tuning. -* Initiator keeps connecting and disconnecting from target in a loop - with constant interval after target reboot. - The problem may be that connection requests from initiator are received - on wrong port/HCA. This can be one due to one (or both) of the following issues: - 1) net.ipv4.conf.all.arp_ignore sysclt is not set to 2 - rdma-cm relies on ARP responses being received on the same interface - that sent the request. Linux default does not do that. - In order to make Linux behave good for rdma-cm, you _MUST_ add - "net.ipv4.conf.all.arp_ignore = 2" to /etc/sysctl.conf - 2) You have more than 1 HCA and PCI mappings to netdev devices is not - persistent between reboots. Possible solution is to have udev rules - for mapping the ibX devices in persistent way. - See below for udev scripts example: +Limitations +----------- -/lib/udev/net.sh -------------------- -#!/bin/sh - -. /etc/sysconfig/net.conf - -type_fd="/sys/${DEVPATH}/type" -if [ ! -f $type_fd ]; then - exit -fi -type=`cat /sys/${DEVPATH}/type` - -if [ "$type" = "32" ]; then # IPoIB interface - i=0 - CONFDEV="DEV${i}" - CONFPCI=${!CONFDEV} - PCI=`basename $PHYSDEVPATH` - while [ -n "$CONFPCI" ]; do - if [ "$CONFPCI" = "$PCI" ]; then - devid=$(printf "%d\n" `cat /sys/$DEVPATH/dev_id`) - let id=$i*2+$devid - DEV="ib$id" - echo "$DEV" - exit - fi - let i=i+1 - CONFDEV="DEV$i" - CONFPCI=${!CONFDEV} - done -fi - -/etc/sysconfig/net.conf ------------------------ -DEV0="0000:01:00.0" -DEV1="0000:02:00.0" - -/etc/udev/rules.d/90-network.rules -------------------------------------- -ACTION=="add", SUBSYSTEM=="net", PROGRAM="/lib/udev/net.sh", RESULT=="?*", NAME="$result" - - -* Login to all targets from initiator sometimes times out. - It may be a network problem (try running tools like ibdiagnet - and rping between target and initiator hosts). The description of those tools - is beyond the scope of this readme. - Another issue may be that you failed to set net.ipv4.conf.all.arp_ignore sysctl - to the value of 2 (see above problem for more detailed explanation). - - -* When running IO, latency is getting higher and higher all the time. - If you have enabled intel_iommu either in kernel command line or in - kernel config (it may be enabled by default), you should specify - iommu=pt on kernel command line to avoid the latency issue. +* Bidirectional SCSI commands are not supported by isert-scst. +Troubleshooting +--------------- +* Confirm that the selected KDIR has InfiniBand enabled and check the build + output for the RDMA header flavor selected by the iscsi-scst Makefile. +* With an external OFED or MOFED stack, ensure that its development headers and + Module.symvers match the RDMA modules for the target kernel. See + README.iser_ofed. +* For discovery across selected interfaces, verify the target's allowed_portal + attributes and the addresses returned to the initiator. +* For connection or latency problems, first verify RDMA connectivity, routing, + source-address selection, device placement, and kernel logs. Do not apply + global arp_ignore, persistent device-naming, or iommu=pt changes from an old + deployment without validating them for the current host. diff --git a/iscsi-scst/README.iser_ofed b/iscsi-scst/README.iser_ofed index 536ea25e2..b2654f158 100644 --- a/iscsi-scst/README.iser_ofed +++ b/iscsi-scst/README.iser_ofed @@ -1,115 +1,33 @@ -iSCSI Extensins for RDMA (iSER) Target driver for Linux -======================================================= +Building the iSER target against external RDMA stacks +====================================================== -Introduction ------------- +Status +------ -The iSER target driver has been designed to work on top of the Linux -InfiniBand kernel drivers. While all recent Linux distributions -include recent versions of the InfiniBand drivers, the only way to -obtain the latest available InfiniBand drivers is by installing the -OFED or MLNX_OFED (for Mellanox drivers) software stack. +This file previously contained an OFED 1.5.1 installation recipe. That recipe +stopped RDMA services, recursively removed distribution kernel modules, and +installed a stack for a specific historical kernel. Do not use those commands +on a current system. -The OFED stack is distributed by the OpenFabrics Alliance (OFA). The -mission of the OpenFabrics Alliance is to is to develop, distribute -and promote a unified, transport-independent, open-source software -stack for RDMA-capable fabrics and networks, including InfiniBand and -Ethernet. +Current build behavior +---------------------- -The MLNX_OFED is distributed by Mellanox and can be obtained from -http://www.mellanox.com/page/products_dyn?product_family=26 +The iscsi-scst Makefile builds isert-scst when the selected kernel build tree +enables InfiniBand support. Set KDIR or KVER as defined by the top-level +Makefile when building for a kernel other than the running kernel. -Note: because during OFED installation the distro-provided InfiniBand -kernel drivers are replaced, doing so voids the support contract -offered by your Linux distributor. +When /usr/bin/ofed_info is absent, the Makefile builds against the RDMA headers +from the selected kernel tree. When it reports MLNX_OFED or OFED-internal, the +Makefile selects its MOFED path and requires matching installed kernel +development packages and headers under /usr/src/ofa_kernel/default. It also +recognizes the legacy RPM-provided kernel-ib-devel and compat-rdma-devel header +layouts. -Please follow the instructions below carefully. Skipping a step may -result in kernel modules that fail to load, a kernel oops or even a -system that does no longer boot. +Install external RDMA stacks and their development packages only according to +the vendor documentation for the exact distribution and kernel. The headers +and Module.symvers used for isert-scst must match the RDMA modules that will be +loaded. Never remove distribution modules from /lib/modules manually to prepare +an SCST build. - -Verifying the kernel version ----------------------------- - -Before installing the OFED distribution, it is very important to check -the OFED release notes. Each OFED distribution has been tested -carefully, but only against the kernel versions specified in -docs/OFED_release_notes.txt (you can find this document in the OFED -distribution). Make sure that you are using a supported kernel / OFED -combination. As an example, if you want to use OFED 1.5.1 on an Ubuntu -system, you will have to start with replacing the Ubuntu kernel by a -kernel from kernel.org since OFED 1.5.1 has not been tested on any -Ubuntu kernel. - - -Compiling iSER against OFED --------------------------- - -Make sure that all necessary packages needed for kernel compilation -have been installed (kernel headers, gcc, binutils, ...). - -Unload any loaded InfiniBand drivers: - - /etc/init.d/opensmd stop - /etc/init.d/openibd stop - -Remove any distro-provided InfiniBand drivers: - - rm -rf /lib/modules/$(uname -r)/kernel/drivers/infiniband - rm -rf /lib/modules/$(uname -r)/kernel/drivers/net/mlx4 - -Next, download and install an OFED package. - -For MLNX_OFED, just run the mlnxofedinstall script inside the MLNX_OFED directory. - -NOTE TO ADVANCED USERS: ------------------------- -If you are installing MLNX_OFED by manually selecting which RPMs/DEBs to install, -make sure ofed_scripts package is one of them, since it is required for correct OFED -version detection by iscsi-scst makefile. - - -For the OFED package.Make sure to enable -at least the kernel-ib and kernel-ib-devel packages (compat-rdma and compat-rdma-devel for OFED 3.5 and above). -An example: - - wget http://www.openfabrics.org/downloads/OFED/ofed-1.5.1/OFED-1.5.1.tgz - tar xzf OFED-1.5.1.tgz - cd OFED-1.5.1 - cat <ofed.conf - libibverbs=y - libibverbs-utils=y - libmthca=y - libmlx4=y - libcxgb3=y - libnes=y - libipathverbs=y - librdmacm=y - librdmacm-utils=y - mstflint=y - ofed-docs=y - ofed-scripts=y - kernel-ib=y - kernel-ib-devel=y - ibvexdmtools=y - qlgc_vnic_daemon=y - core=y - mthca=y - mlx4=y - mlx4_en=y - cxgb3=y - nes=y - ipath=y - ipoib=y - opensm=y - opensm-libs=y - srpt=n - srptools=y - perftest=y - EOF - ./install.pl -c ofed.conf - -Now continue with the installation instructions you can find in the -ISCSI-SCST README file. The Makefile included with ISCSI-SCST detects -whether OFED has been installed, and if so, compiles ISCSIS-SCST with -the OFED kernel headers instead of with the regular kernel headers. +See README.iser for iSER-specific configuration and limitations, and +../INSTALL.md for the repository build and installation workflow. diff --git a/iscsi-scst/doc/SCST_Gentoo_HOWTO.txt b/iscsi-scst/doc/SCST_Gentoo_HOWTO.txt index 6b7f088ea..d9261d0ce 100644 --- a/iscsi-scst/doc/SCST_Gentoo_HOWTO.txt +++ b/iscsi-scst/doc/SCST_Gentoo_HOWTO.txt @@ -1,159 +1,20 @@ ============================================== -SCST 3.x Installation on Gentoo Linux +Historical SCST installation guide for Gentoo ============================================== -This howto will get you a working SCST installation on Gentoo Linux, it -will follow the official howto as much as possible with a few exceptions -because of Gentoo specifics or my preferences. Most of it will also work -on other Linux Distributions so my changes from the official howto might -help you get SCST working on another distro. +This document previously described an SCST 3.x installation on Gentoo. It is +retained only as a pointer and is not a current installation procedure. The +old SVN locations, SCST 3.0 branch, Grsecurity/PaX source workaround, GRUB 1 +assumptions, and host-specific tuning examples were tied to the original +environment and must not be assumed to describe the current repository. -Prerequisites: -- A working and up2date Gentoo Linux installation -- Free storage space -- Installation of the following packages: gentoo-sources, subversion -- User with root privileges -- Read the official iSCSI-SCST howto for more options and information - -Assumptions for the examples used in this howto: -- User logged on as root -- Manual build kernel without initramfs -- Gentoo AMD64 version -- Grub 1 bootloader (use of GRUB2 should be an easy change) -- /boot on a separate partition -- One iSCSI target with 2 LUNS, both LUNS virtual disk files -- One iSCSI target with one LUN, LUN is a real device - -When the above options differ from your environment/setup you will need -to change some of the commands provided in this howto for things to -work. - -1. Download SCST. - -Choose to download a specific branch or the latest development version - -a) Latest development version - cd /root - svn checkout svn://svn.code.sf.net/p/scst/svn/trunk scst - -b) Specific branch (3.0 used in example) - cd /root - svn checkout svn://svn.code.sf.net/p/scst/svn/branches/3.0.x scst - - -2. Build SCST - - ! Important: SCST 3.1.x and trunk: Gentoo Hardened kernels uses - Grsec/PaX patchset. If you happen to use one of those kernels - (sys-kernel/hardened-sources) you need to change every call - "object_is_on_stack" to "object_starts_on_stack" to compile SCST - successfully. At the time of writing this there is a single such call - in scst/src/scst_lib.c - - cd /root/scst - make 2perf - make scst scst_install - make iscsi iscsi_install - make scstadm scstadm_install - - -3. Make a clean config and start SCST - - echo "HANDLER vdisk_fileio {" >> /etc/scst.conf - echo "}" >> /etc/scst.conf - echo "" >> /etc/scst.conf - echo "TARGET_DRIVER iscsi {" >> /etc/scst.conf - echo " enabled 0" >> /etc/scst.conf - echo "}" >> /etc/scst.conf - /etc/init.d/scst start - - -4. Configure SCST - -In this example disk01 is a virtual disk that is stored in the file -/vdisks/vdisk1 and is mounted read_only, disk02 is the same as disk01 -but with file /vdisks/vdisk2 and it has its blocksize increased from 512 -to 4096 (see notes below), and finally disk03 is a real disk used as a -blockdevice. Both disk01 and disk02 are connected to the same target -with different luns and disk03 is connected to a different target as -LUN0, if the allowed_target atttribute is not set the target is open -to all ip's - - scstadmin -open_dev disk01 -handler vdisk_fileio -attributes filename=/vdisks/vdisk1,read_only - scstadmin -open_dev disk02 -handler vdisk_fileio -attributes filename=/vdisks/vdisk2,blocksize=4096 - scstadmin -open_dev disk03 -handler vdisk_blockio -attributes filename=/dev/sdc1 - scstadmin -add_target iqn.2011-08.nl.feka:storage.vdisk1 -driver iscsi - scstadmin -add_target iqn.2011-08.nl.feka:storage.vdisk2 -driver iscsi - scstadmin -add_lun 0 -driver iscsi -target iqn.2011-08.nl.feka:storage.vdisk1 -device disk01 - scstadmin -add_lun 1 -driver iscsi -target iqn.2011-08.nl.feka:storage.vdisk1 -device disk02 - scstadmin -add_lun 0 -driver iscsi -target iqn.2011-08.nl.feka:storage.vdisk2 -device disk03 - scstadmin -enable_target iqn.2011-08.nl.feka:storage.vdisk1 -driver iscsi - scstadmin -enable_target iqn.2011-08.nl.feka:storage.vdisk2 -driver iscsi - scstadmin -set_drv_attr iscsi -attributes enabled=1 - scstadmin -write_config /etc/scst.conf - - -5. Run SCST automatically at startup - - rc-update add scst default - - -6. Using SCST accesscontrol (optional) - -SCST listens on all the targets IP addresses and allows access to any -initiators per default. There are a couple of ways to restrict access and -here I will give some examples. For the full documentation see the -README file in the scst/iscsi-scst/ directory - -To make a target accessible on only one IP address do: - scstadmin -set_tgt_attr iqn.2011-08.nl.feka:storage.vdisk1 -driver iscsi -attributes allowed_portal=192.168.100.15 -(Remember the IP address is the address of the target machine, not of - the allowed initiator, you can add multiple entries on the same target - by repeating the above command, you can also use the wild cards * and ? - where the ? stands for any single digit) - -To make targets only accessible by specific inititators add a group to -each target and define the allowed inititiators on each group and add -the lun to the group instead of the target: -(remove the old lun if it already exists directly under the target) - scstadmin -add_group vCenter -driver iscsi -target iqn.2011-08.nl.feka:storage.vdisk2 - scstadmin -add_group ESXi -driver iscsi -target iqn.2011-08.nl.feka:storage.vdisk1 - scstadmin -add_init iqn.2011-08.nl.feka:vcenterserver -driver iscsi -target iqn.2011-08.nl.feka:storage.vdisk2 -group vCenter - scstadmin -add_init iqn.1998-01.com.vmware:esx1-47f53b1d -driver iscsi -target iqn.2011-08.nl.feka:storage.vdisk1 -group ESXi - scstadmin -add_init iqn.1998-01.com.vmware:esx2-23e64b45 -driver iscsi -target iqn.2011-08.nl.feka:storage.vdisk1 -group ESXi - scstadmin -rem_lun 0 -driver iscsi -target iqn.2011-08.nl.feka:storage.vdisk1 -device disk01 - scstadmin -rem_lun 1 -driver iscsi -target iqn.2011-08.nl.feka:storage.vdisk1 -device disk02 - scstadmin -rem_lun 0 -driver iscsi -target iqn.2011-08.nl.feka:storage.vdisk2 -device disk03 - scstadmin -add_lun 0 -driver iscsi -target iqn.2011-08.nl.feka:storage.vdisk1 -group ESXi -device disk01 - scstadmin -add_lun 1 -driver iscsi -target iqn.2011-08.nl.feka:storage.vdisk1 -group ESXi -device disk02 - scstadmin -add_lun 0 -driver iscsi -target iqn.2011-08.nl.feka:storage.vdisk2 -group vCenter -device disk03 - - -============================================== -Notes -============================================== - -SCST's vdisk_fileio handler will use a default blocksize of 512b, this -is the only safe option for VMware ESXi or older windows versions, for -best performance with newer windows versions a blocksize of 4096 will be -much faster. - -If you have an Intel CPU that supports SSE4.2 make sure to add the -crc32c-intel module to the kernel, so iSCSI-SCST will do all digest -calculations with hardware acceleration. You can find this option in the -kernel Cryptographic API -> CRC32c INTEL hardware acceleration - -It is recommended to increase the read ahead size to at least 512KB on -the initiators and the target. To do this for dev/sdb and /dev/sdc on -the scst target do this: - - nano -w /etc/local.d/blockdev.start - blockdev --setra 1024 /dev/sdb - blockdev --setra 1024 /dev/sdc - chmod 775 /etc/local.d/blockdev.start -(--setra 1024 means 1024x512bytes) - -When running a firewall you need to open TCP Port 3260 on the required -interface, see below example for iptables. -iptables -A INPUT -p TCP -i eth0 --dport 3260 -m state --state NEW -j ACCEPT +Use ../../INSTALL.md for current prerequisites, build, installation, and +service guidance. Use ../README and iscsi-scst-howto.txt for current iSCSI +configuration and access-control guidance. Obtain source and release archives +through the locations documented in ../../README.md and +iscsi-scst-howto.txt. +Consult current Gentoo documentation for package, kernel, bootloader, firewall, +and service integration. Do not apply historical block-device, network, or live +target configuration commands without reviewing them for the exact host, +devices, and deployment. diff --git a/iscsi-scst/resource_agents/README b/iscsi-scst/resource_agents/README index 1e7c40d72..a69b02330 100644 --- a/iscsi-scst/resource_agents/README +++ b/iscsi-scst/resource_agents/README @@ -11,6 +11,19 @@ published by the Free Software Foundation. If you find this program useful or if you modify it, please write me a mail. +STATUS AND SAFETY + +These files are legacy example OCF resource agents. They are not built or +exercised by this repository's CI. Review them against the current SCST sysfs +interface and the OCF implementation used by the cluster before installing +them. + +The agents perform privileged live operations. They load and unload kernel +modules, start and stop iscsi-scstd, write SCST sysfs state, and force-close +sessions. SCSTTarget also clears the active SCST configuration when it starts +its own daemon. Do not run these agents without authorization for the exact +host, targets, sessions, and backing devices. + INSTALLATION Place files SCSTTarget and SCSTLun into pacemaker ocf directory (usually @@ -52,17 +65,16 @@ order OR_DRBD_BEFORE_ISCSI inf: MS_DRBD_VOLUME:promote GR_ISCSI:start -CURRENT LIMITATIONS +ORIGINAL TEST SCOPE -Scripts are tested only with straight drbd or lvm over drbd as backing device, -using vdisk_fileio, with one lun per target. +The original testing covered DRBD or LVM over DRBD as the backing device, +vdisk_fileio, and one LUN per target. This repository does not provide an +automated validation environment for these agents. -CHAP authentication is configurable, but not yet tested. My current version of SCST -throws an error while adding chap parameters. +The agents contain CHAP configuration paths, but these are not validated here. -GIT REPOSITORY - -You can find the latest version in git repository https://github.com/rbicelli/scst-ocf.git +ORIGINAL GIT REPOSITORY +The agents were originally maintained at https://github.com/rbicelli/scst-ocf.git