mirrors/scst
1
0
Fork 0
mirror of https://github.com/SCST-project/scst.git synced 2026-08-22 15:16:33 +00:00
Code Issues Packages Projects Releases Wiki Activity

iscsi-scst: Retire obsolete integration advice

Browse Source 
The iSER and distribution guides prescribe OFED 1.5 installation,
deleting packaged RDMA modules, patching kernel builds and applying
host-wide tuning. The Gentoo procedure targets SVN, while the bundled
OCF agents are neither built nor exercised by repository CI.

Commit 025574018e ("isert: Support building against MOFED without
patching the kernel build system") established the supported
header-selection model. Current Makefile conftests and isert-scst
sources define the remaining compatibility and transport limits.

Replace fixed OFED and tuning recipes with those sources and matching
vendor guidance. Reduce the Gentoo and resource-agent documents to
historical integration notes, and spell out their module, sysfs,
daemon, session and storage side effects before any reuse.
This commit is contained in:
Gleb Chesnokov
2026-08-21 10:05:36 +03:00
parent 9a45498bc7
commit 9bdffe6fe6
4 changed files with 102 additions and 376 deletions
Download Patch File Download Diff File Expand all files Collapse all files
iscsi-scst/README.iser
+41 -106
View File
@@ -1,122 +1,57 @@
iSCSI extensions for RDMA driver
================================
Installation & Configuration:
---------------------------
For installation and configuration, see iscsi README.
There are no specific configuration options for iSER.
See below for performance optimizations as well as troubleshooting.
There is also a HOWTO on http://community.mellanox.com/docs/DOC-1479
Overview
--------
Performance considerations:
---------------------------
isert-scst provides the iSER transport for the iSCSI-SCST target. Target,
authentication, LUN, and access-control configuration uses the same interfaces
described in README. The iscsi-scst Makefile builds isert-scst when the selected
kernel build tree enables InfiniBand support.
In order to achieve better performance, it is recommended to specify
"QueuedCommands 128" parameter per iSER target, since the transport
is very fast and you usually want to connect it to fast backstorage.
For the repository build and installation workflow, see ../INSTALL.md. For the
external RDMA header selection implemented by the iscsi-scst Makefile, see
README.iser_ofed.
For performance tuning of initiator and target machines, see
http://community.mellanox.com/docs/DOC-1483
Note that if you have an SSD controller that is close to a particular
NUMA node, you want the HCA to be close to the same node.
Limitations:
Configuration
-------------
* Bidirectional commands are not supported
* Maximum number of concurent login requests that can be handled is 127 by default.
Note that there may be more connections, but only up to 127 login requests
can be handled at the same time. If you wish to increase this, load isert_scst with
module parameter isert_nr_devs set to the number of login requests you need to handle.
There is no separate iSER target type. Use allowed_portal attributes when a
target must be restricted to particular RDMA-capable addresses; see README for
the current syntax and matching rules.
Troubleshooting:
-----------------
* Initiator fails to connect to target. The following message is seen in dmesg:
Failed to accept conn request, err: -22
The cause of this is often compilation issues if you have OFED or MLNX_OFED installed:
If you are compiling for OFED/MLNX_OFED, make sure OFED is installed for
the kernel you are running. Also, make sure you followed ALL steps described
in README.iser_ofed.
If you are compiling for non-OFED kernel, make sure you don't have
OFED/MLNX_OFED installed.
The read-only isert_nr_devs module parameter controls the pool of concurrent
connection requests handled during login. The source default is 128 and values
greater than 999 are rejected. This limit does not restrict the number of
established connections.
Performance
-----------
* Discovery of iSER targets takes a long time or login to all discovered targets fails.
iSCSI discovery does not have a way to determine between iSCSI and iSER
enabled portals. Thus, initiator tries to connect to all interfaces it
discovered (by default discovery is done over iSCSI TCP).
In order to prevent this behaviour, you should specify
"allowed_portal <target interface IP>" parameter for each target you want
to export through specific RDMA capable adapters.
QueuedCommands defaults to 32 and its effective maximum is limited by the
registered kernel component. Do not assume that 128 is accepted or optimal;
measure the intended workload before changing it.
NUMA locality between backing storage, memory, CPUs, and the RDMA adapter can
affect performance. Use current kernel and adapter-vendor guidance for
system-wide RDMA, IRQ, IOMMU, and network tuning.
* Initiator keeps connecting and disconnecting from target in a loop
with constant interval after target reboot.
The problem may be that connection requests from initiator are received
on wrong port/HCA. This can be one due to one (or both) of the following issues:
1) net.ipv4.conf.all.arp_ignore sysclt is not set to 2
rdma-cm relies on ARP responses being received on the same interface
that sent the request. Linux default does not do that.
In order to make Linux behave good for rdma-cm, you _MUST_ add
"net.ipv4.conf.all.arp_ignore = 2" to /etc/sysctl.conf
2) You have more than 1 HCA and PCI mappings to netdev devices is not
persistent between reboots. Possible solution is to have udev rules
for mapping the ibX devices in persistent way.
See below for udev scripts example:
Limitations
-----------
/lib/udev/net.sh
-------------------
#!/bin/sh
. /etc/sysconfig/net.conf
type_fd="/sys/${DEVPATH}/type"
if [ ! -f $type_fd ]; then
exit
fi
type=`cat /sys/${DEVPATH}/type`
if [ "$type" = "32" ]; then # IPoIB interface
i=0
CONFDEV="DEV${i}"
CONFPCI=${!CONFDEV}
PCI=`basename $PHYSDEVPATH`
while [ -n "$CONFPCI" ]; do
if [ "$CONFPCI" = "$PCI" ]; then
devid=$(printf "%d\n" `cat /sys/$DEVPATH/dev_id`)
let id=$i*2+$devid
DEV="ib$id"
echo "$DEV"
exit
fi
let i=i+1
CONFDEV="DEV$i"
CONFPCI=${!CONFDEV}
done
fi
/etc/sysconfig/net.conf
-----------------------
DEV0="0000:01:00.0"
DEV1="0000:02:00.0"
/etc/udev/rules.d/90-network.rules
-------------------------------------
ACTION=="add", SUBSYSTEM=="net", PROGRAM="/lib/udev/net.sh", RESULT=="?*", NAME="$result"
* Login to all targets from initiator sometimes times out.
It may be a network problem (try running tools like ibdiagnet
and rping between target and initiator hosts). The description of those tools
is beyond the scope of this readme.
Another issue may be that you failed to set net.ipv4.conf.all.arp_ignore sysctl
to the value of 2 (see above problem for more detailed explanation).
* When running IO, latency is getting higher and higher all the time.
If you have enabled intel_iommu either in kernel command line or in
kernel config (it may be enabled by default), you should specify
iommu=pt on kernel command line to avoid the latency issue.
* Bidirectional SCSI commands are not supported by isert-scst.
Troubleshooting
---------------
* Confirm that the selected KDIR has InfiniBand enabled and check the build
output for the RDMA header flavor selected by the iscsi-scst Makefile.
* With an external OFED or MOFED stack, ensure that its development headers and
Module.symvers match the RDMA modules for the target kernel. See
README.iser_ofed.
* For discovery across selected interfaces, verify the target's allowed_portal
attributes and the addresses returned to the initiator.
* For connection or latency problems, first verify RDMA connectivity, routing,
source-address selection, device placement, and kernel logs. Do not apply
global arp_ignore, persistent device-naming, or iommu=pt changes from an old
deployment without validating them for the current host.
iscsi-scst/README.iser_ofed
+26 -108
View File
@@ -1,115 +1,33 @@
iSCSI Extensins for RDMA (iSER) Target driver for Linux
=======================================================
Building the iSER target against external RDMA stacks
======================================================
Introduction
------------
Status
------
The iSER target driver has been designed to work on top of the Linux
InfiniBand kernel drivers. While all recent Linux distributions
include recent versions of the InfiniBand drivers, the only way to
obtain the latest available InfiniBand drivers is by installing the
OFED or MLNX_OFED (for Mellanox drivers) software stack.
This file previously contained an OFED 1.5.1 installation recipe. That recipe
stopped RDMA services, recursively removed distribution kernel modules, and
installed a stack for a specific historical kernel. Do not use those commands
on a current system.
The OFED stack is distributed by the OpenFabrics Alliance (OFA). The
mission of the OpenFabrics Alliance is to is to develop, distribute
and promote a unified, transport-independent, open-source software
stack for RDMA-capable fabrics and networks, including InfiniBand and
Ethernet.
Current build behavior
----------------------
The MLNX_OFED is distributed by Mellanox and can be obtained from
http://www.mellanox.com/page/products_dyn?product_family=26
The iscsi-scst Makefile builds isert-scst when the selected kernel build tree
enables InfiniBand support. Set KDIR or KVER as defined by the top-level
Makefile when building for a kernel other than the running kernel.
Note: because during OFED installation the distro-provided InfiniBand
kernel drivers are replaced, doing so voids the support contract
offered by your Linux distributor.
When /usr/bin/ofed_info is absent, the Makefile builds against the RDMA headers
from the selected kernel tree. When it reports MLNX_OFED or OFED-internal, the
Makefile selects its MOFED path and requires matching installed kernel
development packages and headers under /usr/src/ofa_kernel/default. It also
recognizes the legacy RPM-provided kernel-ib-devel and compat-rdma-devel header
layouts.
Please follow the instructions below carefully. Skipping a step may
result in kernel modules that fail to load, a kernel oops or even a
system that does no longer boot.
Install external RDMA stacks and their development packages only according to
the vendor documentation for the exact distribution and kernel. The headers
and Module.symvers used for isert-scst must match the RDMA modules that will be
loaded. Never remove distribution modules from /lib/modules manually to prepare
an SCST build.
Verifying the kernel version
----------------------------
Before installing the OFED distribution, it is very important to check
the OFED release notes. Each OFED distribution has been tested
carefully, but only against the kernel versions specified in
docs/OFED_release_notes.txt (you can find this document in the OFED
distribution). Make sure that you are using a supported kernel / OFED
combination. As an example, if you want to use OFED 1.5.1 on an Ubuntu
system, you will have to start with replacing the Ubuntu kernel by a
kernel from kernel.org since OFED 1.5.1 has not been tested on any
Ubuntu kernel.
Compiling iSER against OFED
--------------------------
Make sure that all necessary packages needed for kernel compilation
have been installed (kernel headers, gcc, binutils, ...).
Unload any loaded InfiniBand drivers:
/etc/init.d/opensmd stop
/etc/init.d/openibd stop
Remove any distro-provided InfiniBand drivers:
rm -rf /lib/modules/$(uname -r)/kernel/drivers/infiniband
rm -rf /lib/modules/$(uname -r)/kernel/drivers/net/mlx4
Next, download and install an OFED package.
For MLNX_OFED, just run the mlnxofedinstall script inside the MLNX_OFED directory.
NOTE TO ADVANCED USERS:
------------------------
If you are installing MLNX_OFED by manually selecting which RPMs/DEBs to install,
make sure ofed_scripts package is one of them, since it is required for correct OFED
version detection by iscsi-scst makefile.
For the OFED package.Make sure to enable
at least the kernel-ib and kernel-ib-devel packages (compat-rdma and compat-rdma-devel for OFED 3.5 and above).
An example:
wget http://www.openfabrics.org/downloads/OFED/ofed-1.5.1/OFED-1.5.1.tgz
tar xzf OFED-1.5.1.tgz
cd OFED-1.5.1
cat <<EOF >ofed.conf
libibverbs=y
libibverbs-utils=y
libmthca=y
libmlx4=y
libcxgb3=y
libnes=y
libipathverbs=y
librdmacm=y
librdmacm-utils=y
mstflint=y
ofed-docs=y
ofed-scripts=y
kernel-ib=y
kernel-ib-devel=y
ibvexdmtools=y
qlgc_vnic_daemon=y
core=y
mthca=y
mlx4=y
mlx4_en=y
cxgb3=y
nes=y
ipath=y
ipoib=y
opensm=y
opensm-libs=y
srpt=n
srptools=y
perftest=y
EOF
./install.pl -c ofed.conf
Now continue with the installation instructions you can find in the
ISCSI-SCST README file. The Makefile included with ISCSI-SCST detects
whether OFED has been installed, and if so, compiles ISCSIS-SCST with
the OFED kernel headers instead of with the regular kernel headers.
See README.iser for iSER-specific configuration and limitations, and
../INSTALL.md for the repository build and installation workflow.
iscsi-scst/doc/SCST_Gentoo_HOWTO.txt
+15 -154
View File
@@ -1,159 +1,20 @@
==============================================
SCST 3.x Installation on Gentoo Linux
Historical SCST installation guide for Gentoo
==============================================
This howto will get you a working SCST installation on Gentoo Linux, it
will follow the official howto as much as possible with a few exceptions
because of Gentoo specifics or my preferences. Most of it will also work
on other Linux Distributions so my changes from the official howto might
help you get SCST working on another distro.
This document previously described an SCST 3.x installation on Gentoo. It is
retained only as a pointer and is not a current installation procedure. The
old SVN locations, SCST 3.0 branch, Grsecurity/PaX source workaround, GRUB 1
assumptions, and host-specific tuning examples were tied to the original
environment and must not be assumed to describe the current repository.
Prerequisites:
- A working and up2date Gentoo Linux installation
- Free storage space
- Installation of the following packages: gentoo-sources, subversion
- User with root privileges
- Read the official iSCSI-SCST howto for more options and information
Assumptions for the examples used in this howto:
- User logged on as root
- Manual build kernel without initramfs
- Gentoo AMD64 version
- Grub 1 bootloader (use of GRUB2 should be an easy change)
- /boot on a separate partition
- One iSCSI target with 2 LUNS, both LUNS virtual disk files
- One iSCSI target with one LUN, LUN is a real device
When the above options differ from your environment/setup you will need
to change some of the commands provided in this howto for things to
work.
1. Download SCST.
Choose to download a specific branch or the latest development version
a) Latest development version
cd /root
svn checkout svn://svn.code.sf.net/p/scst/svn/trunk scst
b) Specific branch (3.0 used in example)
cd /root
svn checkout svn://svn.code.sf.net/p/scst/svn/branches/3.0.x scst
2. Build SCST
! Important: SCST 3.1.x and trunk: Gentoo Hardened kernels uses
Grsec/PaX patchset. If you happen to use one of those kernels
(sys-kernel/hardened-sources) you need to change every call
"object_is_on_stack" to "object_starts_on_stack" to compile SCST
successfully. At the time of writing this there is a single such call
in scst/src/scst_lib.c
cd /root/scst
make 2perf
make scst scst_install
make iscsi iscsi_install
make scstadm scstadm_install
3. Make a clean config and start SCST
echo "HANDLER vdisk_fileio {" >> /etc/scst.conf
echo "}" >> /etc/scst.conf
echo "" >> /etc/scst.conf
echo "TARGET_DRIVER iscsi {" >> /etc/scst.conf
echo " enabled 0" >> /etc/scst.conf
echo "}" >> /etc/scst.conf
/etc/init.d/scst start
4. Configure SCST
In this example disk01 is a virtual disk that is stored in the file
/vdisks/vdisk1 and is mounted read_only, disk02 is the same as disk01
but with file /vdisks/vdisk2 and it has its blocksize increased from 512
to 4096 (see notes below), and finally disk03 is a real disk used as a
blockdevice. Both disk01 and disk02 are connected to the same target
with different luns and disk03 is connected to a different target as
LUN0, if the allowed_target atttribute is not set the target is open
to all ip's
scstadmin -open_dev disk01 -handler vdisk_fileio -attributes filename=/vdisks/vdisk1,read_only
scstadmin -open_dev disk02 -handler vdisk_fileio -attributes filename=/vdisks/vdisk2,blocksize=4096
scstadmin -open_dev disk03 -handler vdisk_blockio -attributes filename=/dev/sdc1
scstadmin -add_target iqn.2011-08.nl.feka:storage.vdisk1 -driver iscsi
scstadmin -add_target iqn.2011-08.nl.feka:storage.vdisk2 -driver iscsi
scstadmin -add_lun 0 -driver iscsi -target iqn.2011-08.nl.feka:storage.vdisk1 -device disk01
scstadmin -add_lun 1 -driver iscsi -target iqn.2011-08.nl.feka:storage.vdisk1 -device disk02
scstadmin -add_lun 0 -driver iscsi -target iqn.2011-08.nl.feka:storage.vdisk2 -device disk03
scstadmin -enable_target iqn.2011-08.nl.feka:storage.vdisk1 -driver iscsi
scstadmin -enable_target iqn.2011-08.nl.feka:storage.vdisk2 -driver iscsi
scstadmin -set_drv_attr iscsi -attributes enabled=1
scstadmin -write_config /etc/scst.conf
5. Run SCST automatically at startup
rc-update add scst default
6. Using SCST accesscontrol (optional)
SCST listens on all the targets IP addresses and allows access to any
initiators per default. There are a couple of ways to restrict access and
here I will give some examples. For the full documentation see the
README file in the scst/iscsi-scst/ directory
To make a target accessible on only one IP address do:
scstadmin -set_tgt_attr iqn.2011-08.nl.feka:storage.vdisk1 -driver iscsi -attributes allowed_portal=192.168.100.15
(Remember the IP address is the address of the target machine, not of
the allowed initiator, you can add multiple entries on the same target
by repeating the above command, you can also use the wild cards * and ?
where the ? stands for any single digit)
To make targets only accessible by specific inititators add a group to
each target and define the allowed inititiators on each group and add
the lun to the group instead of the target:
(remove the old lun if it already exists directly under the target)
scstadmin -add_group vCenter -driver iscsi -target iqn.2011-08.nl.feka:storage.vdisk2
scstadmin -add_group ESXi -driver iscsi -target iqn.2011-08.nl.feka:storage.vdisk1
scstadmin -add_init iqn.2011-08.nl.feka:vcenterserver -driver iscsi -target iqn.2011-08.nl.feka:storage.vdisk2 -group vCenter
scstadmin -add_init iqn.1998-01.com.vmware:esx1-47f53b1d -driver iscsi -target iqn.2011-08.nl.feka:storage.vdisk1 -group ESXi
scstadmin -add_init iqn.1998-01.com.vmware:esx2-23e64b45 -driver iscsi -target iqn.2011-08.nl.feka:storage.vdisk1 -group ESXi
scstadmin -rem_lun 0 -driver iscsi -target iqn.2011-08.nl.feka:storage.vdisk1 -device disk01
scstadmin -rem_lun 1 -driver iscsi -target iqn.2011-08.nl.feka:storage.vdisk1 -device disk02
scstadmin -rem_lun 0 -driver iscsi -target iqn.2011-08.nl.feka:storage.vdisk2 -device disk03
scstadmin -add_lun 0 -driver iscsi -target iqn.2011-08.nl.feka:storage.vdisk1 -group ESXi -device disk01
scstadmin -add_lun 1 -driver iscsi -target iqn.2011-08.nl.feka:storage.vdisk1 -group ESXi -device disk02
scstadmin -add_lun 0 -driver iscsi -target iqn.2011-08.nl.feka:storage.vdisk2 -group vCenter -device disk03
==============================================
Notes
==============================================
SCST's vdisk_fileio handler will use a default blocksize of 512b, this
is the only safe option for VMware ESXi or older windows versions, for
best performance with newer windows versions a blocksize of 4096 will be
much faster.
If you have an Intel CPU that supports SSE4.2 make sure to add the
crc32c-intel module to the kernel, so iSCSI-SCST will do all digest
calculations with hardware acceleration. You can find this option in the
kernel Cryptographic API -> CRC32c INTEL hardware acceleration
It is recommended to increase the read ahead size to at least 512KB on
the initiators and the target. To do this for dev/sdb and /dev/sdc on
the scst target do this:
nano -w /etc/local.d/blockdev.start
blockdev --setra 1024 /dev/sdb
blockdev --setra 1024 /dev/sdc
chmod 775 /etc/local.d/blockdev.start
(--setra 1024 means 1024x512bytes)
When running a firewall you need to open TCP Port 3260 on the required
interface, see below example for iptables.
iptables -A INPUT -p TCP -i eth0 --dport 3260 -m state --state NEW -j ACCEPT
Use ../../INSTALL.md for current prerequisites, build, installation, and
service guidance. Use ../README and iscsi-scst-howto.txt for current iSCSI
configuration and access-control guidance. Obtain source and release archives
through the locations documented in ../../README.md and
iscsi-scst-howto.txt.
Consult current Gentoo documentation for package, kernel, bootloader, firewall,
and service integration. Do not apply historical block-device, network, or live
target configuration commands without reviewing them for the exact host,
devices, and deployment.
iscsi-scst/resource_agents/README
+20 -8
View File
@@ -11,6 +11,19 @@ published by the Free Software Foundation.
If you find this program useful or if you modify it, please write me a
mail.
STATUS AND SAFETY
These files are legacy example OCF resource agents. They are not built or
exercised by this repository's CI. Review them against the current SCST sysfs
interface and the OCF implementation used by the cluster before installing
them.
The agents perform privileged live operations. They load and unload kernel
modules, start and stop iscsi-scstd, write SCST sysfs state, and force-close
sessions. SCSTTarget also clears the active SCST configuration when it starts
its own daemon. Do not run these agents without authorization for the exact
host, targets, sessions, and backing devices.
INSTALLATION
Place files SCSTTarget and SCSTLun into pacemaker ocf directory (usually
@@ -52,17 +65,16 @@ order OR_DRBD_BEFORE_ISCSI inf: MS_DRBD_VOLUME:promote GR_ISCSI:start
CURRENT LIMITATIONS
ORIGINAL TEST SCOPE
Scripts are tested only with straight drbd or lvm over drbd as backing device,
using vdisk_fileio, with one lun per target.
The original testing covered DRBD or LVM over DRBD as the backing device,
vdisk_fileio, and one LUN per target. This repository does not provide an
automated validation environment for these agents.
CHAP authentication is configurable, but not yet tested. My current version of SCST
throws an error while adding chap parameters.
The agents contain CHAP configuration paths, but these are not validated here.
GIT REPOSITORY
You can find the latest version in git repository https://github.com/rbicelli/scst-ocf.git
ORIGINAL GIT REPOSITORY
The agents were originally maintained at https://github.com/rbicelli/scst-ocf.git