scst/srpt/Testing.txt

ib_srpt Test Procedure
======================

WARNING: This is a maintainer stress and fault-injection checklist, not a
safe automated test suite. Its commands install software, reload modules and
services, modify source, create live fabric sessions, reformat devices and
delete data. Some examples describe old kernels and tools. Revalidate every
step against the current source and documentation, and run it only on an
isolated target and initiator with disposable storage and an explicit
recovery plan.

At least the following tests must be run before releasing a new SRPT version:

* Make sure that SRPT compiles and installs without triggering any
  compiler warning. Use the following command to compile and install SRPT:

  for d in scst srpt; do make -sC $d clean && make -sC $d install; done

* Verify the output of run-regression-tests for the supported kernel versions
  listed in ABT_KERNELS in ../nightly/conf/nightly.conf.

* For each distribution RDMA or OFED environment in the release scope, verify
  the header selection and compatibility conftests performed by Makefile.
  README.ofed documents OFED 1.5.1 only and is not current installation
  guidance.

* Verify that module loading and unloading works fine.

* Verify that rejecting logins does not trigger a memory leak, e.g. as follows:
  * Run the following command on the target system:
      ${SCST_TRUNK}/scripts/monitor-memory-usage | tee memlog.txt
  * Run the following command on the initiator system:
      target_id="$(/usr/sbin/ibsrpdm -c -d /dev/infiniband/umad0 | head -n1)"
      for ((i=0;i<100000;i++)); do
        echo "$target_id" >/sys/class/infiniband_srp/srp-mlx4_0-1/add_target
      done

* Verify that the following I/O stress test does not report any errors even
  when left running for several hours:

    dev=...
    umount /mnt
    mkfs.ext4 -O ^has_journal $dev
    while \
      mount $dev /mnt && \
      rm -rf /mnt/test* && \
      fio --verify=md5 -rw=randwrite --size=10m --bs=4k \
        --loops=1000000 --iodepth=64 --group_reporting --sync=1 --direct=1 \
        --ioengine=aio --directory=/mnt --name=test --thread --numjobs=80 \
	--runtime=30 && \
      fsck -N $dev
    do
      true
    done

* Repeat the above test with SCST_MAX_TGT_DEV_COMMANDS set to 48 and after
  having applied the patch below on ib_srpt.c. The expected result is that no
  data corruption occurs but that on the target error messages are logged
  about a negative req_lim value. The lowest expected value for req_lim is -15.

Index: srpt/src/ib_srpt.c
===================================================================
--- srpt/src/ib_srpt.c  (revision 2412)
+++ srpt/src/ib_srpt.c  (working copy)
@@ -2705,7 +2705,7 @@
 	ch->max_ti_iu_len = it_iu_len;
 	rsp->buf_fmt = cpu_to_be16(SRP_BUF_FORMAT_DIRECT |
 				   SRP_BUF_FORMAT_INDIRECT);
-	rsp->req_lim_delta = cpu_to_be32(ch->rq_size);
+	rsp->req_lim_delta = cpu_to_be32(ch->rq_size + 16);
 	ch->req_lim = ch->rq_size;
 	ch->req_lim_delta = 0;

* Verify that a SCSI reset works properly by running the following command
  on an initiator system (note: with kernel version 2.6.37 and before the
  command below triggers a bug in the Linux SRP initiator -- see also
  https://bugzilla.kernel.org/show_bug.cgi?id=13893):

  (target)
    echo add debug > /sys/kernel/scst_tgt/targets/ib_srpt/trace_level

  (initiator)
    sg_reset -d ${initiator_device}

  Verify that the target logged that it has processed a SRP_TSK_LUN_RESET
  message.

* Run the following command on a target system:

    while true; do                                                     \
      /etc/init.d/scst stop; sleep 3; /etc/init.d/scst start; sleep 5; \
    done

  and the following commands on an initiator system:

    target_id="$(/usr/sbin/ibsrpdm -c -d /dev/infiniband/umad0)"
    while true; do date; rmmod ib_srp; modprobe ib_srp; echo "${target_id}" \
    > /sys/class/infiniband_srp/srp-mlx4_0-1/add_target; sleep 2; done

  and verify that nothing unexpected happens.

* Log in twice from an initiator system, and verify that the first session
  receives a DREQ upon the second login:

    target_id="$(/usr/sbin/ibsrpdm -c -d /dev/infiniband/umad0 | head -n1)"
    dmesg -c
    echo "${target_id}" > /sys/class/infiniband_srp/srp-mlx4_0-1/add_target
    dmesg -c
    echo "${target_id}" > /sys/class/infiniband_srp/srp-mlx4_0-1/add_target
    dmesg -c
    echo "${target_id}" > /sys/class/infiniband_srp/srp-mlx4_0-1/add_target

* Test low memory conditions: load SRPT, reduce the amount of available
  memory by creating a large file on a tmpfs file system and run a stress test
  on an initiator system.

* Test the state machine for SCST commands in SRPT by using SCST's error
  injection mechanism. Add the following to scst/src/Kbuild, log in from
  an initiator system and trigger SRP I/O:
    ccflags-y += -DCONFIG_SCST_DEBUG -g
    ccflags-y += -DCONFIG_SCST_DEBUG_TM -DCONFIG_SCST_TM_DBG_GO_OFFLINE

* Test with multiple values of ib_srp_tablesize in the range 1..128.

* Verify that the initiator does not lock up while running the command below
  (see also http://bugzilla.kernel.org/show_bug.cgi?id=14235):

    dev=...
    fio --bs=512 --buffered=0 --ioengine=libaio --rw=read --invalidate=1 \
        --thread --numjobs=8 --loops=10 --gtod_reduce=1 --group_reporting \
        --name=$dev --filename=$dev

* Test whether queue overflow recovery works correctly as follows:
  - On the target, reload ib_srpt with srpt_sq_size set to 64. Add e.g. the
    following in /etc/modprobe.d/ib_srpt.conf:
      options ib_srpt srpt_sq_size=64
  - On the initiator, run a direct I/O test with large block sizes, e.g.
      dev=/dev/sd...
      scripts/blockdev-perftest -f -d -j -m 12 -M 24 $dev
  - On the initiator, run the following two commands in parallel:
      scripts/blockdev-perftest -r -d -j -m 12 -M 24 $dev
      fio --verify=md5 -rw=randwrite --size=10m --bs=4k \
        --loops=1000000 --iodepth=64 --group_reporting --sync=1 --direct=1 \
        --norandommap --ioengine=aio --directory=/mnt --name=test --thread \
        --numjobs=80 --runtime=30

* Test whether aborting multipart RDMA transfers works correctly as follows:
  - On the target, reload ib_srpt with srpt_sq_size set to 64.
  - On the initiator, run a direct I/O test with large block sizes, e.g. 128 KB.
  - Verify that on the target kernel messages similar to the following are
    logged frequently:
    ib_srpt: ***ERROR***: srpt_perform_rdmas[2966]: ib_post_send() returned
    -12 for 1/2
  - On the target, unload and reload the ib_srpt kernel module.
  - Verify that no kernel crash occurs on the target.
  - Repeat the above a few times.

* Test whether the code for rejecting a login after the completion thread has
  been created works fine. Do that as follows:
  - Insert "ret = -ENOMEM;" after the srpt_ch_qp_rtr() call.
  - Rebuild, reinstall and restart SCST and ib_srpt on a system running a
    debug kernel.
  - Log in a few times from another system.
  - Verify on the initiator system that each login attempt results in
    "write error: Connection reset by peer".
  - Verify on the target that no error messages have been logged in the kernel
    log, that no completion threads remain (output of "ps aux | grep srpt" must
    not list any kernel threads) and that no sessions remain
    (output of "ls /sys/kernel/scst_tgt/targets/ib_srpt/*/sessions" must be
    empty) and that the amount of free memory remains the same.

* Run this script on the target to verify that active sessions are closed
  properly when a target is disabled:
    e=1
    while true; do
      echo $e
      for f in /sys/kernel/scst_tgt/targets/ib_srpt/*/enabled; do
        echo $e > $f
      done
      e=$((1-e))
      sleep 10
    done
  While this script is running, repeatedly try to log in from another system.

* Test as follows that module removal with active sessions works fine (this
  test uses the loopback capability of an IB HCA):

  for ((i=0;i<1;i++)); do
    /etc/init.d/scst restart
    while [ -e /sys/module/ib_srp ]; do
      rmmod ib_srp >/dev/null 2>&1 || sleep 1
    done
    modprobe ib_srp
    cat /sys/kernel/scst_tgt/targets/ib_srpt/*/login_info |
      sed 's/^t//' |
      while read target_info; do
        echo "${target_info}"
        for dev in /sys/class/infiniband_srp/srp-*; do
          echo -n "${target_info}" > $dev/add_target &
        done
        wait
      done
    (cd /sys/class/srp_host && ls)
    sleep 1
  done