diff --git a/doc/scst_user_spec.txt b/doc/scst_user_spec.txt index ceeb4ade8..2a1d51967 100644 --- a/doc/scst_user_spec.txt +++ b/doc/scst_user_spec.txt @@ -87,7 +87,21 @@ where: SCST_USER_REGISTER_DEVICE returns registered device's handler or -1 in case of error, and errno is set appropriately. -In order to unregister the device just call close() on its file descriptor. +In order to unregister the device, either call SCST_USER_UNREGISTER_DEVICE +function, or close on its file descriptor. + + + 2. SCST_USER_UNREGISTER_DEVICE + +SCST_USER_UNREGISTER_DEVICE - unregisters the corresponding virtual user +space device. It doesn't have any parameters. + +During execution of SCST_USER_UNREGISTER_DEVICE at least one another +thread must process all coming subcommands, otherwise after timeout it +will fail with EBUSY error. + +SCST_USER_UNREGISTER_DEVICE returns 0 on success or -1 in case of error, +and errno is set appropriately. 3. SCST_USER_SET_OPTIONS/SCST_USER_GET_OPTIONS @@ -705,7 +719,6 @@ Possible return values are: - SCST_MGMT_STATUS_FAILED - task management function failed - 5. SCST_USER_REPLY_CMD SCST_USER_REPLY_CMD IOCTL function allows the user space handler to @@ -800,6 +813,22 @@ where: - psense_buffer - pointed to sense buffer + 6. SCST_USER_FLUSH_CACHE + +SCST_USER_FLUSH_CACHE - flushes SGV cache for the corresponding virtual +user space device and queues for all cached memory buffers corresponding +SCST_USER_ON_CACHED_MEM_FREE subcommands. + +During execution of SCST_USER_FLUSH_CACHE at least one another thread +must process all coming subcommands, otherwise after timeout it will +fail with EBUSY error. + +SCST_USER_FLUSH_CACHE doesn't have any parameters. + +SCST_USER_FLUSH_CACHE returns 0 on success or -1 in case of error, +and errno is set appropriately. + + IV. Commands processing flow example. As the example consider a simple synchronous VTL, which serves one diff --git a/iscsi-scst/README b/iscsi-scst/README index 4641a01b6..cd785a060 100644 --- a/iscsi-scst/README +++ b/iscsi-scst/README @@ -140,6 +140,24 @@ CAUTION: Working of target and initiator on the same host isn't fully ======= supported. See SCST README file for details. +Troubleshooting +--------------- + +If you have any problems, start troubleshooting from looking at the +kernel and system logs. In the kernel log iSCSI-SCST and SCST core send +their messages, in the system log iscsi-scstd sends its messages. In +most Linux distributions both those logs are put to /var/log/messages +file. + +Then, it might be helpful to increase level of logging. For kernel +modules you should make the debug build, by either running "make +release2debug" if you work with SCST SVN tree, or by enable the +corresponding debug symbols (see below). + +If after looking on the logs the reason of your problem is still unclear +for you, report to SCST mailing list scst-devel@lists.sourceforge.net. + + Work if target's backstorage or link is too slow ------------------------------------------------ diff --git a/iscsi-scst/include/iscsi_scst_ver.h b/iscsi-scst/include/iscsi_scst_ver.h index f9e395a60..30f0b68b2 100644 --- a/iscsi-scst/include/iscsi_scst_ver.h +++ b/iscsi-scst/include/iscsi_scst_ver.h @@ -13,4 +13,4 @@ * GNU General Public License for more details. */ -#define ISCSI_VERSION_STRING "1.0.1/0.4.17r204" +#define ISCSI_VERSION_STRING "1.0.1/0.4.17r206" diff --git a/iscsi-scst/usr/iscsi_adm.c b/iscsi-scst/usr/iscsi_adm.c index 68157fce1..e63b68021 100644 --- a/iscsi-scst/usr/iscsi_adm.c +++ b/iscsi-scst/usr/iscsi_adm.c @@ -155,7 +155,7 @@ static int iscsid_request_send(int fd, struct iscsi_adm_req *req) } static int iscsid_response_recv(int fd, struct iscsi_adm_req *req, void *rsp_data, - size_t rsp_data_sz) + int rsp_data_sz) { int err, ret; struct iovec iov[2]; @@ -173,7 +173,7 @@ static int iscsid_response_recv(int fd, struct iscsi_adm_req *req, void *rsp_dat if (ret != sizeof(rsp) + sizeof(*req)) { err = (ret < 0) ? -errno : -EIO; fprintf(stderr, "readv failed: read %d instead of %d (%s)\n", - ret, sizeof(rsp) + sizeof(*req), strerror(err)); + ret, (int)(sizeof(rsp) + sizeof(*req)), strerror(err)); } else err = rsp.err; @@ -182,7 +182,7 @@ static int iscsid_response_recv(int fd, struct iscsi_adm_req *req, void *rsp_dat if (ret != rsp_data_sz) { err = (ret < 0) ? -errno : -EIO; fprintf(stderr, "read failed: read %d instead of %d (%s)\n", - ret, rsp_data_sz, strerror(err)); + ret, (int)rsp_data_sz, strerror(err)); } } @@ -217,7 +217,7 @@ out: } static int iscsid_request(struct iscsi_adm_req *req, void *rsp_data, - size_t rsp_data_sz) + int rsp_data_sz) { int fd = -1, err = -EIO; diff --git a/qla_isp/README.scst b/qla_isp/README.scst index 4e7358d2a..9ae7c02b9 100644 --- a/qla_isp/README.scst +++ b/qla_isp/README.scst @@ -156,3 +156,6 @@ When all LUNs and channels are disabled, the target role will also become inactive. For N_PORT ID virtualization, when enabling channels, you need to first enable channel 0; when disabling channels, you need to disable channel 0 last. + +Send question about this driver to scst-devel@lists.sourceforge.net, CC +Matthew Jacob and Stanislaw Gruszka . diff --git a/scst/README b/scst/README index c0d6383ff..3b0106817 100644 --- a/scst/README +++ b/scst/README @@ -865,6 +865,20 @@ IMPORTANT: If you use on initiator some versions of Windows (at least W2K) for VDISK FILEIO devices above. +In some cases, for instance working with SSD devices, which consume 100% +of a single CPU load for data transfers in their internal threads, to +maximize IOPS it can be needed to assign for those threads dedicated +CPUs using Linux CPU affinity facilities. No IRQ processing should be +done on those CPUs. Check that using /proc/interrupts. See taskset +command and Documentation/IRQ-affinity.txt in your kernel's source tree +for how to assign IRQ affinity to tasks and IRQs. + +The reason for that is that processing of coming commands in SIRQ +context might be done on the same CPUs as SSD devices' threads doing data +transfers. As the result, those threads won't receive all the processing +power of those CPUs and perform worse. + + Work if target's backstorage or link is too slow ------------------------------------------------ diff --git a/scst/README_in-tree b/scst/README_in-tree index 54faf04db..015b003ac 100644 --- a/scst/README_in-tree +++ b/scst/README_in-tree @@ -799,6 +799,19 @@ IMPORTANT: If you use on initiator some versions of Windows (at least W2K) See also important notes about setting block sizes >512 bytes for VDISK FILEIO devices above. +In some cases, for instance working with SSD devices, which consume 100% +of a single CPU load for data transfers in their internal threads, to +maximize IOPS it can be needed to assign for those threads dedicated +CPUs using Linux CPU affinity facilities. No IRQ processing should be +done on those CPUs. Check that using /proc/interrupts. See taskset +command and Documentation/IRQ-affinity.txt in your kernel's source tree +for how to assign IRQ affinity to tasks and IRQs. + +The reason for that is that processing of coming commands in SIRQ +context might be done on the same CPUs as SSD devices' threads doing data +transfers. As the result, those threads won't receive all the processing +power of those CPUs and perform worse. + Work if target's backstorage or link is too slow ------------------------------------------------ diff --git a/srpt/README b/srpt/README index c966f3d38..59c52df85 100644 --- a/srpt/README +++ b/srpt/README @@ -37,6 +37,7 @@ with the Linux kernel source code, enable and build the SRP target driver in the same way as any other kernel module. Make sure to enable SCST itself and InfiniBand before attempting to enable the SRP target driver. + How-to run ----------- A. On srp target machine @@ -84,7 +85,7 @@ h. echo "add vdisk2 2" >/proc/scsi_tgt/groups/Default/devices 2. modprobe ib_srpt -B. On initiator machines you can manualy do the following steps: +B. On initiator machines you can manually do the following steps: 1. modprobe ib_srp 2. ipsrpdm -c (to discover new SRP target) 3. echo > /sys/class/infiniband_srp/srp-mthca0-1/add_target @@ -108,3 +109,26 @@ automatically ie. set SRP_LOAD=yes, and SRPHA_ENABLE=yes and multipath tool + Please refer to OFED-1.x SRP's user manual for more in-details instructions on how-to enable/use HA feature + + +Performance notes +----------------- + +In some cases, for instance working with SSD devices, which consume 100% +of a single CPU load for data transfers in their internal threads, to +maximize IOPS it can be needed to assign for those threads dedicated +CPUs using Linux CPU affinity facilities. No IRQ processing should be +done on those CPUs. Check that using /proc/interrupts. See taskset +command and Documentation/IRQ-affinity.txt in your kernel's source tree +for how to assign IRQ affinity to tasks and IRQs. + +The reason for that is that processing of coming commands in SIRQ context +might be done on the same CPUs as SSD devices' threads doing data +transfers. As the result, those threads won't receive all the CPU power +and perform worse. + +Alternatively to CPU affinity assignment, you can try to enable SRP +target's internal thread. It will allows Linux CPU scheduler to better +distribute load among available CPUs. To enable SRP target driver's +internal thread you should load ib_srpt module with parameter +"thread=1". diff --git a/srpt/README_in-tree b/srpt/README_in-tree index 327ed0907..60205971d 100644 --- a/srpt/README_in-tree +++ b/srpt/README_in-tree @@ -83,3 +83,26 @@ on how-to enable/use HA feature To minimize QUEUEFULL conditions, you can apply scst_increase_max_tgt_cmds patch from SRPT package from http://sourceforge.net/project/showfiles.php?group_id=110471 + + +Performance notes +----------------- + +In some cases, for instance working with SSD devices, which consume 100% +of a single CPU load for data transfers in their internal threads, to +maximize IOPS it can be needed to assign for those threads dedicated +CPUs using Linux CPU affinity facilities. No IRQ processing should be +done on those CPUs. Check that using /proc/interrupts. See taskset +command and Documentation/IRQ-affinity.txt in your kernel's source tree +for how to assign IRQ affinity to tasks and IRQs. + +The reason for that is that processing of coming commands in SIRQ context +might be done on the same CPUs as SSD devices' threads doing data +transfers. As the result, those threads won't receive all the CPU power +and perform worse. + +Alternatively to CPU affinity assignment, you can try to enable SRP +target's internal thread. It will allows Linux CPU scheduler to better +distribute load among available CPUs. To enable SRP target driver's +internal thread you should load ib_srpt module with parameter +"thread=1". diff --git a/www/index.html b/www/index.html index 3bd3d7f22..8c9c52836 100644 --- a/www/index.html +++ b/www/index.html @@ -30,17 +30,20 @@

Generic SCSI Target Middle Level for Linux

-

The SCSI target mid-level subsystem for Linux (SCST) is a subsystem of the Linux kernel - that provides a standard framework for SCSI target drivers development. - It is designed to provide unified, consistent interface between SCSI target drivers and Linux kernel - and simplify target drivers development as much as possible.

-

A system with a SCSI target device is able to share its local or virtual devices with other systems - on a network with SCSI protocol support, e.g. SCSI bus, SAS, Fibre Channel or iSCSI. - This is commonly used for data storage virtualization.

+

The SCSI target mid-level subsystem for Linux (SCST) is an alternative + implementation of SCSI target subsystem for Linux. It provides unified, + consistent interface between SCSI target drivers and + Linux kernel and simplifies target drivers development. + +

SCST allows creation of sophisticated storage devices, which provide advanced + functionality, like replication, thin provisioning, + deduplication, high availability, automatic backup, etc. + Another class of such devices are Virtual Tape Libraries (VTL) + as well as other disk-based backup solutions.

-

This project consists from a set of subprojects: SCST core itself - as well as target drivers and user space utilities. - They are developed more or less independently and have own maintainers.

+

SCST project consists from a set of subprojects: SCST core itself + with a set of device handlers as well as target + drivers and user space utilities.

Features of SCST Core

    @@ -49,11 +52,12 @@ necessary error recovery.
  • Undertakes most problems, related to execution contexts, thus practically eliminating one of the most complicated problem in the kernel drivers development. For example, a target driver for QLogic - 22xx/23xx cards, which has all necessary features, is only about 2000 lines of code long.
    • + 22xx/23xx cards, which has all necessary features, is only about 2000 lines of code long. + The same is true for InfiniBand SRP target driver.
  • Very low overhead, fine-grained locks and simplest commands processing path, which allow to reach maximum possible performance and scalability. Particularly, incoming requests can be processed in - the caller's context or in one of the internal SCST core's tasklets, therefore no extra context switches - required.
    • + the caller's context or in one of the internal SCST core's tasklets without any + extra context switches.
  • Device handlers, i.e. plugins, architecture provides extra flexibility by allowing various I/O modes in backstorage handling. For example, pass-through device handlers allows to use real SCSI hardware and vdisk device handler allows to use files as virtual disks.
    • diff --git a/www/scstvsstgt.html b/www/scstvsstgt.html index 8d4239e1d..1a4ea9ba9 100644 --- a/www/scstvsstgt.html +++ b/www/scstvsstgt.html @@ -35,28 +35,43 @@

    SCST vs STGT

    STGT is alternative, independent from SCST implementation of SCSI target framework for Linux. It has different architecture, where SCSI target state machine is placed in - the user space, while in SCST all the processing done in the kernel. Such architecture was acknowledged as a - "right" one by the Linux SCSI subsystem maintainers, so kernel's part of STGT quickly found its way to the kernel. - But such architecture has two inherent problems, namely performance and complexity. - See thread - "Relationship between SCST and in-kernel SCSI_TGT" and - this message in thread "Open-FCoE on linux-scsi" for more details.

    + the user space, while in SCST all the processing done in the kernel. Such architecture as STGT has was acknowledged + at the moment by the Linux SCSI subsystem maintainers as a "right" one, so kernel's part of STGT quickly + found its way to the kernel.

    + +

    But such architecture has several inherent problems. Among them performance and complexity. + See description for the set of patches, submitted for the first iteration of in-kernel inclusion review and comments + http://lkml.org/lkml/2008/12/10/245

    + +

    See also the following important discussions: +

      + +
    • + "Performance of SCST versus STGT" for performance comparison.
      • + +
    • + "Question for pass-through target design"
      • -

      See also the following important discussions:

      -

      Also you shouldn't be deceived by the fact that some (small) part of STGT was accepted in the kernel.
      + +

      Also you shouldn't be deceived by the fact that some (small) part of STGT was accepted into the kernel.
      It doesn't mean that STGT has the "kernel quality". In fact, STGT as a whole similarly to any other out-of-tree project lives on its own, hence has its own quality level, which isn't necessary better, than the - quality level of SCST. Actually, from such important aspect of quality as simplicity, it + quality level of SCST. Actually, from such important aspect of quality as simplicity, it might be quite contrary: e.g. SCST isn't required to support HIGHMEM (nowadays it isn't necessary, but required for all in-kernel components), which allowed to simplify memory management a lot.

      diff --git a/www/sponsorship.html b/www/sponsorship.html index a66c8bbd0..f17472406 100644 --- a/www/sponsorship.html +++ b/www/sponsorship.html @@ -34,30 +34,39 @@ sufficiently good to satisfy the real life requirements, and there is a growing demand in a such subsystem. At the same time, SCST has reached mature and stable state and capable to satisfy - all the requirements. - Also is has the widest choice of supported SCSI transports + all the requirements. See + SCST vs STGT page for more details.

      + +

      Also is has the widest choice of supported SCSI transports (iSCSI, Fibre Channel, Infiniband SRP, parallel SCSI, SAS), the best performance and the biggest features set.

      So, the next logical step should be to include SCST in the - mainline kernel. - There is a lot of work to do before it can be achieved in areas of code - preparation, submission, going through review circles, etc. + mainline kernel. The first iteration of in-kernel inclusion review and comments + (see http://lkml.org/lkml/2008/12/10/245) + was quite positive.

      + +

      But still there is a lot of work to do before all the necessary work in areas of code + preparation, submission, going through review circles, etc. is finished. Activities in this direction have been carrying out, but far slower than necessary, because they are mostly done in a spare time.

      Thus, you can speed up a LOT all the works related to SCST inclusion in the mainline kernel by sponsoring it. The needed financial amount is quite low, just for one person for several months. - In this time all current showstoppers, like /proc based interface, as well as - all problems found during future public reviews should be fixed. + In this time all the current showstoppers, like /proc based interface, as well as + all problems found during past and future reviews should be fixed. Even half of this amount would allow to work half full time exclusively on - SCST integration into Linux kernel. Funds can be on one time or per month basis.

      + SCST integration into Linux kernel. Donations can be on one time or per period of time basis, + from companies or individuals. They can be sent by direct bank transfer (preferred) or using + Western Union, Moneygram or similar payment system. Sorry, more convenient options, like PayPal, are + not available in Russia.

      Investing in SCST integration in Linux kernel, you investing not only in making Linux the best storage OS, but, if your company - has a product, based on SCST, you also investing in the secured - future of your product and, hence, your company.

      + has a product or service, based on SCST, you also investing in the secured + future of your product or service and, hence, in the the secured + future of your company.

      If you are interested, please contact Vladislav Bolkhovitin <vst at vlnb net>

diff --git a/www/target_iscsi.html b/www/target_iscsi.html index b3d9d6abb..67880e970 100644 --- a/www/target_iscsi.html +++ b/www/target_iscsi.html @@ -59,7 +59,7 @@

ISCSI-SCST has the following major advantages over the mainline IET:

    -
  • It uses full power of SCST core without loosing any existing feature (except, maybe, illegal +
  • It uses full power of SCST core without loosing any existing IET feature (except, maybe, illegal from SCSI specifications point of view MPIO). Namely, you can additionally use with it:
    • Pass-through mode with one to many relationship, i.e. when multiple initiators can connect to the @@ -83,29 +83,31 @@
  • It has many code improvements and cleanups, including stability and iSCSI RFC violations fixes. - Many IET users use it for ages without problems, so they consider it problemless. But, + Many IET users use it for ages without problems, so they consider it free from any real problem. But, in fact, unfortunately, it isn't so. - IET works well only on "fast" paths and regularly used branches, in many other less used + IET works well only on "fast" paths and regularly used code branches. In many other less used cases IET has various problems, from simply ignoring error processing, as it is with memory allocations, and crashing itself with BUG() macro, as it is for malformed packets from initiators, to possible data corruption. See, for instance, this thread on a VMware forum about in which Russian roulette IET users play using it with VMware. ChangeLog file lists most noticeable fixes, but there were a lot of many other smaller ones.
    • -
  • Due to reworked I/O architecture and SCST backend iSCSI-SCST has - better performance in many cases. In future with upcoming improvements in SCST core, like zero-copy - with Linux cache FILEIO, the performance difference is going to be even bigger. Currently in - tests from a single initiator over a single connection on 1GbE hardware over FILEIO vdisk - iSCSI-SCST with default settings usually outperforms tuned for best - performance IET a on 3-30%. The difference is especially noticeably with real storage, + +
  • Due to reworked I/O architecture and SCST backend iSCSI-SCST has + better performance in many cases. In future with upcoming improvements in SCST core, like zero-copy + with Linux cache FILEIO, the performance difference is going to be even bigger. Currently in + tests from a single initiator over a single connection on 1GbE hardware over FILEIO vdisk + iSCSI-SCST with default settings usually outperforms tuned for best + performance IET a on on up to 100%. The difference is especially noticeably with real storage, not NULLIO or RAM disks. On 10GbE hardware the performance difference is often as high as 100-200% or even more. With higher number of initiators the difference will be even bigger. This is because - iSCSI-SCST has less commands processing overhead per command, hence has smaller processing + iSCSI-SCST has less commands processing overhead per command, hence has smaller processing latency and puts less load on CPU.

-

Also, in contrast to IET, iSCSI-SCST is open for any new development, modifications and + +

Also, in contrast to IET, iSCSI-SCST is open for any new development, modifications and improvements, so people who want to fix or implement something new will not have to keep and maintain separate - patches as it is currently necessary with IET. ISCSI-SCST is actively + patches as it is currently necessary with IET. ISCSI-SCST is actively developed and going to gain in the future new features like support for multiple connections per session, error recovery levels >0, etc., which eventually make it really "Enterprise".