mirrors/scst
1
0
Fork 0
mirror of https://github.com/SCST-project/scst.git synced 2026-08-17 12:46:27 +00:00
Code Issues Packages Projects Releases Wiki Activity

- Docs updates

Browse Source 
- Minor cleanups



git-svn-id: http://svn.code.sf.net/p/scst/svn/trunk@675 d57e44dd-8a1f-0410-8b47-8ef2f437770f
This commit is contained in:
Vladislav Bolkhovitin
2009-02-18 18:52:35 +00:00
parent fe1a79f012
commit e8b67bb755
13 changed files with 210 additions and 56 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/scst_user_spec.txt
+31 -2
View File
@@ -87,7 +87,21 @@ where:
SCST_USER_REGISTER_DEVICE returns registered device's handler or -1
in case of error, and errno is set appropriately.
In order to unregister the device just call close() on its file descriptor.
In order to unregister the device, either call SCST_USER_UNREGISTER_DEVICE
function, or close on its file descriptor.
2. SCST_USER_UNREGISTER_DEVICE
SCST_USER_UNREGISTER_DEVICE - unregisters the corresponding virtual user
space device. It doesn't have any parameters.
During execution of SCST_USER_UNREGISTER_DEVICE at least one another
thread must process all coming subcommands, otherwise after timeout it
will fail with EBUSY error.
SCST_USER_UNREGISTER_DEVICE returns 0 on success or -1 in case of error,
and errno is set appropriately.
3. SCST_USER_SET_OPTIONS/SCST_USER_GET_OPTIONS
@@ -705,7 +719,6 @@ Possible return values are:
- SCST_MGMT_STATUS_FAILED - task management function failed
5. SCST_USER_REPLY_CMD
SCST_USER_REPLY_CMD IOCTL function allows the user space handler to
@@ -800,6 +813,22 @@ where:
- psense_buffer - pointed to sense buffer
6. SCST_USER_FLUSH_CACHE
SCST_USER_FLUSH_CACHE - flushes SGV cache for the corresponding virtual
user space device and queues for all cached memory buffers corresponding
SCST_USER_ON_CACHED_MEM_FREE subcommands.
During execution of SCST_USER_FLUSH_CACHE at least one another thread
must process all coming subcommands, otherwise after timeout it will
fail with EBUSY error.
SCST_USER_FLUSH_CACHE doesn't have any parameters.
SCST_USER_FLUSH_CACHE returns 0 on success or -1 in case of error,
and errno is set appropriately.
IV. Commands processing flow example.
As the example consider a simple synchronous VTL, which serves one
iscsi-scst/README
+18
View File
@@ -140,6 +140,24 @@ CAUTION: Working of target and initiator on the same host isn't fully
======= supported. See SCST README file for details.
Troubleshooting
---------------
If you have any problems, start troubleshooting from looking at the
kernel and system logs. In the kernel log iSCSI-SCST and SCST core send
their messages, in the system log iscsi-scstd sends its messages. In
most Linux distributions both those logs are put to /var/log/messages
file.
Then, it might be helpful to increase level of logging. For kernel
modules you should make the debug build, by either running "make
release2debug" if you work with SCST SVN tree, or by enable the
corresponding debug symbols (see below).
If after looking on the logs the reason of your problem is still unclear
for you, report to SCST mailing list scst-devel@lists.sourceforge.net.
Work if target's backstorage or link is too slow
------------------------------------------------
iscsi-scst/include/iscsi_scst_ver.h
+1 -1
View File
@@ -13,4 +13,4 @@
* GNU General Public License for more details.
*/
#define ISCSI_VERSION_STRING "1.0.1/0.4.17r204"
#define ISCSI_VERSION_STRING "1.0.1/0.4.17r206"
iscsi-scst/usr/iscsi_adm.c
+4 -4
View File
@@ -155,7 +155,7 @@ static int iscsid_request_send(int fd, struct iscsi_adm_req *req)
}
static int iscsid_response_recv(int fd, struct iscsi_adm_req *req, void *rsp_data,
size_t rsp_data_sz)
int rsp_data_sz)
{
int err, ret;
struct iovec iov[2];
@@ -173,7 +173,7 @@ static int iscsid_response_recv(int fd, struct iscsi_adm_req *req, void *rsp_dat
if (ret != sizeof(rsp) + sizeof(*req)) {
err = (ret < 0) ? -errno : -EIO;
fprintf(stderr, "readv failed: read %d instead of %d (%s)\n",
ret, sizeof(rsp) + sizeof(*req), strerror(err));
ret, (int)(sizeof(rsp) + sizeof(*req)), strerror(err));
} else
err = rsp.err;
@@ -182,7 +182,7 @@ static int iscsid_response_recv(int fd, struct iscsi_adm_req *req, void *rsp_dat
if (ret != rsp_data_sz) {
err = (ret < 0) ? -errno : -EIO;
fprintf(stderr, "read failed: read %d instead of %d (%s)\n",
ret, rsp_data_sz, strerror(err));
ret, (int)rsp_data_sz, strerror(err));
}
}
@@ -217,7 +217,7 @@ out:
}
static int iscsid_request(struct iscsi_adm_req *req, void *rsp_data,
size_t rsp_data_sz)
int rsp_data_sz)
{
int fd = -1, err = -EIO;
qla_isp/README.scst
+3
View File
@@ -156,3 +156,6 @@ When all LUNs and channels are disabled, the target role will also become
inactive. For N_PORT ID virtualization, when enabling channels, you need to
first enable channel 0; when disabling channels, you need to disable channel
0 last.
Send question about this driver to scst-devel@lists.sourceforge.net, CC
Matthew Jacob <lydianconcepts@gmail.com> and Stanislaw Gruszka <stf_xl@wp.pl>.
scst/README
+14
View File
@@ -865,6 +865,20 @@ IMPORTANT: If you use on initiator some versions of Windows (at least W2K)
for VDISK FILEIO devices above.
In some cases, for instance working with SSD devices, which consume 100%
of a single CPU load for data transfers in their internal threads, to
maximize IOPS it can be needed to assign for those threads dedicated
CPUs using Linux CPU affinity facilities. No IRQ processing should be
done on those CPUs. Check that using /proc/interrupts. See taskset
command and Documentation/IRQ-affinity.txt in your kernel's source tree
for how to assign IRQ affinity to tasks and IRQs.
The reason for that is that processing of coming commands in SIRQ
context might be done on the same CPUs as SSD devices' threads doing data
transfers. As the result, those threads won't receive all the processing
power of those CPUs and perform worse.
Work if target's backstorage or link is too slow
------------------------------------------------
scst/README_in-tree
+13
View File
@@ -799,6 +799,19 @@ IMPORTANT: If you use on initiator some versions of Windows (at least W2K)
See also important notes about setting block sizes >512 bytes
for VDISK FILEIO devices above.
In some cases, for instance working with SSD devices, which consume 100%
of a single CPU load for data transfers in their internal threads, to
maximize IOPS it can be needed to assign for those threads dedicated
CPUs using Linux CPU affinity facilities. No IRQ processing should be
done on those CPUs. Check that using /proc/interrupts. See taskset
command and Documentation/IRQ-affinity.txt in your kernel's source tree
for how to assign IRQ affinity to tasks and IRQs.
The reason for that is that processing of coming commands in SIRQ
context might be done on the same CPUs as SSD devices' threads doing data
transfers. As the result, those threads won't receive all the processing
power of those CPUs and perform worse.
Work if target's backstorage or link is too slow
------------------------------------------------
srpt/README
+25 -1
View File
@@ -37,6 +37,7 @@ with the Linux kernel source code, enable and build the SRP target driver
in the same way as any other kernel module. Make sure to enable SCST itself
and InfiniBand before attempting to enable the SRP target driver.
How-to run
-----------
A. On srp target machine
@@ -84,7 +85,7 @@ h. echo "add vdisk2 2" >/proc/scsi_tgt/groups/Default/devices
2. modprobe ib_srpt
B. On initiator machines you can manualy do the following steps:
B. On initiator machines you can manually do the following steps:
1. modprobe ib_srp
2. ipsrpdm -c (to discover new SRP target)
3. echo <new target info> > /sys/class/infiniband_srp/srp-mthca0-1/add_target
@@ -108,3 +109,26 @@ automatically ie. set SRP_LOAD=yes, and SRPHA_ENABLE=yes
and multipath tool
+ Please refer to OFED-1.x SRP's user manual for more in-details instructions
on how-to enable/use HA feature
Performance notes
-----------------
In some cases, for instance working with SSD devices, which consume 100%
of a single CPU load for data transfers in their internal threads, to
maximize IOPS it can be needed to assign for those threads dedicated
CPUs using Linux CPU affinity facilities. No IRQ processing should be
done on those CPUs. Check that using /proc/interrupts. See taskset
command and Documentation/IRQ-affinity.txt in your kernel's source tree
for how to assign IRQ affinity to tasks and IRQs.
The reason for that is that processing of coming commands in SIRQ context
might be done on the same CPUs as SSD devices' threads doing data
transfers. As the result, those threads won't receive all the CPU power
and perform worse.
Alternatively to CPU affinity assignment, you can try to enable SRP
target's internal thread. It will allows Linux CPU scheduler to better
distribute load among available CPUs. To enable SRP target driver's
internal thread you should load ib_srpt module with parameter
"thread=1".
srpt/README_in-tree
+23
View File
@@ -83,3 +83,26 @@ on how-to enable/use HA feature
To minimize QUEUEFULL conditions, you can apply scst_increase_max_tgt_cmds
patch from SRPT package from http://sourceforge.net/project/showfiles.php?group_id=110471
Performance notes
-----------------
In some cases, for instance working with SSD devices, which consume 100%
of a single CPU load for data transfers in their internal threads, to
maximize IOPS it can be needed to assign for those threads dedicated
CPUs using Linux CPU affinity facilities. No IRQ processing should be
done on those CPUs. Check that using /proc/interrupts. See taskset
command and Documentation/IRQ-affinity.txt in your kernel's source tree
for how to assign IRQ affinity to tasks and IRQs.
The reason for that is that processing of coming commands in SIRQ context
might be done on the same CPUs as SSD devices' threads doing data
transfers. As the result, those threads won't receive all the CPU power
and perform worse.
Alternatively to CPU affinity assignment, you can try to enable SRP
target's internal thread. It will allows Linux CPU scheduler to better
distribute load among available CPUs. To enable SRP target driver's
internal thread you should load ib_srpt module with parameter
"thread=1".
www/index.html
+17 -13
View File
@@ -30,17 +30,20 @@
<div id="main">
<h1>Generic SCSI Target Middle Level for Linux</h1>
<p>The <strong>SCSI target mid-level subsystem for Linux (SCST)</strong> is a subsystem of the Linux kernel
that provides a standard framework for SCSI target drivers development.
It is designed to provide unified, consistent interface between SCSI target drivers and Linux kernel
and simplify target drivers development as much as possible.</p>
<p>A system with a SCSI target device is able to share its local or virtual devices with other systems
on a network with SCSI protocol support, e.g. <strong>SCSI bus, SAS, Fibre Channel or iSCSI</strong>.
This is commonly used for data storage virtualization.</p>
<p>The <strong>SCSI target mid-level subsystem for Linux (SCST)</strong> is an alternative
implementation of SCSI target subsystem for Linux. It provides unified,
consistent interface between SCSI target drivers and
Linux kernel and simplifies target drivers development.
<p>SCST allows creation of sophisticated storage devices, which provide advanced
functionality, like <EM>replication</EM>, <EM>thin provisioning</EM>,
<EM>deduplication</EM>, <EM>high availability</EM>, <EM>automatic backup</EM>, etc.
Another class of such devices are <EM>Virtual Tape Libraries</EM> (VTL)
as well as other disk-based backup solutions. </p>
<p>This project consists from a set of subprojects: <strong>SCST core</strong> itself
as well as <strong>target drivers</strong> and <strong>user space utilities</strong>.
They are developed more or less independently and have own maintainers.</p>
<p>SCST project consists from a set of subprojects: <strong>SCST core</strong> itself
with a set of <strong>device handlers</strong> as well as <strong>target
drivers</strong> and <strong>user space utilities</strong>.
<h1>Features of SCST Core</h1>
<ul>
@@ -49,11 +52,12 @@
necessary error recovery.</span></li>
<li><span>Undertakes most problems, related to execution contexts, thus practically eliminating one of the most
complicated problem in the kernel drivers development. For example, a target driver for QLogic
22xx/23xx cards, which has all necessary features, is only about 2000 lines of code long.</span></li>
22xx/23xx cards, which has all necessary features, is only about 2000 lines of code long.
The same is true for InfiniBand SRP target driver.</span></li>
<li><span>Very low overhead, fine-grained locks and simplest commands processing path, which allow to reach
maximum possible performance and scalability. Particularly, incoming requests can be processed in
the caller's context or in one of the internal SCST core's tasklets, therefore no extra context switches
required. </span></li>
the caller's context or in one of the internal SCST core's tasklets without any
extra context switches.</span></li>
<li><span>Device handlers, i.e. plugins, architecture provides extra flexibility by allowing various I/O
modes in backstorage handling. For example, pass-through device handlers allows to use real
SCSI hardware and vdisk device handler allows to use files as virtual disks.</span></li>
www/scstvsstgt.html
+28 -13
View File
@@ -35,28 +35,43 @@
<h1>SCST vs STGT</h1>
<p><strong><a href="http://stgt.berlios.de/">STGT</a></strong> is alternative, independent from SCST implementation
of SCSI target framework for Linux. It has different architecture, where SCSI target state machine is placed in
the user space, while in SCST all the processing done in the kernel. Such architecture was acknowledged as a
"right" one by the Linux SCSI subsystem maintainers, so kernel's part of STGT quickly found its way to the kernel.
But such architecture has two inherent problems, namely performance and complexity.
See thread <a href="http://sourceforge.net/mailarchive/forum.php?thread_name=472F7FA4.7040303%40wpkg.org&forum_name=scst-devel">
"Relationship between SCST and in-kernel SCSI_TGT"</a> and <a href="http://thread.gmane.org/gmane.linux.scsi/36417/focus=37273">
this</a> message in thread "Open-FCoE on linux-scsi" for more details.</p>
the user space, while in SCST all the processing done in the kernel. Such architecture as STGT has was acknowledged
at the moment by the Linux SCSI subsystem maintainers as a "right" one, so kernel's part of STGT quickly
found its way to the kernel.</p>
<p>But such architecture has several inherent problems. Among them performance and complexity.
See description for the set of patches, submitted for the first iteration of in-kernel inclusion review and comments
<a href="http://lkml.org/lkml/2008/12/10/245">http://lkml.org/lkml/2008/12/10/245</a></p>
<p>See also the following important discussions:
<ul>
<li><span><a href="http://news.gmane.org/find-root.php?message_id=%3ce2e108260801170127w2937b2afg9bef324efa945e43%40mail.gmail.com%3e">
"Performance of SCST versus STGT"</a> for performance comparison.</span></li>
<li><span><a href="http://news.gmane.org/find-root.php?message_id=%3c463F36AC.3010207%40vlnb.net%3e">
"Question for pass-through target design"</a></span></li>
<p>See also the following important discussions:<ul>
<li><span><a href="http://news.gmane.org/find-root.php?message_id=%3c43987F75.2000301%40vlnb.net%3e">
"Stgt a new version of iscsi target?"</a>, especially pay attention to
this message: <a href="http://article.gmane.org/gmane.linux.scsi/21073">
http://article.gmane.org/gmane.linux.scsi/21073</a> for motivation why STGT's architecture was considered "right".
</span></li>
<li><span><a href="http://news.gmane.org/find-root.php?message_id=%3c463F36AC.3010207%40vlnb.net%3e">
"Question for pass-through target design"</a></span></li>
<li><span><a href="http://news.gmane.org/find-root.php?message_id=%3ce2e108260801170127w2937b2afg9bef324efa945e43%40mail.gmail.com%3e">
"Performance of SCST versus STGT"</a> for performance comparison.</span></li>
<li><span>Thread
<a href="http://sourceforge.net/mailarchive/forum.php?thread_name=472F7FA4.7040303%40wpkg.org&forum_name=scst-devel">
"Relationship between SCST and in-kernel SCSI_TGT"</a>
</span></li>
<li><span><a href="http://thread.gmane.org/gmane.linux.scsi/36417/focus=37273">this</a>
message in thread "Open-FCoE on linux-scsi" for more details.</span></li>
</ul></p>
<p>Also you shouldn't be deceived by the fact that some (small) part of STGT was accepted in the kernel.<br>
<p>Also you shouldn't be deceived by the fact that some (small) part of STGT was accepted into the kernel.<br>
It doesn't mean that STGT has the <strong>"kernel quality"</strong>. In fact, STGT as a whole similarly to any other
out-of-tree project lives on its own, hence has its own quality level, which isn't necessary better, than the
quality level of <strong>SCST</strong>. Actually, from such important aspect of quality as simplicity, it
quality level of SCST. Actually, from such important aspect of quality as simplicity, it
might be quite contrary: e.g. SCST isn't required to support HIGHMEM (nowadays it
isn't necessary, but required for all in-kernel components), which allowed to simplify memory management a lot.</p>
www/sponsorship.html
+19 -10
View File
@@ -34,30 +34,39 @@
sufficiently good to satisfy the real life requirements, and there is a
growing demand in a such subsystem. At the same time, SCST has reached
<strong>mature and stable </strong> state and <strong>capable to satisfy
</strong> all the requirements.
Also is has <strong>the widest choice</strong> of supported SCSI transports
</strong> all the requirements. See
<a href="scstvsstgt.html">SCST vs STGT</a> page for more details.</p>
<p>Also is has <strong>the widest choice</strong> of supported SCSI transports
(iSCSI, Fibre Channel, Infiniband SRP, parallel SCSI, SAS), <strong>the
best performance</strong> and <strong>the biggest features set</strong>.</p>
<p>So, the next logical step should be to <strong>include SCST in the
mainline kernel</strong>.
There is a lot of work to do before it can be achieved in areas of code
preparation, submission, going through review circles, etc.
mainline kernel</strong>. The first iteration of in-kernel inclusion review and comments
(see <a href="http://lkml.org/lkml/2008/12/10/245">http://lkml.org/lkml/2008/12/10/245</a>)
was quite positive.</p>
<p>But still there is a lot of work to do before all the necessary work in areas of code
preparation, submission, going through review circles, etc. is finished.
Activities in this direction have been carrying out, but far slower than
necessary, because they are mostly done in a spare time.</p>
<p>Thus, you can speed up a <strong>LOT</strong> all the works related to
SCST inclusion in the mainline kernel by <strong>sponsoring</strong> it.
The needed financial amount is quite low, just for one person for several months.
In this time all current showstoppers, like /proc based interface, as well as
all problems found during future public reviews should be fixed.
In this time all the current showstoppers, like /proc based interface, as well as
all problems found during past and future reviews should be fixed.
Even half of this amount would allow to work half full time exclusively on
SCST integration into Linux kernel. Funds can be on one time or per month basis.</p>
SCST integration into Linux kernel. Donations can be on one time or per period of time basis,
from companies or individuals. They can be sent by direct bank transfer (preferred) or using
Western Union, Moneygram or similar payment system. Sorry, more convenient options, like PayPal, are
not available in Russia.</p>
<p>Investing in SCST integration in Linux kernel, you investing not only
in making Linux <strong>the best storage OS</strong>, but, if your company
has a product, based on SCST, you also investing in <strong>the secured
future</strong> of your product and, hence, your company.</p>
has a product or service, based on SCST, you also investing in <strong>the secured
future</strong> of your product or service and, hence, in the <strong>the secured
future</strong> of your company.</p>
<p>If you are interested, please contact Vladislav Bolkhovitin <<strong>vst at vlnb net</strong>></p>
</div>
www/target_iscsi.html
+14 -12
View File
@@ -59,7 +59,7 @@
<p><strong>ISCSI-SCST</strong> has the following major advantages over the mainline <strong>IET</strong>:
<ul>
<li><span>It uses full power of <strong>SCST core</strong> without loosing any existing feature (except, maybe, illegal
<li><span>It uses full power of <strong>SCST core</strong> without loosing any existing IET feature (except, maybe, illegal
from SCSI specifications point of view MPIO). Namely, you can additionally use with it:
<ul>
<li><span><strong>Pass-through mode with one to many relationship</strong>, i.e. when multiple initiators can connect to the
@@ -83,29 +83,31 @@
</ul>
</span></li>
<li><span>It has many code improvements and cleanups, including <strong>stability and iSCSI RFC violations fixes</strong>.
Many IET users use it for ages without problems, so they consider it problemless. But,
Many IET users use it for ages without problems, so they consider it free from any real problem. But,
in fact, unfortunately, it isn't so.
IET works well only on "fast" paths and regularly used branches, in many other less used
IET works well only on "fast" paths and regularly used code branches. In many other less used
cases IET has various problems, from simply ignoring error processing, as it is with memory
allocations, and crashing itself with BUG() macro, as it is for malformed packets from initiators, to possible data
corruption. See, for instance, <a href="http://communities.vmware.com/thread/53797?tstart=0&start=15">this</a>
thread on a VMware forum about in which <strong>Russian roulette</strong> IET users play using it with VMware.
ChangeLog file lists most noticeable fixes, but there were a lot of many other smaller ones.</span></li>
<li><span>Due to reworked I/O architecture and <strong>SCST</strong> backend <strong>iSCSI-SCST</strong> has
better performance in many cases. In future with upcoming improvements in <strong>SCST core</strong>, like zero-copy
with Linux cache <strong>FILEIO</strong>, the performance difference is going to be even bigger. Currently in
tests from a single initiator over a single connection on 1GbE hardware over <strong>FILEIO</strong> vdisk
<strong>iSCSI-SCST</strong> with default settings usually outperforms tuned for best
performance <strong>IET</strong> a on <strong>3-30%</strong>. The difference is especially noticeably with real storage,
<li><span>Due to reworked I/O architecture and SCST backend iSCSI-SCST has
better performance in many cases. In future with upcoming improvements in SCST core, like <strong>zero-copy
with Linux cache FILEIO</strong>, the performance difference is going to be even bigger. Currently in
tests from a single initiator over a single connection on 1GbE hardware over FILEIO vdisk
iSCSI-SCST with default settings usually outperforms tuned for best
performance IET a on <strong>on up to 100%</strong>. The difference is especially noticeably with real storage,
not NULLIO or RAM disks.
On 10GbE hardware the performance difference is often as high as <strong>100-200%</strong> or even more.
With higher number of initiators the difference will be even bigger. This is because
<strong>iSCSI-SCST</strong> has less commands processing overhead per command, hence has smaller processing
iSCSI-SCST has less commands processing overhead per command, hence has smaller processing
latency and puts less load on CPU.
</ul></p>
<p>Also, in contrast to <strong>IET, iSCSI-SCST</strong> is open for any new development, modifications and
<p>Also, in contrast to IET, iSCSI-SCST is open for any new development, modifications and
improvements, so people who want to fix or implement something new will not have to keep and maintain separate
patches as it is currently necessary with <strong>IET</strong>. <strong>ISCSI-SCST</strong> is actively
patches as it is currently necessary with IET. ISCSI-SCST is actively
developed and going to gain in the future new features like support for multiple connections per session, error recovery
levels >0, etc., which eventually make it really <i>"Enterprise"</i>.</p>