Not s
locally on the target host. For instance, you can mount your ISO image from emulated by the target
CDROM device locally on the target host.
-
10. "Advanced devices access control" means that different initiators can see different sets
- of devices from the same target. This feature is essential for hardware targets, which don't have ability
+
10. "Advanced devices LUNs control" means that different initiators can see different sets
+ of devices from the same target. This feature is required for hardware targets, which don't have ability
to create virtual targets.
11. Not well tested, because at the moment there is no backend using this functionality.
15. LIO doesn't emulate all the necessary SCSI host functionality to allow to share a SCSI device
in pass-through mode to several initiators. It can only pass SCSI commands from initiator to
- the SCSI device. This is safe only if there is only one initiator. This limitation
- isn't enforced anyhow by LIO. You can find more technical information about that in
+ the SCSI device and response back. This is safe only with a single initiator. This limitation
+ isn't enforced anyhow, so you can accidentally corrupt your data. You can find more technical information about that in
http://www.mail-archive.com/linux-scsi@vger.kernel.org/msg06911.html
19. "Safe restart" means that after the iSCSI target restart, all the connected initiators will seamlessly restore all existing before
the restart connections. "Not safe" means that, most likely, the connected initiators will fail to restore
existing connections with some errors.
-
-
P.S. You can find a brief background history of this comparison page here.
+
+
diff --git a/www/comparison_history.html b/www/comparison_history.html
index c3649a924..0572eacd1 100644
--- a/www/comparison_history.html
+++ b/www/comparison_history.html
@@ -6,7 +6,7 @@
-SCST Comparison History
+SCSI Targets Comparison History
@@ -21,7 +21,7 @@
@@ -53,7 +54,7 @@ document.write("csi.org/index.php/Main_Page#Feature_Overview\">page<\/a>");
wrong about SCST. I asked that page author, Nicholas Bellinger, to
correct the wrong items (see my
e-mail), but Nicholas Bellinger refused to do it without any explanations. Moreover, he
-blocked my access to the LIO mailing list, preventing me from tell that
+blocked my access to the LIO mailing list, preventing me from telling that
to the interested people myself. After all our previous discussions and
with his skills and experience it's nearly impossible to believe that Nicholas Bellinger
didn't know that SCST is a lot more generic than LIO and has zero-copy
@@ -61,32 +62,38 @@ in all the same places, where LIO has. Thus, that comparison page is looking lik
attempt. Seems SCST is so much superior over LIO, so Nicholas Bellinger gave up technical discussions and started
attacking people's perception about SCST, trying to inspire them the opposite.
-
So, we had no choice, except to setup own,
-correct comparison. This comparison table turned out to be very useful,
-so it was extended to cover all the SCSI target areas.
-
Update
-
After our comparison page was published, Nicholas Bellinger corrected the most crying items about SCST. Namely:
-he acknowledged that SCST is a generic engine, marked "zero-copy" item for SCST as "Needs kernel patch" and removed "User Interface" item.
-But "zero-copy" mark for SCST is still cheating, because
+
After our comparison page was published, Nicholas Bellinger corrected in his comparison page the most crying items about SCST. Namely:
+he acknowledged that SCST is a generic engine as well as marked "zero-copy" item for SCST as "Needs kernel patch" and removed "User Interface" item.
+But "zero-copy" mark for SCST is still wrong, because
SCST needs kernel patch to provide send zero-copy in the iSCSI target driver (and only in it) only if it is working with user space backend. LIO doesn't
-support user space backend, so, if it has "X" here, then SCST must also has unconditional "X" here.
+support user space backend, so, if it has "X" in this line, SCST must also have unconditional "X" there.
Moreover, new wrong items where introduced:
-
STGT marked as having Fibre Channel target drivers for Emulex, QLogic and LSI cards.
-
-
STGT marked as having Fibre Channel QLogic target driver in-tree.
-
SCST was marked as "-" in "Additional Header Segment" and "Bidirectional Commands" items.
- In reality, SCST and iSCSI-SCST support bidirectional commands as well as iSCSI-SCST supports other values in the
+ SCST and iSCSI-SCST support bidirectional commands as well as iSCSI-SCST supports other values in the
Additional Header Segment (namely, extended CDB).
+
+
STGT marked as having Fibre Channel target drivers for Emulex, QLogic and LSI cards. Where are they?
+
+
STGT marked as having Fibre Channel QLogic target driver in-tree. Which tree is meant?
+
+
STGT marked as supporting SAS and parallel SCSI targets. But
+ it fundamentally can't do that. Those transports don't provide expecting
+ transfers values (size and direction), so those values must be extracted
+ directly from CDBs. STGT doesn't have the necessary infrastructure for that, see, for instance,
+ "https://lists.berlios.de/pipermail/stgt-devel/2007-March/000436.html.
+ Nothing has changed since time, when that e-mail was sent.
+
+
Interesting coincidence that the above wrong STGT items were added at nearly the same time as LIO started working with STGT's in-kernel target drivers...
The latest stable version of SCST core is 1.0.0. It requires Linux kernel 2.6.16 or higher. Tested mostly on
- i386 and x86_64, but should also work on any other supported by Linux platform. More detail
- information you could find in its README file.
+
+
The latest stable version of SCST core is 1.0.1. It requires Linux kernel 2.6.16 or higher.
+ More detail information you could find in its README file.
+
The latest development version of SCST core, target drivers and
user space utilities is available directly from the
project's SVN. The SCST release policy is to make stable releases once or twice a
year, with the only exception if the current stable version contains a critical bug. But every commit in the
- SVN is carefully tested, so you can consider them as mini releases. Therefore on practice the
- development version in the SVN is usually more stable, than the "stable", i.e. released,
- one.
+ SVN is carefully tested, so you can consider them as mini releases.
+
svn co https://scst.svn.sourceforge.net/svnroot/scst
+
More information about accessing SVN repository may be found
here.
+
History of the pre-SVN SCST development is available in SCST CVS repository, which is accessible using
web-based CVS repository viewer or anonymous CVS access.
The SCSI target mid-level subsystem for Linux (SCST) is an alternative
implementation of SCSI target subsystem for Linux. It provides unified,
consistent interface between SCSI target drivers and
- Linux kernel and simplifies target drivers development.
+ Linux kernel as well as between Linux kernel and storage backend
+ handlers, connecting target drivers with real or emulated storage backend.
SCST allows creation of sophisticated storage devices, which provide advanced
functionality, like replication, thin provisioning,
@@ -42,9 +43,10 @@
automatic backup, etc. Another class of such devices
are Virtual Tape Libraries (VTL)
as well as other disk-based backup solutions. SCST created devices not
- limited by IP network interface only, but they can use any link supporting
+ limited by IP network interface only. They can use any link supporting
SCSI-style data exchange, including Fibre Channel,
- SAS and parallel (Wide) SCSI. It might
+ iSCSI, SAS,
+ InfiniBand and parallel (Wide) SCSI. It might
well be that your favorite storage appliance running SCST in the firmware.
SCST project consists from a set of subprojects: SCST core itself
@@ -54,24 +56,25 @@
Features of SCST Core
Simple, easy to use interface with target drivers.
- Particularly, SCST core performs required pre- and post- processing of incoming requests as well as
+ SCST core performs all required pre- and post- processing of incoming requests as well as
necessary error recovery.
Undertakes most problems, related to execution contexts, thus practically eliminating one of the most
- complicated problem in the kernel drivers development. For example, a target driver for QLogic
- 22xx/23xx cards, which has all necessary features, is only about 2000 lines of code long.
- The same is true for InfiniBand SRP target driver.
-
Very low overhead, fine-grained locks and simplest commands processing path, which allow to reach
+ complicated problem in the kernel drivers development. For example, target drivers for QLogic
+ 22xx/23xx cards and for InfiniBand SRP are only about 2300 lines of code long.
+
Very low overhead, fine-grained locks and simplest commands processing path allow to reach
maximum possible performance and scalability. Particularly, incoming requests can be processed in
the caller's context or in one of the internal SCST core's tasklets without any
extra context switches.
-
Device handlers, i.e. plugins, architecture provides extra flexibility by allowing various I/O
+
Device handlers, i.e. plugins, architecture allows various I/O
modes in backstorage handling. For example, pass-through device handlers allows to use real
SCSI hardware and vdisk device handler allows to use files as virtual disks.
-
Provides advanced per-initiator device visibility management (LUN masking), which allows different
+
Provides advanced per-initiator devices visibility management (LUN masking) allows different
initiators to see different set of devices with different access permissions. For instance,
initiator A could see exported from target T devices X and Y read-writable, and initiator B from
- the same target T could see devices Y read-only and Z read-writable.
-
Emulates necessary functionality of SCSI host adapter, because from remote initiators point of view
+ the same target T could see devices Y read-only and Z read-writable.
+ This feature is required for hardware targets, which don't have ability to create
+ virtual targets (SAS adapters, for instance).
+
Emulates necessary functionality of SCSI host adapter, because from remote initiators' point of view
SCST acts as a SCSI host with its own devices. This is especially important in pass-through mode with
one to many relationship, i.e. when multiple initiators can connect to the exported pass-through
devices. You can find more deep elaboration why it is needed in this
@@ -79,19 +82,23 @@
Generation of necessary UNIT ATTENTIONs, their storage and delivery to all connected
remote initiators.
+
RESERVE/RELEASE functionality.
-
CA/ACA conditions (not implemented yet).
+
All types of RESETs and other task management functions.
+
REPORT LUNS command as well as SCSI address space management in order to have consistent
address space on all remote initiators, since local SCSI devices could not know about each
other to report via REPORT LUNS command. Additionally, SCST core responds with error on all
commands to non-existing devices and provides access control, so different remote
initiators could see different set of devices.
+
Other necessary functionality (task attributes, etc.) as specified in SAM-2, SPC-2, SAM-3,
SPC-3 and other SCSI standards.
-
Multithreaded design and complete SMP support, so, if necessary, all your processors will participate in the commands
+
+
Multithreaded design and complete SMP support, so, if necessary, all your CPU cores will participate in the commands
processing.
Well documented.
@@ -100,11 +107,8 @@
device, for example RESERVE the device, or some of its parameters and that could be done behind SCST, which could
lead to various problems, including data corruption. Thus, RESERVE/RELEASE commands, locally generated
UNIT ATTENTIONs, etc. should be intercepted and processed as if local SCSI initiators act as remote SCSI
- initiators connected to SCST.
-
Pass-through mode with one to many relationship, i.e. when multiple initiators can
@@ -114,12 +118,13 @@
FILEIO mode, which allows to use files on file systems or block devices as virtual
remotely available SCSI disks or CDROMs with benefits of the Linux page cache.
BLOCKIO mode, which performs direct block IO with a block device, bypassing
- page-cache for all operations. This mode works ideally with high-end storage HBAs and for applications that
+ page-cache for all operations. This mode works well with high-end storage HBAs and for applications that
either do not need caching between application and disk or need the large block throughput.
User space mode using scst_user device handler, which allows to implement in the
user space virtual SCSI devices in the SCST environment.
-
"Performance" device handlers, which provide in pseudo pass-through mode a way for
- direct performance measurements without overhead of actual data transferring from/to underlying SCSI device.
+
"Performance" device handlers as well as NULLIO mode provide
+ a way for direct performance measurements without overhead of actual data
+ transferring from/to underlying SCSI device.
MC/S (Multiple Connections per Session) is a feature of iSCSI
+protocol, which allows to combine several connections inside a single
+session for performance and failover purposes. Let's consider what
+practical value this feature has comparing with OS level multipath
+(MPIO) and try to answer why none of Open Source OS'es neither still
+support it, despite of many years since iSCSI protocol started
+being actively used, nor going to implement it in the future.
+
+
MC/S is done on the iSCSI level, while MPIO is done on the higher
+level. Hence, all MPIO infrastructure is shared among all SCSI
+transports, including Fibre Channel and SAS.
+
+
MC/S was designed at time, when most OS'es didn't have OS level
+multipath. Instead, each vendor had its own implementation, which
+created huge interoperability problems. So, one of the goals of MC/S was
+to address this issue and standardize the multipath area. But
+nowadays almost all OS'es has OS level multipath implemented using
+standard SCSI facilities, hence this purpose of MC/S isn't valid anymore.
+
+
Usually it is claimed, than MC/S has the following 2 advantages over MPIO:
+
+
+
Faster failover recovery.
+
+
Better performance.
+
+
+
+
Let's look how much true those claims are.
+
+
Failover recovery time
+
+
Let's consider a single target exporting a single device over 2 links.
+
+
For MC/S failover recovery is pretty simple: all outstanding SCSI
+commands reassigned to another connection. No other actions are
+necessary, because session (i.e. I_T Nexus) remains the same.
+Consequently, all reservations and other SCSI states as well as other
+initiators connected to the device remain unaffected.
+
+
For MPIO failover recovery is much more complicated. This is because
+it involves transferring outstanding commands and SCSI states from one
+I_T Nexus to another. The first thing, which initiator will do for
+failover recovery is to abort all outstanding commands on the faulted
+I_T Nexus. There are 2 approaches for that: CLEAR TASK SET and LUN RESET
+task management functions.
+
+
CLEAR TASK SET function aborts all commands on the device.
+Unfortunately, it has limitations: it isn't always supported by device
+and having single task set shared over initiators isn't always
+appropriate for application.
+
+
LUN RESET function resets the device.
+
+
Both CLEAR TASK SET and LUN RESET functions can somehow harm
+other initiators, because all commands from all initiators, not only
+from one doing the failover recovery, will be aborted. Additionally, LUN
+RESET resets all SCSI settings for all connected initiators to the
+initial state and, if device had reservation from any initiator, it will
+be cleared.
+
+
But the harm will be minimal:
+
+
+
With TAS bit set on Control Mode page, all the aborted commands will
+ be returned to all affected initiators with TASK ABORTED status, so they
+ can simply immediately retry them. For CLEAR TASK SET if TAS isn't set
+ all affected initiators will be notified by Unit Attention COMMANDS
+ CLEARED BY ANOTHER INITIATOR, so they also can immediately retry all
+ outstanding commands.
+
+
In case of device reset the affected initiators will be notified via
+ the corresponding Unit Attention about the device reset, i.e. about reset of
+ all SCSI settings to the initial state. Then they can do the necessary
+ recovery actions. Usually no recovery actions are needed, except for the
+ reservation holder, whose reservation was cleared. For it recovery might
+ be not trivial. But Persistent Reservations solve this issue, because
+ they are not cleared by the device reset.
+
+
+
Thus, with Persistent Reservations or using CLEAR TASK SET function
+additional failover recovery time, which MPIO has comparing to MC/S,
+is time to wait for reset or commands abort finished and time to
+retry all the aborted commands. On a properly configured system it
+should be less than few seconds, which is well acceptable on practice.
+If Linux storage stack improved to allow to abort all submitted to it
+commands (currently only wait for their completion is possible), then
+time to abort all commands can be decreased to a fraction of second.
+
+
Performance
+
+
At first, neither MC/S, nor MPIO can improve performance if there is
+only one SCSI command sent to target at time. For instance, for tape
+backup and restore. Both MC/S and MPIO work on the commands level, so
+can't split data transfers for a single command over several links. Only
+bonding can improve performance in this case, because it works on the
+link level.
+
+
MC/S over several links preserves commands execution order, i.e. with
+it commands executed in the same order as they were submitted. MPIO
+can't preserve this order, because it can't see, which command on which
+link was submitted earlier. Delays in links processing can change
+commands order in the place where target receives them.
+
+
Since initiators usually send commands in the optimal for performance
+order, reordering can hurt performance. But this can happen only with
+naive implementation, which can't recover the optimal commands execution
+order. Currently Linux is not naive and quite good on it. See, for
+instance, section "SEQUENTIAL ACCESS OVER MPIO" in those measurements. Don't look at the absolute
+numbers, look at %% of performance improvement using the second link.
+The result equivalent to 200 MB/s over 2 1Gbps links, which is close to
+possible maximum.
+
+
If free commands reorder is forbidden for a device, either
+by use of ORDERED tag, or if the Queue Algorithm Modifier in the Control
+Mode Page is set to 0, then MPIO will have to maintain commands order by
+sending them over only a single link. But on practice this case is
+really rare and 99.(9)% of OS'es and applications allow free commands
+reorder and it is enabled by default.
+
+
However strictly preserving commands order as MC/S does has a
+downside as well. It can lead to so called "commands ordering
+bottleneck", when newer commands have to wait before one or more older
+commands get executed, although it would be better for performance to
+reorder them. As result, MPIO sometimes has better performance, than
+MC/S, especially in setups, where maximum IOPS number is important. See,
+for instance,
+
+http://article.gmane.org/gmane.linux.scsi/16311.
+
+
+
Conclusion
+
+
Thus:
+
+
+
Benefits of MC/S are marginal and with future MPIO
+ improvements can be made negligible.
+
+
MPIO allows to utilize existing infrastructure for all
+ transports, not only iSCSI.
+
+
+
All transports can benefit from improvements in MPIO.
+
+
With MPIO there is no need to create multiple layers doing very similar
+ functionality.
+
+
MPIO doesn't have commands ordering bottleneck, which MC/S has.
+
+
+
+
Hence, what's the point to implement MC/S, which is a very tricky
+task? Better to spend this effort on improving MPIO. Simply, MC/S is
+done on the wrong level. No surprise then that no Open Source OS'es
+neither support, nor going to implement it. Moreover, when back to 2005
+there was an attempt to add MC/S in Linux, it was rejected. See for more details:
+http://article.gmane.org/gmane.linux.scsi/15769
+and http://article.gmane.org/gmane.linux.scsi/16301.
+
+
+
But for sake of completeness, there are cases, where MPIO can't be
+used or will not provide any benefit, but MC/S can provide benefits:
+
+
+
When strict commands order is required.
+
+
When aborted commands can't be retried.
+
+
+
+
For disks both of them are always false. However for some tape drives and
+backup applications one or both can be true. But on practice:
+
+
+
+
There are neither known tape drives, nor backup
+ applications, which can use multiple outstanding commands at
+ time. All them support and use only one single outstanding
+ command at time. MC/S can't increase performance for them, only
+ bonding can. So, in this case there no difference between MC/S
+ and MPIO.
+
+
The lack of ability to retry commands is rather a
+ limitation of legacy tape drives, which support only implicit
+ address commands, not of MPIO. Modern tape drives and backup
+ applications can use explicit address commands, which you can
+ abort and then retry, hence compatible with MPIO.
+
+
+
+
If in future SCSI standards have possibility to group several I_T nexuses
+with possibilities to reassign commands between them and preserve commands
+order among them, the above the only advantages of MC/S over MPIO will be
+removed and, hence, all investments in it will be voided.
SCST administration utility "scstadmin" is developed by Mark Buechler.
With it you can
+
SCST administration utility "scstadmin" developed by Mark Buechler.
With it you can
manually or automatically using either plain text config file, or MySQL database configure every aspect of SCST.
-
Especially useful feature of scstadmin is ability to figure out and apply on the currently
+
Especially useful feature of scstadmin is ability to figure out and apply on the currently
running system changes in scst.conf file. In other words, you can have SCST subsystem running with configuration from file
scst.conf, then you edit this file, e.g. add new devices, then scstadmin will figure out that you added those devices
and add them to SCST.
@@ -117,13 +118,13 @@
are divided (as it should be) between the target driver (iSCSI-SCST) and the SCST core: target driver is
responsible for handling targets and their parameters, SCST core is responsible for handling backstorage.
-
The latest stable version is 1.0.0. Requires Linux kernel version 2.6.16.x or higher and SCST version 1.0.0 or higher.
+
The latest stable version is 1.0.1. Requires Linux kernel version 2.6.16.x or higher and SCST version 1.0.1 or higher.
Tested mostly on i386 and x86_64, but should work on any other supported by Linux platform.
You can find the latest development version of this driver in the SCST SVN. See the download page how to setup
access to it.
This driver allows you to access devices that are exported via SCST
+ directly on the same Linux system that they are exported from.
+
+
It makes no assumptions in the code about the device types on the target, so
+ any device handlers that you load in SCST will be visible, including tapes
+ and so forth.
+
+
This driver was made by Richard Sharpe.
+
+
It is on the beta stage. The latest released version is 0.9. You can download
+ the latest development version from the SCST SVN repository. See the
+ download page how to setup access to it.
This is target driver for QLogic 22xx/23xx Fibre Channel cards. It is stable and well tested.
-
The latest stable version is 1.0.0. Requires Linux kernel version 2.6.16.x or higher and SCST version 1.0.0 or higher.
- Tested mostly on i386 and x86_64, but should work on any other supported by Linux platform.
+
+
The latest stable version is 1.0.1. Requires Linux kernel version 2.6.16.x or higher and
+ SCST version 1.0.1 or higher.
+
You can find the latest development version of this driver in the SCST SVN. See the download page how to setup
access to it.
This is target driver for QLogic 22xx/23xx Fibre Channel cards. It is stable and well tested.