Features comparison between Linux SCSI targets
As on July 2009
| SCST | STGT | IET | LIO | |
|---|---|---|---|---|
| General | ||||
| Upstream kernel | - | Kernel part | - | - |
| Generic Target Engine | + | + | iSCSI only | Under development |
| Architecture | Kernel only | User space only 1 | - | Kernel only |
| Stability | + | + 3 | + | Under development |
| Performance 4 | ***** 5 | *** | **** | **** 6 |
| Zero-copy passing data between target and backend drivers | + 7 | + 8 | + | + |
| Support for transports without expecting transfer values (parallel SCSI, SAS) | + | - | - | - |
| Interface with user space | ProcFS | Custom | - | ConfigFS/IOCTL/ProcFS |
| Major features | ||||
| Target drivers in kernel space | + | + | - | Under development |
| Target drivers in user space | + | + | - | - |
| Backstorage handlers in kernel space | + | - | - | + |
| Backstorage handlers in user space | + | + | - | - |
| Local access to emulated backstorage devices 9 | + | - | - | - |
| Advanced devices access control 10 | + | - | - | - |
| Support for Asynchronous Event Notifications (AEN) | + | - | - | - |
| Notification for devices added/removed or resized through AENs or Unit Attentions | + | - | - | - |
| Bidirectional Commands | + 11 | + | - | - |
| Extended CDB (size >16 bytes) | + 11 | + | - | - |
| Descriptor sense support | + | + | - | - | RESERVE/RELEASE (Windows 2003 clustering) | + | + | + | + | Safe RESERVE/RELEASE implementation according to SCSI requirements 12 | Safe | Safe | Not safe 13 | Not safe | Safe implementation of Task Management commands 14 | Safe | Not safe | Not safe | Not safe |
| Support for SCSI task attributes, including ORDERED commands | + | + | -, data corruption possible 15 | -, data corruption possible 15 |
| Persistent Reservations (Windows 2008 clustering) | - | - | - | Under development |
| ALUA | - | - | - | Under development |
| Failover Clustering | + | ? | + | + |
| SCSI MIBs | - | - | - | - 16 |
| Cluster Storage Integration | - | - | - | VHACS |
| Supported transports and hardware | ||||
| iSCSI | + | + | + | + |
| QLogic (Fibre Channel) | + | - | - | - | Emulex (Fibre Channel and FCoE) | + | - | - | - |
| SRP | + | - | - | - |
| iSER | - | + 3 | - | - | Marvell (SAS) | Beta | - | - | - |
| FCoE | Under development | Under development | - | - |
| LSI (Parallel (Wide) SCSI and Fibre Channel) | Alpha | - | - | - | LSI (SAS) | Preliminary | - | - | - |
| IBM pSeries Virtual SCSI | - | + | - | - |
| Supported backstorage | ||||
| Kernel side FILEIO | + | - | + | + |
| Kernel side BLOCKIO | + | - | + | + |
| User space FILEIO | + | + | - | - |
| SCSI pass-through 17 | + | Disks only, single initiator only, not enforced 18 | - | Single initiator only, not enforced 18 |
| Zero-copy data read/write to/from backstorage | BLOCKIO, user space FILEIO in O_DIRECT mode, pass-through 19 | - 8 | BLOCKIO | BLOCKIO, pass- through |
| 4k sectors support in pass-through mode | + | - | - | ? |
| 4k, 2k, 1k and 512 byte sectors emulation in modes, other than pass-through | + | - | - | + |
| Virtual CD devices emulation from ISO files | + | + | - | - |
| Possibility to write to emulated from ISO files CD devices | - | + | - | - |
| Emulation of virtual tape and media changer devices | - | Experimental | - | - |
| iSCSI Target | ||||
| Architecture | Split 2 | User space only | Split 2 | Kernel only |
| Interface with user space | IOCTL/ProcFS/ Netlink | - | IOCTL/ProcFS/ Netlink | ConfigFS/IOCTL/ProcFS |
| Zero-copy data send/receive | Send only 20 | None 8 | Send only | Send only |
| Multiple connections per session (MS/C) | - | - | - | + |
| Max ErrorRecoveryLevel | 0 | 0 | 0 | 2 |
| Bidirectional Commands | + 11 | + | - | - |
| Extended CDB (size >16 bytes) | + 11 | + | - | - |
| Support for Asynchronous Event Notifications (AEN) | + | - | - | - | Safe implementation of connections and sessions reinstatement 21 | Safe | Not safe | Not safe | Not safe | Safe restart 22 | Safe | ? | Not safe | ? |
| iSCSI MIBs | - | - | - | - 16 |
REMARKS:
1. STGT has SCSI target engine and memory management in user space with small hooks in the kernel to interact with in-kernel target drivers. As a direct consequence, fully user space STGT target (e.g. iSCSI) can run without any kernel modules needed.
2. All iSCSI management implemented in user space and actual data transfers in kernel space without user space involved.
3. ISER target driver has long known (since Feb 2008) data corruption problem, which localization hasn't been determined yet and might be in the STGT core. See here and here.
4. The result "in average" is listed. One target can be better somewhere, another one somewhere else. Although manual tuning of target and system parameters tends the restore the difference listed in the comparison. You can find example measurements here, here and here.
5. All SCST and its drivers' kernel patches supposed to be applied and SCST with the drivers built in the release or performance build. Without the kernel patches SCST performance will be at "****+" level, except for the case, when user space backstorage handler used with iSCSI-SCST iSCSI target driver, where performance will be at "***+" level.
6. There is no performance data for LIO, which allow to make a direct comparison with other targets. The conclusion was made by source code study only. LIO should have performance on the IET level or less, because of more processing overhead. It might be much less for small block sizes.
7. In SCST data are always passed in zero-copy manner between target and backend drivers without need for any additional kernel patches, except in case, when local access (scst_local) used with user space backend.
8. Some zero-copy functionality isn't available from user space, sometimes fundamentally. For instance, zero-copy FILEIO with page cache or zero-copy send to a socket. Also STGT can't use splice() for in-kernel target drivers, because it has memory management in user space. To use splice() with socket-based user space target drivers STGT would need a deep redesign of internal interactions between target drivers, core and backend handlers.
9. Local access to emulated backstorage devices" means that you can access emulated by a SCSI target devices locally on the target host. For instance, you can mount your ISO image from emulated by the target CDROM device locally on the target host.
10. "Advanced devices access control" means that different initiators can see different sets of devices from the same target. This feature is required for hardware targets, which don't have ability to create virtual targets.
11. Not well tested, because at the moment there is no backend using this functionality.
12. SCSI requires that if an initiator clears reservation held by another initiator, the reservation holder must be notified about the reservation clearance. Otherwise, several initiators can at the same time change supposed to be protected by the reservation data, which can corrupt them. This is what was called "Russian roulette with your data" on the VMware community forum by someone working for VMware. But, sure, it can affect not only VMware, but also any other cluster implementation, relying on this functionality.
13. The problem in IET was confirmed in IET mailing list.
14. After a task management command completed and before the corresponding response was sent to the initiator, who sent that task management command, all the affected SCSI commands must get into a state, where they can't affect following after the tasks management response commands from this initiator. This is the safe implementation. The unsafe implementation only marks all the affected SCSI commands as aborted and then immediately send task management response to the initiator. This implementation only guarantees that the initiator will never receive responses from those commands, but it doesn't guarantee that none of those commands will get executed by backstorage *AFTER* any SCSI command, which initiator will send after it received the task management response thinking that all the aborted commands actually fully aborted. This could lead to a data corruption.
15. Both IET and LIO report in INQUIRY command response support for full task management model. But they process ORDERED commands the same way as SIMPLE commands, i.e. allow free reorder of them before they get executed. That violates SCSI standard and can lead to a data corruption to any application relying on commands order provided by ORDERED attribute.
16. The IETF SCSI-MIB (RFC 4455) is not supported by LIO, only proprietary MIBs SBE-SCSI-MIB, SBE-ISCSI-MIB and SBE-IPS-AUTH-MIB are supported.
17. SCSI pass-through mode allows to export your local SCSI-capable device. For instance with it you can share your parallel SCSI tape or SATA DVD-RW device to your iSCSI network.
18. STGT and LIO don't emulate all the necessary SCSI host functionality to allow to share SCSI devices in pass-through mode to several initiators. They can only pass SCSI commands from initiators to SCSI devices and responses back. This is safe only with a single initiator. This limitation isn't enforced anyhow and both STGT and LIO don't issue any warning about it, so an user will not be notified about this limitation and can quietly corrupt his/her data. You can find more technical information about it here.
19. You can find a proposal how to implement zero-copy FILEIO in SCST on the Contributing page.
20. Doesn't need any kernel patch, except in the case, when used with user space backend.
21. Connections and sessions reinstatement is, basically, a kind of Task Management command, because it implies commands aborting. For instance, open-iscsi uses it as a less intrusive substistute for target reset in eh_target_reset_handler() callback. So, similarly to the safe task management above, a safe implementation of connections and sessions reinstatement must not accept SCSI commands from new connection/session until all the SCSI commands in being reinstated connection/session get into a state, where they can't affect new commands.
22. "Safe restart" means that after the iSCSI target restart, all the connected initiators will seamlessly restore all existing before the restart connections. "Not safe" means that, most likely, the connected initiators will fail to restore existing connections with some errors. However, your iSCSI initiator also should be able to handle the safe restart. For instance, old (pre-CentOS/RHEL 5) open-iscsi has problems in this area. But the latest versions do it pretty well.