diff --git a/doc/Makefile b/doc/Makefile index d625a55d4..7016d41d1 100644 --- a/doc/Makefile +++ b/doc/Makefile @@ -10,6 +10,8 @@ all: html txt pdf tex dvi ps info lyx rtf txt: $(SOURCE_NAME).txt +ptxt: $(SOURCE_NAME).ptxt + html: $(SOURCE_NAME).html tex: $(SOURCE_NAME).tex @@ -26,9 +28,15 @@ lyx: $(SOURCE_NAME).lyx rtf: $(SOURCE_NAME).rtf +check: + $(COMMAND)check $(SOURCE) + $(SOURCE_NAME).txt: $(SOURCE) $(COMMAND)txt $(SOURCE) +$(SOURCE_NAME).ptxt: $(SOURCE) + $(COMMAND)txt -f $(SOURCE) + $(SOURCE_NAME).html: $(SOURCE) $(COMMAND)html --split=0 $(SOURCE) @@ -54,13 +62,11 @@ $(SOURCE_NAME).rtf: $(SOURCE) $(COMMAND)rtf $(SOURCE) clean: - @mv scst_user_spec.txt scst_user_spec.tx @mv sgv_cache.txt sgv_cache.tx rm -f *.txt *.html *.tex *.dvi *.ps *.pdf *.info *.lyx *.rtf - @mv scst_user_spec.tx scst_user_spec.txt @mv sgv_cache.tx sgv_cache.txt extraclean: clean rm -f *.orig *.rej -.PHONY: all default html txt pdf tex dvi ps info lyx rtf clean extraclean +.PHONY: all default html txt ptxt pdf tex dvi ps info lyx rtf check clean extraclean diff --git a/doc/scst_user_spec.txt b/doc/scst_user_spec.sgml similarity index 59% rename from doc/scst_user_spec.txt rename to doc/scst_user_spec.sgml index 1f4ed23e1..2d09fa4e0 100644 --- a/doc/scst_user_spec.txt +++ b/doc/scst_user_spec.sgml @@ -1,16 +1,27 @@ - SCST USER SPACE DEVICE HANDLER MODULE. + - USER SPACE INTERFACE DESCRIPTION. +
- Version 2.1.0 + +SCST user space device handler module interface descrition + + + Vladislav Bolkhovitin + - I. Description. +Version 2.1.0 + + +Introduction + +

SCST user space device handler module scst_user is a device handler for SCST, which provides a way to implement in the user space complete, full feature virtual SCSI devices in the SCST environment. +

This document assumes that the reader is familiar with the SCST architecture and the states through which SCSI commands go during processing in SCST. Module scst_user basically only provides hooks to @@ -18,21 +29,24 @@ them. Their description could be found on the SCST web page on http://scst.st.net. - II. User space API. + User space API +

Module scst_user provides /dev/scst_user character device with the following system calls available: - - open() - allows to open the device and get a file handle, which + + Device /dev/scst_user could be opened in blocking or non-blocking mode using O_NONBLOCK flag. In the blocking mode ioctl() @@ -45,19 +59,20 @@ default. The module scst_user API is defined in scst_user.h file. + IOCTL() functions - III. IOCTL() functions. - +

There are following IOCTL functions available. All of them has one argument. They all, except SCST_USER_REGISTER_DEVICE return 0 for success or -1 in case of error, and errno is set appropriately. + SCST_USER_REGISTER_DEVICE - 1. SCST_USER_REGISTER_DEVICE - -SCST_USER_REGISTER_DEVICE - registers new virtual user space device. The +

+SCST_USER_REGISTER_DEVICE registers new virtual user space device. The argument is: + struct scst_user_dev_desc { aligned_u64 version_str; @@ -73,23 +88,26 @@ struct scst_user_dev_desc char name[SCST_MAX_NAME]; char sgv_name[SCST_MAX_NAME]; }, + where: - - version_str - protocol version, shall be DEV_USER_VERSION. + - - license_str - license of this module, for instance, "GPL", "GPL v2", or + 0, then the SGV cache will work in the fixed size buffers mode. In this case it sets the size of each buffer in pages. See the SGV cache documentation (http://scst.sourceforge.net/sgv_cache.txt) for more details. - - sgv_purge_interval - sets the SGV cache purging interval. I.e. an SG + SCST_USER_REGISTER_DEVICE returns registered device's handler or -1 in case of error, and errno is set appropriately. @@ -129,9 +148,10 @@ In order to unregister the device, either call SCST_USER_UNREGISTER_DEVICE function, or close on its file descriptor. - 2. SCST_USER_UNREGISTER_DEVICE + SCST_USER_UNREGISTER_DEVICE -SCST_USER_UNREGISTER_DEVICE - unregisters the corresponding virtual user +

+SCST_USER_UNREGISTER_DEVICE unregisters the corresponding virtual user space device. It doesn't have any parameters. During execution of SCST_USER_UNREGISTER_DEVICE at least one another @@ -143,14 +163,16 @@ SCST_USER_UNREGISTER_DEVICE returns 0 on success or -1 in case of error, and errno is set appropriately. - 3. SCST_USER_SET_OPTIONS/SCST_USER_GET_OPTIONS + SCST_USER_SET_OPTIONS/SCST_USER_GET_OPTIONS +

SCST_USER_SET_OPTIONS/SCST_USER_GET_OPTIONS allows to set or get correspondingly various options that control various aspects of SCSI commands processing. The argument is: + struct scst_user_opt { uint8_t parse_type; @@ -167,51 +189,59 @@ struct scst_user_opt uint8_t has_own_order_mgmt; }, + where: - - parse_type - defines how the user space handler wants to process + + + - - on_free_cmd_type - defines how the user space handler wants to + + - - memory_reuse_type - defines how memory allocated by the user space + + - - partial_transfers_type - defines if the user space handler supports + + - - tst, queue_alg, tas, swp, d_sense - set values for TST, QUEUE ALGORITHM MODIFIER, - TAS, SWP and D_SENSE fields from control mode page correspondingly, + -Flags parse_type and on_free_cmd_type are designed to improve +Flags SCST_USER_REPLY_AND_GET_CMD +

SCST_USER_REPLY_AND_GET_CMD allows at one call reply on the current subcommand from SCST and get the next one. If 0 is returned by ioctl(), SCST_USER_REPLY_AND_GET_CMD returns a SCST subcommand in the argument, which is defined as the following: + struct scst_user_get_cmd { uint32_t cmd_h; @@ -280,16 +315,19 @@ struct scst_user_get_cmd struct scst_user_tm tm_cmd; }; }, + where: - - cmd_h - command handle used to identify the command in the reply. + + Other union members contain command's specific payload. @@ -299,15 +337,16 @@ that the subcommand's processing is finished, although some subcommands don't return a value. - 4.1. Possible subcommands: +Possible subcommands: + SCST_USER_ATTACH_SESS - 4.1.1. SCST_USER_ATTACH_SESS - +

SCST_USER_ATTACH_SESS notifies the user space handler that a new initiator's session is about to be attached to the device. Payload contains struct scst_user_sess, which is defined as the following: + struct scst_user_sess { uint64_t sess_h; @@ -319,28 +358,31 @@ struct scst_user_sess char initiator_name[SCST_MAX_NAME]; char target_name[SCST_MAX_NAME]; }, + where: - - sess_h - session's handle, may not be 0 + + When SCST_USER_ATTACH_SESS is returned, it is guaranteed that there are no other commands are being executed or pending. @@ -350,8 +392,9 @@ handler shall reply using "result" field of the corresponding reply command. - 4.1.2. SCST_USER_DETACH_SESS + SCST_USER_DETACH_SESS +

SCST_USER_DETACH_SESS notifies the user space handler that the corresponding initiator is about to be detached from the particular device. Payload contains struct scst_user_sess, where only handle field @@ -365,8 +408,9 @@ SCST_USER_REPLY_AND_GET_CMD or SCST_USER_REPLY_CMD function must be called. - 4.1.3. SCST_USER_PARSE + SCST_USER_PARSE +

SCST_USER_PARSE returns SCSI command on PARSE state of the SCST processing. The PARSE state is intended to check validity of the command, determine data transfer type and the necessary data buffer @@ -378,6 +422,7 @@ job. Payload contains struct scst_user_scsi_cmd_parse, which is defined as the following: + struct scst_user_scsi_cmd_parse { uint64_t sess_h; @@ -401,65 +446,70 @@ struct scst_user_scsi_cmd_parse uint32_t sn; }, + where: - - sess_h - corresponding session handler + + Bits of scst_cdb_flags can be: - - SCST_TRANSFER_LEN_TYPE_FIXED - this command uses fixed blocks addressing + + In the PARSE state of SCSI commands processing the user space device handler shall check and provide SCST values for command data buffer @@ -471,8 +521,9 @@ SCST_USER_EXEC subcommand, where the appropriate SAM status and sense shall be set. - 4.4.4. SCST_USER_ALLOC_MEM + SCST_USER_ALLOC_MEM +

SCST_USER_ALLOC_MEM returns SCSI command on memory allocation state of the SCST processing. On this state the user space device handler shall allocate the command's data buffer with bufflen length and then return @@ -489,6 +540,7 @@ function. Payload contains struct scst_user_scsi_cmd_alloc_mem, which is defined as the following: + struct scst_user_scsi_cmd_alloc_mem { uint64_t sess_h; @@ -503,29 +555,32 @@ struct scst_user_scsi_cmd_alloc_mem uint32_t sn; }, + where: - - sess_h - corresponding session handler + + Memory allocation, preparation and freeing are ones of the most complicated and expensive operations during SCSI commands processing. Module scst_user provides a way to almost completely eliminate those operations by reusing once allocated memory for subsequent SCSI -commands. It is controlled by memory_reuse_type option, which could be +commands. It is controlled by for more details. Since the SGV cache caches SG vectors, which can be bigger, than actual @@ -552,26 +607,29 @@ cached ones are busy. Some important notes: -1. If the user space handler needs to call fork(), it must call + + If the user space handler needs to call fork(), it must call madvise() with MADV_DONTFORK flag for all allocated data buffers, otherwise parent or child process could loose the connection with them, -which could lead to data corruption. See http://lwn.net/Articles/171941/ +which could lead to data corruption. See for details. -2. The interface assumes that all allocated memory by the user space + The interface assumes that all allocated memory by the user space handler is DMA'able by the target hardware. This is almost always true for most modern systems, except if the target hardware isn't capable of using 64-bit address space and the system has >4GB of memory or the memory addresses are in address space, which is unavailable with 32-bit addresses. + In case of any error the error reporting should be deferred until SCST_USER_EXEC subcommand, where the appropriate SAM status and sense should be set. - 4.4.5. SCST_USER_EXEC + SCST_USER_EXEC +

SCST_USER_EXEC returns SCSI command on execution state of the SCST processing. The user space handler should execute the SCSI command and reply using the corresponding reply command. @@ -588,6 +646,7 @@ SCST_USER_ALLOC_MEM, apply to SCST_USER_EXEC as well. Payload contains struct scst_user_scsi_cmd_exec, which is defined as the following: + struct scst_user_scsi_cmd_exec { uint64_t sess_h; @@ -613,58 +672,63 @@ struct scst_user_scsi_cmd_exec int32_t parent_cmd_data_len; uint32_t partial_offset; }, + where: - - sess_h - corresponding session handler + + + - - timeout - CDB execution timeout + It is guaranteed that only commands of the same queue_type per session can be returned simultaneously. @@ -676,19 +740,21 @@ the user space handler or will be passed in the future, will be aborted by scst_user, the user space handler should ignore them. - 4.4.6. SCST_USER_ON_FREE_CMD + SCST_USER_ON_FREE_CMD +

SCST_USER_ON_FREE_CMD returns SCSI command when the command is about to be freed. At this stage, the user space device handler could do any necessary cleanups, for instance, free allocated for data buffer memory. -NOTE! If the memory reusage is enabled, then the data buffer must not be + struct scst_user_scsi_on_free_cmd { uint64_t pbuf; @@ -698,33 +764,39 @@ struct scst_user_scsi_on_free_cmd uint8_t status; uint8_t delivery_status; }, + where: - - pbuf - pointer to command's data buffer or 0 for SCSI commands + + + + The user space handler should reply using the corresponding reply command. No error code is needed. - 4.4.7. SCST_USER_ON_CACHED_MEM_FREE + SCST_USER_ON_CACHED_MEM_FREE +

SCST_USER_ON_CACHED_MEM_FREE subcommand is returned, when SGV cache decided that this buffer isn't needed anymore. This happens after some time of inactivity or when the system is under memory pressure. @@ -732,21 +804,27 @@ time of inactivity or when the system is under memory pressure. Payload contains struct scst_user_on_cached_mem_free, which is defined as the following: + struct scst_user_scsi_cmd_alloc_mem { uint64_t pbuf; }, + where: - - pbuf - pointer to buffer, which should be freed + + - 4.4.8. SCST_USER_TASK_MGMT + SCST_USER_TASK_MGMT +

SCST_USER_TASK_MGMT subcommand returns a task management functions. Payload contains struct scst_user_tm, which is defined as the following: + struct scst_user_tm { uint64_t sess_h; @@ -755,21 +833,24 @@ struct scst_user_tm uint32_t cmd_sn; uint8_t cmd_sn_set; }, + where: - - sess_h - corresponding session handler + + After TM function is completed, the user space device handler shall reply using "result" field of the corresponding reply command. It isn't @@ -778,47 +859,53 @@ function shall be returned immediately. SCST core will take care that the reply to the TM function isn't sent before all affection SCSI commands finished. -Possible values of "fn" field: +Possible values of + Possible return values are: - - SCST_MGMT_STATUS_SUCCESS - success + + - 5. SCST_USER_REPLY_CMD + SCST_USER_REPLY_CMD +

SCST_USER_REPLY_CMD IOCTL function allows the user space handler to return the result of a command's execution. Its argument is defined as: + struct scst_user_reply_cmd { uint32_t cmd_h; @@ -830,18 +917,23 @@ struct scst_user_reply_cmd struct scst_user_scsi_cmd_reply_exec exec_reply; }; }, + where: - - cmd_h - command handle used to identify the command in the reply. + + -Union contains the subcommand's specific payload: + struct scst_user_scsi_cmd_reply_parse { uint8_t queue_type; @@ -852,43 +944,53 @@ struct scst_user_scsi_cmd_reply_parse int32_t bufflen; int32_t out_bufflen; }, + where: - - queue_type - SCSI task attribute (queue type). NOTE! In current + + + struct scst_user_scsi_cmd_reply_alloc_mem { uint64_t pbuf; }, + where: - - pbuf - pointer to command's data buffer + + + struct scst_user_scsi_cmd_reply_exec { int32_t resp_data_len; @@ -900,34 +1002,40 @@ struct scst_user_scsi_cmd_reply_exec uint8_t sense_len; aligned_u64 psense_buffer; }, + where: - - resp_data_len - length of the response data + + + - - status - SAM status of the commands execution + - 6. SCST_USER_FLUSH_CACHE + SCST_USER_FLUSH_CACHE +

SCST_USER_FLUSH_CACHE - flushes SGV cache for the corresponding virtual user space device and queues for all cached memory buffers corresponding SCST_USER_ON_CACHED_MEM_FREE subcommands. @@ -942,8 +1050,9 @@ SCST_USER_FLUSH_CACHE returns 0 on success or -1 in case of error, and errno is set appropriately. - 7. SCST_USER_DEVICE_CAPACITY_CHANGED + SCST_USER_DEVICE_CAPACITY_CHANGED +

SCST_USER_DEVICE_CAPACITY_CHANGED - queues CAPACITY DATA HAS CHANGED Unit Attention or corresponding Asynchronous Event to the corresponding virtual device. It will notify remote initiators, connected to the @@ -956,153 +1065,181 @@ SCST_USER_DEVICE_CAPACITY_CHANGED returns 0 on success or -1 in case of error, and errno is set appropriately. - 8. SCST_USER_GET_EXTENDED_CDB + SCST_USER_GET_EXTENDED_CDB +

SCST_USER_GET_EXTENDED_CDB - requests extended CDB, if CDB size is more than SCST_MAX_CDB_SIZE bytes. In this case SCST_USER_GET_EXTENDED_CDB returns additional CDB data beyond SCST_MAX_CDB_SIZE bytes. SCST_USER_GET_EXTENDED_CDB has the following arguments: + struct scst_user_get_ext_cdb { uint32_t cmd_h; aligned_u64 ext_cdb_buffer; }, + where: - - cmd_h - command handle used to identify the command in the reply. + + SCST_USER_GET_EXTENDED_CDB returns 0 on success or -1 in case of error, and errno is set appropriately. - 9. SCST_USER_PREALLOC_BUFFER + SCST_USER_PREALLOC_BUFFER +

SCST_USER_PREALLOC_BUFFER - asks to preallocate a buffer. It has the following arguments: + union scst_user_prealloc_buffer { struct scst_user_prealloc_buffer_in in; struct scst_user_prealloc_buffer_out out; }, + where: - - in - provides data about buffer to preallocate + + -Structure scst_user_prealloc_buffer_in has the following definition: +Structure +struct scst_user_prealloc_buffer_in { aligned_u64 pbuf; uint32_t bufflen; uint8_t for_clust_pool; }, + where: - - pbuf - pointer to the buffer + + -Structure scst_user_prealloc_buffer_out has the following definition: +Structure struct scst_user_prealloc_buffer_out { uint32_t cmd_h; } + where: - - cmd_h - handle used to identify the buffer in + + SCST_USER_PREALLOC_BUFFER returns 0 on success or -1 in case of error, and errno is set appropriately. - IV. Commands processing flow example. + Commands processing flow example. +

As the example consider a simple synchronous VTL, which serves one virtual SCSI tape device and can process only one command at time from any initiator. -At the beginning the VTL opens using open() call /dev/scst_user in the + + + At the beginning the VTL opens using Then it using Then it prepares struct + -Then it prepares struct scst_user_get_cmd with reply set to the prepared -scst_user_reply_cmd structure, calls SCST_USER_REPLY_AND_GET_CMD ioctl() + Then it prepares If the received SCSI command is READ-type one, SCST does the necessary +preparations, then the VTL receives + -Then it prepares struct scst_user_get_cmd with reply set to the prepared -scst_user_reply_cmd structure, calls SCST_USER_REPLY_AND_GET_CMD ioctl() -and waits for the next SCSI command arrives from the initiator. That's -all for this SCSI command. For the next one the used data buffer will be -reused. + Then it prepares That's all for this SCSI command. For the next command the used +data buffer will be reused. + + For WRITE-type SCSI commands the processing is the same, but -SCST_USER_ALLOC_MEM will be returned before SCST_USER_EXEC, since the +