- Some reference leaks fixed

- Updated to compile on pre-2.6.25 kernels
 - Small docs updates



git-svn-id: http://svn.code.sf.net/p/scst/svn/trunk@1570 d57e44dd-8a1f-0410-8b47-8ef2f437770f
This commit is contained in:
Vladislav Bolkhovitin
2010-03-30 11:28:20 +00:00
parent f2b01c8595
commit 4e53e166b4
11 changed files with 267 additions and 77 deletions

View File

@@ -618,10 +618,7 @@ as well.
3. ISCSI initiators from pre-CentOS/RHEL 5 reported to have some
performance problems. If you use it, it is strongly advised to upgrade.
4. Pay attention to have io_grouping_type option set correctly. See SCST
core's README for more details.
5. If you are going to use your target in an VM environment, for
4. If you are going to use your target in an VM environment, for
instance as a shared storage with VMware, make sure all your VMs
connected to the target via *separate* sessions, i.e. each VM has own
connection to the target, not all VMs connected using a single
@@ -629,7 +626,7 @@ connection. You can check it using SCST proc or sysfs interface. If you
miss it, you can greatly loose performance of parallel access to your
target from different VMs.
6. Many dual port network adapters are not able to transfer data
5. Many dual port network adapters are not able to transfer data
simultaneousy on both ports, i.e. they transfer data via both ports on
the same speed as via any single port. Thus, using such adapters in MPIO
configuration can't improve performance. To allow MPIO to have double
@@ -638,6 +635,9 @@ dual-port adapter capable to to transfer data simultaneousy on both
ports. You can check it by running 2 iperf's through both ports in
parallel.
6. See SCST core's README for more advices. Especially pay attention to
have io_grouping_type option set correctly.
Compilation options
-------------------

View File

@@ -527,10 +527,7 @@ as well.
3. ISCSI initiators built in pre-CentOS/RHEL 5 reported to have some
performance problems. If you use it, it is strongly advised to upgrade.
4. Pay attention to have io_grouping_type option set correctly. See SCST
core's README for more details.
5. If you are going to use your target in an VM environment, for
4. If you are going to use your target in an VM environment, for
instance as a shared storage with VMware, make sure all your VMs
connected to the target via *separate* sessions, i.e. each VM has own
connection to the target, not all VMs connected using a single
@@ -538,7 +535,7 @@ connection. You can check it using SCST proc or sysfs interface. If you
miss it, you can greatly loose performance of parallel access to your
target from different VMs.
6. Many dual port network adapters are not able to transfer data
5. Many dual port network adapters are not able to transfer data
simultaneousy on both ports, i.e. they transfer data via both ports on
the same speed as via any single port. Thus, using such adapters in MPIO
configuration can't improve performance. To allow MPIO to have double
@@ -547,6 +544,9 @@ dual-port adapter capable to to transfer data simultaneousy on both
ports. You can check it by running 2 iperf's through both ports in
parallel.
6. See SCST core's README for more advices. Especially pay attention to
have io_grouping_type option set correctly.
Compilation options
-------------------

View File

@@ -424,10 +424,7 @@ The resulting overall SCST sysfs hierarchy with initiator
Performance advices
-------------------
1. Pay attention to have io_grouping_type option set correctly. See
SCST core's README for more details.
2. If you are going to use your target in an VM environment, for
1. If you are going to use your target in an VM environment, for
instance as a shared storage with VMware, make sure all your VMs
connected to the target via *separate* sessions. You can check it using
SCST proc or sysfs interface. You should use available facilities, like
@@ -435,6 +432,9 @@ NPIV, to make separate sessions for each VM. If you miss it, you can
greatly loose performance of parallel access to your target from
different VMs.
2. See SCST core's README for more advices. Especially pay attention to
have io_grouping_type option set correctly.
Credits
-------

View File

@@ -726,7 +726,7 @@ Every target should have at least the following entries:
transports in MPIO configurations you should either use value
"this_group_only", or an explicit I/O group number. This attribute is
also available in the initiators security groups, so you can assign
the I/O groupping on per-initiator basis. See below for more info how
the I/O grouping on per-initiator basis. See below for more info how
to use this attribute.
- rel_tgt_id - allows to read or write SCSI Relative Target Port
@@ -1446,6 +1446,12 @@ you, so the resulting performance will, in average, be better
(sometimes, much better) than with other SCSI targets. But in some cases
you can by manual tuning improve it even more.
If you want to get maximum performance from your target, RHEL/CentOS 5.x
kernels are not recommended, because they are based on very outdated
2.6.18 kernel, hence, missed >3 years of important improvements in the
kernel's storage area. You should use at least long maintained vanilla
2.6.27.x kernel, although 2.6.29+ would be even better.
Before doing any performance measurements note that performance results
are very much dependent from your type of load, so it is crucial that
you choose access mode (FILEIO, BLOCKIO, O_DIRECT, pass-through), which

View File

@@ -656,7 +656,7 @@ Every target should have at least the following entries:
transports in MPIO configurations you should either use value
"this_group_only", or an explicit I/O group number. This attribute is
also available in the initiators security groups, so you can assign
the I/O groupping on per-initiator basis. See below for more info how
the I/O grouping on per-initiator basis. See below for more info how
to use this attribute.
- rel_tgt_id - allows to read or write SCSI Relative Target Port

View File

@@ -1447,6 +1447,14 @@ struct scst_cmd_threads {
struct io_context *io_context;
#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 25)
/*
* Those kernels don't support ref counting based IO context sharing
* between threads/processes, so need own ref counting.
*/
struct kref *io_context_kref;
#endif
int nr_threads;
struct list_head threads_list;
@@ -1962,6 +1970,16 @@ struct scst_thr_data_hdr {
void (*free_fn) (struct scst_thr_data_hdr *data);
};
/*
* Used to clearly dispose async io_context
*/
struct scst_async_io_context_keeper {
struct kref aic_keeper_kref;
struct io_context *aic;
struct task_struct *aic_keeper_thr;
wait_queue_head_t aic_keeper_waitQ;
};
/*
* Used to store per-session specific device information
*/
@@ -2017,8 +2035,18 @@ struct scst_tgt_dev {
/* Pointer to lists of commands with the lock */
struct scst_cmd_threads *active_cmd_threads;
/* Lists of commands with lock, if dedicated threads are used */
struct scst_cmd_threads tgt_dev_cmd_threads;
/* Union to save some CPU cache footprint */
union {
struct {
/* Copy to save fast path dereference */
struct io_context *async_io_context;
struct scst_async_io_context_keeper *aic_keeper;
};
/* Lists of commands with lock, if dedicated threads are used */
struct scst_cmd_threads tgt_dev_cmd_threads;
};
spinlock_t tgt_dev_lock; /* per-session device lock */

View File

@@ -29,6 +29,7 @@
#include <linux/string.h>
#include <asm/kmap_types.h>
#include <linux/ctype.h>
#include <linux/delay.h>
#include "scst.h"
#include "scst_priv.h"
@@ -45,6 +46,19 @@ struct scsi_io_context {
static struct kmem_cache *scsi_io_context_cache;
#endif
#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 21)
static int strncasecmp(const char *s1, const char *s2, size_t n)
{
int c1, c2;
do {
c1 = tolower(*s1++);
c2 = tolower(*s2++);
} while ((--n > 0) && c1 == c2 && c1 != 0);
return c1 - c2;
}
#endif
/* get_trans_len_x extract x bytes from cdb as length starting from off */
static int get_trans_len_1(struct scst_cmd *cmd, uint8_t off);
static int get_trans_len_1_256(struct scst_cmd *cmd, uint8_t off);
@@ -2269,10 +2283,10 @@ struct scst_acg *scst_tgt_find_acg(struct scst_tgt *tgt, const char *name)
}
/* scst_mutex supposed to be held */
static struct io_context *scst_find_shared_io_context(
static struct scst_tgt_dev *scst_find_shared_io_tgt_dev(
struct scst_tgt_dev *tgt_dev)
{
struct io_context *res = NULL;
struct scst_tgt_dev *res = NULL;
struct scst_acg *acg = tgt_dev->acg_dev->acg;
struct scst_tgt_dev *t;
@@ -2343,23 +2357,24 @@ out:
found:
if (t->active_cmd_threads == &scst_main_cmd_threads) {
res = t->tgt_dev_cmd_threads.io_context;
TRACE_MGMT_DBG("Going to share async IO context %p (t %p, "
"ini %s, dev %s, cmd_threads %p, grouping type %d)",
res, t, t->sess->initiator_name, t->dev->virt_name,
&t->tgt_dev_cmd_threads,
res = t;
TRACE_MGMT_DBG("Going to share async IO context %p (res %p, "
"ini %s, dev %s, grouping type %d)",
t->aic_keeper->aic, res, t->sess->initiator_name,
t->dev->virt_name,
t->acg_dev->acg->acg_io_grouping_type);
} else {
res = t->active_cmd_threads->io_context;
if (res == NULL) {
res = t;
if (res->active_cmd_threads->io_context == NULL) {
TRACE_MGMT_DBG("IO context for t %p not yet "
"initialized, waiting...", t);
msleep(100);
barrier();
goto found;
}
TRACE_MGMT_DBG("Going to share IO context %p (t %p, ini %s, "
"dev %s, cmd_threads %p, grouping type %d)", res, t,
TRACE_MGMT_DBG("Going to share IO context %p (res %p, ini %s, "
"dev %s, cmd_threads %p, grouping type %d)",
res->active_cmd_threads->io_context, res,
t->sess->initiator_name, t->dev->virt_name,
t->active_cmd_threads,
t->acg_dev->acg->acg_io_grouping_type);
@@ -2388,11 +2403,50 @@ enum scst_dev_type_threads_pool_type scst_parse_threads_pool_type(const char *p,
return res;
}
static int scst_ioc_keeper_thread(void *arg)
{
struct scst_async_io_context_keeper *aic_keeper =
(struct scst_async_io_context_keeper *)arg;
TRACE_ENTRY();
TRACE_MGMT_DBG("AIC %p keeper thread %s (PID %d) started", aic_keeper,
current->comm, current->pid);
current->flags |= PF_NOFREEZE;
sBUG_ON(aic_keeper->aic != NULL);
#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 25)
aic_keeper->aic = get_io_context(GFP_KERNEL);
#else
aic_keeper->aic = get_io_context(GFP_KERNEL, -1);
#endif
TRACE_MGMT_DBG("Alloced new async IO context %p (aic %p)",
aic_keeper->aic, aic_keeper);
/* We have our own ref counting */
put_io_context(aic_keeper->aic);
/* We are ready */
wake_up_all(&aic_keeper->aic_keeper_waitQ);
wait_event_interruptible(aic_keeper->aic_keeper_waitQ,
kthread_should_stop());
TRACE_MGMT_DBG("AIC %p keeper thread %s (PID %d) finished", aic_keeper,
current->comm, current->pid);
TRACE_EXIT();
return 0;
}
/* scst_mutex supposed to be held */
int scst_tgt_dev_setup_threads(struct scst_tgt_dev *tgt_dev)
{
int res = 0;
struct scst_device *dev = tgt_dev->dev;
struct scst_async_io_context_keeper *aic_keeper;
TRACE_ENTRY();
@@ -2400,31 +2454,52 @@ int scst_tgt_dev_setup_threads(struct scst_tgt_dev *tgt_dev)
tgt_dev->active_cmd_threads = &scst_main_cmd_threads;
if (dev->threads_num == 0) {
struct io_context *shared_io_context;
struct scst_tgt_dev *shared_io_tgt_dev;
shared_io_tgt_dev = scst_find_shared_io_tgt_dev(tgt_dev);
if (shared_io_tgt_dev != NULL) {
aic_keeper = shared_io_tgt_dev->aic_keeper;
kref_get(&aic_keeper->aic_keeper_kref);
shared_io_context = scst_find_shared_io_context(tgt_dev);
if (shared_io_context != NULL) {
TRACE_MGMT_DBG("Linking async io context %p "
"for shared tgt_dev %p (cmd_threads "
"%p, dev %s)", shared_io_context,
tgt_dev, &tgt_dev->tgt_dev_cmd_threads,
"for shared tgt_dev %p (dev %s)",
aic_keeper->aic, tgt_dev,
tgt_dev->dev->virt_name);
tgt_dev->tgt_dev_cmd_threads.io_context =
ioc_task_link(shared_io_context);
} else {
/* Create new context */
struct io_context *io_context = current->io_context;
current->io_context = NULL;
tgt_dev->tgt_dev_cmd_threads.io_context =
ioc_task_link(get_io_context(GFP_KERNEL, -1));
current->io_context = io_context;
aic_keeper = kzalloc(sizeof(*aic_keeper),
GFP_KERNEL);
if (aic_keeper == NULL) {
PRINT_ERROR("Unable to alloc aic_keeper "
"(size %d)", sizeof(*aic_keeper));
res = -ENOMEM;
goto out;
}
kref_init(&aic_keeper->aic_keeper_kref);
init_waitqueue_head(&aic_keeper->aic_keeper_waitQ);
aic_keeper->aic_keeper_thr =
kthread_run(scst_ioc_keeper_thread,
aic_keeper, "aic_keeper");
if (IS_ERR(aic_keeper->aic_keeper_thr)) {
PRINT_ERROR("Error running ioc_keeper "
"thread (tgt_dev %p)", tgt_dev);
res = PTR_ERR(aic_keeper->aic_keeper_thr);
goto out_free_keeper;
}
wait_event(aic_keeper->aic_keeper_waitQ,
aic_keeper->aic != NULL);
TRACE_MGMT_DBG("Created async io context %p "
"for not shared tgt_dev %p "
"(cmd_threads %p, dev %s)",
tgt_dev->tgt_dev_cmd_threads.io_context,
tgt_dev, &tgt_dev->tgt_dev_cmd_threads,
"for not shared tgt_dev %p (dev %s)",
aic_keeper->aic, tgt_dev,
tgt_dev->dev->virt_name);
}
tgt_dev->async_io_context = aic_keeper->aic;
tgt_dev->aic_keeper = aic_keeper;
}
res = scst_add_threads(tgt_dev->active_cmd_threads, NULL, NULL,
@@ -2435,23 +2510,60 @@ int scst_tgt_dev_setup_threads(struct scst_tgt_dev *tgt_dev)
switch (dev->threads_pool_type) {
case SCST_THREADS_POOL_PER_INITIATOR:
{
struct io_context *shared_io_context;
struct scst_tgt_dev *shared_io_tgt_dev;
scst_init_threads(&tgt_dev->tgt_dev_cmd_threads);
tgt_dev->active_cmd_threads = &tgt_dev->tgt_dev_cmd_threads;
shared_io_context = scst_find_shared_io_context(tgt_dev);
if (shared_io_context != NULL) {
shared_io_tgt_dev = scst_find_shared_io_tgt_dev(tgt_dev);
if (shared_io_tgt_dev != NULL) {
TRACE_MGMT_DBG("Linking io context %p for "
"shared tgt_dev %p (cmd_threads %p)",
shared_io_context, tgt_dev,
tgt_dev->active_cmd_threads);
shared_io_tgt_dev->active_cmd_threads->io_context,
tgt_dev, tgt_dev->active_cmd_threads);
/* It's ref counted via threads */
tgt_dev->active_cmd_threads->io_context =
ioc_task_link(shared_io_context);
shared_io_tgt_dev->active_cmd_threads->io_context;
#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 25)
tgt_dev->active_cmd_threads->io_context_kref =
shared_io_tgt_dev->active_cmd_threads->io_context_kref;
#endif
}
#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 25)
else {
struct kref *io_context_kref;
io_context_kref = kmalloc(sizeof(*io_context_kref),
GFP_KERNEL);
if (io_context_kref == NULL) {
PRINT_ERROR("Unable to alloc io_context_kref "
"(size %d)", sizeof(*io_context_kref));
res = -ENOMEM;
goto out;
}
kref_init(io_context_kref);
tgt_dev->tgt_dev_cmd_threads.io_context_kref =
io_context_kref;
}
#endif
res = scst_add_threads(tgt_dev->active_cmd_threads, NULL,
tgt_dev,
dev->threads_num + tgt_dev->sess->tgt->tgtt->threads_num);
if (res != 0) {
/* Let's clear here, because no threads could be run */
tgt_dev->active_cmd_threads->io_context = NULL;
#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 25)
if (shared_io_tgt_dev == NULL) {
if (tgt_dev->active_cmd_threads->io_context_kref != NULL) {
kfree(tgt_dev->active_cmd_threads->io_context_kref);
tgt_dev->active_cmd_threads->io_context_kref = NULL;
}
}
#endif
}
break;
}
case SCST_THREADS_POOL_SHARED:
@@ -2473,6 +2585,27 @@ out:
TRACE_EXIT_RES(res);
return res;
out_free_keeper:
kfree(aic_keeper);
goto out;
}
static void scst_aic_keeper_release(struct kref *kref)
{
struct scst_async_io_context_keeper *aic_keeper;
TRACE_ENTRY();
aic_keeper = container_of(kref, struct scst_async_io_context_keeper,
aic_keeper_kref);
kthread_stop(aic_keeper->aic_keeper_thr);
kfree(aic_keeper);
TRACE_EXIT();
return;
}
/* scst_mutex supposed to be held */
@@ -2482,10 +2615,10 @@ void scst_tgt_dev_stop_threads(struct scst_tgt_dev *tgt_dev)
if (tgt_dev->active_cmd_threads == &scst_main_cmd_threads) {
/* Global async threads */
scst_del_threads(tgt_dev->active_cmd_threads,
tgt_dev->sess->tgt->tgtt->threads_num);
put_io_context(tgt_dev->tgt_dev_cmd_threads.io_context);
tgt_dev->tgt_dev_cmd_threads.io_context = NULL;
kref_put(&tgt_dev->aic_keeper->aic_keeper_kref,
scst_aic_keeper_release);
tgt_dev->async_io_context = NULL;
tgt_dev->aic_keeper = NULL;
} else if (tgt_dev->active_cmd_threads == &tgt_dev->dev->dev_cmd_threads) {
/* Per device shared threads */
scst_del_threads(tgt_dev->active_cmd_threads,
@@ -2493,6 +2626,7 @@ void scst_tgt_dev_stop_threads(struct scst_tgt_dev *tgt_dev)
} else if (tgt_dev->active_cmd_threads == &tgt_dev->tgt_dev_cmd_threads) {
/* Per tgt_dev threads */
scst_del_threads(tgt_dev->active_cmd_threads, -1);
scst_deinit_threads(&tgt_dev->tgt_dev_cmd_threads);
} /* else no threads (not yet initialized, e.g.) */
tm_dbg_deinit_tgt_dev(tgt_dev);
@@ -2576,8 +2710,6 @@ static struct scst_tgt_dev *scst_alloc_add_tgt_dev(struct scst_session *sess,
for (i = 0; i < (int)ARRAY_SIZE(tgt_dev->sn_slots); i++)
atomic_set(&tgt_dev->sn_slots[i], 0);
scst_init_threads(&tgt_dev->tgt_dev_cmd_threads);
if (dev->handler->parse_atomic &&
(sess->tgt->tgtt->preprocessing_done == NULL)) {
if (sess->tgt->tgtt->rdy_to_xfer_atomic)
@@ -2700,8 +2832,6 @@ static void scst_free_tgt_dev(struct scst_tgt_dev *tgt_dev)
scst_tgt_dev_stop_threads(tgt_dev);
scst_deinit_threads(&tgt_dev->tgt_dev_cmd_threads);
sBUG_ON(!list_empty(&tgt_dev->thr_data_list));
kmem_cache_free(scst_tgtd_cachep, tgt_dev);

View File

@@ -1407,8 +1407,7 @@ void scst_del_threads(struct scst_cmd_threads *cmd_threads, int num)
struct scst_tgt_dev *tgt_dev;
list_for_each_entry(tgt_dev, &dev->dev_tgt_dev_list,
dev_tgt_dev_list_entry) {
if (scst_del_thr_data(tgt_dev, ct->cmd_thread))
break;
scst_del_thr_data(tgt_dev, ct->cmd_thread);
}
}
@@ -1421,10 +1420,8 @@ void scst_del_threads(struct scst_cmd_threads *cmd_threads, int num)
break;
}
if (cmd_threads->nr_threads == 0) {
put_io_context(cmd_threads->io_context);
cmd_threads->io_context = NULL;
}
EXTRACHECKS_BUG_ON((cmd_threads->nr_threads == 0) &&
(cmd_threads->io_context != NULL));
out:
TRACE_EXIT();

View File

@@ -200,13 +200,13 @@ static inline bool scst_set_io_context(struct scst_cmd *cmd,
if (cmd->cmd_threads == &scst_main_cmd_threads) {
EXTRACHECKS_BUG_ON(in_interrupt());
/*
* No need to call ioc_task_link(), because io_context
* No need for any ref counting action, because io_context
* supposed to be cleared in the end of the caller function.
*/
current->io_context = cmd->tgt_dev->tgt_dev_cmd_threads.io_context;
current->io_context = cmd->tgt_dev->async_io_context;
res = true;
TRACE_DBG("io_context %p (cmd_threads %p)", current->io_context,
&cmd->tgt_dev->tgt_dev_cmd_threads);
TRACE_DBG("io_context %p (tgt_dev %p)", current->io_context,
cmd->tgt_dev);
EXTRACHECKS_BUG_ON(current->io_context == NULL);
} else
res = false;

View File

@@ -899,7 +899,7 @@ static ssize_t scst_device_sysfs_threads_num_store(struct kobject *kobj,
goto out;
}
if (newtn <= 0) {
if (newtn < 0) {
PRINT_ERROR("Illegal threads num value %ld", newtn);
res = -EINVAL;
goto out;

View File

@@ -3739,6 +3739,13 @@ static inline int test_cmd_threads(struct scst_cmd_threads *p_cmd_threads)
return res;
}
#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 25)
static void kref_free(struct kref *kref)
{
kfree(kref);
}
#endif
int scst_cmd_thread(void *arg)
{
struct scst_cmd_threads *p_cmd_threads = (struct scst_cmd_threads *)arg;
@@ -3760,19 +3767,30 @@ int scst_cmd_thread(void *arg)
if (p_cmd_threads != &scst_main_cmd_threads) {
if (p_cmd_threads->io_context == NULL) {
p_cmd_threads->io_context = ioc_task_link(
get_io_context(GFP_KERNEL, -1));
#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 25)
p_cmd_threads->io_context = get_io_context(GFP_KERNEL);
#else
p_cmd_threads->io_context = get_io_context(GFP_KERNEL, -1);
#endif
TRACE_MGMT_DBG("Alloced new IO context %p "
"(p_cmd_threads %p)",
p_cmd_threads->io_context,
p_cmd_threads);
/* It's ref counted via threads */
put_io_context(p_cmd_threads->io_context);
#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 25)
/* p_cmd_threads->io_context_kref is already 1 */
#endif
} else {
put_io_context(current->io_context);
current->io_context = ioc_task_link(
p_cmd_threads->io_context);
#if LINUX_VERSION_CODE >= KERNEL_VERSION(2, 6, 25)
current->io_context = ioc_task_link(p_cmd_threads->io_context);
#else
current->io_context = p_cmd_threads->io_context;
kref_get(p_cmd_threads->io_context_kref);
#endif
TRACE_MGMT_DBG("Linked IO context %p "
"(p_cmd_threads %p)",
p_cmd_threads->io_context,
"(p_cmd_threads %p)", p_cmd_threads->io_context,
p_cmd_threads);
}
}
@@ -3814,6 +3832,17 @@ int scst_cmd_thread(void *arg)
EXTRACHECKS_BUG_ON((p_cmd_threads->nr_threads == 1) &&
!list_empty(&p_cmd_threads->active_cmd_list));
if ((p_cmd_threads->nr_threads == 1) &&
(p_cmd_threads != &scst_main_cmd_threads))
p_cmd_threads->io_context = NULL;
#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 25)
if (!kref_put(p_cmd_threads->io_context_kref, kref_free))
current->io_context = NULL;
else
p_cmd_threads->io_context_kref = NULL;
#endif
PRINT_INFO("Processing thread %s (PID %d) finished", current->comm,
current->pid);