mirrors/scst
1
0
Fork 0
mirror of https://github.com/SCST-project/scst.git synced 2026-08-17 04:36:22 +00:00
Code Issues Packages Projects Releases Wiki Activity

- Possibility to operate with single size entries as well as control of the reclamation time added to the SGV caache

Browse Source 
- Fileio_tgt updated to be able to use those neww facilities

 - Docs updated. Particulary, a new dociment describing the SGV cache added



git-svn-id: http://svn.code.sf.net/p/scst/svn/trunk@986 d57e44dd-8a1f-0410-8b47-8ef2f437770f
This commit is contained in:
Vladislav Bolkhovitin
2009-07-29 12:49:17 +00:00
parent e2b8393690
commit d257a9541d
12 changed files with 655 additions and 231 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/scst_user_spec.txt
+30 -21
View File
@@ -63,6 +63,8 @@ struct scst_user_dev_desc
uint8_t version;
uint8_t type;
uint8_t sgv_shared;
int32_t sgv_single_alloc_pages;
int32_t sgv_purge_interval
uint8_t has_own_order_mgmt;
struct scst_user_opt opt;
uint32_t block_size;
@@ -76,9 +78,22 @@ where:
- type - SCSI type of the device
- sgv_shared - true, if SGV cache for this device should be shared with
- sgv_shared - true, if the SGV cache for this device should be shared with
other devices. False, if the SGV cache should be dedicated.
- sgv_single_alloc_pages - if 0, then the SGV cache for this device will
work in the set of power 2 size buffers mode. If >0, then the SGV
cache will work in the fixed size buffers mode. In this case it sets
the size of each buffer in pages. See the SGV cache documentation
(http://scst.sourceforge.net/sgv_cache.txt) for more details.
- sgv_purge_interval - sets the SGV cache purging interval. I.e. an SG
buffer will be freed if it's unused for time t purge_interval <= t <
2*purge_interval. If purge_interval is 0, then the default interval
will be used (60 seconds). If purge_interval <0, then the automatic
purging will be disabled. Shrinking by the system's demand will also
be disabled.
- has_own_order_mgmt - set it in non-zero, if device implements own
ORDERED commands management, i.e. guarantees commands execution order
requirements, specified by SAM.
@@ -469,23 +484,17 @@ allocated and fully built SG vectors for subsequent commands of this
type, so for them SCST_USER_ALLOC_MEM subfunction will not be called and
in SCST_USER_EXEC pbuf pointer will point to that reused buffer.
SGV cache is a backend cache made on top of Linux kernel SLAB cache. It
caches SG vectors with power of 2 data sizes: 4K, 8K, ..., 4M. So, if
there is a 5K SCSI command coming, then the user space handler will be
asked for 8K allocation, from which only 5K will be used for this
particular SCSI command. Then that SG vector will be cached and
subsequently reused for all SCSI commands between 4K and 8K. For a 1M
SCSI command the user space handler will be asked for another buffer
(while the previous 5K one will be kept), which will not be then reused
for 5K SCSI commands, since 1M and 5K vectors belong to different cache
data sizes, it will be reused only for commands between 512K and 1M.
SGV cache is a backend cache made on top of Linux kernel kmem cache. It
caches unused SG vectors for future allocations to improve performance.
Then, after some time of inactivity or when the system is under memory
pressure, the cache entries will be freed and the user space handler
will be notified using SCST_USER_ON_CACHED_MEM_FREE.
will be notified using SCST_USER_ON_CACHED_MEM_FREE. See the SGV cache
documentation (http://scst.sourceforge.net/sgv_cache.txt) for more
details.
Since SGV cache caches SG vectors with power of 2 data sizes, alloc_len
field could be up to 2 times more, than actually required by the SCSI
command.
Since the SGV cache caches SG vectors, which can be bigger, than actual
data sizes of SCSI commands, alloc_len field could also be bigger, than
actually required by the SCSI command.
The memory reuse could be used in both SCSI tagged and untagged queuing
environments. In the SCSI tagged queuing environment the SGV cache will
@@ -521,12 +530,12 @@ reply using the corresponding reply command.
In some cases for performance reasons for READ-type SCSI commands
SCST_USER_ALLOC_MEM subcommand isn't returned before SCST_USER_EXEC.
Thus, if pbuf pointer is 0 and the SCSI command needs data transfer,
the user space handler should be prepared to allocate the data buffer
with size alloc_len, which could be up to 2 times more, than actually
required by the SCSI command. But field bufflen will contain the correct
value. All the memory reusage rules, described for SCST_USER_ALLOC_MEM,
apply to SCST_USER_EXEC as well.
Thus, if pbuf pointer is 0 and the SCSI command needs data transfer, the
user space handler should be prepared to allocate the data buffer with
size alloc_len, which could be bigger (due to the SGV cache), than
actually required by the SCSI command. But field bufflen will contain
the correct value. All the memory reusage rules, described for
SCST_USER_ALLOC_MEM, apply to SCST_USER_EXEC as well.
Payload contains struct scst_user_scsi_cmd_exec, which is defined as the
following:
doc/sgv_cache.txt
+228
View File
@@ -0,0 +1,228 @@
SCST SGV CACHE.
PROGRAMMING INTERFACE DESCRIPTION.
For SCST version 1.0.2
SCST SGV cache is a memory management subsystem in SCST. One can call it
a "memory pool", but Linux kernel already have a mempool interface,
which serves different purposes. SGV cache provides to SCST core, target
drivers and backend dev handlers facilities to allocate and build SG
vectors for data buffers. The main advantage of it is that it doesn't
free to the system each vector, which is not used anymore, but keeps it
for a while to let it be reused by the next consecutive command. This
allows to reduce commands processing latency and, hence, improve
performance. The freed SG vectors are kept by the SGV cache either for
some predefined time, or until the system needs more memory and asks to
free some using the set_shrinker() interface. Also the SGV cache allows
to:
- Cluster pages together. "Cluster" means merging adjacent pages in a
single SG entry. It allows to have less SG entries in the resulting SG
vector, hence improve performance handling it as well as allow to
work with bigger buffers on hardware with limited SG capabilities.
- Set custom page allocator functions. For instance, scst_user device
handler uses this facility to eliminate unneeded mapping/unmapping of
user space pages and avoid unneeded IOCTL calls for buffers allocations.
In fileio_tgt application, which uses a regular malloc() function to
allocate data buffers, this facility allows ~30% less CPU load and
considerable performance increase.
- Prevent each initiator or all initiators altogether to allocate too
much memory and DoS the target. Consider 10 initiators, which can have
access to 10 devices each. Any of them can queue up to 64 commands, each
can transfer up to 1MB of data. So, all of them in a peak can allocate
up to 10*10*64 = ~6.5GB of memory for data buffers. This amount must be
limited somehow and the SGV cache performs this function.
From implementation POV the SGV cache is a simple extension of the kmem
cache. It can work in 2 modes:
1. With fixed size buffers.
2. With a set of power 2 size buffers. In this mode each SGV cache
(struct sgv_pool) has SGV_POOL_ELEMENTS (11 currently) of kmem caches.
Each of those kmem caches keeps SGV cache objects (struct sgv_pool_obj)
corresponding to SG vectors with size of order X pages. For instance,
request to allocate 4 pages will be served from kmem cache[2], since the
order of the of number of requested pages is 2. If later request to
allocate 11KB comes, the same SG vector with 4 pages will be reused (see
below). This mode is in average allows less memory overhead comparing
with the fixed size buffers mode.
Consider how the SGV cache works in the set of buffers mode. When a
request to allocate new SG vector comes, sgv_pool_alloc() via
sgv_get_obj() checks if there is already a cached vector with that
order. If yes, then that vector will be reused and its length, if
necessary, will be modified to match the requested size. In the above
example request for 11KB buffer, 4 pages vector will be reused and
modified using trans_tbl to contain 3 pages and the last entry will be
modified to contain the requested length - 2*PAGE_SIZE. If there is no
cached object, then a new sgv_pool_obj will be allocated from the
corresponding kmem cache, chosen by the order of number of requested
pages. Then that vector will be filled by pages and returned.
In the fixed size buffers mode the SGV cache works similarly, except
that it always allocate buffer with the predefined fixed size. I.e.
even for 4K request the whole buffer with predefined size, say, 1MB,
will be used.
In both modes, if size of a request exceeds the maximum allowed for
caching buffer size, the requested buffer will be allocated, but not
cached.
Freed cached sgv_pool_obj objects are actually freed to the system
either by the purge work, which is scheduled once in 60 seconds, or in
sgv_shrink() called by system, when it's asking for memory.
Interface.
struct sgv_pool *sgv_pool_create(const char *name,
enum sgv_clustering_types clustered, int single_alloc_pages,
bool shared, int purge_interval)
This function creates and initializes an SGV cache. It has the following
arguments:
- name - the name of the SGV cache
- clustered - sets type of the pages clustering. The type can be:
* sgv_no_clustering - no clustering performed.
* sgv_tail_clustering - a page will only be merged with the latest
previously allocated page, so the order of pages in the SG will be
preserved
* sgv_full_clustering - free merging of pages at any place in
the SG is allowed. This mode usually provides the best merging
rate.
- single_alloc_pages - if 0, then the SGV cache will work in the set of
power 2 size buffers mode. If >0, then the SGV cache will work in the
fixed size buffers mode. In this case single_alloc_pages sets the
size of each buffer in pages.
- shared - sets if the SGV cache can be shared between devices or not.
The cache sharing allowed only between devices created inside the same
address space. If an SGV cache is shared, each subsequent call of
sgv_pool_create() with the same cache name will not create a new cache,
but instead return a reference to it.
- purge_interval - sets the cache purging interval. I.e. an SG buffer
will be freed if it's unused for time t purge_interval <= t <
2*purge_interval. If purge_interval is 0, then the default interval
will be used (60 seconds). If purge_interval <0, then the automatic
purging will be disabled. Shrinking by the system's demand will also
be disabled.
Returns the resulting SGV cache or NULL in case of any error.
void sgv_pool_del(struct sgv_pool *pool)
This function deletes the corresponding SGV cache. If the cache is
shared, it will decrease its reference counter. If the reference counter
reaches 0, the cache will be destroyed.
void sgv_pool_flush(struct sgv_pool *pool)
This function flushes, i.e. frees, all the cached entries in the SGV
cache.
void sgv_pool_set_allocator(struct sgv_pool *pool,
struct page *(*alloc_pages_fn)(struct scatterlist *sg, gfp_t gfp, void *priv),
void (*free_pages_fn)(struct scatterlist *sg, int sg_count, void *priv));
This function allows to set for the SGV cache a custom pages allocator. For
instance, scst_user uses such function to supply to the cache mapped from
user space pages.
alloc_pages_fn() has the following parameters:
- sg - SG entry, to which the allocated page should be added.
- gfp - the allocation GFP flags
- priv - pointer to a private data supplied to sgv_pool_alloc()
This function should return the allocated page or NULL, if no page was
allocated.
free_pages_fn() has the following parameters:
- sg - SG vector to free
- sg_count - number of SG entries in the sg
- priv - pointer to a private data supplied to the corresponding sgv_pool_alloc()
struct scatterlist *sgv_pool_alloc(struct sgv_pool *pool, unsigned int size,
gfp_t gfp_mask, int flags, int *count,
struct sgv_pool_obj **sgv, struct scst_mem_lim *mem_lim, void *priv)
This function allocates an SG vector from the SGV cache. It has the
following parameters:
- pool - the cache to alloc from
- size - size of the resulting SG vector in bytes
- gfp_mask - the allocation mask
- flags - the allocation flags. The following flags are possible and
can be set using OR operation:
* SGV_POOL_ALLOC_NO_CACHED - the SG vector must not be cached.
* SGV_POOL_NO_ALLOC_ON_CACHE_MISS - don't do an allocation on a
cache miss.
* SGV_POOL_RETURN_OBJ_ON_ALLOC_FAIL - return an empty SGV object,
i.e. without the SG vector, if the allocation can't be completed.
For instance, because SGV_POOL_NO_ALLOC_ON_CACHE_MISS flag set.
- count - the resulting count of SG entries in the resulting SG vector.
- sgv - the resulting SGV object. It should be used to free the
resulting SG vector.
- mem_lim - memory limits, see below.
- priv - pointer to private for this allocation data. This pointer will
be supplied to alloc_pages_fn() and free_pages_fn() and can be
retrieved by sgv_get_priv().
This function returns pointer to the resulting SG vector or NULL in case
of any error.
void sgv_pool_free(struct sgv_pool_obj *sgv, struct scst_mem_lim *mem_lim)
This function frees previously allocated SG vector, referenced by SGV
cache object sgv.
void *sgv_get_priv(struct sgv_pool_obj *sgv)
This function allows to get the allocation private data for this SGV
cache object sgv. The private data are set by sgv_pool_alloc().
void scst_init_mem_lim(struct scst_mem_lim *mem_lim)
This function initializes memory limits structure mem_lim according to
the current system configuration. This structure should be latter used
to track and limit allocated by one or more SGV caches memory.
Runtime information and statistics.
Runtime information and statistics is available in /proc/scsi_tgt/sgv.
scst/include/scst_debug.h
+7 -7
View File
@@ -199,7 +199,7 @@ do { \
#ifdef CONFIG_SCST_DEBUG
#define __TRACE(trace, format, args...) \
#define TRACE_DBG_FLAG(trace, format, args...) \
do { \
if (trace_flag & (trace)) { \
char *__tflag = LOG_FLAG; \
@@ -211,13 +211,13 @@ do { \
} \
} while (0)
#define TRACE_MEM(args...) __TRACE(TRACE_MEMORY, args)
#define TRACE_SG(args...) __TRACE(TRACE_SG_OP, args)
#define TRACE_DBG(args...) __TRACE(TRACE_DEBUG, args)
#define TRACE_DBG_SPECIAL(args...) __TRACE(TRACE_DEBUG|TRACE_SPECIAL, args)
#define TRACE_MGMT_DBG(args...) __TRACE(TRACE_MGMT_DEBUG, args)
#define TRACE_MEM(args...) TRACE_DBG_FLAG(TRACE_MEMORY, args)
#define TRACE_SG(args...) TRACE_DBG_FLAG(TRACE_SG_OP, args)
#define TRACE_DBG(args...) TRACE_DBG_FLAG(TRACE_DEBUG, args)
#define TRACE_DBG_SPECIAL(args...) TRACE_DBG_FLAG(TRACE_DEBUG|TRACE_SPECIAL, args)
#define TRACE_MGMT_DBG(args...) TRACE_DBG_FLAG(TRACE_MGMT_DEBUG, args)
#define TRACE_MGMT_DBG_SPECIAL(args...) \
__TRACE(TRACE_MGMT_DEBUG|TRACE_SPECIAL, args)
TRACE_DBG_FLAG(TRACE_MGMT_DEBUG|TRACE_SPECIAL, args)
#define TRACE_BUFFER(message, buff, len) \
do { \
scst/include/scst_sgv.h
+108 -5
View File
@@ -19,16 +19,16 @@
#ifndef __SCST_SGV_H
#define __SCST_SGV_H
/* SGV pool routines and flag bits */
/** SGV pool routines and flag bits **/
/* Set if the allocated object must be not from the cache */
#define SCST_POOL_ALLOC_NO_CACHED 1
#define SGV_POOL_ALLOC_NO_CACHED 1
/* Set if there should not be any memory allocations on a cache miss */
#define SCST_POOL_NO_ALLOC_ON_CACHE_MISS 2
#define SGV_POOL_NO_ALLOC_ON_CACHE_MISS 2
/* Set an object should be returned even if it doesn't have SG vector built */
#define SCST_POOL_RETURN_OBJ_ON_ALLOC_FAIL 4
#define SGV_POOL_RETURN_OBJ_ON_ALLOC_FAIL 4
struct sgv_pool_obj;
struct sgv_pool;
@@ -46,27 +46,130 @@ struct scst_mem_lim {
/* Types of clustering */
enum sgv_clustering_types {
/* No clustering performed */
sgv_no_clustering = 0,
/*
* A page will only be merged with the latest previously allocated
* page, so the order of pages in the SG will be preserved.
*/
sgv_tail_clustering,
/*
* Free merging of pages at any place in the SG is allowed. This mode
* usually provides the best merging rate.
*/
sgv_full_clustering,
};
/**
* sgv_pool_create - creates and initializes an SGV cache
* @name: the name of the SGV cache
* @clustered: sets type of the pages clustering.
* @single_alloc_pages: if 0, then the SGV cache will work in the set of
* power 2 size buffers mode. If >0, then the SGV cache will
* work in the fixed size buffers mode. In this case
* single_alloc_pages sets the size of each buffer in pages.
* @shared: sets if the SGV cache can be shared between devices or not.
* The cache sharing allowed only between devices created inside
* the same address space. If an SGV cache is shared, each
* subsequent call of sgv_pool_create() with the same cache name
* will not create a new cache, but instead return a reference
* to it.
* @purge_interval sets the cache purging interval. I.e., an SG buffer
* will be freed if it's unused for time t
* purge_interval <= t < 2*purge_interval. If purge_interval
* is 0, then the default interval will be used (60 seconds).
* If purge_interval <0, then the automatic purging will be
* disabled.
*
* Description:
* Returns the resulting SGV cache or NULL in case of any error.
*/
struct sgv_pool *sgv_pool_create(const char *name,
enum sgv_clustering_types clustered, bool shared);
enum sgv_clustering_types clustered, int single_alloc_pages,
bool shared, int purge_interval);
/**
* sgv_pool_del - deletes the corresponding SGV cache
* @:pool the cache to delete.
*
* Description:
* If the cache is shared, it will decrease its reference counter.
* If the reference counter reaches 0, the cache will be destroyed.
*/
void sgv_pool_del(struct sgv_pool *pool);
/**
* sgv_pool_flush - flushes the SGV cache
* @:pool the cache to flush
*
* Description:
* Flushes, i.e. frees, all the cached entries in the SGV cache.
*/
void sgv_pool_flush(struct sgv_pool *pool);
/**
* sgv_pool_set_allocator - allows to set a custom pages allocator
* @:pool the cache
* @:alloc_pages_fn pages allocation function
* @:free_pages_fn pages freeing function
*
* Description:
* See the SGV cache documentation for more details.
*/
void sgv_pool_set_allocator(struct sgv_pool *pool,
struct page *(*alloc_pages_fn)(struct scatterlist *, gfp_t, void *),
void (*free_pages_fn)(struct scatterlist *, int, void *));
/**
* sgv_pool_alloc - allocates an SG vector from the SGV cache
* @:pool the cache to alloc from
* @:size size of the resulting SG vector in bytes
* @:gfp_mask the allocation mask
* @:flags the allocation flags
* @:count the resulting count of SG entries in the resulting SG vector
* @:sgv the resulting SGV object
* @:mem_lim memory limits
* @:priv pointer to private for this allocation data
*
* Description:
* Returns pointer to the resulting SG vector or NULL in case
* of any error. See the SGV cache documentation for more details.
*/
struct scatterlist *sgv_pool_alloc(struct sgv_pool *pool, unsigned int size,
gfp_t gfp_mask, int flags, int *count,
struct sgv_pool_obj **sgv, struct scst_mem_lim *mem_lim, void *priv);
/**
* sgv_pool_free - frees previously allocated SG vector
* @:sgv the SGV object to free
* @:mem_lim memory limits
*
* Description:
* Frees previously allocated SG vector, referenced by SGV cache object sgv
*/
void sgv_pool_free(struct sgv_pool_obj *sgv, struct scst_mem_lim *mem_lim);
/**
* sgv_get_priv - returns the private allocation data
* @:sgv the SGV object
*
* Description:
* Allows to get the allocation private data for this SGV
* cache object sgv. The private data are set by sgv_pool_alloc().
*/
void *sgv_get_priv(struct sgv_pool_obj *sgv);
/**
* scst_init_mem_lim - initializes memory limits
* @:mem_lim memory limits
*
* Description:
* Initializes memory limits structure mem_lim according to
* the current system configuration. This structure should be latter used
* to track and limit allocated by one or more SGV caches memory.
*/
void scst_init_mem_lim(struct scst_mem_lim *mem_lim);
#endif /* __SCST_SGV_H */
scst/include/scst_user.h
+2
View File
@@ -91,6 +91,8 @@ struct scst_user_dev_desc {
aligned_u64 version_str;
uint8_t type;
uint8_t sgv_shared;
int32_t sgv_single_alloc_pages;
int32_t sgv_purge_interval;
struct scst_user_opt opt;
uint32_t block_size;
char name[SCST_MAX_NAME];
scst/src/dev_handlers/scst_user.c
+23 -12
View File
@@ -522,12 +522,12 @@ static int dev_user_alloc_sg(struct scst_user_cmd *ucmd, int cached_buff)
EXTRACHECKS_BUG_ON(bufflen == 0);
if (cached_buff) {
flags |= SCST_POOL_RETURN_OBJ_ON_ALLOC_FAIL;
flags |= SGV_POOL_RETURN_OBJ_ON_ALLOC_FAIL;
if (ucmd->ubuff == 0)
flags |= SCST_POOL_NO_ALLOC_ON_CACHE_MISS;
flags |= SGV_POOL_NO_ALLOC_ON_CACHE_MISS;
} else {
TRACE_MEM("%s", "Not cached buff");
flags |= SCST_POOL_ALLOC_NO_CACHED;
flags |= SGV_POOL_ALLOC_NO_CACHED;
if (ucmd->ubuff == 0) {
res = 1;
goto out;
@@ -2640,7 +2640,7 @@ out:
static int dev_user_register_dev(struct file *file,
const struct scst_user_dev_desc *dev_desc)
{
int res = -ENOMEM, i;
int res, i;
struct scst_user_dev *dev, *d;
int block;
@@ -2673,12 +2673,15 @@ static int dev_user_register_dev(struct file *file,
if (!try_module_get(THIS_MODULE)) {
PRINT_ERROR("%s", "Fail to get module");
res = -ETXTBSY;
goto out;
}
dev = kzalloc(sizeof(*dev), GFP_KERNEL);
if (dev == NULL)
if (dev == NULL) {
res = -ENOMEM;
goto out_put;
}
init_rwsem(&dev->dev_rwsem);
spin_lock_init(&dev->cmd_lists.cmd_list_lock);
@@ -2702,9 +2705,13 @@ static int dev_user_register_dev(struct file *file,
(dev_desc->sgv_name[0] == '\0') ? dev->name :
dev_desc->sgv_name);
dev->pool = sgv_pool_create(dev->devtype.name, sgv_no_clustering,
dev_desc->sgv_shared);
if (dev->pool == NULL)
dev_desc->sgv_single_alloc_pages,
dev_desc->sgv_shared,
dev_desc->sgv_purge_interval);
if (dev->pool == NULL) {
res = -ENOMEM;
goto out_free_dev;
}
sgv_pool_set_allocator(dev->pool, dev_user_alloc_pages,
dev_user_free_sg_entries);
@@ -2712,9 +2719,14 @@ static int dev_user_register_dev(struct file *file,
(dev_desc->sgv_name[0] == '\0') ? dev->name :
dev_desc->sgv_name);
dev->pool_clust = sgv_pool_create(dev->devtype.name,
sgv_tail_clustering, dev_desc->sgv_shared);
if (dev->pool_clust == NULL)
sgv_tail_clustering,
dev_desc->sgv_single_alloc_pages,
dev_desc->sgv_shared,
dev_desc->sgv_purge_interval);
if (dev->pool_clust == NULL) {
res = -ENOMEM;
goto out_free0;
}
sgv_pool_set_allocator(dev->pool_clust, dev_user_alloc_pages,
dev_user_free_sg_entries);
@@ -2739,6 +2751,8 @@ static int dev_user_register_dev(struct file *file,
dev->def_block = block;
res = __dev_user_set_opt(dev, &dev_desc->opt);
if (res != 0)
goto out_free;
TRACE_MEM("dev %p, name %s", dev, dev->name);
@@ -2758,9 +2772,6 @@ static int dev_user_register_dev(struct file *file,
spin_unlock(&dev_list_lock);
if (res != 0)
goto out_del_free;
res = scst_register_virtual_dev_driver(&dev->devtype);
if (res < 0)
goto out_del_free;
scst/src/scst_lib.c
+2 -2
View File
@@ -2457,9 +2457,9 @@ int scst_alloc_space(struct scst_cmd *cmd)
gfp_mask = tgt_dev->gfp_mask | (atomic ? GFP_ATOMIC : GFP_KERNEL);
flags = atomic ? SCST_POOL_NO_ALLOC_ON_CACHE_MISS : 0;
flags = atomic ? SGV_POOL_NO_ALLOC_ON_CACHE_MISS : 0;
if (cmd->no_sgv)
flags |= SCST_POOL_ALLOC_NO_CACHED;
flags |= SGV_POOL_ALLOC_NO_CACHED;
cmd->sg = sgv_pool_alloc(tgt_dev->pool, cmd->bufflen, gfp_mask, flags,
&cmd->sg_cnt, &cmd->sgv, &cmd->dev->dev_mem_lim, NULL);
scst/src/scst_mem.c
+208 -172
View File
@@ -30,9 +30,8 @@
#include "scst_priv.h"
#include "scst_mem.h"
#define PURGE_INTERVAL (60 * HZ)
#define PURGE_TIME_AFTER PURGE_INTERVAL
#define SHRINK_TIME_AFTER (1 * HZ)
#define SGV_DEFAULT_PURGE_INTERVAL (60 * HZ)
#define SGV_MIN_SHRINK_INTERVAL (1 * HZ)
/* Max pages freed from a pool per shrinking iteration */
#define MAX_PAGES_PER_POOL 50
@@ -45,7 +44,7 @@ static atomic_t sgv_pages_total = ATOMIC_INIT(0);
static int sgv_hi_wmk;
static int sgv_lo_wmk;
static int sgv_max_local_order, sgv_max_trans_order;
static int sgv_max_local_pages, sgv_max_trans_pages;
static DEFINE_SPINLOCK(sgv_pools_lock); /* inner lock for sgv_pool_lock! */
static DEFINE_MUTEX(sgv_pools_mutex);
@@ -123,7 +122,7 @@ static void sgv_dtor_and_free(struct sgv_pool_obj *obj)
kfree(obj->sg_entries);
}
kmem_cache_free(pool->caches[obj->order_or_pages], obj);
kmem_cache_free(pool->caches[obj->cache_num], obj);
return;
}
@@ -175,7 +174,7 @@ static void sgv_dec_cached_entries(struct sgv_pool *pool, int pages)
/* Must be called under sgv_pool_lock held */
static void __sgv_purge_from_cache(struct sgv_pool_obj *obj)
{
int pages = 1 << obj->order_or_pages;
int pages = obj->pages;
struct sgv_pool *pool = obj->owner_pool;
TRACE_MEM("Purging sgv obj %p from pool %p (new cached_entries %d)",
@@ -193,16 +192,16 @@ static void __sgv_purge_from_cache(struct sgv_pool_obj *obj)
}
/* Must be called under sgv_pool_lock held */
static bool sgv_purge_from_cache(struct sgv_pool_obj *obj, int after,
static bool sgv_purge_from_cache(struct sgv_pool_obj *obj, int min_interval,
unsigned long cur_time)
{
EXTRACHECKS_BUG_ON(after < 0);
EXTRACHECKS_BUG_ON(min_interval < 0);
TRACE_MEM("Checking if sgv obj %p should be purged (cur time %ld, "
"obj time %ld, time to purge %ld)", obj, cur_time,
obj->time_stamp, obj->time_stamp + after);
obj->time_stamp, obj->time_stamp + min_interval);
if (time_after_eq(cur_time, (obj->time_stamp + after))) {
if (time_after_eq(cur_time, (obj->time_stamp + min_interval))) {
__sgv_purge_from_cache(obj);
return true;
}
@@ -210,15 +209,20 @@ static bool sgv_purge_from_cache(struct sgv_pool_obj *obj, int after,
}
/* No locks */
static int sgv_shrink_pool(struct sgv_pool *pool, int nr, int after,
static int sgv_shrink_pool(struct sgv_pool *pool, int nr, int min_interval,
unsigned long cur_time)
{
int freed = 0;
TRACE_ENTRY();
TRACE_MEM("Trying to shrink pool %p (nr %d, after %d)", pool, nr,
after);
TRACE_MEM("Trying to shrink pool %p (nr %d, min_interval %d)",
pool, nr, min_interval);
if (pool->purge_interval < 0) {
TRACE_MEM("Not shrinkable pool %p, skipping", pool);
goto out;
}
spin_lock_bh(&pool->sgv_pool_lock);
@@ -228,8 +232,8 @@ static int sgv_shrink_pool(struct sgv_pool *pool, int nr, int after,
pool->sorted_recycling_list.next,
struct sgv_pool_obj, sorted_recycling_list_entry);
if (sgv_purge_from_cache(obj, after, cur_time)) {
int pages = 1 << obj->order_or_pages;
if (sgv_purge_from_cache(obj, min_interval, cur_time)) {
int pages = obj->pages;
freed += pages;
nr -= pages;
@@ -253,12 +257,13 @@ static int sgv_shrink_pool(struct sgv_pool *pool, int nr, int after,
spin_unlock_bh(&pool->sgv_pool_lock);
out:
TRACE_EXIT_RES(nr);
return nr;
}
/* No locks */
static int __sgv_shrink(int nr, int after)
static int __sgv_shrink(int nr, int min_interval)
{
struct sgv_pool *pool;
unsigned long cur_time = jiffies;
@@ -267,8 +272,8 @@ static int __sgv_shrink(int nr, int after)
TRACE_ENTRY();
TRACE_MEM("Trying to shrink %d pages from all sgv pools (after %d)",
nr, after);
TRACE_MEM("Trying to shrink %d pages from all sgv pools "
"(min_interval %d)", nr, min_interval);
while (nr > 0) {
struct list_head *next;
@@ -307,7 +312,7 @@ static int __sgv_shrink(int nr, int after)
spin_unlock_bh(&sgv_pools_lock);
nr = sgv_shrink_pool(pool, nr, after, cur_time);
nr = sgv_shrink_pool(pool, nr, min_interval, cur_time);
sgv_pool_put(pool);
}
@@ -330,24 +335,26 @@ static int sgv_shrink(int nr, gfp_t gfpm)
{
TRACE_ENTRY();
if (nr > 0)
nr = __sgv_shrink(nr, SHRINK_TIME_AFTER);
else {
if (nr > 0) {
nr = __sgv_shrink(nr, SGV_MIN_SHRINK_INTERVAL);
TRACE_MEM("Left %d", nr);
} else {
struct sgv_pool *pool;
int inactive_pages = 0;
spin_lock_bh(&sgv_pools_lock);
list_for_each_entry(pool, &sgv_active_pools_list,
sgv_active_pools_list_entry) {
inactive_pages += pool->inactive_cached_pages;
if (pool->purge_interval > 0)
inactive_pages += pool->inactive_cached_pages;
}
spin_unlock_bh(&sgv_pools_lock);
nr = max((int)0, inactive_pages - sgv_lo_wmk);
TRACE_MEM("Can free %d (total %d)", nr,
atomic_read(&sgv_pages_total));
}
TRACE_MEM("Returning %d", nr);
TRACE_EXIT_RES(nr);
return nr;
}
@@ -379,7 +386,7 @@ static void sgv_purge_work_fn(struct delayed_work *work)
pool->sorted_recycling_list.next,
struct sgv_pool_obj, sorted_recycling_list_entry);
if (sgv_purge_from_cache(obj, PURGE_TIME_AFTER, cur_time)) {
if (sgv_purge_from_cache(obj, pool->purge_interval, cur_time)) {
spin_unlock_bh(&pool->sgv_pool_lock);
sgv_dtor_and_free(obj);
spin_lock_bh(&pool->sgv_pool_lock);
@@ -390,10 +397,10 @@ static void sgv_purge_work_fn(struct delayed_work *work)
* to reclaim buffers quickier.
*/
TRACE_MEM("Rescheduling purge work for pool %p (delay "
"%d HZ/%d sec)", pool, PURGE_INTERVAL,
PURGE_INTERVAL/HZ);
"%d HZ/%d sec)", pool, pool->purge_interval,
pool->purge_interval/HZ);
schedule_delayed_work(&pool->sgv_purge_work,
PURGE_INTERVAL);
pool->purge_interval);
pool->purge_work_scheduled = true;
break;
}
@@ -639,7 +646,7 @@ out_no_mem:
}
static int sgv_alloc_arrays(struct sgv_pool_obj *obj,
int pages_to_alloc, int order, gfp_t gfp_mask)
int pages_to_alloc, gfp_t gfp_mask)
{
int sz, tsz = 0;
int res = 0;
@@ -659,7 +666,7 @@ static int sgv_alloc_arrays(struct sgv_pool_obj *obj,
sg_init_table(obj->sg_entries, pages_to_alloc);
if (sgv_pool_clustered(obj->owner_pool)) {
if (order <= sgv_max_trans_order) {
if (pages_to_alloc <= sgv_max_trans_pages) {
obj->trans_tbl =
(struct trans_tbl_ent *)obj->sg_entries_data;
/*
@@ -678,9 +685,9 @@ static int sgv_alloc_arrays(struct sgv_pool_obj *obj,
}
}
TRACE_MEM("pages_to_alloc %d, order %d, sz %d, tsz %d, obj %p, "
"sg_entries %p, trans_tbl %p", pages_to_alloc, order,
sz, tsz, obj, obj->sg_entries, obj->trans_tbl);
TRACE_MEM("pages_to_alloc %d, sz %d, tsz %d, obj %p, sg_entries %p, "
"trans_tbl %p", pages_to_alloc, sz, tsz, obj, obj->sg_entries,
obj->trans_tbl);
out:
TRACE_EXIT_RES(res);
@@ -692,15 +699,14 @@ out_free:
goto out;
}
static struct sgv_pool_obj *sgv_get_obj(struct sgv_pool *pool, int order,
gfp_t gfp_mask)
static struct sgv_pool_obj *sgv_get_obj(struct sgv_pool *pool, int cache_num,
int pages, gfp_t gfp_mask)
{
struct sgv_pool_obj *obj;
int pages = 1 << order;
spin_lock_bh(&pool->sgv_pool_lock);
if (likely(!list_empty(&pool->recycling_lists[order]))) {
obj = list_entry(pool->recycling_lists[order].next,
if (likely(!list_empty(&pool->recycling_lists[cache_num]))) {
obj = list_entry(pool->recycling_lists[cache_num].next,
struct sgv_pool_obj, recycling_list_entry);
list_del(&obj->sorted_recycling_list_entry);
@@ -709,8 +715,6 @@ static struct sgv_pool_obj *sgv_get_obj(struct sgv_pool *pool, int order,
pool->inactive_cached_pages -= pages;
spin_unlock_bh(&pool->sgv_pool_lock);
EXTRACHECKS_BUG_ON(obj->order_or_pages != order);
goto out;
}
@@ -730,11 +734,12 @@ static struct sgv_pool_obj *sgv_get_obj(struct sgv_pool *pool, int order,
TRACE_MEM("New cached entries %d (pool %p)", pool->cached_entries,
pool);
obj = kmem_cache_alloc(pool->caches[order],
obj = kmem_cache_alloc(pool->caches[cache_num],
gfp_mask & ~(__GFP_HIGHMEM|GFP_DMA));
if (likely(obj)) {
memset(obj, 0, sizeof(*obj));
obj->order_or_pages = order;
obj->cache_num = cache_num;
obj->pages = pages;
obj->owner_pool = pool;
} else {
spin_lock_bh(&pool->sgv_pool_lock);
@@ -750,15 +755,13 @@ static void sgv_put_obj(struct sgv_pool_obj *obj)
{
struct sgv_pool *pool = obj->owner_pool;
struct list_head *entry;
struct list_head *list = &pool->recycling_lists[obj->order_or_pages];
int pages = 1 << obj->order_or_pages;
EXTRACHECKS_BUG_ON(obj->order_or_pages < 0);
struct list_head *list = &pool->recycling_lists[obj->cache_num];
int pages = obj->pages;
spin_lock_bh(&pool->sgv_pool_lock);
TRACE_MEM("sgv %p, order %d, sg_count %d", obj, obj->order_or_pages,
obj->sg_count);
TRACE_MEM("sgv %p, cache num %d, pages %d, sg_count %d", obj,
obj->cache_num, pages, obj->sg_count);
if (sgv_pool_clustered(pool)) {
/* Make objects with less entries more preferred */
@@ -766,8 +769,8 @@ static void sgv_put_obj(struct sgv_pool_obj *obj)
struct sgv_pool_obj *tmp = list_entry(entry,
struct sgv_pool_obj, recycling_list_entry);
TRACE_MEM("tmp %p, order %d, sg_count %d", tmp,
tmp->order_or_pages, tmp->sg_count);
TRACE_MEM("tmp %p, cache num %d, pages %d, sg_count %d",
tmp, tmp->cache_num, tmp->pages, tmp->sg_count);
if (obj->sg_count <= tmp->sg_count)
break;
@@ -789,7 +792,8 @@ static void sgv_put_obj(struct sgv_pool_obj *obj)
if (!pool->purge_work_scheduled) {
TRACE_MEM("Scheduling purge work for pool %p", pool);
pool->purge_work_scheduled = true;
schedule_delayed_work(&pool->sgv_purge_work, PURGE_INTERVAL);
schedule_delayed_work(&pool->sgv_purge_work,
pool->purge_interval);
}
spin_unlock_bh(&pool->sgv_pool_lock);
@@ -880,11 +884,10 @@ struct scatterlist *sgv_pool_alloc(struct sgv_pool *pool, unsigned int size,
struct sgv_pool_obj **sgv, struct scst_mem_lim *mem_lim, void *priv)
{
struct sgv_pool_obj *obj;
int order, pages, cnt;
int cache_num, pages, cnt;
struct scatterlist *res = NULL;
int pages_to_alloc;
struct kmem_cache *cache;
int no_cached = flags & SCST_POOL_ALLOC_NO_CACHED;
int no_cached = flags & SGV_POOL_ALLOC_NO_CACHED;
bool allowed_mem_checked = false, hiwmk_checked = false;
TRACE_ENTRY();
@@ -895,20 +898,24 @@ struct scatterlist *sgv_pool_alloc(struct sgv_pool *pool, unsigned int size,
sBUG_ON((gfp_mask & __GFP_NOFAIL) == __GFP_NOFAIL);
pages = ((size + PAGE_SIZE - 1) >> PAGE_SHIFT);
order = get_order(size);
if (pool->single_alloc_pages == 0) {
int pages_order = get_order(size);
cache_num = pages_order;
pages_to_alloc = (1 << pages_order);
} else {
cache_num = 0;
pages_to_alloc = max(pool->single_alloc_pages, pages);
}
TRACE_MEM("size=%d, pages=%d, order=%d, flags=%x, *sgv %p", size, pages,
order, flags, *sgv);
TRACE_MEM("size=%d, pages=%d, pages_to_alloc=%d, cache num=%d, "
"flags=%x, no_cached=%d, *sgv=%p", size, pages,
pages_to_alloc, cache_num, flags, no_cached, *sgv);
if (*sgv != NULL) {
obj = *sgv;
pages_to_alloc = (1 << order);
cache = pool->caches[obj->order_or_pages];
TRACE_MEM("Supplied obj %p, sgv_order %d", obj,
obj->order_or_pages);
TRACE_MEM("Supplied obj %p, cache num %d", obj, obj->cache_num);
EXTRACHECKS_BUG_ON(obj->order_or_pages != order);
EXTRACHECKS_BUG_ON(obj->sg_count != 0);
if (unlikely(!sgv_check_allowed_mem(mem_lim, pages_to_alloc)))
@@ -918,15 +925,12 @@ struct scatterlist *sgv_pool_alloc(struct sgv_pool *pool, unsigned int size,
if (unlikely(sgv_hiwmk_check(pages_to_alloc) != 0))
goto out_fail_free_sg_entries;
hiwmk_checked = true;
} else if ((order < SGV_POOL_ELEMENTS) && !no_cached) {
pages_to_alloc = (1 << order);
cache = pool->caches[order];
} else if ((pages_to_alloc <= pool->max_cached_pages) && !no_cached) {
if (unlikely(!sgv_check_allowed_mem(mem_lim, pages_to_alloc)))
goto out_fail;
allowed_mem_checked = true;
obj = sgv_get_obj(pool, order, gfp_mask);
obj = sgv_get_obj(pool, cache_num, pages_to_alloc, gfp_mask);
if (unlikely(obj == NULL)) {
TRACE(TRACE_OUT_OF_MEM, "Allocation of "
"sgv_pool_obj failed (size %d)", size);
@@ -935,19 +939,18 @@ struct scatterlist *sgv_pool_alloc(struct sgv_pool *pool, unsigned int size,
if (obj->sg_count != 0) {
TRACE_MEM("Cached obj %p", obj);
EXTRACHECKS_BUG_ON(obj->order_or_pages != order);
atomic_inc(&pool->cache_acc[order].hit_alloc);
atomic_inc(&pool->cache_acc[cache_num].hit_alloc);
goto success;
}
if (flags & SCST_POOL_NO_ALLOC_ON_CACHE_MISS) {
if (!(flags & SCST_POOL_RETURN_OBJ_ON_ALLOC_FAIL))
if (flags & SGV_POOL_NO_ALLOC_ON_CACHE_MISS) {
if (!(flags & SGV_POOL_RETURN_OBJ_ON_ALLOC_FAIL))
goto out_fail_free;
}
TRACE_MEM("Brand new obj %p", obj);
if (order <= sgv_max_local_order) {
if (pages_to_alloc <= sgv_max_local_pages) {
obj->sg_entries = obj->sg_entries_data;
sg_init_table(obj->sg_entries, pages_to_alloc);
TRACE_MEM("sg_entries %p", obj->sg_entries);
@@ -958,17 +961,17 @@ struct scatterlist *sgv_pool_alloc(struct sgv_pool *pool, unsigned int size,
/*
* No need to clear trans_tbl, if needed, it
* will be fully rewritten in
* sgv_alloc_sg_entries(),
* sgv_alloc_sg_entries().
*/
}
} else {
if (unlikely(sgv_alloc_arrays(obj, pages_to_alloc,
order, gfp_mask) != 0))
gfp_mask) != 0))
goto out_fail_free;
}
if ((flags & SCST_POOL_NO_ALLOC_ON_CACHE_MISS) &&
(flags & SCST_POOL_RETURN_OBJ_ON_ALLOC_FAIL))
if ((flags & SGV_POOL_NO_ALLOC_ON_CACHE_MISS) &&
(flags & SGV_POOL_RETURN_OBJ_ON_ALLOC_FAIL))
goto out_return;
obj->allocator_priv = priv;
@@ -985,11 +988,10 @@ struct scatterlist *sgv_pool_alloc(struct sgv_pool *pool, unsigned int size,
goto out_fail;
allowed_mem_checked = true;
if (flags & SCST_POOL_NO_ALLOC_ON_CACHE_MISS)
if (flags & SGV_POOL_NO_ALLOC_ON_CACHE_MISS)
goto out_return2;
cache = NULL;
sz = sizeof(*obj) + pages*sizeof(obj->sg_entries[0]);
sz = sizeof(*obj) + pages * sizeof(obj->sg_entries[0]);
obj = kmalloc(sz, gfp_mask);
if (unlikely(obj == NULL)) {
@@ -1000,7 +1002,9 @@ struct scatterlist *sgv_pool_alloc(struct sgv_pool *pool, unsigned int size,
memset(obj, 0, sizeof(*obj));
obj->owner_pool = pool;
obj->order_or_pages = -pages_to_alloc;
cache_num = -1;
obj->cache_num = cache_num;
obj->pages = pages_to_alloc;
obj->allocator_priv = priv;
obj->sg_entries = obj->sg_entries_data;
@@ -1010,7 +1014,7 @@ struct scatterlist *sgv_pool_alloc(struct sgv_pool *pool, unsigned int size,
goto out_fail_free_sg_entries;
hiwmk_checked = true;
TRACE_MEM("Big or no_cached obj %p (size %d)", obj, sz);
TRACE_MEM("Big or no_cached obj %p (size %d)", obj, sz);
}
obj->sg_count = sgv_alloc_sg_entries(obj->sg_entries,
@@ -1018,15 +1022,16 @@ struct scatterlist *sgv_pool_alloc(struct sgv_pool *pool, unsigned int size,
obj->trans_tbl, &pool->alloc_fns, priv);
if (unlikely(obj->sg_count <= 0)) {
obj->sg_count = 0;
if ((flags & SCST_POOL_RETURN_OBJ_ON_ALLOC_FAIL) && cache)
if ((flags & SGV_POOL_RETURN_OBJ_ON_ALLOC_FAIL) &&
(cache_num >= 0))
goto out_return1;
else
goto out_fail_free_sg_entries;
}
if (cache) {
if (cache_num >= 0) {
atomic_add(pages_to_alloc - obj->sg_count,
&pool->cache_acc[order].merged);
&pool->cache_acc[cache_num].merged);
} else {
if (no_cached) {
atomic_add(pages_to_alloc,
@@ -1042,9 +1047,9 @@ struct scatterlist *sgv_pool_alloc(struct sgv_pool *pool, unsigned int size,
}
success:
if (cache) {
if (cache_num >= 0) {
int sg;
atomic_inc(&pool->cache_acc[order].total_alloc);
atomic_inc(&pool->cache_acc[cache_num].total_alloc);
if (sgv_pool_clustered(pool))
cnt = obj->trans_tbl[pages-1].sg_num;
else
@@ -1106,12 +1111,12 @@ out_fail_free_sg_entries:
}
out_fail_free:
if (cache) {
if (cache_num >= 0) {
spin_lock_bh(&pool->sgv_pool_lock);
sgv_dec_cached_entries(pool, pages_to_alloc);
spin_unlock_bh(&pool->sgv_pool_lock);
kmem_cache_free(pool->caches[obj->order_or_pages], obj);
kmem_cache_free(pool->caches[obj->cache_num], obj);
} else
kfree(obj);
@@ -1138,26 +1143,22 @@ EXPORT_SYMBOL(sgv_get_priv);
void sgv_pool_free(struct sgv_pool_obj *obj, struct scst_mem_lim *mem_lim)
{
int pages;
int pages = (obj->sg_count != 0) ? obj->pages : 0;
TRACE_MEM("Freeing obj %p, order %d, sg_entries %p, "
"sg_count %d, allocator_priv %p", obj, obj->order_or_pages,
TRACE_MEM("Freeing obj %p, cache num %d, pages %d, sg_entries %p, "
"sg_count %d, allocator_priv %p", obj, obj->cache_num, pages,
obj->sg_entries, obj->sg_count, obj->allocator_priv);
if (obj->order_or_pages >= 0) {
if (obj->cache_num >= 0) {
obj->sg_entries[obj->orig_sg].length = obj->orig_length;
pages = (obj->sg_count != 0) ? 1 << obj->order_or_pages : 0;
sgv_put_obj(obj);
} else {
obj->owner_pool->alloc_fns.free_pages_fn(obj->sg_entries,
obj->sg_count, obj->allocator_priv);
pages = (obj->sg_count != 0) ? -obj->order_or_pages : 0;
kfree(obj);
sgv_hiwmk_uncheck(pages);
}
sgv_uncheck_allowed_mem(mem_lim, pages);
return;
}
EXPORT_SYMBOL(sgv_pool_free);
@@ -1236,9 +1237,59 @@ void scst_free(struct scatterlist *sg, int count)
}
EXPORT_SYMBOL(scst_free);
/* Must be called under sgv_pools_mutex */
static void sgv_pool_init_cache(struct sgv_pool *pool, int cache_num)
{
int size;
int pages;
struct sgv_pool_obj *obj;
atomic_set(&pool->cache_acc[cache_num].total_alloc, 0);
atomic_set(&pool->cache_acc[cache_num].hit_alloc, 0);
atomic_set(&pool->cache_acc[cache_num].merged, 0);
if (pool->single_alloc_pages == 0)
pages = 1 << cache_num;
else
pages = pool->single_alloc_pages;
if (pages <= sgv_max_local_pages) {
size = sizeof(*obj) + pages *
(sizeof(obj->sg_entries[0]) +
((pool->clustering_type != sgv_no_clustering) ?
sizeof(obj->trans_tbl[0]) : 0));
} else if (pages <= sgv_max_trans_pages) {
/*
* sg_entries is allocated outside object,
* but trans_tbl is still embedded.
*/
size = sizeof(*obj) + pages *
(((pool->clustering_type != sgv_no_clustering) ?
sizeof(obj->trans_tbl[0]) : 0));
} else {
size = sizeof(*obj);
/* both sgv and trans_tbl are kmalloc'ed() */
}
TRACE_MEM("pages=%d, size=%d", pages, size);
scnprintf(pool->cache_names[cache_num],
sizeof(pool->cache_names[cache_num]),
"%s-%uK", pool->name, (pages << PAGE_SHIFT) >> 10);
pool->caches[cache_num] = kmem_cache_create(
pool->cache_names[cache_num], size, 0, SCST_SLAB_FLAGS, NULL
#if (LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 23))
, NULL);
#else
);
#endif
return;
}
/* Must be called under sgv_pools_mutex */
int sgv_pool_init(struct sgv_pool *pool, const char *name,
enum sgv_clustering_types clustering_type)
enum sgv_clustering_types clustering_type, int single_alloc_pages,
int purge_interval)
{
int res = -ENOMEM;
int i;
@@ -1246,6 +1297,13 @@ int sgv_pool_init(struct sgv_pool *pool, const char *name,
TRACE_ENTRY();
if (single_alloc_pages < 0) {
PRINT_ERROR("Wrong single_alloc_pages value %d",
single_alloc_pages);
res = -EINVAL;
goto out;
}
memset(pool, 0, sizeof(*pool));
atomic_set(&pool->big_alloc, 0);
@@ -1256,56 +1314,40 @@ int sgv_pool_init(struct sgv_pool *pool, const char *name,
atomic_set(&pool->other_merged, 0);
pool->clustering_type = clustering_type;
pool->single_alloc_pages = single_alloc_pages;
if (purge_interval != 0) {
pool->purge_interval = purge_interval;
if (purge_interval < 0) {
/* Let's pretend that it's always scheduled */
pool->purge_work_scheduled = 1;
}
} else
pool->purge_interval = SGV_DEFAULT_PURGE_INTERVAL;
if (single_alloc_pages == 0) {
pool->max_caches = SGV_POOL_ELEMENTS;
pool->max_cached_pages = 1 << SGV_POOL_ELEMENTS;
} else {
pool->max_caches = 1;
pool->max_cached_pages = single_alloc_pages;
}
pool->alloc_fns.alloc_pages_fn = sgv_alloc_sys_pages;
pool->alloc_fns.free_pages_fn = sgv_free_sys_sg_entries;
TRACE_MEM("name %s, sizeof(*obj)=%zd, clustering_type=%d", name,
sizeof(*obj), clustering_type);
TRACE_MEM("name %s, sizeof(*obj)=%zd, clustering_type=%d, "
"single_alloc_pages=%d, max_caches=%d, max_cached_pages=%d",
name, sizeof(*obj), clustering_type, single_alloc_pages,
pool->max_caches, pool->max_cached_pages);
strncpy(pool->name, name, sizeof(pool->name)-1);
pool->name[sizeof(pool->name)-1] = '\0';
pool->owner_mm = current->mm;
for (i = 0; i < SGV_POOL_ELEMENTS; i++) {
int size;
atomic_set(&pool->cache_acc[i].total_alloc, 0);
atomic_set(&pool->cache_acc[i].hit_alloc, 0);
atomic_set(&pool->cache_acc[i].merged, 0);
if (i <= sgv_max_local_order) {
size = sizeof(*obj) + (1 << i) *
(sizeof(obj->sg_entries[0]) +
((clustering_type != sgv_no_clustering) ?
sizeof(obj->trans_tbl[0]) : 0));
} else if (i <= sgv_max_trans_order) {
/*
* sgv ie sg_entries is allocated outside object, but
* ttbl is still embedded.
*/
size = sizeof(*obj) + (1 << i) *
(((clustering_type != sgv_no_clustering) ?
sizeof(obj->trans_tbl[0]) : 0));
} else {
size = sizeof(*obj);
/* both sgv and ttbl are kallocated() */
}
TRACE_MEM("pages=%d, size=%d", 1 << i, size);
scnprintf(pool->cache_names[i], sizeof(pool->cache_names[i]),
"%s-%luK", name, (PAGE_SIZE >> 10) << i);
pool->caches[i] = kmem_cache_create(pool->cache_names[i],
size, 0, SCST_SLAB_FLAGS, NULL
#if (LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 23))
, NULL);
#else
);
#endif
for (i = 0; i < pool->max_caches; i++) {
sgv_pool_init_cache(pool, i);
if (pool->caches[i] == NULL) {
TRACE(TRACE_OUT_OF_MEM, "Allocation of sgv_pool cache "
"%s(%d) failed", name, i);
TRACE(TRACE_OUT_OF_MEM, "Allocation of sgv_pool "
"cache %s(%d) failed", name, i);
goto out_free;
}
}
@@ -1313,7 +1355,7 @@ int sgv_pool_init(struct sgv_pool *pool, const char *name,
atomic_set(&pool->sgv_pool_ref, 1);
spin_lock_init(&pool->sgv_pool_lock);
INIT_LIST_HEAD(&pool->sorted_recycling_list);
for (i = 0; i < SGV_POOL_ELEMENTS; i++)
for (i = 0; i < pool->max_caches; i++)
INIT_LIST_HEAD(&pool->recycling_lists[i]);
#if (LINUX_VERSION_CODE >= KERNEL_VERSION(2, 6, 20))
@@ -1334,7 +1376,7 @@ out:
return res;
out_free:
for (i = 0; i < SGV_POOL_ELEMENTS; i++) {
for (i = 0; i < pool->max_caches; i++) {
if (pool->caches[i]) {
kmem_cache_destroy(pool->caches[i]);
pool->caches[i] = NULL;
@@ -1344,31 +1386,17 @@ out_free:
goto out;
}
static void sgv_evaluate_local_order(void)
static void sgv_evaluate_local_max_pages(void)
{
int space4sgv_ttbl = PAGE_SIZE - sizeof(struct sgv_pool_obj);
sgv_max_local_order = get_order(
(((space4sgv_ttbl /
(sizeof(struct trans_tbl_ent) + sizeof(struct scatterlist))) *
PAGE_SIZE) & PAGE_MASK)) - 1;
sgv_max_local_pages = space4sgv_ttbl /
(sizeof(struct trans_tbl_ent) + sizeof(struct scatterlist));
sgv_max_trans_order = get_order(
(((space4sgv_ttbl / sizeof(struct trans_tbl_ent)) * PAGE_SIZE)
& PAGE_MASK)) - 1;
sgv_max_trans_pages = space4sgv_ttbl / sizeof(struct trans_tbl_ent);
TRACE_MEM("sgv_max_local_order %d, sgv_max_trans_order %d",
sgv_max_local_order, sgv_max_trans_order);
TRACE_MEM("max object size with embedded sgv & ttbl %zd",
(1 << sgv_max_local_order) * (sizeof(struct trans_tbl_ent) +
sizeof(struct scatterlist)) +
sizeof(struct sgv_pool_obj));
TRACE_MEM("max object size with embedded sgv (!clustered) %zd",
(1 << sgv_max_local_order) * sizeof(struct scatterlist) +
sizeof(struct sgv_pool_obj));
TRACE_MEM("max object size with embedded ttbl %zd",
(1 << sgv_max_trans_order) * sizeof(struct trans_tbl_ent)
+ sizeof(struct sgv_pool_obj));
TRACE_MEM("sgv_max_local_pages %d, sgv_max_trans_pages %d",
sgv_max_local_pages, sgv_max_trans_pages);
return;
}
@@ -1378,7 +1406,7 @@ void sgv_pool_flush(struct sgv_pool *pool)
TRACE_ENTRY();
for (i = 0; i < SGV_POOL_ELEMENTS; i++) {
for (i = 0; i < pool->max_caches; i++) {
struct sgv_pool_obj *obj;
spin_lock_bh(&pool->sgv_pool_lock);
@@ -1420,7 +1448,7 @@ void sgv_pool_deinit(struct sgv_pool *pool)
spin_unlock_bh(&sgv_pools_lock);
mutex_unlock(&sgv_pools_mutex);
for (i = 0; i < SGV_POOL_ELEMENTS; i++) {
for (i = 0; i < pool->max_caches; i++) {
if (pool->caches[i])
kmem_cache_destroy(pool->caches[i]);
pool->caches[i] = NULL;
@@ -1441,7 +1469,8 @@ void sgv_pool_set_allocator(struct sgv_pool *pool,
EXPORT_SYMBOL(sgv_pool_set_allocator);
struct sgv_pool *sgv_pool_create(const char *name,
enum sgv_clustering_types clustering_type, bool shared)
enum sgv_clustering_types clustering_type,
int single_alloc_pages, bool shared, int purge_interval)
{
struct sgv_pool *pool;
int rc;
@@ -1473,7 +1502,8 @@ struct sgv_pool *sgv_pool_create(const char *name,
goto out_unlock;
}
rc = sgv_pool_init(pool, name, clustering_type);
rc = sgv_pool_init(pool, name, clustering_type, single_alloc_pages,
purge_interval);
if (rc != 0)
goto out_free_unlock;
@@ -1541,20 +1571,20 @@ int scst_sgv_pools_init(unsigned long mem_hwmark, unsigned long mem_lwmark)
sgv_hi_wmk = mem_hwmark;
sgv_lo_wmk = mem_lwmark;
sgv_evaluate_local_order();
sgv_evaluate_local_max_pages();
mutex_lock(&sgv_pools_mutex);
res = sgv_pool_init(&sgv_norm_pool, "sgv", sgv_no_clustering);
res = sgv_pool_init(&sgv_norm_pool, "sgv", sgv_no_clustering, 0, 0);
if (res != 0)
goto out_unlock;
res = sgv_pool_init(&sgv_norm_clust_pool, "sgv-clust",
sgv_full_clustering);
sgv_full_clustering, 0, 0);
if (res != 0)
goto out_free_norm;
res = sgv_pool_init(&sgv_dma_pool, "sgv-dma", sgv_no_clustering);
res = sgv_pool_init(&sgv_dma_pool, "sgv-dma", sgv_no_clustering, 0, 0);
if (res != 0)
goto out_free_clust;
@@ -1608,7 +1638,7 @@ static void sgv_do_proc_read(struct seq_file *seq, const struct sgv_pool *pool)
int i, total = 0, hit = 0, merged = 0, allocated = 0;
int oa, om;
for (i = 0; i < SGV_POOL_ELEMENTS; i++) {
for (i = 0; i < pool->max_caches; i++) {
int t;
hit += atomic_read(&pool->cache_acc[i].hit_alloc);
@@ -1616,7 +1646,10 @@ static void sgv_do_proc_read(struct seq_file *seq, const struct sgv_pool *pool)
t = atomic_read(&pool->cache_acc[i].total_alloc) -
atomic_read(&pool->cache_acc[i].hit_alloc);
allocated += t * (1 << i);
if (pool->single_alloc_pages == 0)
allocated += t * (1 << i);
else
allocated += t * pool->single_alloc_pages;
merged += atomic_read(&pool->cache_acc[i].merged);
}
@@ -1625,10 +1658,13 @@ static void sgv_do_proc_read(struct seq_file *seq, const struct sgv_pool *pool)
pool->cached_pages, pool->inactive_cached_pages,
pool->cached_entries);
for (i = 0; i < SGV_POOL_ELEMENTS; i++) {
for (i = 0; i < pool->max_caches; i++) {
int t = atomic_read(&pool->cache_acc[i].total_alloc) -
atomic_read(&pool->cache_acc[i].hit_alloc);
allocated = t * (1 << i);
if (pool->single_alloc_pages == 0)
allocated = t * (1 << i);
else
allocated = t * pool->single_alloc_pages;
merged = atomic_read(&pool->cache_acc[i].merged);
seq_printf(seq, " %-28s %-11d %-11d %d\n",
scst/src/scst_mem.h
+10 -3
View File
@@ -32,8 +32,8 @@ struct trans_tbl_ent {
};
struct sgv_pool_obj {
/* if <0 - pages, >0 - order */
int order_or_pages;
int cache_num;
int pages;
/* jiffies, protected by sgv_pool_lock */
unsigned long time_stamp;
@@ -65,6 +65,8 @@ struct sgv_pool_alloc_fns {
struct sgv_pool {
enum sgv_clustering_types clustering_type;
int single_alloc_pages;
int max_cached_pages;
struct sgv_pool_alloc_fns alloc_fns;
@@ -73,6 +75,8 @@ struct sgv_pool {
spinlock_t sgv_pool_lock; /* outer lock for sgv_pools_lock! */
int purge_interval;
/* Protected by sgv_pool_lock */
unsigned int purge_work_scheduled:1;
@@ -101,6 +105,8 @@ struct sgv_pool {
atomic_t sgv_pool_ref;
int max_caches;
/* SCST_MAX_NAME + few more bytes to match scst_user expectations */
char cache_names[SGV_POOL_ELEMENTS][SCST_MAX_NAME + 10];
char name[SCST_MAX_NAME + 10];
@@ -111,7 +117,8 @@ struct sgv_pool {
};
int sgv_pool_init(struct sgv_pool *pool, const char *name,
enum sgv_clustering_types clustering_type);
enum sgv_clustering_types clustering_type, int single_alloc_pages,
int purge_interval);
void sgv_pool_deinit(struct sgv_pool *pool);
static inline struct scatterlist *sgv_pool_sg(struct sgv_pool_obj *obj)
scst/src/scst_priv.h
+6 -6
View File
@@ -58,12 +58,12 @@ extern unsigned long scst_trace_flag;
TRACE_LINE | TRACE_FUNCTION | TRACE_SPECIAL | TRACE_MGMT | \
/*TRACE_MGMT_MINOR |*/ TRACE_MGMT_DEBUG | TRACE_RTRY)
#define TRACE_RETRY(args...) __TRACE(TRACE_RTRY, args)
#define TRACE_SN(args...) __TRACE(TRACE_SCSI_SERIALIZING, args)
#define TRACE_SEND_TOP(args...) __TRACE(TRACE_SND_TOP, args)
#define TRACE_RECV_TOP(args...) __TRACE(TRACE_RCV_TOP, args)
#define TRACE_SEND_BOT(args...) __TRACE(TRACE_SND_BOT, args)
#define TRACE_RECV_BOT(args...) __TRACE(TRACE_RCV_BOT, args)
#define TRACE_RETRY(args...) TRACE_DBG_FLAG(TRACE_RTRY, args)
#define TRACE_SN(args...) TRACE_DBG_FLAG(TRACE_SCSI_SERIALIZING, args)
#define TRACE_SEND_TOP(args...) TRACE_DBG_FLAG(TRACE_SND_TOP, args)
#define TRACE_RECV_TOP(args...) TRACE_DBG_FLAG(TRACE_RCV_TOP, args)
#define TRACE_SEND_BOT(args...) TRACE_DBG_FLAG(TRACE_SND_BOT, args)
#define TRACE_RECV_BOT(args...) TRACE_DBG_FLAG(TRACE_RCV_BOT, args)
#else /* CONFIG_SCST_DEBUG */
usr/fileio/fileio.c
+30 -3
View File
@@ -91,7 +91,7 @@ static int wt_flag, rd_only_flag, o_direct_flag, nullio, nv_cache;
#if defined(DEBUG_TM_IGNORE) || defined(DEBUG_TM_IGNORE_ALL)
static int debug_tm_ignore;
#endif
static int non_blocking, sgv_shared;
static int non_blocking, sgv_shared, sgv_single_alloc_pages, sgv_purge_interval;
static void *(*alloc_fn)(size_t size) = align_alloc;
static struct option const long_options[] =
@@ -111,6 +111,8 @@ static struct option const long_options[] =
{"flush", required_argument, 0, 'F'},
{"unreg_before_close", no_argument, 0, 'u'},
{"sgv_shared", no_argument, 0, 's'},
{"sgv_single_cache", required_argument, 0, 'S'},
{"sgv_purge_interval", required_argument, 0, 'P'},
#if defined(DEBUG) || defined(TRACING)
{"debug", required_argument, 0, 'd'},
#endif
@@ -124,7 +126,7 @@ static struct option const long_options[] =
static void usage(void)
{
printf("Usage: %s [OPTION] name path [name path] ...\n", app_name);
printf("Usage: %s [OPTIONS] name path [name path] ...\n", app_name);
printf("\nFILEIO disk target emulator for SCST\n");
printf(" -b, --block=size Block size, must be power of 2 and >=512\n");
printf(" -e, --threads=count Number of threads, %d by default\n", THREADS);
@@ -143,6 +145,8 @@ static void usage(void)
printf(" -I, --vdisk_id=ID Vdisk ID (used in multi-targets setups)\n");
printf(" -F, --flush=n Flush SGV cache each n seconds\n");
printf(" -s, --sgv_shared Use shared SGV cache\n");
printf(" -S, --sgv_single_cache=n Use single entry SGV cache with n pages/entry\n");
printf(" -P, --sgv_purge_interval=n Use SGV cache purge interval n seconds\n");
printf(" -u, --unreg_before_close Unregister before close\n");
#if defined(DEBUG) || defined(TRACING)
printf(" -d, --debug=level Debug tracing level\n");
@@ -317,6 +321,8 @@ int start(int argc, char **argv)
strncpy(desc.sgv_name, devs[0].name, sizeof(desc.sgv_name)-1);
desc.sgv_name[sizeof(desc.sgv_name)-1] = '\0';
}
desc.sgv_single_alloc_pages = sgv_single_alloc_pages;
desc.sgv_purge_interval = sgv_purge_interval;
desc.type = devs[i].type;
desc.block_size = devs[i].block_size;
@@ -439,7 +445,7 @@ int main(int argc, char **argv)
memset(devs, 0, sizeof(devs));
while ((ch = getopt_long(argc, argv, "+b:e:trongluF:I:cp:f:m:d:vh", long_options,
while ((ch = getopt_long(argc, argv, "+b:e:trongluF:I:cp:f:m:d:vsS:P:h", long_options,
&longindex)) >= 0) {
switch (ch) {
case 'b':
@@ -496,6 +502,12 @@ int main(int argc, char **argv)
case 's':
sgv_shared = 1;
break;
case 'S':
sgv_single_alloc_pages = atoi(optarg);
break;
case 'P':
sgv_purge_interval = atoi(optarg);
break;
case 'm':
if (strncmp(optarg, "all", 3) == 0)
memory_reuse_type = SCST_USER_MEM_REUSE_ALL;
@@ -602,6 +614,21 @@ int main(int argc, char **argv)
sBUG();
}
if (sgv_shared)
PRINT_INFO(" %s", "SGV shared");
if (sgv_single_alloc_pages != 0)
PRINT_INFO(" Use single entry SGV cache with %d pages/entry",
sgv_single_alloc_pages);
if (sgv_purge_interval != 0) {
if (sgv_purge_interval > 0)
PRINT_INFO(" Use SGV cache purge interval %d seconds",
sgv_purge_interval);
else
PRINT_INFO(" %s", "SGV cache purging disabled");
}
if (!o_direct_flag && (memory_reuse_type == SCST_USER_MEM_NO_REUSE)) {
PRINT_INFO(" %s", "Using unaligned buffers");
alloc_fn = malloc;
www/index.html
+1
View File
@@ -137,6 +137,7 @@
<p><a href="qla2x00t-howto.html">HOWTO For QLogic Target Driver</a></p>
<p><a href="http://lpfcxxxx.sourceforge.net/HOWTO.lpfc">HOWTO For Emulex lpfc Target Driver</a></p>
<p><a href="scst_user_spec.txt">SCST User Interface Description</a></p>
<p><a href="sgv_cache.txt">SCST SGV Cache Description</a></p>
<h1>SCST 0.9.6 graphs</h1>
<p><a href=images/init_scst.png>init_scst</a></p>
<p><a href=images/scst_cmd_thread.png>scst_cmd_thread</a></p>