mirrors/scst
1
0
Fork 0
mirror of https://github.com/SCST-project/scst.git synced 2026-05-14 01:01:27 +00:00
Code Issues Packages Projects Releases Wiki Activity

SGV cache doc converted to LinuxDoc

Browse Source 
git-svn-id: http://svn.code.sf.net/p/scst/svn/trunk@3316 d57e44dd-8a1f-0410-8b47-8ef2f437770f
This commit is contained in:
Vladislav Bolkhovitin
2011-03-28 11:31:49 +00:00
parent c469a9745e
commit 0649cbfc8d
2 changed files with 155 additions and 56 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
doc/Makefile
View File

@@ -36,9 +36,7 @@ rtf: $(RTFS)
$(COMMAND)rtf $(<)
clean:
@mv sgv_cache.txt sgv_cache.tx
rm -f *.txt *.html *.tex *.dvi *.ps *.pdf *.info *.lyx *.rtf
@mv sgv_cache.tx sgv_cache.txt
extraclean: clean
rm -f *.orig *.rej

209
doc/sgv_cache.txt → doc/sgv_cache.sgml
View File

@@ -1,9 +1,22 @@
SCST SGV CACHE.
<!doctype linuxdoc system>
PROGRAMMING INTERFACE DESCRIPTION.
<article>
For SCST version 1.0.2
<title>
SCST SGV cache description
</title>
<author>
<name>Vladislav Bolkhovitin</name>
</author>
<date>Version 2.1.0</date>
<toc>
<sect>Introduction
<p>
SCST SGV cache is a memory management subsystem in SCST. One can call it
a "memory pool", but Linux kernel already have a mempool interface,
which serves different purposes. SGV cache provides to SCST core, target
@@ -13,50 +26,65 @@ facility, when it doesn't free to the system each vector, which is not
used anymore, but keeps it for a while (possibly indefinitely) to let it
be reused by the next consecutive command. This allows to:
- Reduce commands processing latencies and, hence, improve performance;
<itemize>
- Make commands processing latencies predictable, which is essential
<item> Reduce commands processing latencies and, hence, improve performance;
<item> Make commands processing latencies predictable, which is essential
for RT applications.
</itemize>
The freed SG vectors are kept by the SGV cache either for some (possibly
indefinite) time, or, optionally, until the system needs more memory and
asks to free some using the set_shrinker() interface. Also the SGV cache
allows to:
- Cluster pages together. "Cluster" means merging adjacent pages in a
<itemize>
<item> Cluster pages together. "Cluster" means merging adjacent pages in a
single SG entry. It allows to have less SG entries in the resulting SG
vector, hence improve performance handling it as well as allow to
work with bigger buffers on hardware with limited SG capabilities.
- Set custom page allocator functions. For instance, scst_user device
<item> Set custom page allocator functions. For instance, scst_user device
handler uses this facility to eliminate unneeded mapping/unmapping of
user space pages and avoid unneeded IOCTL calls for buffers allocations.
In fileio_tgt application, which uses a regular malloc() function to
allocate data buffers, this facility allows ~30% less CPU load and
considerable performance increase.
- Prevent each initiator or all initiators altogether to allocate too
<item> Prevent each initiator or all initiators altogether to allocate too
much memory and DoS the target. Consider 10 initiators, which can have
access to 10 devices each. Any of them can queue up to 64 commands, each
can transfer up to 1MB of data. So, all of them in a peak can allocate
up to 10*10*64 = ~6.5GB of memory for data buffers. This amount must be
limited somehow and the SGV cache performs this function.
</itemize>
<sect> Implementation
<p>
From implementation POV the SGV cache is a simple extension of the kmem
cache. It can work in 2 modes:
1. With fixed size buffers.
<enum>
2. With a set of power 2 size buffers. In this mode each SGV cache
<item> With fixed size buffers.
<item> With a set of power 2 size buffers. In this mode each SGV cache
(struct sgv_pool) has SGV_POOL_ELEMENTS (11 currently) of kmem caches.
Each of those kmem caches keeps SGV cache objects (struct sgv_pool_obj)
corresponding to SG vectors with size of order X pages. For instance,
request to allocate 4 pages will be served from kmem cache[2], since the
request to allocate 4 pages will be served from kmem cache&lsqb;2&rsqb, since the
order of the of number of requested pages is 2. If later request to
allocate 11KB comes, the same SG vector with 4 pages will be reused (see
below). This mode is in average allows less memory overhead comparing
with the fixed size buffers mode.
</enum>
Consider how the SGV cache works in the set of buffers mode. When a
request to allocate new SG vector comes, sgv_pool_alloc() via
sgv_get_obj() checks if there is already a cached vector with that
@@ -82,153 +110,226 @@ Freed cached sgv_pool_obj objects are actually freed to the system
either by the purge work, which is scheduled once in 60 seconds, or in
sgv_shrink() called by system, when it's asking for memory.
<sect> Interface
Interface.
<sect1> sgv_pool *sgv_pool_create()
struct sgv_pool *sgv_pool_create(const char *name,
<p>
<verb>
struct sgv_pool *sgv_pool_create(
const char *name,
enum sgv_clustering_types clustered, int single_alloc_pages,
bool shared, int purge_interval)
</verb>
This function creates and initializes an SGV cache. It has the following
arguments:
- name - the name of the SGV cache
<itemize>
- clustered - sets type of the pages clustering. The type can be:
<item> <bf/name/ - the name of the SGV cache
* sgv_no_clustering - no clustering performed.
<item> <bf/clustered/ - sets type of the pages clustering. The type can be:
* sgv_tail_clustering - a page will only be merged with the latest
previously allocated page, so the order of pages in the SG will be
preserved
<itemize>
* sgv_full_clustering - free merging of pages at any place in
the SG is allowed. This mode usually provides the best merging
rate.
<item> <bf/sgv_no_clustering/ - no clustering performed.
- single_alloc_pages - if 0, then the SGV cache will work in the set of
<item> <bf/sgv_tail_clustering/ - a page will only be merged with the latest
previously allocated page, so the order of pages in the SG will be
preserved
<item> <bf/sgv_full_clustering/ - free merging of pages at any place in
the SG is allowed. This mode usually provides the best merging
rate.
</itemize>
<item> <bf/single_alloc_pages/ - if 0, then the SGV cache will work in the set of
power 2 size buffers mode. If >0, then the SGV cache will work in the
fixed size buffers mode. In this case single_alloc_pages sets the
size of each buffer in pages.
- shared - sets if the SGV cache can be shared between devices or not.
<item> <bf/shared/ - sets if the SGV cache can be shared between devices or not.
The cache sharing allowed only between devices created inside the same
address space. If an SGV cache is shared, each subsequent call of
sgv_pool_create() with the same cache name will not create a new cache,
but instead return a reference to it.
- purge_interval - sets the cache purging interval. I.e. an SG buffer
<item> <bf/purge_interval/ - sets the cache purging interval. I.e. an SG buffer
will be freed if it's unused for time t purge_interval <= t <
2*purge_interval. If purge_interval is 0, then the default interval
will be used (60 seconds). If purge_interval <0, then the automatic
purging will be disabled. Shrinking by the system's demand will also
be disabled.
</itemize>
Returns the resulting SGV cache or NULL in case of any error.
<sect1> void sgv_pool_del()
void sgv_pool_del(struct sgv_pool *pool)
<p>
<verb>
void sgv_pool_del(
struct sgv_pool *pool)
</verb>
This function deletes the corresponding SGV cache. If the cache is
shared, it will decrease its reference counter. If the reference counter
reaches 0, the cache will be destroyed.
<sect1> void sgv_pool_flush()
void sgv_pool_flush(struct sgv_pool *pool)
<p>
<verb>
void sgv_pool_flush(
struct sgv_pool *pool)
</verb>
This function flushes, i.e. frees, all the cached entries in the SGV
cache.
<sect1> void sgv_pool_set_allocator()
void sgv_pool_set_allocator(struct sgv_pool *pool,
<p>
<verb>
void sgv_pool_set_allocator(
struct sgv_pool *pool,
struct page *(*alloc_pages_fn)(struct scatterlist *sg, gfp_t gfp, void *priv),
void (*free_pages_fn)(struct scatterlist *sg, int sg_count, void *priv));
</verb>
This function allows to set for the SGV cache a custom pages allocator. For
instance, scst_user uses such function to supply to the cache mapped from
user space pages.
alloc_pages_fn() has the following parameters:
<bf/alloc_pages_fn()/ has the following parameters:
- sg - SG entry, to which the allocated page should be added.
<itemize>
- gfp - the allocation GFP flags
<item> <bf/sg/ - SG entry, to which the allocated page should be added.
- priv - pointer to a private data supplied to sgv_pool_alloc()
<item> <bf/gfp/ - the allocation GFP flags
<item> <bf/priv/ - pointer to a private data supplied to sgv_pool_alloc()
</itemize>
This function should return the allocated page or NULL, if no page was
allocated.
free_pages_fn() has the following parameters:
<bf/free_pages_fn()/ has the following parameters:
- sg - SG vector to free
<itemize>
- sg_count - number of SG entries in the sg
<item> <bf/sg/ - SG vector to free
- priv - pointer to a private data supplied to the corresponding sgv_pool_alloc()
<item> <bf/sg_count/ - number of SG entries in the sg
<item> <bf/priv/ - pointer to a private data supplied to the
corresponding sgv_pool_alloc()
struct scatterlist *sgv_pool_alloc(struct sgv_pool *pool, unsigned int size,
gfp_t gfp_mask, int flags, int *count,
struct sgv_pool_obj **sgv, struct scst_mem_lim *mem_lim, void *priv)
</itemize>
<sect1> struct scatterlist *sgv_pool_alloc()
<p>
<verb>
struct scatterlist *sgv_pool_alloc(
struct sgv_pool *pool,
unsigned int size,
gfp_t gfp_mask,
int flags,
int *count,
struct sgv_pool_obj **sgv,
struct scst_mem_lim *mem_lim,
void *priv)
</verb>
This function allocates an SG vector from the SGV cache. It has the
following parameters:
- pool - the cache to alloc from
<itemize>
- size - size of the resulting SG vector in bytes
<item> <bf/pool/ - the cache to alloc from
- gfp_mask - the allocation mask
<item> <bf/size/ - size of the resulting SG vector in bytes
- flags - the allocation flags. The following flags are possible and
<item> <bf/gfp_mask/ - the allocation mask
<item> <bf/flags/ - the allocation flags. The following flags are possible and
can be set using OR operation:
* SGV_POOL_ALLOC_NO_CACHED - the SG vector must not be cached.
<enum>
* SGV_POOL_NO_ALLOC_ON_CACHE_MISS - don't do an allocation on a
<item> <bf/SGV_POOL_ALLOC_NO_CACHED/ - the SG vector must not be cached.
<item> <bf/SGV_POOL_NO_ALLOC_ON_CACHE_MISS/ - don't do an allocation on a
cache miss.
* SGV_POOL_RETURN_OBJ_ON_ALLOC_FAIL - return an empty SGV object,
<item> <bf/SGV_POOL_RETURN_OBJ_ON_ALLOC_FAIL/ - return an empty SGV object,
i.e. without the SG vector, if the allocation can't be completed.
For instance, because SGV_POOL_NO_ALLOC_ON_CACHE_MISS flag set.
- count - the resulting count of SG entries in the resulting SG vector.
</enum>
- sgv - the resulting SGV object. It should be used to free the
<item> <bf/count/ - the resulting count of SG entries in the resulting SG vector.
<item> <bf/sgv/ - the resulting SGV object. It should be used to free the
resulting SG vector.
- mem_lim - memory limits, see below.
<item> <bf/mem_lim/ - memory limits, see below.
- priv - pointer to private for this allocation data. This pointer will
<item> <bf/priv/ - pointer to private for this allocation data. This pointer will
be supplied to alloc_pages_fn() and free_pages_fn() and can be
retrieved by sgv_get_priv().
</itemize>
This function returns pointer to the resulting SG vector or NULL in case
of any error.
<sect1> void sgv_pool_free()
void sgv_pool_free(struct sgv_pool_obj *sgv, struct scst_mem_lim *mem_lim)
<p>
<verb>
void sgv_pool_free(
struct sgv_pool_obj *sgv,
struct scst_mem_lim *mem_lim)
</verb>
This function frees previously allocated SG vector, referenced by SGV
cache object sgv.
<sect1> void *sgv_get_priv(struct sgv_pool_obj *sgv)
void *sgv_get_priv(struct sgv_pool_obj *sgv)
<p>
<verb>
void *sgv_get_priv(
struct sgv_pool_obj *sgv)
</verb>
This function allows to get the allocation private data for this SGV
cache object sgv. The private data are set by sgv_pool_alloc().
<sect1> void scst_init_mem_lim()
void scst_init_mem_lim(struct scst_mem_lim *mem_lim)
<p>
<verb>
void scst_init_mem_lim(
struct scst_mem_lim *mem_lim)
</verb>
This function initializes memory limits structure mem_lim according to
the current system configuration. This structure should be latter used
to track and limit allocated by one or more SGV caches memory.
Runtime information and statistics.
<sect> Runtime information and statistics.
<p>
Runtime information and statistics is available in /sys/kernel/scst_tgt/sgv.
</article>