The system is running an iSCSI write test with header and data digests
enabled. On the command that resulted in the assert, all of the write
data was sent as immediate and unsolicited. The unsolicited data
consisted of a single DataOut PDU and it was the iscsi_cmnd resource
allocated for receiving the unsolicited DataOut that caused the assert,
not the iscsi_cmnd resource tracking the iSCSI write command. As the
Recv thread is finishing up with the iscsi_cmnd struct for receiving the
final unsolicited DataOut, SCST is call the iscsi_pre_exec function to
restart the command and this routine will process the DataOut
iscsi_cmnd as it is queued onto the rx_digest_list of the iscsi_cmnd for
Write.
ffff8a984150c600 is the iscsi_cmnd allocated to track the iSCSI Write command
ffff8a984150d680 is the iscsi_cmnd allocated for receiving the Data-out PDU
In the logs below, recv thread 3677 just completed receiving the unsolicited DataOut.
[ 977.563046] [3677]: do_recv:808:nr_segs 0, bytes_left 0, res 196608
[ 977.563048] [31728]: iscsi_make_conn_rd_active:351:conn ffff8a97d0df1600, rd_state 2, rd_data_ready 1
[ 977.563065] [3677]: do_recv:808:nr_segs 0, bytes_left 0, res 4
[ 977.563069] [31728]: iscsi_make_conn_rd_active:351:conn ffff8a97d0df1600, rd_state 2, rd_data_ready 1
[ 977.563076] [3677]: cmnd_rx_end:3475:cmnd ffff8a984150d680, opcode 5
[ 977.563078] [31728]: iscsi_make_conn_rd_active:351:conn ffff8a97d0df1600, rd_state 2, rd_data_ready 1
[ 977.563081] [3677]: cmnd_rx_end:3478:Updated last_rcv_time 4295644274
[ 977.563091] [3677]: data_out_end:2455:cmnd ffff8a984150d680, req ffff8a984150c600
[ 977.563093] [3677]: cmd_add_on_rx_ddigest_list:820:Adding RX ddigest cmd ffff8a984150d680 to digest list of req ffff8a984150c600
[ 977.563096] [3677]: cmnd_get:764:cmnd ffff8a984150d680, new cmnd->ref_cnt 2
Here we see no more data is pending to be received on this write, so
iscsi_restart_cmnd is called by recv thread which should trigger SCST to
start processing the command:
[ 977.563106] [3677]: data_out_end:2495:req ffff8a984150c600, FINAL 80, outstanding_r2t 0, r2t_len_to_receive 0, r2t_len_to_send 0
[ 977.563108] [3677]: req_del_from_write_timeout_list:141:Deleting cmd ffff8a984150c600 from conn ffff8a97d0df1600 write_timeout_list
[ 977.563111] [3677]: cmnd_remove_data_wait_hash:1606:Deleting cmnd ffff8a984150c600 from the hash (ITT 1e230400)
SCST thread 24654 calls the pre execute routine which verifies the data
digest on immediate and unsolicited data:
[ 977.563157] [24654]: iscsi_pre_exec:1917:Checking digest of RX ddigest cmd ffff8a984150c600
[ 977.563161] [24654]: digest_data:125:req ffff8a984150c600, idx 0, count 16, sg_cnt 64, size 65536, offset 0
[ 977.563170] [24654]: digest_rx_data:219:RX data digest OK for cmd ffff8a984150c600
[ 977.563173] [3677]: cmnd_put:775:cmnd ffff8a984150c600, new ref_cnt 2
[ 977.563176] [24654]: cmd_del_from_rx_ddigest_list:830:Deleting RX digest cmd ffff8a984150c600 from digest list
Recv thread calls pre release on iscsi_cmnd allocated for unsolicited
DataOut PDU:
[ 977.563178] [3677]: req_cmnd_pre_release:820:req ffff8a984150d680
[ 977.563181] [24654]: cmnd_put:775:cmnd ffff8a984150c600, new ref_cnt 1
SCST thread 24654 processing data digest on unsolicited DataOut PDU.
Here we have two threads accessing the same iscsi_cmnd struct. SCST
thread will set on_rx_digest_list flag to 0. Recv thread will set
release_called flag to 1.
[ 977.563184] [24654]: iscsi_pre_exec:1917:Checking digest of RX ddigest cmd ffff8a984150d680
[ 977.563186] [24654]: cmd_del_from_rx_ddigest_list:830:Deleting RX digest cmd ffff8a984150d680 from digest list
[ 977.563189] [24654]: cmnd_put:775:cmnd ffff8a984150d680, new ref_cnt 0
[ 977.563191] [3677]: conn_get:868:conn ffff8a97d0df1600, new conn_ref_cnt 4
[ 977.563194] [24654]: cmnd_done:589:cmnd ffff8a984150d680
[ 977.563196] BUG at ./iscsi-scst/kernel/iscsi.c:598 (cmnd->on_rx_digest_list)
[ 977.563212] ------------[ cut here ]------------
[ 977.563215] kernel BUG at ./iscsi-scst/kernel/iscsi.c:598!
Reported and analyzed by Adam Hutchinson <ajhutchin@gmail.com>
git-svn-id: http://svn.code.sf.net/p/scst/svn/trunk@7338 d57e44dd-8a1f-0410-8b47-8ef2f437770f
Avoid that building SCST with W=1 against kernel v4.15 triggers warnings about
kernel-doc header issues.
git-svn-id: http://svn.code.sf.net/p/scst/svn/trunk@7298 d57e44dd-8a1f-0410-8b47-8ef2f437770f
This makes it easier to add support for new kernel versions, e.g. kernel
version v4.14.
git-svn-id: http://svn.code.sf.net/p/scst/svn/trunk@7260 d57e44dd-8a1f-0410-8b47-8ef2f437770f
This is a "marginal" warning, but there are only six of them in the SCST code
and this fixes two of those.
session.c:105:4: warning: cast from function call of type void * to non-matching type _Bool [-Wbad-function-cast]
target.c:668:14: warning: cast from function call of type void * to non-matching type _Bool [-Wbad-function-cast]
Signed-off-by: David Butterfield <dab21774@gmail.com>
git-svn-id: http://svn.code.sf.net/p/scst/svn/trunk@7200 d57e44dd-8a1f-0410-8b47-8ef2f437770f
Waiting for other threads to release an object using code like
"while (object->refcnt > 0) msleep(100)" without holding a reference
on 'object' is wrong because the memory object points at may be freed
before or while this loop is in progress. Hence introduce a global
portal object count and wait on that count instead of waiting for
the per-portal reference count to reach zero.
The use-after-free was introduced in r6952 ("isert: faster release
of isert_scst module").
See also https://sourceforge.net/p/scst/tickets/2/.
git-svn-id: http://svn.code.sf.net/p/scst/svn/trunk@7184 d57e44dd-8a1f-0410-8b47-8ef2f437770f
This change helped a secondary problem I had under valgrind (due to some other
bug) with initiators timing out and reconnecting their sessions faster than the
threads were getting cleaned up (for one thing, valgrind only ever runs one
thread at a time).
It seems logical that the code in conn_lookup() would want to skip any
connection that's already known closing in this case, for the same reason it
searches the list in reverse. And the dropping connections don't have to drop
in order, so searching in reverse doesn't seem sufficient to avoid finding a
wrong (stale, closing) connection structure.
Signed-off-by: David Butterfield <dab21774@gmail.com>
git-svn-id: http://svn.code.sf.net/p/scst/svn/trunk@7144 d57e44dd-8a1f-0410-8b47-8ef2f437770f
In an error path in iscsi_threads_pool_get(), when a new pool cannot be
allocated, if there is a pool on iscsi_thread_pools_list, it passes that back as
an alternative, so return zero in that case.
Signed-off-by: David Butterfield <dab21774@gmail.com>
git-svn-id: http://svn.code.sf.net/p/scst/svn/trunk@7143 d57e44dd-8a1f-0410-8b47-8ef2f437770f
In scsi_cmnd_start() conn->rx_task is intended to be set only during the
call to _stage1_done, but there was no code to reset it. It doesn't
actually matter in execution, but it was a little confusing to wonder
and try to understand why it wasn't reset.
Signed-off-by: David Butterfield <dab21774@gmail.com>
git-svn-id: http://svn.code.sf.net/p/scst/svn/trunk@7142 d57e44dd-8a1f-0410-8b47-8ef2f437770f
Change local names of ioctl() and open() to not conflict with C library names.
This isn't really a bug in a strictly kernel-resident build, but the change
avoids symbol conflicts with libc when building for usermode.
Signed-off-by: David Butterfield <dab21774@gmail.com>
with some changes
git-svn-id: http://svn.code.sf.net/p/scst/svn/trunk@7138 d57e44dd-8a1f-0410-8b47-8ef2f437770f
Add "XXX" comments in a few places about potential problems seen in SCST code,
for future investigation and possible repair.
Signed-off-by: David Butterfield <dab21774@gmail.com>
git-svn-id: http://svn.code.sf.net/p/scst/svn/trunk@7123 d57e44dd-8a1f-0410-8b47-8ef2f437770f
casting const to non-const
uninitialized structure members
non-static local function
missing enumerated switch-value cases
Signed-off-by: David Butterfield <dab21774@gmail.com>
git-svn-id: http://svn.code.sf.net/p/scst/svn/trunk@7122 d57e44dd-8a1f-0410-8b47-8ef2f437770f
If this attrubute is set, each iSCSI session for this initiator has
dedicated, i.e. not shared with other sessions, pool of the iscsi{wr,rd}
kernel threads.
Useful to control per-session CPU affinity to improve performance.
git-svn-id: http://svn.code.sf.net/p/scst/svn/trunk@7069 d57e44dd-8a1f-0410-8b47-8ef2f437770f
It adds target_ip attribute containing target IP address in each
/sys/kernel/scst_tgt/targets/iscsi/TARGET/sessions/INITIATOR/INITIATOR_IP subdirectory.
It's useful to simplify monitoring management utilities
git-svn-id: http://svn.code.sf.net/p/scst/svn/trunk@7067 d57e44dd-8a1f-0410-8b47-8ef2f437770f
It's false positive, because responses live in the parent request's list
until last put for the parent.
git-svn-id: http://svn.code.sf.net/p/scst/svn/trunk@7009 d57e44dd-8a1f-0410-8b47-8ef2f437770f
