sstables: Close promoted index readers when advancing to next summary index
Problem fixed on master since 5ed559c. So branch-4.5 and up aren't affected.
Index reader fails to close input streams of promoted index readers when advancing
to next summary entry, so Scylla can abort as a result of a stream being destroyed
while there were reads in progress. This problem was seen when row cache issued
a fast forward, so index reader was asked to advance to next summary entry while
the previous one still had reads in progress.
By closing the list of index readers when there's only one owner holding it,
the problem is safely fixed, because it cannot happen that an index_bound like
_lower_bound or _upper_bound will be left with a list that's already closed.
Fixes #9049.
test: mode(dev, debug).
No observable perf regression:
BEFORE:
read skip time (s) iterations frags frag/s mad f/s max f/s min f/s avg aio aio (KiB) blocked dropped idx hit idx miss idx blk c hit c miss c blk cpu
-> 1 0 8.168640 4 100000 12242 108 12262 11982 50032.2 50049 6403116 20707 0 0 8 8 0 0 0 83.3%
-> 1 1 22.257916 4 50000 2246 3 2249 2238 150025.0 150025 6454272 100001 0 49999 100000 149999 0 0 0 54.7%
-> 1 8 9.384961 4 11112 1184 5 1184 1178 77781.2 77781 1439328 66618 11111 1 33334 44444 0 0 0 44.0%
-> 1 16 4.976144 4 5883 1182 6 1184 1173 41180.0 41180 762053 35264 5882 0 17648 23530 0 0 0 44.1%
-> 1 32 2.582744 4 3031 1174 4 1175 1167 21216.0 21216 392619 18176 3031 0 9092 12122 0 0 0 43.8%
-> 1 64 1.308410 4 1539 1176 2 1178 1173 10772.0 10772 199353 9233 1539 0 4616 6154 0 0 0 44.0%
-> 1 256 0.331037 4 390 1178 12 1190 1165 2729.0 2729 50519 2338 390 0 1169 1558 0 0 0 44.0%
-> 1 1024 0.085108 4 98 1151 7 1155 1141 685.0 685 12694 587 98 0 293 390 0 0 0 42.9%
-> 1 4096 0.024393 6 25 1025 5 1029 1020 174.0 174 3238 149 25 0 74 98 0 0 0 37.4%
-> 64 1 8.765446 4 98462 11233 16 11236 11182 54642.0 54648 6405470 23632 1 1538 4615 4615 0 0 0 79.3%
-> 64 8 8.456430 4 88896 10512 48 10582 10464 55578.0 55578 6405971 24031 4166 0 5553 5553 0 0 0 77.3%
-> 64 16 7.798197 4 80000 10259 108 10299 10077 51248.0 51248 5922500 22160 4996 0 4998 4998 0 0 0 74.8%
-> 64 32 6.605148 4 66688 10096 64 10168 10033 42715.0 42715 4936359 18796 4164 0 4165 4165 0 0 0 75.5%
-> 64 64 4.933287 4 50016 10138 28 10189 10111 32039.0 32039 3702428 14106 3124 0 3125 3125 0 0 0 75.3%
-> 64 256 1.971701 4 20032 10160 57 10347 10103 12831.0 12831 1482993 5731 1252 0 1250 1250 0 0 0 74.1%
-> 64 1024 0.587026 4 5888 10030 84 10277 9946 3770.0 3770 435895 1635 368 0 366 366 0 0 0 74.6%
-> 64 4096 0.157401 4 1600 10165 69 10202 9698 1023.0 1023 118449 455 100 0 98 98 0 0 0 73.9%
AFTER:
read skip time (s) iterations frags frag/s mad f/s max f/s min f/s avg aio aio (KiB) blocked dropped idx hit idx miss idx blk c hit c miss c blk cpu
-> 1 0 8.191639 4 100000 12208 46 12279 12161 50031.2 50025 6403108 20243 0 0 0 0 0 0 0 87.0%
-> 1 1 22.933121 4 50000 2180 36 2198 2115 150025.0 150025 6454272 100001 0 49999 100000 149999 0 0 0 54.9%
-> 1 8 9.471735 4 11112 1173 5 1178 1168 77781.2 77781 1439328 66663 11111 0 33334 44445 0 0 0 44.6%
-> 1 16 5.001569 4 5883 1176 2 1176 1170 41180.0 41180 762053 35296 5882 1 17648 23529 0 0 0 44.6%
-> 1 32 2.587069 4 3031 1172 1 1173 1164 21216.0 21216 392619 18185 3031 1 9092 12121 0 0 0 44.8%
-> 1 64 1.310747 4 1539 1174 3 1177 1171 10772.0 10772 199353 9233 1539 0 4616 6154 0 0 0 44.9%
-> 1 256 0.335490 4 390 1162 2 1167 1161 2729.0 2729 50519 2338 390 0 1169 1558 0 0 0 45.7%
-> 1 1024 0.081944 4 98 1196 21 1210 1162 685.0 685 12694 585 98 0 293 390 0 0 0 46.2%
-> 1 4096 0.022266 6 25 1123 3 1125 1105 174.0 174 3238 149 24 0 74 98 0 0 0 41.9%
-> 64 1 8.731741 4 98462 11276 45 11417 11231 54642.0 54640 6405470 23686 0 1538 4615 4615 0 0 0 80.2%
-> 64 8 8.396247 4 88896 10588 19 10596 10560 55578.0 55578 6405971 24275 4166 0 5553 5553 0 0 0 77.6%
-> 64 16 7.700995 4 80000 10388 88 10405 10221 51248.0 51248 5922500 22100 5000 0 4998 4998 0 0 0 76.4%
-> 64 32 6.517276 4 66688 10232 31 10342 10201 42715.0 42715 4936359 19013 4164 0 4165 4165 0 0 0 75.3%
-> 64 64 4.898669 4 50016 10210 60 10291 10150 32039.0 32039 3702428 14110 3124 0 3125 3125 0 0 0 74.4%
-> 64 256 1.969972 4 20032 10169 22 10173 10091 12831.0 12831 1482993 5660 1252 0 1250 1250 0 0 0 74.3%
-> 64 1024 0.575180 4 5888 10237 84 10316 10028 3770.0 3770 435895 1656 368 0 366 366 0 0 0 74.6%
-> 64 4096 0.158503 4 1600 10094 81 10195 10014 1023.0 1023 118449 460 100 0 98 98 0 0 0 73.5%
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <20210722180302.64675-1-raphaelsc@scylladb.com>
This commit is contained in:
Raphael S. Carvalho
committed by
Avi Kivity
Avi Kivity
parent
99b8c04a40
commit
9dce1e4b2b
@@ -367,6 +367,7 @@ class index_reader {
|
||||
const io_priority_class& _pc;
|
||||
tracing::trace_state_ptr _trace_state;
|
||||
shared_index_lists _index_lists;
|
||||
future<> _background_closes = make_ready_future<>();
|
||||
|
||||
struct reader {
|
||||
index_consumer _consumer;
|
||||
@@ -472,6 +473,16 @@ private:
|
||||
};
|
||||
|
||||
return _index_lists.get_or_load(summary_idx, loader).then([this, &bound, summary_idx] (shared_index_lists::list_ptr ref) {
|
||||
// to make sure list is not closed when another bound is still using it, index list will only be closed when there's only one owner holding it
|
||||
if (bound.current_list && bound.current_list.use_count() == 1) {
|
||||
// a new background close will only be initiated when previous ones terminate, so as to limit the concurrency.
|
||||
_background_closes = _background_closes.then_wrapped([current_list = std::move(bound.current_list)] (future<>&& f) mutable {
|
||||
f.ignore_ready_future();
|
||||
return do_with(std::move(current_list), [] (shared_index_lists::list_ptr& current_list) mutable {
|
||||
return close_index_list(current_list);
|
||||
});
|
||||
});
|
||||
}
|
||||
bound.current_list = std::move(ref);
|
||||
bound.current_summary_idx = summary_idx;
|
||||
bound.current_index_idx = 0;
|
||||
@@ -841,6 +852,8 @@ public:
|
||||
return close_index_list(_upper_bound->current_list);
|
||||
}
|
||||
return make_ready_future<>();
|
||||
}).then([this] () mutable {
|
||||
return std::move(_background_closes);
|
||||
});
|
||||
}
|
||||
};
|
||||
|
||||
@@ -173,6 +173,10 @@ public:
|
||||
return res;
|
||||
}
|
||||
|
||||
long use_count() const noexcept {
|
||||
return _e ? _e.use_count() : 0;
|
||||
}
|
||||
|
||||
friend class loading_shared_values;
|
||||
};
|
||||
|
||||
|
||||
Reference in New Issue
Block a user