Commit Graph

369 Commits

Author SHA1 Message Date
Robert Baldyga
2cba3f4473
Merge pull request #673 from jfckm/settle-in-tests
Settle in start_stop tests before checking stats
2022-03-28 09:47:18 +02:00
Adam Rutkowski
4a839cd332 Verify standby/active cache state in OCF entry points
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2022-03-28 09:42:02 +02:00
Robert Baldyga
aa4622cc01 Make core remove error recoverable
First try to clean only the mapping. This operation does not require any
rollback, so even if flushing collision fails, core object is still
intact. In case of error we inform user that core was not removed by
returning new error code (-OCF_ERR_CORE_NOT_REMOVED).

After flushing collision succeeds we remove core from metadata and
flush superblock at the end. At that point the core is fully removed
from OCF and even if superblock flush error occurs there is nothing we
can do about it, so we just return the error code.

Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2022-03-25 21:32:55 +01:00
Jan Musial
4d3abc37d5 Settle in start_stop tests before checking stats
Signed-off-by: Jan Musial <jan.musial@intel.com>
2022-03-18 10:56:46 +01:00
Robert Baldyga
d46845b97a Remove unused error code
Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2022-03-09 11:43:35 +01:00
Robert Baldyga
d5b2c65a39 Remove "metadata_layout" parameter of the cache
This feature is replaced with LRU list shuffling.

Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2022-03-07 17:48:25 +01:00
Robert Baldyga
73f3b73f0f
Merge pull request #663 from pdebski21/1023_xfail
Test for load with changed core size
2022-03-02 21:03:00 +01:00
Robert Baldyga
765d1c5d77 rio: fix size comparison in
Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2022-02-24 15:02:46 +01:00
Jan Musial
cd0551e72e Use rio in promotion policy tests
Signed-off-by: Jan Musial <jan.musial@intel.com>
2022-02-24 14:56:09 +01:00
Adam Rutkowski
a6d8bd0470 rio: accept queue list in Rio constructor
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
Signed-off-by: Jan Musial <jan.musial@intel.com>
2022-02-24 14:56:09 +01:00
Adam Rutkowski
372b1f1e1c remove unnecessary callback decorator in rio
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2022-02-24 14:56:09 +01:00
Adam Rutkowski
9baf9a3876 rio: convert Size to bytes for extent calculation
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2022-02-24 14:56:09 +01:00
Jan Musial
9360e55851 Update basic pyOCF tests to use rio
Signed-off-by: Jan Musial <jan.musial@intel.com>
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2022-02-24 14:56:09 +01:00
Adam Rutkowski
17eddab094 pyocf: queue settle functionality
Queue settle is a mechanism to wait for all OCF processing
on a given queue to finish.

In some tests simply waiting for I/O to finish is not enough. Most
notably some statistics are potentially incremented after user triggered
I/O is finished. This is due to asynchronous nature of I/O operations
and OCF approach to statistics update, where only eventual consistency
is guaranteed without explicit mechanism available to query whether
the final state is reached yet. However it is fully in the adapter power
to determine this, as OCF executes in context of API calls from the
adapter (like I/O submission, ocf_queue_run, ocf_cleaner_run, management
operations) and I/O completion callbacks. Queue settle is a mechanism to
assure ocf_queue_run is not being executed by the thread associated with
a queue.

With queue settle mechanism it is straightforward for the adapter to
wait for cache statistics to reach fixed values:
1. wait for all I/O to OCF to finish
2. settle all I/O queues
3. make sure background cleaner is not active

Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2022-02-24 14:56:09 +01:00
Jan Musial
41732090ae Implement rio io tester for pyocf
rio stands for Rigid IO tester and is a simple mechanism for testing
OCF cache IO.

Signed-off-by: Jan Musial <jan.musial@intel.com>
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2022-02-24 14:56:09 +01:00
Jan Musial
96124ca87a Enable Size comparisons in pyocf
Signed-off-by: Jan Musial <jan.musial@intel.com>
2022-02-24 14:56:09 +01:00
Jan Musial
9fcc7bfbb7 Use new capabilities in surprise shutdown tests
Signed-off-by: Jan Musial <jan.musial@intel.com>
2022-02-24 12:27:10 +01:00
Jan Musial
5d31dd87d0 Test for open_cores load
Signed-off-by: Jan Musial <jan.musial@intel.com>
2022-02-24 12:27:10 +01:00
Jan Musial
973a474b37 Make open_cores load viable by adding getter for cores
Signed-off-by: Jan Musial <jan.musial@intel.com>
2022-02-24 12:27:10 +01:00
Jan Musial
53d2c5a197 Fix test_start_stop_noqueue
Signed-off-by: Jan Musial <jan.musial@intel.com>
2022-02-24 12:27:10 +01:00
Jan Musial
bfee654770 Add wrapper for cache_config_set_default
Signed-off-by: Jan Musial <jan.musial@intel.com>
2022-02-24 12:27:10 +01:00
Jan Musial
2038c3aa68 Add basic load test with core reattachment and cleanup structures
Signed-off-by: Jan Musial <jan.musial@intel.com>
2022-02-24 12:27:10 +01:00
Jan Musial
30fe2eb783 Handle try_add core adding mode
Signed-off-by: Jan Musial <jan.musial@intel.com>
2022-02-24 12:27:10 +01:00
Piotr Debski
08907dae5d test for load with changed core size
removing xfail mark and update core oprations after merged ocf patch

Signed-off-by: Piotr Debski <piotr.debski@intel.com>
2022-02-18 23:15:47 +01:00
Jan Musial
1273e9ddfe Fix leaking Volumes and OcfCtx's
Signed-off-by: Jan Musial <jan.musial@intel.com>
2022-02-16 07:06:05 +01:00
Jan Musial
8f47cfa5bf Make multiple pyocf loggers viable
Signed-off-by: Jan Musial <jan.musial@intel.com>
2022-02-16 07:06:05 +01:00
Jan Musial
441e3c20d7 Clean-up pyocf c-python shared objects instance tracking
Signed-off-by: Jan Musial <jan.musial@intel.com>
2022-02-16 07:06:05 +01:00
Jan Musial
4c9d4fe24e Fix pyocf Ctx ownership
Signed-off-by: Jan Musial <jan.musial@intel.com>
2022-02-16 07:06:05 +01:00
Robert Baldyga
da59a1c9aa
Merge pull request #634 from arutk/mopft_21.12
pyocf: management operation power failure tests
2022-02-14 14:10:52 +01:00
Robert Baldyga
49abe816ce
Merge pull request #649 from pdebski21/1023
fix for issue #1023
2022-02-07 16:17:14 +01:00
Robert Baldyga
76684ed8a9
Merge pull request #642 from robertbaldyga/parallelize
Parallelize metadata initialization
2022-02-07 13:53:45 +01:00
Robert Baldyga
53f6cffe7f tests: Add unit tests for bisect generator
Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2022-01-31 06:59:28 +01:00
Robert Baldyga
b70492ad3d Parallelize ALRU recovery
Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2022-01-31 06:59:28 +01:00
Piotr Debski
9b980d3f22 fix for issue #1023
Better error for core size mismatch during activation/load

adding pyocf test for new error code

Signed-off-by: Piotr Debski <piotr.debski@intel.com>
2022-01-25 05:18:16 +01:00
Adam Rutkowski
a97bc61010 pyocf: mark most surprise shutdown tests as long
.. to skip by default

Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2022-01-13 17:42:42 +01:00
Adam Rutkowski
2b7a249143 pyocf: document custom markers in pytest.ini
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2022-01-13 17:42:42 +01:00
Adam Rutkowski
ae240f5aa8 pyocf: test update to handle zeroed metadata in attach
With atomic superblock commit during cache attach, it is possible
that power failure interrupts attach operation at a point where
neither new or old superblock is present - right after the superblock
is cleared.

Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2022-01-13 17:41:04 +01:00
Adam Rutkowski
366d89a9c4 pyocf: management operation power failure handling tests
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2022-01-13 17:40:44 +01:00
Adam Rutkowski
ce0bbc260e pycof: explicitly free Volume::_uuid_ dictionary
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2022-01-13 17:35:33 +01:00
Adam Rutkowski
7f60aa0dd6 pyocf: fix error handling in Cache::stop()
Write error in cache stop means the instance was in fact stopped.

Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2022-01-13 17:35:33 +01:00
Adam Rutkowski
ed5185e870 pyocf: make io class config more usable
Unify field param names and include io class id in info struct.

Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2022-01-13 17:35:33 +01:00
Adam Rutkowski
5fe217d96f pyocf: set proper OCF error codes in Volume
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2022-01-13 17:35:33 +01:00
Adam Rutkowski
c43059a4cf pyocf: add get_bytes() function to Volume and Data
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2022-01-13 17:35:33 +01:00
Adam Rutkowski
683174c78f pyocf: add option to load cache without openning cores
... this is useful to workaround current pyocf limitations and
load cache with manual core insertion

Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2022-01-13 17:35:33 +01:00
Adam Rutkowski
58dac85f7b pyocf: add ocf_core_get_uuid() wrapper
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2022-01-13 17:35:29 +01:00
Adam Rutkowski
c7144decf8 pyocf: extend error volume capabilities
Adding option to
1. inject error based on I/O number
2. arm/disarm error injection for easier testing

Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2022-01-13 16:40:54 +01:00
Adam Rutkowski
cf72664c2c pyocf: fix cache device config
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2022-01-13 16:40:54 +01:00
Piotr Debski
609a22cfda added ERROR code for superblock mismatch
Signed-off-by: Piotr Debski <piotr.debski@intel.com>
2022-01-08 23:06:10 +01:00
Robert Baldyga
716b5751d6 Redesign failover standby API
Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2021-12-29 20:31:40 +01:00
Robert Baldyga
cac5869406
Merge pull request #603 from robertbaldyga/metadata-flapping
Introduce flapping of metadata config sections
2021-12-15 17:11:15 +01:00
Robert Baldyga
98cb9bff70 pyocf: Use bigger cache device size in tests
As amount of fixed size metadata allocated by OCF grows, we need to adjust
test to not try to start cache on device that is too small.

Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2021-12-15 15:30:34 +01:00
Robert Baldyga
6d1a5c733e pyocf: Add missing error codes
Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2021-12-13 15:58:59 +01:00
Rafal Stefanowski
decfc8dea6 Remove executable bit from Makefile
Signed-off-by: Rafal Stefanowski <rafal.stefanowski@intel.com>
2021-11-24 17:01:03 +01:00
Rafal Stefanowski
f22da1cde7 Fix license
Change license to BSD-3-Clause

Signed-off-by: Rafal Stefanowski <rafal.stefanowski@intel.com>
2021-10-28 13:08:50 +02:00
Krzysztof Majzerowicz-Jaszcz
71262d5097 Cache standby mode API changes
Error for an invalid cache operation while in passive mode added

Signed-off-by: Krzysztof Majzerowicz-Jaszcz <krzysztof.majzerowicz-jaszcz@intel.com>

Error name correction

Signed-off-by: Krzysztof Majzerowicz-Jaszcz <krzysztof.majzerowicz-jaszcz@intel.com>

API changes for passive cache mode

Moved the passive cache error return source to the api for flush and
set_param

Signed-off-by: Krzysztof Majzerowicz-Jaszcz <krzysztof.majzerowicz-jaszcz@intel.com>

Further API changes for passive cache mode

Signed-off-by: Krzysztof Majzerowicz-Jaszcz <krzysztof.majzerowicz-jaszcz@intel.com>

 Passive api - review changes

Signed-off-by: Krzysztof Majzerowicz-Jaszcz <krzysztof.majzerowicz-jaszcz@intel.com>
2021-10-22 15:10:53 +02:00
Michal Mielewczyk
ad536d9ad9 pyocf: fix cache attach config struct
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
Signed-off-by: ajrutkow <adam.j.rutkowski@intel.com>
2021-10-08 16:04:14 +02:00
Robert Baldyga
7d43aae3a0 pyocf: Add new error codes
Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2021-09-06 20:44:40 +02:00
Robert Baldyga
865d29d0cb
Merge pull request #527 from arutk/clc_ut_fix
fix debug logs in cacheline concurrency UT tests
2021-07-29 23:18:16 +02:00
Michal Mielewczyk
a072e071e2 Remove metadata updated from security tests
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
2021-07-29 09:51:29 +02:00
Michal Mielewczyk
1620c544a1 pyocf: update cleaning policy switching test
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
2021-07-27 15:44:33 +02:00
Michal Mielewczyk
a94677f433 Fix cleaning UT
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
2021-07-27 15:44:33 +02:00
Michal Mielewczyk
92f9c3d532 UT framework: fix misspelled variable name
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
2021-07-27 15:44:18 +02:00
Kozlowski Mateusz
f494448f97 Align structures to cacheline
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
2021-07-08 12:46:39 +02:00
Michal Mielewczyk
98d2ce0264 pyocf: test for removing dirty core
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
2021-06-28 17:40:49 +02:00
Adam Rutkowski
3775ab0e37 fix debug logs in cacheline concurrency UT tests
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2021-06-26 16:55:25 +02:00
Adam Rutkowski
1e1955b833 lru refactor
rearanging lru implementation for easier journaling

Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2021-06-21 22:32:57 +02:00
Adam Rutkowski
07cbba32f6 remove stale references to eviction
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2021-06-18 12:19:08 +02:00
Adam Rutkowski
33e2beac24 Rename "evp_lru*" functions to "ocf_lru*"
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2021-06-18 12:19:08 +02:00
Adam Rutkowski
b1143374a8 Move eviction files to new locations
src/eviction/lru.c -> src/ocf_lru.c
src/eviction/lru.h -> src/ocf_lru.h
src/eviction/lru_structs.h -> src/ocf_lru_structs.h
src/eviction/eviction.c -> src/ocf_space.c
src/eviction/eviction.h -> src/ocf_space.h

.. as well as corresponding UT files.

Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>

... in UT as well

Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2021-06-18 12:19:08 +02:00
Adam Rutkowski
7c0f940876 Replace eviction with lru in metadata structs
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2021-06-18 12:19:08 +02:00
Adam Rutkowski
88e04a4204 Remove eviction policy abstraction
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2021-06-18 12:19:08 +02:00
Adam Rutkowski
77bccba036 do not track hotness on free lru list
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2021-06-18 12:19:08 +02:00
Adam Rutkowski
fac35d34a2 Rename "evict" to "remap" across the entire repo
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2021-06-18 12:19:08 +02:00
Adam Rutkowski
4f217b91a5 Remove partition list
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2021-06-18 12:19:08 +02:00
Adam Rutkowski
87f834c793 Move common user and freelist partition data to a new struct
New structure ocf_part is added to contain all the data common for both
user partitions and freelist partition: part_runtime and part_id.
ocf_user_part now contains ocf_part structure as well as pointer to
cleaning partition runtime metadata (moved out from part_runtime) and
user partition config (no change here).

Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2021-06-18 12:07:10 +02:00
Kozlowski Mateusz
367fcbfe4e Update debug prints and methods
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
2021-06-16 13:48:37 +02:00
Kozlowski Mateusz
c17b587444 Update cache line concurrency unit tests
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
2021-06-16 13:48:37 +02:00
Kozlowski Mateusz
ce316cc67c Change alock API to include slow/fast lock callbacks
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
2021-06-16 13:48:35 +02:00
Adam Rutkowski
f589341c9a remove metadata updater
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2021-06-15 10:25:05 +02:00
Jan Musial
9f1df6e380 Update pyOCF and tests
Signed-off-by: Jan Musial <jan.musial@intel.com>
2021-05-26 07:57:37 +02:00
Michal Mielewczyk
d5ae764175 pyocf: fix valid seq cutoff threshold range
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
2021-04-15 11:09:26 +02:00
Michal Mielewczyk
0bfa9ed870 pyocf: Seq-cutoff promotion count security tests
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
2021-04-14 15:40:29 +02:00
Michal Mielewczyk
8e0bb49493 functional test for eviction between cores
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2021-04-01 18:01:28 -05:00
Kozlowski Mateusz
3f9af8bd82 Update pyocf types to new field order
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
2021-03-26 08:57:04 +01:00
Michal Mielewczyk
92a5ddd524 ut framework: don't mock env functions
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
2021-03-19 16:08:27 +01:00
Michal Mielewczyk
0d3f3cde14 Return error when modifying default ioclass rule
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
2021-03-19 16:06:23 +01:00
Robert Baldyga
8020e7fd67
Merge pull request #457 from Ostrokrzew/false_stats
Fix broken 'dirty_for' stats
2021-03-18 10:24:02 +01:00
Michał Mielewczyk
d2b5de7970
Merge pull request #448 from robertbaldyga/perqueue-seq-cutoff
Per-queue multi-stream sequential cutoff
2021-03-05 14:38:21 +01:00
Adam Rutkowski
81fc7ab5c5 Parallel eviction
Eviction changes allowing to evict (remap) cachelines while
holding hash bucket write lock instead of global metadata
write lock.

As eviction (replacement) is now tightly coupled with request,
each request uses eviction size equal to number of its
unmapped cachelines.

Evicting without global metadata write lock is possible
thanks to the fact that remaping is always performed
while exclusively holding cacheline (read or write) lock.
So for a cacheline on LRU list we acquire cacheline lock,
safely resolve hash and consequently write-lock hash bucket.
Since cacheline lock is acquired under hash bucket (everywhere
except for new eviction implementation), we are certain that
noone acquires cacheline lock behind our back. Concurrent
eviction threads are eliminated by holding eviction list
lock for the duration of critial locking operations.

Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2021-03-05 11:20:47 +01:00
Robert Baldyga
9352c881ab tests: Update sequential cutoff tests
Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2021-03-04 16:38:31 +01:00
Robert Baldyga
ac9bd5b094
Merge pull request #453 from arutk/no_cl_gl_lock
Skip cacheline concurrency global lock in fast path
2021-03-04 12:33:50 +01:00
Michal Mielewczyk
3a26bc56cd pyocf: improve test logging
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
2021-03-02 18:37:58 +01:00
Michal Mielewczyk
7f862c3080 pyocf: improve random string generator
Set of random characters may be exteded with a custom list.

Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
2021-03-02 18:37:58 +01:00
Michal Mielewczyk
a81be31dd4 pyocf: default range for int16
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
2021-03-02 18:37:58 +01:00
Michal Mielewczyk
c4a2dc4cad pyocf: security tests for ioclass api
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
2021-03-02 18:37:58 +01:00
Michal Mielewczyk
fa556247d7 pyocf: change encoding of ioclass name
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
2021-03-02 14:46:57 +01:00
Michal Mielewczyk
080e13a071 pyocf: valid ranges for ioclass config values
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
2021-03-02 14:46:57 +01:00
Michal Mielewczyk
06edc48717 pyocf: remove min_size from ioclass config
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
2021-03-02 14:46:57 +01:00
Michal Mielewczyk
95d756de91 Remove ioclass min_size from public API
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
2021-03-02 14:46:57 +01:00
Michal Mielewczyk
5c053ad964 pyocf: security test for seq cutoff threshold
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
2021-02-26 08:24:41 -05:00