github/ocf
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,693 Commits 1 Branch 1 Tag 4.1 MiB
f8a0f88892
Commit Graph

227 Commits

Author SHA1 Message Date
Adam Rutkowski
83bb7317bf pyocf: format all .py files with black -l 100 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-05-16 16:44:19 +02:00
Adam Rutkowski
3a1b6fd718
Merge pull request #695 from arutk/failover_test_5 
pyocf: failover functional and power failure recovery tests
 2022-05-16 16:37:45 +02:00
Adam Rutkowski
fcfbd860c4 pyocf: fix error volume stats reset 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-05-16 15:40:18 +02:00
Adam Rutkowski
a504821302 pyocf: separate get_conf() from stats in Cache 
Configuration parameters are available in standby mode, while
stats are not. Need to separate the two.

Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-05-16 15:40:18 +02:00
Adam Rutkowski
3d83e1f004 pyocf: more generic erorr device volume 
Make ErrorDevice a wraper over any type of volume rather than
inheriting from RamVolume. This way error injection layer can be added
over any type of volume.

Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-05-16 15:40:18 +02:00
Adam Rutkowski
d8e519855b pyocf: more options to standby load 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-05-16 15:40:18 +02:00
Adam Rutkowski
b78c36ccdd pyocf: add missing import Volume in cache 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-05-16 15:40:18 +02:00
Adam Rutkowski
6a03145acf pyocf: cache/core object volume fixes 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-05-16 15:40:18 +02:00
Adam Rutkowski
2dde108cd9 pyocf: collision addr helper 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-05-16 15:40:18 +02:00
Adam Rutkowski
978620f9e3 pyocf: move C wrappers to newly added "c" diretory 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-05-16 15:40:18 +02:00
Adam Rutkowski
f66eefb3bd pyocf: remember queue name 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-05-16 15:40:18 +02:00
Adam Rutkowski
7c186f5d0e pyocf: make queues a required rio param 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-05-16 15:40:18 +02:00
Adam Rutkowski
c37ffea4c2 pyocf: add from_page() method to Size class 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-05-16 15:40:16 +02:00
Robert Baldyga
ed012411d3 Add disable_cleaner field to CacheAttachConfig 
Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
 2022-04-28 13:04:27 +02:00
Robert Baldyga
94aca1e8e4 pyocf: Remove non-existing field from CacheAttachConfig 
This field has been moved to CacheDeviceConfig.

Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
 2022-04-28 13:04:27 +02:00
Adam Rutkowski
650511df4e pyocf: remove Queue objects from Cache::start() params 
... since a started cache is required to construct a queue object.
Instead add params to control whether to construct and register
management and default I/O queue in start().

Additionally introducing Cache method to register additional I/O
queues.

Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-04-13 10:38:57 +02:00
Robert Baldyga
a64fc61413
Merge pull request #666 from arutk/pyocf_vol_cleaniup 
pyocf: volume changes required for failover tests
 2022-03-29 10:02:01 +02:00
Robert Baldyga
cf2d4327a1
Merge pull request #667 from arutk/pyocf_failover_api 
pyocf: failover API in Cache
 2022-03-29 09:32:00 +02:00
Robert Baldyga
af43a240d3 Return more specific error on CRC mismatch 
Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
 2022-03-28 22:42:59 +02:00
Adam Rutkowski
5c39af81ca pyocf: failover api 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-03-28 16:24:43 +02:00
Adam Rutkowski
0e8d02235a pyocf: Move core/cache md5() logic to volume 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-03-28 10:37:11 +02:00
Adam Rutkowski
16c85c1560 Remove ocf_core_new_io 
This function is equivalent to calling ocf_volume_new_io()
on core front volume

Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-03-28 10:36:19 +02:00
Adam Rutkowski
9e88afad62 pyocf: cache and core exported object volumes 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-03-28 10:35:18 +02:00
Adam Rutkowski
d2164e4ffd pyocf: replicated volume 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-03-28 10:29:02 +02:00
Adam Rutkowski
dcf3129ff8 Extract Volume::open() class method 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-03-28 10:29:02 +02:00
Adam Rutkowski
d60e48ee26 pyocf: ocf_volume_new_io wrapper 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-03-28 10:29:02 +02:00
Adam Rutkowski
e0c762387c pyocf: implement get_(front_)volume() in Cache and Core 
This common interface is later going to be used for generic code
implementing cache/core exported object I/O.

Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-03-28 10:29:02 +02:00
Adam Rutkowski
2672f5460a pyocf: unique volume ops for each volume class 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-03-28 10:29:02 +02:00
Adam Rutkowski
7cb9523d8c pyocf: volume offlining 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-03-28 10:29:02 +02:00
Adam Rutkowski
33b134242b pyocf: remove trace volume 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-03-28 10:29:02 +02:00
Adam Rutkowski
838870fa10 pyocf: Use abstract Volume in generic code paths 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-03-28 10:29:02 +02:00
Adam Rutkowski
9e13364896 pyocf: Extract generic volume logic from RamVolume 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-03-28 10:29:02 +02:00
Adam Rutkowski
16f9d58f28 pyocf: Rename Volume to RamVolume 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-03-28 10:29:02 +02:00
Adam Rutkowski
062f63e4ff pyocf: Volume not to inherit from ctypes.Structure 
Volume object is never referenced from C code, so there
is no need to inherit from ctypes.Structure.

Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-03-28 10:29:02 +02:00
Adam Rutkowski
9975706488 pyocf: Move poison out of Volume class 
.. so that abstract Volume does not need to be imported to access
the poison value.

Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-03-28 10:29:02 +02:00
Adam Rutkowski
232133302b pyocf: use generic volume submit in io class 
.. instead of submit_core(). This will enable to reuse Io class
for cache exported object I/O.

Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-03-28 10:29:02 +02:00
Adam Rutkowski
2de7b9d8f2 pyocf: submit_flush() and submit_dicard() 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-03-28 10:29:02 +02:00
Adam Rutkowski
efd6389e74 pyocf: add get_default_queue to core object 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-03-28 10:29:02 +02:00
Adam Rutkowski
4a839cd332 Verify standby/active cache state in OCF entry points 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-03-28 09:42:02 +02:00
Robert Baldyga
aa4622cc01 Make core remove error recoverable 
First try to clean only the mapping. This operation does not require any
rollback, so even if flushing collision fails, core object is still
intact. In case of error we inform user that core was not removed by
returning new error code (-OCF_ERR_CORE_NOT_REMOVED).

After flushing collision succeeds we remove core from metadata and
flush superblock at the end. At that point the core is fully removed
from OCF and even if superblock flush error occurs there is nothing we
can do about it, so we just return the error code.

Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
 2022-03-25 21:32:55 +01:00
Robert Baldyga
d46845b97a Remove unused error code 
Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
 2022-03-09 11:43:35 +01:00
Robert Baldyga
d5b2c65a39 Remove "metadata_layout" parameter of the cache 
This feature is replaced with LRU list shuffling.

Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
 2022-03-07 17:48:25 +01:00
Robert Baldyga
765d1c5d77 rio: fix size comparison in 
Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
 2022-02-24 15:02:46 +01:00
Adam Rutkowski
a6d8bd0470 rio: accept queue list in Rio constructor 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
Signed-off-by: Jan Musial <jan.musial@intel.com>
 2022-02-24 14:56:09 +01:00
Adam Rutkowski
372b1f1e1c remove unnecessary callback decorator in rio 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-02-24 14:56:09 +01:00
Adam Rutkowski
9baf9a3876 rio: convert Size to bytes for extent calculation 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-02-24 14:56:09 +01:00
Adam Rutkowski
17eddab094 pyocf: queue settle functionality 
Queue settle is a mechanism to wait for all OCF processing
on a given queue to finish.

In some tests simply waiting for I/O to finish is not enough. Most
notably some statistics are potentially incremented after user triggered
I/O is finished. This is due to asynchronous nature of I/O operations
and OCF approach to statistics update, where only eventual consistency
is guaranteed without explicit mechanism available to query whether
the final state is reached yet. However it is fully in the adapter power
to determine this, as OCF executes in context of API calls from the
adapter (like I/O submission, ocf_queue_run, ocf_cleaner_run, management
operations) and I/O completion callbacks. Queue settle is a mechanism to
assure ocf_queue_run is not being executed by the thread associated with
a queue.

With queue settle mechanism it is straightforward for the adapter to
wait for cache statistics to reach fixed values:
1. wait for all I/O to OCF to finish
2. settle all I/O queues
3. make sure background cleaner is not active

Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-02-24 14:56:09 +01:00
Jan Musial
41732090ae Implement rio io tester for pyocf 
rio stands for Rigid IO tester and is a simple mechanism for testing
OCF cache IO.

Signed-off-by: Jan Musial <jan.musial@intel.com>
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-02-24 14:56:09 +01:00
Jan Musial
96124ca87a Enable Size comparisons in pyocf 
Signed-off-by: Jan Musial <jan.musial@intel.com>
 2022-02-24 14:56:09 +01:00
Jan Musial
973a474b37 Make open_cores load viable by adding getter for cores 
Signed-off-by: Jan Musial <jan.musial@intel.com>
 2022-02-24 12:27:10 +01:00
Jan Musial
bfee654770 Add wrapper for cache_config_set_default 
Signed-off-by: Jan Musial <jan.musial@intel.com>
 2022-02-24 12:27:10 +01:00
Jan Musial
2038c3aa68 Add basic load test with core reattachment and cleanup structures 
Signed-off-by: Jan Musial <jan.musial@intel.com>
 2022-02-24 12:27:10 +01:00
Jan Musial
30fe2eb783 Handle try_add core adding mode 
Signed-off-by: Jan Musial <jan.musial@intel.com>
 2022-02-24 12:27:10 +01:00
Jan Musial
1273e9ddfe Fix leaking Volumes and OcfCtx's 
Signed-off-by: Jan Musial <jan.musial@intel.com>
 2022-02-16 07:06:05 +01:00
Jan Musial
8f47cfa5bf Make multiple pyocf loggers viable 
Signed-off-by: Jan Musial <jan.musial@intel.com>
 2022-02-16 07:06:05 +01:00
Jan Musial
441e3c20d7 Clean-up pyocf c-python shared objects instance tracking 
Signed-off-by: Jan Musial <jan.musial@intel.com>
 2022-02-16 07:06:05 +01:00
Jan Musial
4c9d4fe24e Fix pyocf Ctx ownership 
Signed-off-by: Jan Musial <jan.musial@intel.com>
 2022-02-16 07:06:05 +01:00
Robert Baldyga
da59a1c9aa
Merge pull request #634 from arutk/mopft_21.12 
pyocf: management operation power failure tests
 2022-02-14 14:10:52 +01:00
Robert Baldyga
49abe816ce
Merge pull request #649 from pdebski21/1023 
fix for issue #1023
 2022-02-07 16:17:14 +01:00
Piotr Debski
9b980d3f22 fix for issue #1023 
Better error for core size mismatch during activation/load

adding pyocf test for new error code

Signed-off-by: Piotr Debski <piotr.debski@intel.com>
 2022-01-25 05:18:16 +01:00
Adam Rutkowski
ce0bbc260e pycof: explicitly free Volume::_uuid_ dictionary 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-01-13 17:35:33 +01:00
Adam Rutkowski
7f60aa0dd6 pyocf: fix error handling in Cache::stop() 
Write error in cache stop means the instance was in fact stopped.

Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-01-13 17:35:33 +01:00
Adam Rutkowski
ed5185e870 pyocf: make io class config more usable 
Unify field param names and include io class id in info struct.

Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-01-13 17:35:33 +01:00
Adam Rutkowski
5fe217d96f pyocf: set proper OCF error codes in Volume 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-01-13 17:35:33 +01:00
Adam Rutkowski
c43059a4cf pyocf: add get_bytes() function to Volume and Data 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-01-13 17:35:33 +01:00
Adam Rutkowski
683174c78f pyocf: add option to load cache without openning cores 
... this is useful to workaround current pyocf limitations and
load cache with manual core insertion

Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-01-13 17:35:33 +01:00
Adam Rutkowski
58dac85f7b pyocf: add ocf_core_get_uuid() wrapper 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-01-13 17:35:29 +01:00
Adam Rutkowski
c7144decf8 pyocf: extend error volume capabilities 
Adding option to
1. inject error based on I/O number
2. arm/disarm error injection for easier testing

Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-01-13 16:40:54 +01:00
Adam Rutkowski
cf72664c2c pyocf: fix cache device config 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2022-01-13 16:40:54 +01:00
Piotr Debski
609a22cfda added ERROR code for superblock mismatch 
Signed-off-by: Piotr Debski <piotr.debski@intel.com>
 2022-01-08 23:06:10 +01:00
Robert Baldyga
716b5751d6 Redesign failover standby API 
Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
 2021-12-29 20:31:40 +01:00
Robert Baldyga
6d1a5c733e pyocf: Add missing error codes 
Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
 2021-12-13 15:58:59 +01:00
Rafal Stefanowski
f22da1cde7 Fix license 
Change license to BSD-3-Clause

Signed-off-by: Rafal Stefanowski <rafal.stefanowski@intel.com>
 2021-10-28 13:08:50 +02:00
Michal Mielewczyk
ad536d9ad9 pyocf: fix cache attach config struct 
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
Signed-off-by: ajrutkow <adam.j.rutkowski@intel.com>
 2021-10-08 16:04:14 +02:00
Robert Baldyga
7d43aae3a0 pyocf: Add new error codes 
Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
 2021-09-06 20:44:40 +02:00
Michal Mielewczyk
1620c544a1 pyocf: update cleaning policy switching test 
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
 2021-07-27 15:44:33 +02:00
Kozlowski Mateusz
f494448f97 Align structures to cacheline 
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
 2021-07-08 12:46:39 +02:00
Adam Rutkowski
88e04a4204 Remove eviction policy abstraction 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2021-06-18 12:19:08 +02:00
Adam Rutkowski
f589341c9a remove metadata updater 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2021-06-15 10:25:05 +02:00
Jan Musial
9f1df6e380 Update pyOCF and tests 
Signed-off-by: Jan Musial <jan.musial@intel.com>
 2021-05-26 07:57:37 +02:00
Michal Mielewczyk
d5ae764175 pyocf: fix valid seq cutoff threshold range 
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
 2021-04-15 11:09:26 +02:00
Michal Mielewczyk
0bfa9ed870 pyocf: Seq-cutoff promotion count security tests 
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
 2021-04-14 15:40:29 +02:00
Kozlowski Mateusz
3f9af8bd82 Update pyocf types to new field order 
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
 2021-03-26 08:57:04 +01:00
Robert Baldyga
8020e7fd67
Merge pull request #457 from Ostrokrzew/false_stats 
Fix broken 'dirty_for' stats
 2021-03-18 10:24:02 +01:00
Robert Baldyga
9352c881ab tests: Update sequential cutoff tests 
Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
 2021-03-04 16:38:31 +01:00
Michal Mielewczyk
fa556247d7 pyocf: change encoding of ioclass name 
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
 2021-03-02 14:46:57 +01:00
Michal Mielewczyk
080e13a071 pyocf: valid ranges for ioclass config values 
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
 2021-03-02 14:46:57 +01:00
Michal Mielewczyk
06edc48717 pyocf: remove min_size from ioclass config 
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
 2021-03-02 14:46:57 +01:00
Michal Mielewczyk
73d6fb33de pyocf: api for setting core seq cutoff threshold 
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
 2021-02-26 08:24:41 -05:00
Michal Mielewczyk
9aebf57efa pyocf: valid ranges for seq cutoff threshold 
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
 2021-02-26 08:24:41 -05:00
Michal Mielewczyk
92f67c65f0 pyocf: valid ranges for cleaning policy params 
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
 2021-02-26 07:56:38 -05:00
Slawomir Jankowski
eeda1f3f0f Unify type of dirty_for in info structs 
Reformat function that calculates how long cache/core is dirty
Update `dirty_for` types in functional tests

Values stored in info structs fields (both in cache and core structs)
are unsigned 64-bits ints but `dirty_for`s were unsigned 32-bits ints.

Use existing function to transform returned value to seconds.
Replace seconds stored in metadata with seconds.
Replacement was done if old value of replaced field was equal to zero.
Acquiring monotonic high precission timestamp is potentially
slow and it makes sense to compare the field's value
to zero before calling atomic function.

Signed-off-by: Slawomir Jankowski <slawomir.jankowski@intel.com>
 2021-02-25 14:51:53 +01:00
Rafal Stefanowski
6ed4cf8a24 Update copyright statements (2021) 
Signed-off-by: Rafal Stefanowski <rafal.stefanowski@intel.com>
 2021-01-21 13:17:34 +01:00
Michal Mielewczyk
5c2a5c5e79 pyocf: add cache detach api 
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
 2020-12-21 08:23:34 -05:00
Michal Mielewczyk
93318696cd pyocf: ioclass api 
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
 2020-12-21 08:23:34 -05:00
Adam Rutkowski
693c07b865 pyocf: fix cache save method 
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
 2020-09-30 10:40:14 +02:00
Robert Baldyga
1c9312842a
Merge pull request #369 from rafalste/copyright_update 
Update copyright statements
 2020-05-06 12:42:10 +02:00
Rafal Stefanowski
38e7e19290 Update copyright statements 
Signed-off-by: Rafal Stefanowski <rafal.stefanowski@intel.com>
 2020-04-28 13:37:54 +02:00
Michal Rakowski
0f85a73669 pyocf: test_eviction fixup 
Signed-off-by: Michal Rakowski <michal.rakowski@intel.com>
 2020-04-24 15:34:59 +02:00
Robert Baldyga
188559416c
Merge pull request #354 from robertbaldyga/multistream-seq-cutoff 
Introduce multi-stream seqential cutoff
 2020-04-22 15:35:42 +02:00