Commit Graph

293 Commits

Author SHA1 Message Date
Kamil Lepek
bdfd086d81 Fix wb xfailed test, update exported obj md5
Signed-off-by: Kamil Lepek <kamil.lepek94@gmail.com>
2019-06-18 15:46:15 +02:00
Kamil Lepek
70142eef7f Pyocf minor api fixes
Signed-off-by: Kamil Lepek <kamil.lepek94@gmail.com>
2019-06-14 18:15:22 +02:00
Adam Rutkowski
34a5b8e882
Merge pull request #187 from KamilLepek/config
Create config which excludes tests marked as "long" and security tests by default
2019-06-14 11:23:36 -04:00
Kamil Lepek
e52d34c1c8 Adapt all python code to PEP8 style standards
Signed-off-by: Kamil Lepek <kamil.lepek94@gmail.com>
2019-06-13 17:55:41 +02:00
Adam Rutkowski
4a548d26cf functional tests: only send sector-aligned IO
... to satisfy OCF assumptions.

Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2019-06-12 12:07:02 -04:00
Adam Rutkowski
31ab2b3fe6 pyocf: add sector size logic to Size class
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2019-06-12 12:07:02 -04:00
Adam Rutkowski
641fba1708 Functional tests for WO cache mode
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2019-06-12 12:07:02 -04:00
Adam Rutkowski
a6312eb8aa Update management tests for WO (Write-only) cache mode
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2019-06-12 12:07:02 -04:00
Adam Rutkowski
7b7d65e092 pyocf: fix setting data offset in IO
When setting data in IO, the offset should be stored in
IO itself, not data object.

Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2019-06-12 12:07:01 -04:00
Kamil Lepek
d416d166f2 Create config which excludes tests marked as "long" and security tests by default
Signed-off-by: Kamil Lepek <kamil.lepek94@gmail.com>
2019-06-10 13:04:49 +02:00
Michal Mielewczyk
3e2ee7a7c8 UT framerwok: Auto generating wrap functions
Test code have to contain wraps only for those functions, which are used by
test itself. Empty wraps are generated automatically by UT framework.

Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
2019-06-06 08:20:25 -04:00
Michal Mielewczyk
e6bedb692c Unified management functions prefix.
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
2019-06-05 09:31:59 -04:00
Michal Rakowski
9f4536c6e3 Error codes in IO path changed to OCF-specific
Signed-off-by: Michal Rakowski <michal.rakowski@intel.com>
2019-06-05 09:10:54 +02:00
Robert Baldyga
711de86bff Associate core metadata with core object
Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2019-05-28 14:49:59 +02:00
Robert Baldyga
46ae8b2ff1 Remove unused OCF_ERR_CACHE_IN_USE error code
Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2019-05-28 11:20:48 +02:00
Robert Baldyga
8a82be339f Introduce asynchronous cache lock
Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2019-05-28 11:20:48 +02:00
Robert Baldyga
bdcd4df0ef Remove utils_device.h
Move core mngt related code to ocf_mngt_core.c

Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2019-05-27 17:00:39 +02:00
Robert Baldyga
7de56940a4 Move ocf_request from utils
ocf_request has always been first class citizen in OCF,
so lets place it along with another essential objects.

Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2019-05-27 15:51:27 +02:00
Robert Baldyga
ab2fc6d3c3 Rename utils_allocator to utils_realloc
Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2019-05-27 13:10:17 +02:00
Michał Mielewczyk
b1321edf69
Merge pull request #170 from robertbaldyga/convert-ut-to-python3
Convert UT scripts to python3
2019-05-27 09:58:26 +02:00
Jan Musiał
93a06686a7
Merge pull request #165 from robertbaldyga/ctx-get-put
Give ocf_ctx get/put semantics
2019-05-24 14:29:04 +02:00
Robert Baldyga
a671a28282 Convert UT scripts to python3
Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2019-05-24 10:13:50 +02:00
Robert Baldyga
c903d13ad2 Give ocf_ctx get/put semantics
Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2019-05-23 15:50:10 +02:00
Michał Mielewczyk
16a01d7698
Merge pull request #157 from KamilLepek/management
Added a few management tests (changing configuration)
2019-05-22 16:20:16 +02:00
Kamil Lepek
54c0e48fce Pyocf test checking if default values are correct
Signed-off-by: Kamil Lepek <kamil.lepek94@gmail.com>
2019-05-22 16:09:02 +02:00
Kamil Lepek
ecd492e562 Added a few management tests (changing configuration after start and checking statistics)
Fixed core/cache stats pyocf API

Signed-off-by: Kamil Lepek <kamil.lepek94@gmail.com>
2019-05-22 15:34:07 +02:00
Kamil Lepek
827a115fa9 Gather all unit tests logs in single directory
Signed-off-by: Kamil Lepek <kamil.lepek94@gmail.com>
2019-05-17 11:35:33 +02:00
Daniel Madej
3da9059e50
Merge pull request #143 from imjfckm/master
Fix loading of cache in pyocf
2019-05-17 10:59:04 +02:00
Kamil Lepek
382c31a8e0 Save unit tests output to file
Seperate output of different script stages

Signed-off-by: Kamil Lepek <kamil.lepek94@gmail.com>
2019-05-16 10:26:02 +02:00
Jan Musial
991bcf3491 Fix loading of cache in pyocf
Flush/load metadata paths are heavily dependend on Data behaving
correctly in terms of seeks/position and that needed to be fixed.

Signed-off-by: Jan Musial <jan.musial@intel.com>
2019-05-10 15:34:23 +02:00
Michal Mielewczyk
7165bc16c3 Add kick function for cleaner
ocf_kick_cleaner() allows to perfom cleaning immediately.

Nop cleaning policy now returns new 'OCF_CLEANER_DISABLE' macro which indicates
that cleaing shouldn't be performed. To enable it back, ocf_kick_cleaner()
should be called.

Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
2019-05-10 08:09:56 -04:00
Robert Bałdyga
1c9fe96663
Merge pull request #133 from arutk/ajrutkow_async_counters
Extended reference counting
2019-05-08 14:23:50 +02:00
Adam Rutkowski
1e812b4eaf Unit tests for reference counter utility
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2019-05-07 12:40:06 -04:00
Robert Baldyga
7b88aac56f Remove "interruption" argument from flush() functions
As non-interruptible flushes are no longer triggered from OCF
internals, we can get rid of "interruption" argument and let
adapters handle it themselves.

Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2019-05-07 17:23:19 +02:00
Adam Rutkowski
c8e6b96ece pyocf: remove put from cache stop to match get count
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2019-05-06 19:37:51 -04:00
Adam Rutkowski
2485584487 Remove obsolete ocf_cache_wait_for_io_finish from pyocf
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2019-05-06 19:37:51 -04:00
Adam Rutkowski
84f5454b29 Update unit tests to work with extedned ref counters
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2019-05-06 19:37:51 -04:00
Adam Rutkowski
93c8932a17 pyocf: fix management queue stop/put
Queue is stopped when reference counter drops to 0, so no need to
manually stop management queue after cache stop - simple put is
enough.

Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2019-04-25 13:21:25 -04:00
Katarzyna Łapińska
bf879c4b09
Merge pull request #113 from Deixx/tests_start_stop
pyocf tests for start/stop
2019-04-23 12:10:17 +02:00
Adam Rutkowski
e11edcacd8 Rename makefile ENV* variables to OCF_ENV*
'ENV' variable is set and used by OS utilities  on some Linux
distros, interfering with OCF build system. Renaming ENV and
other ENV* variables to OCF_ENV* to avoid conflicts.

Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2019-04-20 16:54:04 +02:00
Daniel Madej
1aa5e99173 Pyocf tests for start/stop
Tests for starting cache and checking if mode works properly
Tests for stopping cache
Negative and stress tests

Signed-off-by: Daniel Madej <daniel.madej@intel.com>
2019-04-18 17:00:14 +02:00
Adam Rutkowski
79cb39e273
Merge pull request #129 from imjfckm/master
Fix cores volumes dropping before close
2019-04-18 09:41:33 -04:00
Jan Musial
be88300071 Fix cores volumes dropping before close
* Add references to Cores in Cache, ctx holds caches, caches hold cores,
  everything gets cleaned up nicely

* GC in Python seems to be a bit lazy, if we want to run CI on
  low-memory machines we need to make sure it does run in between tests
  'cause we don't want no huge Volumes hanging around for long

Signed-off-by: Jan Musial <jan.musial@intel.com>
2019-04-18 14:11:59 +02:00
Daniel Madej
99c448aae6
Merge pull request #124 from katlapinka/security_config
Security tests for start cache command
2019-04-18 14:07:32 +02:00
Kamil Łepek
92837ec2f3
Merge pull request #119 from Deixx/tests_neg_io
Negative tests for IO path
2019-04-18 14:06:15 +02:00
klapinsk
f9eab4c749 Security tests for start cache command 2019-04-18 13:43:27 +02:00
Daniel Madej
876294f603 Negative tests for IO path
Signed-off-by: Daniel Madej <daniel.madej@intel.com>
2019-04-18 12:58:02 +02:00
Jan Musial
75c3948f6d Secure erase tests
Implement simple secure erase tests. Perform IO that will trigger
copying of Data buffers and make sure OCF calls secure erase on them.

Signed-off-by: Jan Musial <jan.musial@intel.com>
2019-04-18 12:51:59 +02:00
Michał Mielewczyk
d4e929140e
Merge pull request #127 from arutk/pytest_wait_io
pyocf: wait for pending I/O before cache stop
2019-04-18 09:21:50 +02:00
Katarzyna Łapińska
c80ca4a9a8
Merge pull request #122 from KamilLepek/security_tests
Add security fuzzy tests and improve pyocf api
2019-04-18 07:33:11 +02:00
Adam Rutkowski
d607402f2b pyocf: wait for pending I/O before cache stop
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2019-04-17 18:12:56 -04:00
Michal Mielewczyk
5e52ac4ef2 Rename 'printf' field name in logger ops to 'print'.
Due the aggresive security checks in compiler 'printf' might be substituded with
'__printf_chk'. However it does not differentiate whether substituted string is
library function call whether field in structure.

By renaming field we prevent it to be unintentionally subustituted by the
preprocessor.

Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
2019-04-17 11:37:47 -04:00
Kamil Lepek
d55195ecf5 Add security fuzzy tests and improve pyocf api
Fuzzy tests added for:
changing cache mode
attaching cache device with fuzzed cache line size
setting cleaning policy
setting alru params
setting acp params
setting seq cut off policy per cache
setting seq cut off policy per core
2019-04-17 12:46:25 +02:00
Kamil Łepek
00c434fb66
Merge pull request #115 from imjfckm/leaks
Fix leaky tests
2019-04-16 10:20:56 +02:00
Jan Musial
003d6505aa Fix leaky tests
Fix some instance tracking to prevent Python-side objects from leaking
buffers.
Also, reduce min size of Cache instance (real minimum should be around ~~19MiB,
but we need to make it more deterministic and 20 MiB seems to be reasonable).

Still some stuff left to do, but it needs more investigation and, for
now, this should suffice just to enable some form of CI.

Signed-off-by: Jan Musial <jan.musial@intel.com>
2019-04-16 09:55:02 +02:00
Kamil Lepek
465e9acd2c Add randomizers for ctypes and string in pyocf 2019-04-16 09:51:02 +02:00
Kamil Lepek
5ceb69e9c8 Added cache mode change in pyocf and basic test 2019-04-11 10:57:25 +02:00
Daniel Madej
f9c22262fb
Merge pull request #109 from KamilLepek/add_remove
Pyocf add/remove tests
2019-04-10 15:02:22 +02:00
Jan Musial
db46a9decb Stop using mgmt queue for IO in pyocf
Signed-off-by: Jan Musial <jan.musial@intel.com>
2019-04-10 11:25:39 +02:00
Kamil Lepek
c453bddc28 Pyocf add/remove tests 2019-04-09 10:17:39 +02:00
Daniel Madej
1fa52e84a8 Extract cache.flush() method from cache.stop()
Signed-off-by: Daniel Madej <daniel.madej@intel.com>
2019-04-04 11:13:45 +02:00
Jan Musial
be773a45af Add remove_core fcn
Signed-off-by: Jan Musial <jan.musial@intel.com>
2019-04-04 10:58:29 +02:00
Daniel Madej
d9d4f2fab1
Merge pull request #99 from katlapinka/ini
Ini
2019-04-04 10:50:39 +02:00
klapinsk
a8780655e2 *.ini files added 2019-04-04 10:29:20 +02:00
Jan Musial
753ab00331 Fix core_new_io
Signed-off-by: Jan Musial <jan.musial@intel.com>
2019-04-03 15:48:14 +02:00
Jan Musial
59178eef01 Make libocf.so loading CWD independent
Signed-off-by: Jan Musial <jan.musial@intel.com>
2019-04-03 12:28:15 +02:00
Jan Musial
545d5b8aac Make pyocf work with async API
Signed-off-by: Jan Musial <jan.musial@intel.com>
2019-04-03 11:13:27 +02:00
Adam Rutkowski
e06082c4a3 Treat warnings as errors in tests/build/Makefile
Signed-off-by: Adam Rutkowski <adam.j.rutkowski@intel.com>
2019-04-01 12:38:43 -04:00
Robert Baldyga
6cd84476f6 Handle metadata asynchronously
Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2019-03-26 12:49:23 +01:00
Robert Baldyga
72b6f35f52 Implement asynchronous save, add core and remove core
Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2019-03-21 09:03:59 +01:00
Robert Baldyga
23b0a32aec Parametrize pipeline steps
This allows to reuse same step functions giving them different parameters
on each step.

Additionally move pipeline to utils, to make it accessible to other
subsystems of OCF (e.g. metadata).

Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2019-03-20 17:06:56 +01:00
Robert Baldyga
91e0345b78 Implement asynchronous attach, load, detach and stop
NOTE: This is still not the real asynchronism. Metadata interfaces
are still not fully asynchronous.

Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
2019-03-18 15:06:57 +01:00
Robert Baldyga
c5df82f2cb Make management API asynchronous
NOTE: This patch only changes API that pretends to be asynchronous.
Most of management operations are still performed synchronously.
The real asynchronism will be introduced in the next patches.

Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2019-03-12 10:33:26 +01:00
Michal Mielewczyk
d06f376627 Alru cleaning policy improvement
Instead of calling flush separatly for each IO class, it is called after
collecting number of dirty cache lines defined by user or after iterating
through all IO classes.

Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
2019-03-11 04:08:16 -04:00
Michal Rakowski
e5227cef89 Implement pyocf adapter with sample OCF test
PyOCF is a tool written with testing OCF functionality in mind.
It is a Python3 (3.6 version required) package which wraps OCF
by providing Python objects in place of OCF objects (volumes, queues,
etc). Thin layer of translation between OCF objects and PyOCF objects
enables using customized behaviors for OCF primitives by subclassing
PyOCF classes.

This initial version implements only WT and WI modes and single,
synchronously operating Queue.

TO DO:

  - Queues/Cleaner/MetadataUpdater implemented as Python threads
  - Loading of caches from PyOCF Volumes (fix bugs in OCF)
  - Make sure it works multi-threaded for more sophisticated tests

Co-authored-by: Jan Musial <jan.musial@intel.com>
Signed-off-by: Michal Rakowski <michal.rakowski@intel.com>
Signed-off-by: Jan Musial <jan.musial@intel.com>
2019-03-07 12:48:40 +01:00
Robert Baldyga
9b9b965b55 Add cache locking functions in trylock variant
- Add cache trylock and read trylock functions.
- Introduce new error code -OCF_ERR_NO_LOCK.
- Change trylock functions in env to return this code in case of
  lock contention.

[ENV CHANGES REQUIRED]
Following functions should return 0 on success or -OCF_ERR_NO_LOCK
in case of lock contention:
- env_mutex_trylock()
- env_rwsem_up_read_trylock()
- env_rwsem_up_write_trylock()

Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2019-03-01 17:31:32 +01:00
Michal Mielewczyk
e53944d472 Dynamic I/O queue management
- Queue allocation is now separated from starting cache.
- Queue can be created and destroyed in runtime.
- All queue ops accept queue handle instead of queue id.
- Cache stores queues as list instead of array.

Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2019-02-26 17:36:19 +01:00
Robert Baldyga
404b976109 Make cleaner asynchronous
Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2019-02-19 16:10:24 +01:00
Robert Baldyga
97300b1137 Rename "data object" to "volume"
Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2019-02-08 14:13:05 +01:00
Michal Mielewczyk
2dfd6248c9 Configure all io classes at once.
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
2019-01-28 08:51:26 -05:00
Michal Mielewczyk
5e97795630 Fixed script for generating UT templates
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
2019-01-24 05:16:16 -05:00
Robert Baldyga
6860cdbd2c Reorganize context ops
Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2019-01-22 13:14:35 +01:00
Robert Baldyga
03c2f142b3 Cache lock cleanup and API improvements
Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2019-01-21 13:46:28 +01:00
Kamil Łepek
92b2fa362e Refactor tests directory
Signed-off-by: Kamil Łepek <kamil.lepek94@gmail.com>
2019-01-18 03:12:49 -06:00
Robert Baldyga
f933246dba Introduce basic OCF build test with posix environment
Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2019-01-16 14:10:29 +01:00
Robert Baldyga
ad10d58a2c Update posix environment
Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2019-01-15 09:41:33 +01:00
Michal Rakowski
ad3467ddd4 Added Makefile for ocf lib to be used in functional tests
Signed-off-by: Michal Rakowski <michal.rakowski@intel.com>
2019-01-08 12:58:28 +01:00
Robert Baldyga
4be6761a18 Core data object
Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2019-01-04 08:33:34 +01:00
Mariusz Barczak
15ed5b4450 OCF provides own macros for MIN, MAX, and DIV_ROUND_UP
Signed-off-by: Mariusz Barczak <mariusz.barczak@intel.com>
2018-12-13 10:59:23 -05:00
Robert Baldyga
db92083432 Unify req naming convention (rq -> req)
Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2018-12-12 13:36:34 +01:00
wolszews
0cb11b0d3c Remove OCF prefix 2018-12-06 16:04:30 +01:00
Michal Mielewczyk
35ae74e3fa Fix existing unit tests
Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
2018-12-06 06:30:50 -05:00
Robert Baldyga
a8e1ce8cc5 Initial commit
Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
2018-11-29 15:14:21 +01:00