Commit Graph

129 Commits

Author SHA1 Message Date
Samuel Karp
9665a2650a *: replace 3600 seconds with 1 hour
Signed-off-by: Samuel Karp <skarp@amazon.com>
2018-08-02 11:40:03 -07:00
Phil Estes
4249f44d81
Merge pull request #2493 from dmcgowan/sync-lease-removal
Add sync option to lease removal
2018-07-31 11:31:55 -04:00
Derek McGowan
dfc9991135
Add content ingests to lease and gc
Allow content ingests to be cleaned up during gc.
Use a default expiration on content ingests or make
use of the lease expiration when provided.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-07-25 16:54:14 -07:00
Derek McGowan
b760cee65a
Update lease service errors
Ensure delete returns a typed error on not found

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-07-25 14:56:35 -07:00
Derek McGowan
94e132fd07
Add sync option on lease removal
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-07-19 14:37:55 -07:00
Derek McGowan
29b72d4ff0
Support lease filters
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-07-18 10:43:37 -07:00
Derek McGowan
8cf3fad8d4
Add leases manager interface
Add leases manager to the leases package and use the
interface on the client and service.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-07-18 10:43:37 -07:00
Derek McGowan
c77c89b3d1
Add lease expiration to garbage collection
Allow setting an expiration label to have the garbage
collector remove an item after the specified time.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-07-18 00:45:06 -07:00
sudeesh john
5a4f007e48 Fix the formatting directives error during compilation
Signed-off-by: Sudeesh John <sudeesh@linux.vnet.ibm.com>
2018-07-05 20:40:21 +05:30
Akihiro Suda
d88de4a34f content: change Writer/ReaderAt to take OCI
This change allows implementations to resolve the location of the actual data
using OCI descriptor fields such as MediaType.

No OCI descriptor field is written to the store.

No change on gRPC API.

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-06-01 11:51:43 +09:00
Michael Crosby
009ba4d797 Move testutils to pkg
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2018-05-22 17:08:38 -04:00
Michael Crosby
2b565da7ec Add restart monitor
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2018-05-16 13:07:23 -04:00
Derek McGowan
f5e3e67dad
gc: add support for multiple snapshot labels
Allows linking to multiple snapshots within the same snapshotter.
Adds support for using slash to separate content as well for consistency.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-04-11 15:56:01 -07:00
Michael Crosby
9af8d56cdd Allow image and snapshotkey to be updated
This allows a container's image and snapshot key to be updated via the
Update APIs.

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2018-04-05 17:01:07 -04:00
Akihiro Suda
83e35b3d3a enable native (formerly naive) snapshotter by default
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-04-02 13:29:19 +09:00
Phil Estes
dd1085c922
Fix typo in metadata test typeurl string
Incorrectly spelled "containerd" in test string.

Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com>
2018-03-28 12:35:42 -04:00
Derek McGowan
43d0a5cb60
Pass in context to lease done function in client
Allows the client to choose the context to finish the lease.
This allows the client to switch contexts when the main context
used to the create the lease may have been cancelled.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-03-22 14:09:24 -07:00
Stephen Day
220a479292
Merge pull request #1833 from dmcgowan/snapshot-gc-3rd-phase
snapshots/overlay: add overlay cleanup
2018-03-12 17:49:56 -07:00
Stephen Day
b3b95c0a2a
Merge pull request #2154 from dmcgowan/shared-content-ingests
content: shared content across namespaces
2018-03-12 16:11:32 -07:00
Lantao Liu
1128b3d664 Add service plugin and support in process integration.
Signed-off-by: Lantao Liu <lantaol@google.com>
2018-03-12 18:03:50 +00:00
Derek McGowan
a1a67899f8
Shared content across namespaces
Update content ingests to use content from another namespace.
Ingests must be committed to make content available and the
client will see the sharing as an ingest which has already
been fully written to, but not completed.

Updated the database version to change the ingest record in
the database from a link key to an object with a link and
expected value. This expected value is used to indicate that
the content already exists and an underlying writer may
not yet exist.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-02-22 14:45:10 -08:00
Derek McGowan
b3aeba7062
Allow test runners to wrap contexts
Let the test runners choose the namespaces and
wrap the contexts. This allows the test suite to create
multiple contexts without worrying about namespacing
or leasing in the contexts.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-02-22 14:45:10 -08:00
Kunal Kushwaha
b12c3215a0 Licence header added
Signed-off-by: Kunal Kushwaha <kushwaha_kunal_v7@lab.ntt.co.jp>
2018-02-19 10:32:26 +09:00
Derek McGowan
14d402e289
Add overlay cleanup
Updates overlay remove to simply remove the reference, adds
a cleanup method for discarding the directory.
Updates snapshot create to setup the directory structure while
in the transaction, to prevent cleanup from removing directories
which are part of a create.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-01-30 16:17:41 -08:00
Stephen J Day
3295bca5bf
content/testsuite: pass context to hold lease
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2018-01-29 16:29:51 -08:00
Daniel Nephin
1cd32fa68d Cleanup extra returns in tests
Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2018-01-23 14:09:22 -05:00
Stephen Day
acc6011ac1
Merge pull request #1960 from dmcgowan/images-removal-dirty
metadata: image removal triggers GC
2018-01-17 13:58:39 -08:00
Michael Crosby
45fbc3b3c2
Merge pull request #1975 from dnephin/add-unconvert-linter
Add unconvert linter
2018-01-10 10:54:00 -05:00
Daniel Nephin
184bc25629 Add unconvert linter
This linter checks for unnecessary type convertions.

Some convertions are whitelisted because their type is different
on 32bit platforms

Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2018-01-09 17:36:44 -05:00
Derek McGowan
0d8b093361
metadata: add content lease on existing content
When a writer is requested for an object that already
exists, add that object to the provided any lease to prevent
other operations from affecting the current lease's use of
that content.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-01-09 14:33:30 -08:00
Derek McGowan
89fa154efd
Update metadata image store to be initialized once
The boltdb image store now manages its own transactions when
one is not provided, but allows the caller to pass in a
transaction through the context. This makes the image store
more similar to the content and snapshot stores. Additionally,
use the reference to the metadata database to mark the content
store as dirty after an image has been deleted. The deletion
of an image means a reference to a piece of content is gone
and therefore garbage collection should be run to check if
any resources can be cleaned up as a result.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-01-05 13:38:45 -08:00
Derek McGowan
5a54862ae5
Update namespace empty check to use buckets
Directly get and check whether a bucket is empty.
This prevents unnecessarily loading all records of
the buckets into memory just to check existence.
Also added checks for content and snapshots.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-01-05 13:38:45 -08:00
Daniel Nephin
06edd193ef Small refactor of gc/scheduler to remove import of metadata
Replace metadata.GCStats with an interface for exposing elapsed time

Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2018-01-03 14:36:11 -05:00
Derek McGowan
b28d7cdf1b
Update snapshot and content related log messages
Make the tense and casing consistent.
Add useful log messages in image service.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-12-01 14:57:34 -08:00
Akihiro Suda
222156d598
Merge pull request #1847 from jessvalarezo/metadata-errors
metadata: update container error messages
2017-12-02 04:42:40 +09:00
Jess Valarezo
b873ae8b55 metadata: update container error messages
Signed-off-by: Jess Valarezo <valarezo.jessica@gmail.com>
2017-12-01 10:32:48 -08:00
yason
4f1c0e067e improve error message
Signed-off-by: yason <yan.xuean@zte.com.cn>
2017-12-01 15:45:15 +08:00
Jess Valarezo
9885edfc44 rename snapshot->snapshots pkg
Signed-off-by: Jess Valarezo <valarezo.jessica@gmail.com>
2017-11-29 14:55:02 -08:00
Stephen Day
372cdfac3b
Merge pull request #1638 from dmcgowan/gc-policy
gc: add policy plugin
2017-11-27 18:20:10 -08:00
Derek McGowan
f011bec35e
Merge pull request #1800 from yanxuean/image-store-redandunt
remove redandunt check for imagestore create
2017-11-27 17:46:47 -08:00
yanxuean
3b670ebc94 remove redundant check for imagestore create
correct check object for imagestore update
Signed-off-by: yanxuean <yan.xuean@zte.com.cn>
2017-11-28 09:24:46 +08:00
Derek McGowan
92d40b9a2d
Add test for container as gc root
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-11-21 16:26:36 -08:00
Daniel Nephin
f74862a0dd Add structcheck, unused, and varcheck linters.
Warn on unused and dead code

Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2017-11-21 11:14:37 -05:00
Derek McGowan
3f1a61f76a
Add synchronous image delete
Synchronous image delete provides an option image delete to wait
until the next garbage collection deletes after an image is removed
before returning success to the caller.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-11-20 17:08:35 -08:00
Derek McGowan
00596f400e
Add gc policy plugin
Add garbage collection as a background process and policy
configuration for configuring when to run garbage collection.
By default garbage collection will run when deletion occurs
and no more than 20ms out of every second.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-11-20 16:57:39 -08:00
Stephen Day
9cb91f8bfa
Merge pull request #1765 from dmcgowan/rm-lease-resources
Add support for removing leases on resource removal
2017-11-16 18:53:53 -08:00
Kenfe-Mickaël Laventure
cc7e5934c8
Merge pull request #1760 from dnephin/run-unit-tests-on-windows
Re-enable unit tests on appveyor
2017-11-15 18:53:16 -08:00
Daniel Nephin
2e7f7318cc Normalize 'already exists' errors
and fix some error messages where they were wrong or redundant

Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2017-11-15 16:46:27 -05:00
Jess Valarezo
f3a63f52dc
Add support for removing leases on resource removal
Signed-off-by: Jessica Valarezo <valarezo.jessica@gmail.com>
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-11-15 12:09:59 -08:00
Daniel Nephin
a72279e53d Skip some tests on windows where the implementation is missing
Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2017-11-15 14:52:00 -05:00