Commit Graph

116 Commits

Author SHA1 Message Date
Michael Crosby
9af8d56cdd Allow image and snapshotkey to be updated
This allows a container's image and snapshot key to be updated via the
Update APIs.

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2018-04-05 17:01:07 -04:00
Akihiro Suda
83e35b3d3a enable native (formerly naive) snapshotter by default
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-04-02 13:29:19 +09:00
Phil Estes
dd1085c922
Fix typo in metadata test typeurl string
Incorrectly spelled "containerd" in test string.

Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com>
2018-03-28 12:35:42 -04:00
Derek McGowan
43d0a5cb60
Pass in context to lease done function in client
Allows the client to choose the context to finish the lease.
This allows the client to switch contexts when the main context
used to the create the lease may have been cancelled.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-03-22 14:09:24 -07:00
Stephen Day
220a479292
Merge pull request #1833 from dmcgowan/snapshot-gc-3rd-phase
snapshots/overlay: add overlay cleanup
2018-03-12 17:49:56 -07:00
Stephen Day
b3b95c0a2a
Merge pull request #2154 from dmcgowan/shared-content-ingests
content: shared content across namespaces
2018-03-12 16:11:32 -07:00
Lantao Liu
1128b3d664 Add service plugin and support in process integration.
Signed-off-by: Lantao Liu <lantaol@google.com>
2018-03-12 18:03:50 +00:00
Derek McGowan
a1a67899f8
Shared content across namespaces
Update content ingests to use content from another namespace.
Ingests must be committed to make content available and the
client will see the sharing as an ingest which has already
been fully written to, but not completed.

Updated the database version to change the ingest record in
the database from a link key to an object with a link and
expected value. This expected value is used to indicate that
the content already exists and an underlying writer may
not yet exist.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-02-22 14:45:10 -08:00
Derek McGowan
b3aeba7062
Allow test runners to wrap contexts
Let the test runners choose the namespaces and
wrap the contexts. This allows the test suite to create
multiple contexts without worrying about namespacing
or leasing in the contexts.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-02-22 14:45:10 -08:00
Kunal Kushwaha
b12c3215a0 Licence header added
Signed-off-by: Kunal Kushwaha <kushwaha_kunal_v7@lab.ntt.co.jp>
2018-02-19 10:32:26 +09:00
Derek McGowan
14d402e289
Add overlay cleanup
Updates overlay remove to simply remove the reference, adds
a cleanup method for discarding the directory.
Updates snapshot create to setup the directory structure while
in the transaction, to prevent cleanup from removing directories
which are part of a create.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-01-30 16:17:41 -08:00
Stephen J Day
3295bca5bf
content/testsuite: pass context to hold lease
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2018-01-29 16:29:51 -08:00
Daniel Nephin
1cd32fa68d Cleanup extra returns in tests
Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2018-01-23 14:09:22 -05:00
Stephen Day
acc6011ac1
Merge pull request #1960 from dmcgowan/images-removal-dirty
metadata: image removal triggers GC
2018-01-17 13:58:39 -08:00
Michael Crosby
45fbc3b3c2
Merge pull request #1975 from dnephin/add-unconvert-linter
Add unconvert linter
2018-01-10 10:54:00 -05:00
Daniel Nephin
184bc25629 Add unconvert linter
This linter checks for unnecessary type convertions.

Some convertions are whitelisted because their type is different
on 32bit platforms

Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2018-01-09 17:36:44 -05:00
Derek McGowan
0d8b093361
metadata: add content lease on existing content
When a writer is requested for an object that already
exists, add that object to the provided any lease to prevent
other operations from affecting the current lease's use of
that content.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-01-09 14:33:30 -08:00
Derek McGowan
89fa154efd
Update metadata image store to be initialized once
The boltdb image store now manages its own transactions when
one is not provided, but allows the caller to pass in a
transaction through the context. This makes the image store
more similar to the content and snapshot stores. Additionally,
use the reference to the metadata database to mark the content
store as dirty after an image has been deleted. The deletion
of an image means a reference to a piece of content is gone
and therefore garbage collection should be run to check if
any resources can be cleaned up as a result.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-01-05 13:38:45 -08:00
Derek McGowan
5a54862ae5
Update namespace empty check to use buckets
Directly get and check whether a bucket is empty.
This prevents unnecessarily loading all records of
the buckets into memory just to check existence.
Also added checks for content and snapshots.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-01-05 13:38:45 -08:00
Daniel Nephin
06edd193ef Small refactor of gc/scheduler to remove import of metadata
Replace metadata.GCStats with an interface for exposing elapsed time

Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2018-01-03 14:36:11 -05:00
Derek McGowan
b28d7cdf1b
Update snapshot and content related log messages
Make the tense and casing consistent.
Add useful log messages in image service.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-12-01 14:57:34 -08:00
Akihiro Suda
222156d598
Merge pull request #1847 from jessvalarezo/metadata-errors
metadata: update container error messages
2017-12-02 04:42:40 +09:00
Jess Valarezo
b873ae8b55 metadata: update container error messages
Signed-off-by: Jess Valarezo <valarezo.jessica@gmail.com>
2017-12-01 10:32:48 -08:00
yason
4f1c0e067e improve error message
Signed-off-by: yason <yan.xuean@zte.com.cn>
2017-12-01 15:45:15 +08:00
Jess Valarezo
9885edfc44 rename snapshot->snapshots pkg
Signed-off-by: Jess Valarezo <valarezo.jessica@gmail.com>
2017-11-29 14:55:02 -08:00
Stephen Day
372cdfac3b
Merge pull request #1638 from dmcgowan/gc-policy
gc: add policy plugin
2017-11-27 18:20:10 -08:00
Derek McGowan
f011bec35e
Merge pull request #1800 from yanxuean/image-store-redandunt
remove redandunt check for imagestore create
2017-11-27 17:46:47 -08:00
yanxuean
3b670ebc94 remove redundant check for imagestore create
correct check object for imagestore update
Signed-off-by: yanxuean <yan.xuean@zte.com.cn>
2017-11-28 09:24:46 +08:00
Derek McGowan
92d40b9a2d
Add test for container as gc root
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-11-21 16:26:36 -08:00
Daniel Nephin
f74862a0dd Add structcheck, unused, and varcheck linters.
Warn on unused and dead code

Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2017-11-21 11:14:37 -05:00
Derek McGowan
3f1a61f76a
Add synchronous image delete
Synchronous image delete provides an option image delete to wait
until the next garbage collection deletes after an image is removed
before returning success to the caller.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-11-20 17:08:35 -08:00
Derek McGowan
00596f400e
Add gc policy plugin
Add garbage collection as a background process and policy
configuration for configuring when to run garbage collection.
By default garbage collection will run when deletion occurs
and no more than 20ms out of every second.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-11-20 16:57:39 -08:00
Stephen Day
9cb91f8bfa
Merge pull request #1765 from dmcgowan/rm-lease-resources
Add support for removing leases on resource removal
2017-11-16 18:53:53 -08:00
Kenfe-Mickaël Laventure
cc7e5934c8
Merge pull request #1760 from dnephin/run-unit-tests-on-windows
Re-enable unit tests on appveyor
2017-11-15 18:53:16 -08:00
Daniel Nephin
2e7f7318cc Normalize 'already exists' errors
and fix some error messages where they were wrong or redundant

Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2017-11-15 16:46:27 -05:00
Jess Valarezo
f3a63f52dc
Add support for removing leases on resource removal
Signed-off-by: Jessica Valarezo <valarezo.jessica@gmail.com>
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-11-15 12:09:59 -08:00
Daniel Nephin
a72279e53d Skip some tests on windows where the implementation is missing
Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2017-11-15 14:52:00 -05:00
Akihiro Suda
4feb6f228a snapshot: add Close()
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-11-14 02:00:33 +00:00
Derek McGowan
e13894bb7a
Add leases api
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-11-07 12:54:22 -08:00
Stephen Day
c933088381
Merge pull request #1663 from dmcgowan/content-local-labels
content: add support for label storage in local content store
2017-11-01 11:24:15 -07:00
Derek McGowan
7b08bcdb65
Add support for label storage in local content store
Allows running tests which require labels on the content store

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-10-31 12:59:52 -07:00
Derek McGowan
432670237c
Fix race in gc sweep
Removes extra goroutine and calls removal and scan in same thread

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-10-30 13:56:16 -07:00
Michael Crosby
e74c423f78 Fix races with concurrent GC
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-10-27 14:47:23 -04:00
Michael Crosby
b6e0c4f321 Fix go lint errors
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-10-25 15:26:44 -04:00
Michael Crosby
5fd0415985 Add comments and fix common lint issues
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-10-20 13:19:14 -04:00
Derek McGowan
923a5c873b
Update content store garbage collection
Use single instance of content store instead of
creating new one for each collection. Using new
instance and wrapping causes failures.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-10-18 15:30:12 -07:00
Derek McGowan
295cce278f
Use single instance of snapshot and content stores
Ensures that all callers and the garbage collector are using
the same lock instances to prevent cleanup of objects
during creation.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-10-16 11:57:50 -07:00
Derek McGowan
17471d5592
Metadata garbage collection
Marks and sweeps unreferenced objects.
Add snapshot cleanup to metadata.
Add content garbage collection

Add dirty flags for snapshotters and content store which
are set on deletion and used during the next garbage collection.
Cleanup content store backend when content metadata is removed.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-10-11 10:42:47 -07:00
Derek McGowan
e4defbcb6d
Add documentation for metadata database
Add documentation and explanation for migrations.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-10-06 13:33:07 -07:00
Derek McGowan
7f657ce3de
Add database migrations
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-10-05 16:35:58 -07:00