Commit Graph

151 Commits

Author SHA1 Message Date
Daniel Nephin
184bc25629 Add unconvert linter
This linter checks for unnecessary type convertions.

Some convertions are whitelisted because their type is different
on 32bit platforms

Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2018-01-09 17:36:44 -05:00
Derek McGowan
0d8b093361 metadata: add content lease on existing content
When a writer is requested for an object that already
exists, add that object to the provided any lease to prevent
other operations from affecting the current lease's use of
that content.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-01-09 14:33:30 -08:00
Derek McGowan
89fa154efd Update metadata image store to be initialized once
The boltdb image store now manages its own transactions when
one is not provided, but allows the caller to pass in a
transaction through the context. This makes the image store
more similar to the content and snapshot stores. Additionally,
use the reference to the metadata database to mark the content
store as dirty after an image has been deleted. The deletion
of an image means a reference to a piece of content is gone
and therefore garbage collection should be run to check if
any resources can be cleaned up as a result.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-01-05 13:38:45 -08:00
Derek McGowan
5a54862ae5 Update namespace empty check to use buckets
Directly get and check whether a bucket is empty.
This prevents unnecessarily loading all records of
the buckets into memory just to check existence.
Also added checks for content and snapshots.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-01-05 13:38:45 -08:00
Daniel Nephin
06edd193ef Small refactor of gc/scheduler to remove import of metadata
Replace metadata.GCStats with an interface for exposing elapsed time

Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2018-01-03 14:36:11 -05:00
Derek McGowan
b28d7cdf1b Update snapshot and content related log messages
Make the tense and casing consistent.
Add useful log messages in image service.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-12-01 14:57:34 -08:00
Akihiro Suda
222156d598 Merge pull request #1847 from jessvalarezo/metadata-errors
metadata: update container error messages
2017-12-02 04:42:40 +09:00
Jess Valarezo
b873ae8b55 metadata: update container error messages
Signed-off-by: Jess Valarezo <valarezo.jessica@gmail.com>
2017-12-01 10:32:48 -08:00
yason
4f1c0e067e improve error message
Signed-off-by: yason <yan.xuean@zte.com.cn>
2017-12-01 15:45:15 +08:00
Jess Valarezo
9885edfc44 rename snapshot->snapshots pkg
Signed-off-by: Jess Valarezo <valarezo.jessica@gmail.com>
2017-11-29 14:55:02 -08:00
Stephen Day
372cdfac3b Merge pull request #1638 from dmcgowan/gc-policy
gc: add policy plugin
2017-11-27 18:20:10 -08:00
Derek McGowan
f011bec35e Merge pull request #1800 from yanxuean/image-store-redandunt
remove redandunt check for imagestore create
2017-11-27 17:46:47 -08:00
yanxuean
3b670ebc94 remove redundant check for imagestore create
correct check object for imagestore update
Signed-off-by: yanxuean <yan.xuean@zte.com.cn>
2017-11-28 09:24:46 +08:00
Derek McGowan
92d40b9a2d Add test for container as gc root
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-11-21 16:26:36 -08:00
Daniel Nephin
f74862a0dd Add structcheck, unused, and varcheck linters.
Warn on unused and dead code

Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2017-11-21 11:14:37 -05:00
Derek McGowan
3f1a61f76a Add synchronous image delete
Synchronous image delete provides an option image delete to wait
until the next garbage collection deletes after an image is removed
before returning success to the caller.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-11-20 17:08:35 -08:00
Derek McGowan
00596f400e Add gc policy plugin
Add garbage collection as a background process and policy
configuration for configuring when to run garbage collection.
By default garbage collection will run when deletion occurs
and no more than 20ms out of every second.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-11-20 16:57:39 -08:00
Stephen Day
9cb91f8bfa Merge pull request #1765 from dmcgowan/rm-lease-resources
Add support for removing leases on resource removal
2017-11-16 18:53:53 -08:00
Kenfe-Mickaël Laventure
cc7e5934c8 Merge pull request #1760 from dnephin/run-unit-tests-on-windows
Re-enable unit tests on appveyor
2017-11-15 18:53:16 -08:00
Daniel Nephin
2e7f7318cc Normalize 'already exists' errors
and fix some error messages where they were wrong or redundant

Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2017-11-15 16:46:27 -05:00
Jess Valarezo
f3a63f52dc Add support for removing leases on resource removal
Signed-off-by: Jessica Valarezo <valarezo.jessica@gmail.com>
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-11-15 12:09:59 -08:00
Daniel Nephin
a72279e53d Skip some tests on windows where the implementation is missing
Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2017-11-15 14:52:00 -05:00
Akihiro Suda
4feb6f228a snapshot: add Close()
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-11-14 02:00:33 +00:00
Derek McGowan
e13894bb7a Add leases api
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-11-07 12:54:22 -08:00
Stephen Day
c933088381 Merge pull request #1663 from dmcgowan/content-local-labels
content: add support for label storage in local content store
2017-11-01 11:24:15 -07:00
Derek McGowan
7b08bcdb65 Add support for label storage in local content store
Allows running tests which require labels on the content store

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-10-31 12:59:52 -07:00
Derek McGowan
432670237c Fix race in gc sweep
Removes extra goroutine and calls removal and scan in same thread

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-10-30 13:56:16 -07:00
Michael Crosby
e74c423f78 Fix races with concurrent GC
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-10-27 14:47:23 -04:00
Michael Crosby
b6e0c4f321 Fix go lint errors
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-10-25 15:26:44 -04:00
Michael Crosby
5fd0415985 Add comments and fix common lint issues
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-10-20 13:19:14 -04:00
Derek McGowan
923a5c873b Update content store garbage collection
Use single instance of content store instead of
creating new one for each collection. Using new
instance and wrapping causes failures.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-10-18 15:30:12 -07:00
Derek McGowan
295cce278f Use single instance of snapshot and content stores
Ensures that all callers and the garbage collector are using
the same lock instances to prevent cleanup of objects
during creation.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-10-16 11:57:50 -07:00
Derek McGowan
17471d5592 Metadata garbage collection
Marks and sweeps unreferenced objects.
Add snapshot cleanup to metadata.
Add content garbage collection

Add dirty flags for snapshotters and content store which
are set on deletion and used during the next garbage collection.
Cleanup content store backend when content metadata is removed.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-10-11 10:42:47 -07:00
Derek McGowan
e4defbcb6d Add documentation for metadata database
Add documentation and explanation for migrations.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-10-06 13:33:07 -07:00
Derek McGowan
7f657ce3de Add database migrations
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-10-05 16:35:58 -07:00
Derek McGowan
8d892a651b Update metadata plugin initialization
Updates metadata plugin to require content and
snapshotter plugins be loaded and initializes with
those plugins, keeping the metadata database structure
static after initialization. Service plugins now only
require metadata plugin access snapshotter or content
stores through metadata, which was already required
behavior of the services.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-10-04 11:25:46 -07:00
Derek McGowan
2ab70f21ac Add content and snapshot store references
Update database object to hold reference to the
data stores.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-10-04 11:16:22 -07:00
Derek McGowan
447a0a9452 Add children bucket to back reference snapshots
Adds back links from parent to children in order to prevent
deletion of a referenced snapshot in a namespace.
Avoid removing snapshot during metadata delete to
prevent shared namespaces from being mistakenly deleted.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-10-04 11:16:22 -07:00
Derek McGowan
56c1b79a4c Create metadata db object
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-10-04 11:16:22 -07:00
Stephen Day
f6d88927d5 Merge pull request #1583 from stevvooe/image-storage-validation
metadata: validation and testing of image store
2017-10-03 16:48:38 -07:00
Stephen J Day
698b6d15f7 metadata: validation and testing of image store
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-10-03 16:34:00 -07:00
Michael Crosby
451421b615 Comment more packages to pass go lint
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-10-02 13:54:56 -04:00
Jess Valarezo
18c4322bb3 Labels are consistently validated across services
* The combined size of a key/value pair cannot exceed 4096 bytes

Signed-off-by: Jess Valarezo <valarezo.jessica@gmail.com>
2017-09-21 15:11:46 -07:00
Michael Crosby
d22160c28e Vendor typeurl package
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-09-19 09:43:55 -04:00
Stephen J Day
f7d31e2be4 api, metadata: use map type for Container.Extensions
To allow for updating extensions without collisions, we have moved to
using a map type that can be explicitly selected via the field path for
updates. This ensures that multiple parties can operate on their
extensions without stepping on each other's toes or incurring an
inordinate number of round trips.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-09-08 19:34:26 -07:00
Brian Goff
3552ce5688 Add field to Container for client-defined data
This field allows a client to store specialized information in the
container metadata rather than having to store this itself and keep
the data in sync with containerd.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2017-09-07 16:54:47 -04:00
Akihiro Suda
e30e0c8b75 api: RootFS -> SnapshotKey
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-09-06 15:22:19 -07:00
Kenfe-Mickaël Laventure
e1eeb0e0a2 Merge pull request #1475 from dmcgowan/content-commit-context
Add context to content commit
2017-09-06 11:04:31 -07:00
Derek McGowan
9613acb2ed Add context to content commit
Content commit is updated to take in a context, allowing
content to be committed within the same context the writer
was in. This is useful when commit may be able to use more
context to complete the action rather than creating its own.
An example of this being useful is for the metadata implementation
of content, having a context allows tests to fully create
content in one database transaction by making use of the context.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-09-06 10:19:12 -07:00
Kenfe-Mickael Laventure
e0d8cb1366 Fix retrieval of container Runtime.Options field
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2017-08-31 12:05:39 -07:00