488 Commits

Author SHA1 Message Date
Kir Kolyshkin
9d0d4b806c context pkg: untangle
Since Go 1.7, "context" is a standard package, superceding the
"x/net/context". Since Go 1.9, the latter only provides type aliases
from the former. Therefore, it makes sense to switch to the standard
package, and the change is not disruptive in any sense.

This commit deals with a few cases where both packages happened to be
imported by the same source file. A choice between "context" and
"gocontext" was made for each file in order to minimize the patch.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2018-04-24 14:33:34 -07:00
Michael Wan
7fd6d5e2be fix words misspell
Signed-off-by: Michael Wan <zirenwan@gmail.com>
2018-04-23 00:09:42 -04:00
Stephen J Day
caac3bca3e protobuf: remove generated google/rpc files
We move from having our own generated version of the googleapis files to
an upstream version that is present in gogo. As part of this, we update
the protobuf package to 1.0 and make some corrections for slight
differences in the generated code.

The impact of this change is very low.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2018-04-05 14:36:32 -07:00
Akihiro Suda
7b323b1402 services/content: fix reading a blob which is smaller than the read buffer.
The newly added test fails without this fix in services/content/service.go:

    $ go test -c . && sudo ./containerd.test -test.v -test.root -test.run TestContentClient
    ...
        --- FAIL: TestContentClient/SmallBlob (0.02s)
            provideringester.go:62: rpc error: code = OutOfRange desc = read
    past object length 6 bytes
            helpers.go:67: drwx------       4096
    /tmp/content-suite-ContentClient-286788688
    FAIL

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-03-23 12:51:28 +09:00
Stephen Day
b3b95c0a2a Merge pull request #2154 from dmcgowan/shared-content-ingests
content: shared content across namespaces
2018-03-12 16:11:32 -07:00
Lantao Liu
1128b3d664 Add service plugin and support in process integration.
Signed-off-by: Lantao Liu <lantaol@google.com>
2018-03-12 18:03:50 +00:00
Derek McGowan
a1a67899f8 Shared content across namespaces
Update content ingests to use content from another namespace.
Ingests must be committed to make content available and the
client will see the sharing as an ingest which has already
been fully written to, but not completed.

Updated the database version to change the ingest record in
the database from a link key to an object with a link and
expected value. This expected value is used to indicate that
the content already exists and an underlying writer may
not yet exist.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-02-22 14:45:10 -08:00
Kunal Kushwaha
b12c3215a0 Licence header added
Signed-off-by: Kunal Kushwaha <kushwaha_kunal_v7@lab.ntt.co.jp>
2018-02-19 10:32:26 +09:00
Derek McGowan
b763777288 diff: rename differ to comparer
Remove combined interface and split implementations.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-01-26 16:32:09 -08:00
Akihiro Suda
b580441f91 diff: resplit Applier from Differ
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-01-25 13:58:34 -08:00
Darren Stahl
a5a9f91832 Implement Windows snapshotter and differ
This implements the Windows snapshotter and diff Apply function.
This allows for Windows layers to be created, and layers to be pulled
from the hub.

Signed-off-by: Darren Stahl <darst@microsoft.com>
2018-01-23 14:40:23 -08:00
Stephen Day
acc6011ac1 Merge pull request #1960 from dmcgowan/images-removal-dirty
metadata: image removal triggers GC
2018-01-17 13:58:39 -08:00
Phil Estes
0b449a4a51 Merge pull request #1982 from stevvooe/deletion-race-handling
services/tasks, linux: ignore shutdown tasks
2018-01-10 16:53:42 -05:00
Stephen J Day
c80ca4f4a2 services/tasks, linux: ignore shutdown tasks
Because tasks may be deleted while listing containers, we need to ignore
errors from state requests that are due to a closed error. All of these
get mapped to ErrNotFound, which can be used to filter the entries.

There may be a better fix that does a better job of keeping track of the
intended state of a backend task. The current condition of assuming that
a closed client is a shutdown task may be too naive.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2018-01-10 11:45:33 -08:00
Daniel Nephin
184bc25629 Add unconvert linter
This linter checks for unnecessary type convertions.

Some convertions are whitelisted because their type is different
on 32bit platforms

Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2018-01-09 17:36:44 -05:00
Derek McGowan
89fa154efd Update metadata image store to be initialized once
The boltdb image store now manages its own transactions when
one is not provided, but allows the caller to pass in a
transaction through the context. This makes the image store
more similar to the content and snapshot stores. Additionally,
use the reference to the metadata database to mark the content
store as dirty after an image has been deleted. The deletion
of an image means a reference to a piece of content is gone
and therefore garbage collection should be run to check if
any resources can be cleaned up as a result.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-01-05 13:38:45 -08:00
Daniel Nephin
06edd193ef Small refactor of gc/scheduler to remove import of metadata
Replace metadata.GCStats with an interface for exposing elapsed time

Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2018-01-03 14:36:11 -05:00
Derek McGowan
b28d7cdf1b Update snapshot and content related log messages
Make the tense and casing consistent.
Add useful log messages in image service.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-12-01 14:57:34 -08:00
Jess Valarezo
1dd6f339a2 rename snapshot->snapshots proto pkg and service
Signed-off-by: Jess Valarezo <valarezo.jessica@gmail.com>
2017-11-29 14:55:24 -08:00
Jess Valarezo
9885edfc44 rename snapshot->snapshots pkg
Signed-off-by: Jess Valarezo <valarezo.jessica@gmail.com>
2017-11-29 14:55:02 -08:00
Stephen Day
90a6b79bfc Merge pull request #1810 from dnephin/add-staticcheck-linter
Fix usage of sync.Pool
2017-11-28 15:46:41 -08:00
Daniel Nephin
dab3ba2aff Replace calls to deprecated grpc functions
Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2017-11-28 15:12:05 -05:00
Daniel Nephin
ee04cfa3f9 Add staticcheck linter
Fix issues with sync.Pool being passed an array and not a pointer.
See https://github.com/dominikh/go-tools/blob/master/cmd/staticcheck/docs/checks/SA6002

Add missing tests for content.Copy

Fix T.Fatal being called in a goroutine

Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2017-11-28 13:05:30 -05:00
Stephen Day
372cdfac3b Merge pull request #1638 from dmcgowan/gc-policy
gc: add policy plugin
2017-11-27 18:20:10 -08:00
Derek McGowan
bae47820d7 Document defaults
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-11-21 16:12:10 -08:00
Daniel Nephin
f74862a0dd Add structcheck, unused, and varcheck linters.
Warn on unused and dead code

Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2017-11-21 11:14:37 -05:00
Derek McGowan
3f1a61f76a Add synchronous image delete
Synchronous image delete provides an option image delete to wait
until the next garbage collection deletes after an image is removed
before returning success to the caller.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-11-20 17:08:35 -08:00
Derek McGowan
00596f400e Add gc policy plugin
Add garbage collection as a background process and policy
configuration for configuring when to run garbage collection.
By default garbage collection will run when deletion occurs
and no more than 20ms out of every second.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-11-20 16:57:39 -08:00
Phil Estes
2edc475818 Merge pull request #1776 from stevvooe/decouple-events-grpc
events: decouple events package from grpc
2017-11-17 11:44:46 -05:00
Michael Crosby
a4968710fd Merge pull request #1777 from jessvalarezo/migrate-client-code
client: migrate client code to containerd
2017-11-17 09:32:34 -05:00
Stephen J Day
2d05c4aa1a events: decouple events package from grpc
By defining a concrete, non-protobuf type for the events interface, we
can completely decouple it from the grpc packages that are expensive at
runtime. This does requires some allocation cost for converting between
types, but the saving for the size of the shim are worth it.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-11-16 18:49:46 -08:00
Jess Valarezo
8c5dede74e client: move content store client to containerd package
Signed-off-by: Jess Valarezo <valarezo.jessica@gmail.com>
2017-11-16 18:14:21 -08:00
Jess Valarezo
aefaeeacb0 client: move diff client to containerd package
Signed-off-by: Jess Valarezo <valarezo.jessica@gmail.com>
2017-11-16 18:06:14 -08:00
Jess Valarezo
8d23e5ef9c client: move image store client to containerd package
Signed-off-by: Jess Valarezo <valarezo.jessica@gmail.com>
2017-11-16 17:47:54 -08:00
Jess Valarezo
6cd9962c7f client: move namespaces client to containerd package
Signed-off-by: Jess Valarezo <valarezo.jessica@gmail.com>
2017-11-16 17:47:38 -08:00
Derek McGowan
5c6e789dde Merge pull request #1769 from stevvooe/split-events-types
api/events: split event types from events service
2017-11-16 16:29:28 -08:00
Jess Valarezo
5b64f7030d client: move snapshot client to containerd package
Signed-off-by: Jess Valarezo <valarezo.jessica@gmail.com>
2017-11-16 15:45:07 -08:00
Stephen J Day
09b5ca1072 api/events: split event types from events service
To avoid importing all of grpc when consuming events, the types of
events have been split in to a separate package. This should allow a
reduction in memory usage in cases where a package is consuming events
but not using the gprc service directly.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-11-16 15:20:46 -08:00
Stephen J Day
3e5e2ecc0a content/local: consistent handling of data and locks
The locks now retry on the backend side to prevent clients from having
to round trip on locks that might be momentarily held. This exposed some
timing errors in the updated_at fields for content ingest, so we've had
to move that to a separate file to export the monotonic go runtime
timestamps.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-11-16 16:13:19 -05:00
Stephen J Day
c5022ad92d protobuf: use the gogo/types package for empty
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-11-15 19:08:54 -08:00
Akihiro Suda
4feb6f228a snapshot: add Close()
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-11-14 02:00:33 +00:00
Jess Valarezo
807f4d2ec7 expose exec-id on ctr task ps
Signed-off-by: Jess Valarezo <valarezo.jessica@gmail.com>
2017-11-10 11:38:51 -08:00
Stephen Day
27d450a01b Merge pull request #1690 from dmcgowan/metadata-transactions
gc: resource leases
2017-11-07 16:11:11 -08:00
Derek McGowan
dce27d8c62 Remove client use of gc root label
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-11-07 12:54:22 -08:00
Derek McGowan
e13894bb7a Add leases api
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-11-07 12:54:22 -08:00
Michael Crosby
a70b95b202 Move events exchange into subpackage
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-11-07 10:51:12 -05:00
Michael Crosby
b6e0c4f321 Fix go lint errors
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-10-25 15:26:44 -04:00
Michael Crosby
5fd0415985 Add comments and fix common lint issues
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-10-20 13:19:14 -04:00
Michael Crosby
9bd1dc78cb Unexport grpc service types
Since these are registered and the interface is what matters, these
Service types do not need to be exported.

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-10-20 13:19:14 -04:00
Derek McGowan
ffb03c4fe2 Add checkpoint and userns gc labels
Prevent checkpoints from getting garbage collected by
adding root labels to unreferenced checkpoint objects.
Mark checkpoints as gc roots.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-10-11 11:26:46 -07:00