Commit Graph

275 Commits

Author SHA1 Message Date
John Howard
40d898a820 Windows: Allow running as a service
Signed-off-by: John Howard <jhoward@microsoft.com>

Allows containerd.exe to run as a Windows service. eg

Register: `.\containerd.exe --register-service`

Start: `net start containerd`
...
Stop: `net stop containerd`

Unregister: `.\containerd.exe --unregister-service`

When running as a service, logs will go to the Windows application
event log.
2018-11-27 17:28:04 -08:00
Phil Estes
f0b081bb44 Merge pull request #2734 from Ace-Tang/start_early_error
check task existence before new task
2018-11-07 10:02:32 -05:00
Justin Terry (VM)
8329a491b9 Build Windows V1 and V2 runtimes always
Signed-off-by: Justin Terry (VM) <juterry@microsoft.com>
2018-11-02 11:59:03 -07:00
Ace-Tang
124194b98b check task existence before new task
make user know correct error when new task.

Signed-off-by: Ace-Tang <aceapril@126.com>
2018-11-03 02:55:12 +08:00
Michael Crosby
35a32a8778 Merge pull request #2747 from fuweid/bugfix_wrap_error_in_closeIO
bugfix: CloseIO should return correct status code
2018-10-29 11:25:37 -04:00
Wei Fu
00ba92364c bugfix: CloseIO should return correct status code
Signed-off-by: Wei Fu <fuweid89@gmail.com>
2018-10-28 12:08:59 +08:00
Michael Crosby
16aaf6c065 Change unsupported snapshot warnings to INFO
Since there is no real action the user can do, these can safely be
informative that the underlying filesystem does not support a snapshot
plugin at boot.

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2018-10-26 12:44:14 -04:00
Wei Fu
06616dab00 enhance: split config from server package
The github.com/containerd/containerd/services/server has a lot of
dependencies, like content, snapshots services implementation and
docker-metrics.

For the client side, it uses the config struct from server package
to start up the containerd in background. It will import a lot of
useless packages which might be conflict with existing vendor's package.

It makes integration easier with single config package.

Signed-off-by: Wei Fu <fuweid89@gmail.com>
2018-10-17 16:34:39 +08:00
Derek McGowan
0f756495a9 Fix writer deadlock in local store
The local store could end up in a state where the writer is
closed but the reference is locked after a commit on an
existing object.
Cleans up Commit logic to always close the writer even after
an error occurs, guaranteeing the reference is unlocked after commit.
Adds a test to the content test suite to verify this behavior.
Updates the content store interface definitions to clarify the behavior.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-10-15 16:33:45 -07:00
Phil Estes
05984a966d Merge pull request #2642 from dmcgowan/fix-commit-already-exists
Fix content store bug when already exists
2018-09-14 09:09:22 -04:00
Derek McGowan
6875d3df3a Always check exists on commit error
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-09-14 01:26:03 -07:00
John Howard
2586f3fbb9 boltdb/bolt --> go.etcd.io/bbolt
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-09-12 15:23:57 -07:00
Justin Terry (VM)
0110b3c0bc Introduce the Windows lcow diff/snaphotter
Implements the Windows lcow differ/snapshotter responsible for managing
the creation and lifetime of lcow containers on Windows.

Signed-off-by: Justin Terry (VM) <juterry@microsoft.com>
2018-08-28 13:51:04 -07:00
Michael Crosby
e5115c3919 Export content service New
Closes #2441

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2018-08-21 15:38:51 -04:00
Derek McGowan
dd97a11b6f Merge pull request #2523 from jterry75/windows_v2_tasks_service
Refactors the TasksService requires per platform
2018-08-06 10:33:46 -07:00
Michael Crosby
5a47c5ec1d Add lib support as an option
Some images like `criu` will have extra libs that it requires.  This
adds lib support via LD_LIBRARY_PATH and InstallOpts

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2018-08-02 17:11:29 -04:00
Evan Hazlett
1537f31381 Add install support for binary images
This adds a way for users to programatically install containerd binary
dependencies.

With runtime v2 and new shim's being built, it will be a challenge to
get those onto machines.  Users would have to find the link, download,
place it in their path, yada yada yada.

With this functionality of a managed `/opt` directory, containerd can
use existing image and distribution infra. to get binarys, shims, etc
onto the system.

Configuration:

*default:* `/opt/containerd`

*containerd config:*
```toml
[plugins.opt]
	path = "/opt/mypath"

```

Usage:

*code:*

```go
image, err := client.Pull(ctx, "docker.io/crosbymichael/runc:latest")
client.Install(ctx, image)
```

*ctr:*

```bash
ctr content fetch docker.io/crosbymichael/runc:latest
ctr install docker.io/crosbymichael/runc:latest
```

You can manage versions and see what is running via standard image
commands.

Images:

These images MUST be small and only contain binaries.

```Dockerfile
FROM scratch
Add runc /bin/runc
```

Containerd will only extract files in `/bin` of the image.

Later on, we can add support for `/lib`.

The code adds a service to manage an `/opt/containerd` directory and
provide that path to callers via the introspection service.

How to Test:

Delete runc from your system.

```bash
> sudo ctr run --rm  docker.io/library/redis:alpine redis
ctr: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v1.linux/default/redis/log.json: no such file or directory): exec: "runc": executable file not found in $PATH: unknown

> sudo ctr content fetch docker.io/crosbymichael/runc:latest
> sudo ctr  install docker.io/crosbymichael/runc:latest

> sudo ctr run --rm  docker.io/library/redis:alpine redis
1:C 01 Aug 15:59:52.864 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 01 Aug 15:59:52.864 # Redis version=4.0.10, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 01 Aug 15:59:52.864 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
1:M 01 Aug 15:59:52.866 # You requested maxclients of 10000 requiring at least 10032 max file descriptors.
1:M 01 Aug 15:59:52.866 # Server can't set maximum open files to 10032 because of OS error: Operation not permitted.
1:M 01 Aug 15:59:52.866 # Current maximum open files is 1024. maxclients has been reduced to 992 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
1:M 01 Aug 15:59:52.870 * Running mode=standalone, port=6379.
1:M 01 Aug 15:59:52.870 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 01 Aug 15:59:52.870 # Server initialized
1:M 01 Aug 15:59:52.870 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1:M 01 Aug 15:59:52.870 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
1:M 01 Aug 15:59:52.870 * Ready to accept connections
^C1:signal-handler (1533139193) Received SIGINT scheduling shutdown...
1:M 01 Aug 15:59:53.472 # User requested shutdown...
1:M 01 Aug 15:59:53.472 * Saving the final RDB snapshot before exiting.
1:M 01 Aug 15:59:53.484 * DB saved on disk
1:M 01 Aug 15:59:53.484 # Redis is now ready to exit, bye bye...
```

Signed-off-by: Evan Hazlett <ejhazlett@gmail.com>
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2018-08-02 17:11:29 -04:00
Justin Terry (VM)
9936370fef Refactors the TasksService requires per platform
Removes the start dependency on V1 runtimes in the TasksService for:
// +build windows_v2. For unix and windows (v1) this code remains to load all
v1 runtimes as expected.

Signed-off-by: Justin Terry (VM) <juterry@microsoft.com>
2018-08-02 11:20:35 -07:00
Phil Estes
4249f44d81 Merge pull request #2493 from dmcgowan/sync-lease-removal
Add sync option to lease removal
2018-07-31 11:31:55 -04:00
Michael Crosby
400f16fc46 Add containers streaming API
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2018-07-26 13:55:20 -04:00
Derek McGowan
b760cee65a Update lease service errors
Ensure delete returns a typed error on not found

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-07-25 14:56:35 -07:00
Michael Crosby
0d52c71c80 Merge pull request #2474 from dmcgowan/lease-expiration
Improved lease management
2018-07-20 16:54:17 -04:00
Derek McGowan
94e132fd07 Add sync option on lease removal
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-07-19 14:37:55 -07:00
Derek McGowan
94cfce62ba Merge pull request #2434 from crosbymichael/shimv2
Runtime v2 (shim API)
2018-07-18 13:14:10 -07:00
Derek McGowan
8cf3fad8d4 Add leases manager interface
Add leases manager to the leases package and use the
interface on the client and service.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-07-18 10:43:37 -07:00
Akihiro Suda
02579c8c3f Merge pull request #2415 from dmcgowan/proxy-plugins
Add support for proxy plugins
2018-07-18 11:17:24 +09:00
Michael Crosby
7e49c601a8 Add shim start for shim creation
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2018-07-17 10:21:29 -04:00
Michael Crosby
da1b5470cd Runtime v2
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2018-07-17 10:21:29 -04:00
Dave Henderson
9a97ab34ce Switching from crypto/rand to math/rand to avoid blocking
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
2018-07-11 16:24:51 -04:00
Derek McGowan
7049671465 Add support for proxy plugins in configuration
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-06-21 18:57:42 -07:00
Vincent Demeester
832b05ae67 Update tests to use gotest.tools angel
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
2018-06-08 21:02:01 +02:00
Derek McGowan
993d4b8fc5 Ignore zero max message size in grpc config
Fixes upgrade issue where existing configs cause
the max message size to get set to 0.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-06-01 16:21:53 -07:00
Akihiro Suda
d88de4a34f content: change Writer/ReaderAt to take OCI
This change allows implementations to resolve the location of the actual data
using OCI descriptor fields such as MediaType.

No OCI descriptor field is written to the store.

No change on gRPC API.

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-06-01 11:51:43 +09:00
Michael Crosby
8e97da0958 Move server to services pkg
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2018-05-22 17:15:58 -04:00
Michael Crosby
ceae112f7d Rename Runtime to PlatformRuntime
This renames the runtime interface to PlatformRuntime to denote the
layer at which the runtime is being abstracted.  This should be used to
abstract different platforms that vary greatly and do not have full
compat with OCI based binary runtimes.

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2018-05-16 09:36:05 -04:00
Tom Godkin
fc8bce59b9 Use user-specific temp directory if set
This allows non-privileged users to use containerd. This is part of a
larger track of work integrating containerd into Cloudfoundry's garden
with support for rootless.

[#156343575]

Signed-off-by: Claudia Beresford <cberesford@pivotal.io>
2018-05-04 10:27:58 +01:00
Kir Kolyshkin
bbe14f0a2e Switch from x/net/context to context
Since Go 1.7, context is a standard package, superceding the
"x/net/context". Since Go 1.9, the latter only provides a few type
aliases from the former. Therefore, it makes sense to switch to the
standard package.

This commit was generated by the following script (with a couple of
minor fixups to remove extra changes done by goimports):

	#!/bin/bash

	if [ $# -ge 1 ]; then
		FILES=$*
	else
		FILES=$(git ls-files \*.go | grep -vF ".pb.go" | grep -v
	^vendor/)
	fi

	for f in $FILES; do
		printf .
		sed -i -e 's|"golang.org/x/net/context"$|"context"|' $f
		goimports -w $f
		awk '	/^$/ {e=1; next;}
			/[[:space:]]"context"$/ {e=0;}
			{if (e) {print ""; e=0}; print;}' < $f > $f.new && \
				mv $f.new $f
		goimports -w $f
	done
	echo

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2018-04-24 14:33:34 -07:00
Kir Kolyshkin
9d0d4b806c context pkg: untangle
Since Go 1.7, "context" is a standard package, superceding the
"x/net/context". Since Go 1.9, the latter only provides type aliases
from the former. Therefore, it makes sense to switch to the standard
package, and the change is not disruptive in any sense.

This commit deals with a few cases where both packages happened to be
imported by the same source file. A choice between "context" and
"gocontext" was made for each file in order to minimize the patch.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2018-04-24 14:33:34 -07:00
Michael Wan
7fd6d5e2be fix words misspell
Signed-off-by: Michael Wan <zirenwan@gmail.com>
2018-04-23 00:09:42 -04:00
Stephen J Day
caac3bca3e protobuf: remove generated google/rpc files
We move from having our own generated version of the googleapis files to
an upstream version that is present in gogo. As part of this, we update
the protobuf package to 1.0 and make some corrections for slight
differences in the generated code.

The impact of this change is very low.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2018-04-05 14:36:32 -07:00
Akihiro Suda
7b323b1402 services/content: fix reading a blob which is smaller than the read buffer.
The newly added test fails without this fix in services/content/service.go:

    $ go test -c . && sudo ./containerd.test -test.v -test.root -test.run TestContentClient
    ...
        --- FAIL: TestContentClient/SmallBlob (0.02s)
            provideringester.go:62: rpc error: code = OutOfRange desc = read
    past object length 6 bytes
            helpers.go:67: drwx------       4096
    /tmp/content-suite-ContentClient-286788688
    FAIL

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-03-23 12:51:28 +09:00
Stephen Day
b3b95c0a2a Merge pull request #2154 from dmcgowan/shared-content-ingests
content: shared content across namespaces
2018-03-12 16:11:32 -07:00
Lantao Liu
1128b3d664 Add service plugin and support in process integration.
Signed-off-by: Lantao Liu <lantaol@google.com>
2018-03-12 18:03:50 +00:00
Derek McGowan
a1a67899f8 Shared content across namespaces
Update content ingests to use content from another namespace.
Ingests must be committed to make content available and the
client will see the sharing as an ingest which has already
been fully written to, but not completed.

Updated the database version to change the ingest record in
the database from a link key to an object with a link and
expected value. This expected value is used to indicate that
the content already exists and an underlying writer may
not yet exist.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-02-22 14:45:10 -08:00
Kunal Kushwaha
b12c3215a0 Licence header added
Signed-off-by: Kunal Kushwaha <kushwaha_kunal_v7@lab.ntt.co.jp>
2018-02-19 10:32:26 +09:00
Derek McGowan
b763777288 diff: rename differ to comparer
Remove combined interface and split implementations.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-01-26 16:32:09 -08:00
Akihiro Suda
b580441f91 diff: resplit Applier from Differ
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-01-25 13:58:34 -08:00
Darren Stahl
a5a9f91832 Implement Windows snapshotter and differ
This implements the Windows snapshotter and diff Apply function.
This allows for Windows layers to be created, and layers to be pulled
from the hub.

Signed-off-by: Darren Stahl <darst@microsoft.com>
2018-01-23 14:40:23 -08:00
Stephen Day
acc6011ac1 Merge pull request #1960 from dmcgowan/images-removal-dirty
metadata: image removal triggers GC
2018-01-17 13:58:39 -08:00
Phil Estes
0b449a4a51 Merge pull request #1982 from stevvooe/deletion-race-handling
services/tasks, linux: ignore shutdown tasks
2018-01-10 16:53:42 -05:00