Commit Graph

293 Commits

Author SHA1 Message Date
Derek McGowan
acca107732 Merge pull request #3204 from crosbymichael/fix-forward
Fix API forward events for shims
2019-04-11 11:46:46 -07:00
Phil Estes
5703f415c6 Merge pull request #3197 from Random-Liu/add-required-plugins
Add support for required plugins.
2019-04-11 12:02:16 +02:00
Lantao Liu
4b3b99ea11 Add support for required plugins.
Signed-off-by: Lantao Liu <lantaol@google.com>
2019-04-10 11:31:59 -07:00
Michael Crosby
4ba756edda Fix API forward events for shims
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2019-04-10 13:34:34 -04:00
Michael Crosby
a6f587e4c4 Use ttrpc to publish runtime v2 events
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2019-04-09 14:38:50 -04:00
Michael Crosby
a8a805cad3 Add ttrpc server to containerd
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2019-04-09 13:08:23 -04:00
Michael Crosby
559dfa59a1 Add configurable plugin path
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2019-04-01 16:54:27 -04:00
Michael Crosby
9b882c44f8 Merge pull request #3000 from stefanberger/descriptor_annotations.pr
Add missing annotations map to Descriptor for gRPC transfer
2019-03-22 14:05:44 -04:00
John Howard
6034c1950a Windows:Create root/state with ACL
Signed-off-by: John Howard <jhoward@microsoft.com>
2019-03-21 18:47:34 -07:00
Akihiro Suda
f3ff95ab72 decouple api.ContentServer implementation package from bbolt dependency
For moby/buildkit#886

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2019-03-19 16:09:45 +09:00
Stefan Berger
0b711d616a Copy annotations around where necessary
Make sure that the newly added annotations are copied around appropriately.

Signed-off-by: Stefan Berger <stefanb@linux.ibm.com>
2019-03-06 12:26:23 -05:00
Michael Crosby
84a24711e8 Add runc.v2 multi-shim
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2019-02-21 11:09:46 -05:00
Wei Fu
132ee9b826 fix: linter issue
megacheck, gosimple and unused has been deprecated and subsumed by
staticcheck. And staticcheck also has been upgraded. we need to update
code for the linter issue.

close: #2945

Signed-off-by: Wei Fu <fuweid89@gmail.com>
2019-01-23 22:54:51 +08:00
Derek McGowan
8706a355dd Merge pull request #2889 from linxiulei/isolated_content
metadata: define content sharing policy
2019-01-14 13:15:38 -08:00
Eric Lin
4247f2684d metadata: define content sharing policy
This changeset modifies the metadata store to allow one to set a
"content sharing policy" that defines how blobs are shared between
namespaces in the content store.

The default mode "shared" will make blobs available in all namespaces
once it is pulled into any namespace.  The blob will be pulled into
the namespace if a writer is opened with the "Expected" digest that
is already present in the backend.

The alternative mode, "isolated" requires that clients prove they have
access to the content by providing all of the content to the ingest
before the blob is added to the namespace.

Both modes share backing data, while "shared" will reduce total
bandwidth across namespaces, at the cost of allowing access to any
blob just by knowing its digest.

Note: Most functional codes and changelog of this commit originate from
Stephen J Day <stephen.day@docker.com>, see
40455aade8

Fixes #1713 Fixes #2865

Signed-off-by: Eric Lin <linxiulei@gmail.com>
2018-12-21 15:02:21 +08:00
Phil Estes
06e04bc5a9 Merge pull request #2830 from Ace-Tang/support_cr_without_image
cr: support checkpoint/restore without image
2018-12-20 13:24:37 -05:00
zhangyue
6ccb0d0629 fix: remove dead code
Signed-off-by: zhangyue <zy675793960@yeah.net>
2018-12-13 09:55:08 +08:00
Ace-Tang
6593399e9f cr: support checkpoint/restore without image
support checkpoint without committing a checkpoint dir into a
checkpoint image and restore without untar image into checkpoint
directory. support for both v1 and v2 runtime

Signed-off-by: Ace-Tang <aceapril@126.com>
2018-11-29 10:19:39 +08:00
John Howard
40d898a820 Windows: Allow running as a service
Signed-off-by: John Howard <jhoward@microsoft.com>

Allows containerd.exe to run as a Windows service. eg

Register: `.\containerd.exe --register-service`

Start: `net start containerd`
...
Stop: `net stop containerd`

Unregister: `.\containerd.exe --unregister-service`

When running as a service, logs will go to the Windows application
event log.
2018-11-27 17:28:04 -08:00
Phil Estes
f0b081bb44 Merge pull request #2734 from Ace-Tang/start_early_error
check task existence before new task
2018-11-07 10:02:32 -05:00
Justin Terry (VM)
8329a491b9 Build Windows V1 and V2 runtimes always
Signed-off-by: Justin Terry (VM) <juterry@microsoft.com>
2018-11-02 11:59:03 -07:00
Ace-Tang
124194b98b check task existence before new task
make user know correct error when new task.

Signed-off-by: Ace-Tang <aceapril@126.com>
2018-11-03 02:55:12 +08:00
Michael Crosby
35a32a8778 Merge pull request #2747 from fuweid/bugfix_wrap_error_in_closeIO
bugfix: CloseIO should return correct status code
2018-10-29 11:25:37 -04:00
Wei Fu
00ba92364c bugfix: CloseIO should return correct status code
Signed-off-by: Wei Fu <fuweid89@gmail.com>
2018-10-28 12:08:59 +08:00
Michael Crosby
16aaf6c065 Change unsupported snapshot warnings to INFO
Since there is no real action the user can do, these can safely be
informative that the underlying filesystem does not support a snapshot
plugin at boot.

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2018-10-26 12:44:14 -04:00
Wei Fu
06616dab00 enhance: split config from server package
The github.com/containerd/containerd/services/server has a lot of
dependencies, like content, snapshots services implementation and
docker-metrics.

For the client side, it uses the config struct from server package
to start up the containerd in background. It will import a lot of
useless packages which might be conflict with existing vendor's package.

It makes integration easier with single config package.

Signed-off-by: Wei Fu <fuweid89@gmail.com>
2018-10-17 16:34:39 +08:00
Derek McGowan
0f756495a9 Fix writer deadlock in local store
The local store could end up in a state where the writer is
closed but the reference is locked after a commit on an
existing object.
Cleans up Commit logic to always close the writer even after
an error occurs, guaranteeing the reference is unlocked after commit.
Adds a test to the content test suite to verify this behavior.
Updates the content store interface definitions to clarify the behavior.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-10-15 16:33:45 -07:00
Phil Estes
05984a966d Merge pull request #2642 from dmcgowan/fix-commit-already-exists
Fix content store bug when already exists
2018-09-14 09:09:22 -04:00
Derek McGowan
6875d3df3a Always check exists on commit error
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-09-14 01:26:03 -07:00
John Howard
2586f3fbb9 boltdb/bolt --> go.etcd.io/bbolt
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-09-12 15:23:57 -07:00
Justin Terry (VM)
0110b3c0bc Introduce the Windows lcow diff/snaphotter
Implements the Windows lcow differ/snapshotter responsible for managing
the creation and lifetime of lcow containers on Windows.

Signed-off-by: Justin Terry (VM) <juterry@microsoft.com>
2018-08-28 13:51:04 -07:00
Michael Crosby
e5115c3919 Export content service New
Closes #2441

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2018-08-21 15:38:51 -04:00
Derek McGowan
dd97a11b6f Merge pull request #2523 from jterry75/windows_v2_tasks_service
Refactors the TasksService requires per platform
2018-08-06 10:33:46 -07:00
Michael Crosby
5a47c5ec1d Add lib support as an option
Some images like `criu` will have extra libs that it requires.  This
adds lib support via LD_LIBRARY_PATH and InstallOpts

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2018-08-02 17:11:29 -04:00
Evan Hazlett
1537f31381 Add install support for binary images
This adds a way for users to programatically install containerd binary
dependencies.

With runtime v2 and new shim's being built, it will be a challenge to
get those onto machines.  Users would have to find the link, download,
place it in their path, yada yada yada.

With this functionality of a managed `/opt` directory, containerd can
use existing image and distribution infra. to get binarys, shims, etc
onto the system.

Configuration:

*default:* `/opt/containerd`

*containerd config:*
```toml
[plugins.opt]
	path = "/opt/mypath"

```

Usage:

*code:*

```go
image, err := client.Pull(ctx, "docker.io/crosbymichael/runc:latest")
client.Install(ctx, image)
```

*ctr:*

```bash
ctr content fetch docker.io/crosbymichael/runc:latest
ctr install docker.io/crosbymichael/runc:latest
```

You can manage versions and see what is running via standard image
commands.

Images:

These images MUST be small and only contain binaries.

```Dockerfile
FROM scratch
Add runc /bin/runc
```

Containerd will only extract files in `/bin` of the image.

Later on, we can add support for `/lib`.

The code adds a service to manage an `/opt/containerd` directory and
provide that path to callers via the introspection service.

How to Test:

Delete runc from your system.

```bash
> sudo ctr run --rm  docker.io/library/redis:alpine redis
ctr: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v1.linux/default/redis/log.json: no such file or directory): exec: "runc": executable file not found in $PATH: unknown

> sudo ctr content fetch docker.io/crosbymichael/runc:latest
> sudo ctr  install docker.io/crosbymichael/runc:latest

> sudo ctr run --rm  docker.io/library/redis:alpine redis
1:C 01 Aug 15:59:52.864 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 01 Aug 15:59:52.864 # Redis version=4.0.10, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 01 Aug 15:59:52.864 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
1:M 01 Aug 15:59:52.866 # You requested maxclients of 10000 requiring at least 10032 max file descriptors.
1:M 01 Aug 15:59:52.866 # Server can't set maximum open files to 10032 because of OS error: Operation not permitted.
1:M 01 Aug 15:59:52.866 # Current maximum open files is 1024. maxclients has been reduced to 992 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
1:M 01 Aug 15:59:52.870 * Running mode=standalone, port=6379.
1:M 01 Aug 15:59:52.870 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 01 Aug 15:59:52.870 # Server initialized
1:M 01 Aug 15:59:52.870 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1:M 01 Aug 15:59:52.870 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
1:M 01 Aug 15:59:52.870 * Ready to accept connections
^C1:signal-handler (1533139193) Received SIGINT scheduling shutdown...
1:M 01 Aug 15:59:53.472 # User requested shutdown...
1:M 01 Aug 15:59:53.472 * Saving the final RDB snapshot before exiting.
1:M 01 Aug 15:59:53.484 * DB saved on disk
1:M 01 Aug 15:59:53.484 # Redis is now ready to exit, bye bye...
```

Signed-off-by: Evan Hazlett <ejhazlett@gmail.com>
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2018-08-02 17:11:29 -04:00
Justin Terry (VM)
9936370fef Refactors the TasksService requires per platform
Removes the start dependency on V1 runtimes in the TasksService for:
// +build windows_v2. For unix and windows (v1) this code remains to load all
v1 runtimes as expected.

Signed-off-by: Justin Terry (VM) <juterry@microsoft.com>
2018-08-02 11:20:35 -07:00
Phil Estes
4249f44d81 Merge pull request #2493 from dmcgowan/sync-lease-removal
Add sync option to lease removal
2018-07-31 11:31:55 -04:00
Michael Crosby
400f16fc46 Add containers streaming API
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2018-07-26 13:55:20 -04:00
Derek McGowan
b760cee65a Update lease service errors
Ensure delete returns a typed error on not found

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-07-25 14:56:35 -07:00
Michael Crosby
0d52c71c80 Merge pull request #2474 from dmcgowan/lease-expiration
Improved lease management
2018-07-20 16:54:17 -04:00
Derek McGowan
94e132fd07 Add sync option on lease removal
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-07-19 14:37:55 -07:00
Derek McGowan
94cfce62ba Merge pull request #2434 from crosbymichael/shimv2
Runtime v2 (shim API)
2018-07-18 13:14:10 -07:00
Derek McGowan
8cf3fad8d4 Add leases manager interface
Add leases manager to the leases package and use the
interface on the client and service.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-07-18 10:43:37 -07:00
Akihiro Suda
02579c8c3f Merge pull request #2415 from dmcgowan/proxy-plugins
Add support for proxy plugins
2018-07-18 11:17:24 +09:00
Michael Crosby
7e49c601a8 Add shim start for shim creation
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2018-07-17 10:21:29 -04:00
Michael Crosby
da1b5470cd Runtime v2
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2018-07-17 10:21:29 -04:00
Dave Henderson
9a97ab34ce Switching from crypto/rand to math/rand to avoid blocking
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
2018-07-11 16:24:51 -04:00
Derek McGowan
7049671465 Add support for proxy plugins in configuration
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-06-21 18:57:42 -07:00
Vincent Demeester
832b05ae67 Update tests to use gotest.tools angel
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
2018-06-08 21:02:01 +02:00
Derek McGowan
993d4b8fc5 Ignore zero max message size in grpc config
Fixes upgrade issue where existing configs cause
the max message size to get set to 0.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-06-01 16:21:53 -07:00