From c2081369c504e799e355e3e107519994894f9915 Mon Sep 17 00:00:00 2001 From: ktock Date: Sat, 27 Jun 2020 19:02:33 +0900 Subject: [PATCH] Add doc about remote snapshotter Signed-off-by: Kohei Tokunaga --- docs/remote-snapshotter.md | 129 +++++++++++++++++++++++++++++++++++++ 1 file changed, 129 insertions(+) create mode 100644 docs/remote-snapshotter.md diff --git a/docs/remote-snapshotter.md b/docs/remote-snapshotter.md new file mode 100644 index 000000000..2e9bdee06 --- /dev/null +++ b/docs/remote-snapshotter.md @@ -0,0 +1,129 @@ +# Remote Snapshotter + +Containerd allows snapshotters to reuse snapshots existing somewhere managed by them. + +_Remote Snapshotter_ is a snapshotter that leverages this functionality and reuses snapshots that are stored in a remotely shared place. +These remotely shared snapshots are called _remote snapshots_. +Remote snapshotter allows containerd to prepare these remote snapshots without pulling layers from registries, which hopefully shorten the time to take for image pull. + +One of the remote snapshotter implementations is [Stargz Snapshotter](https://github.com/containerd/stargz-snapshotter). +This enables containerd to lazily pull images from standard-compliant registries leveraging remote snapshotter functionality and stargz images by google/crfs. + +## The containerd client API + +The containerd client's `Pull` API with unpacking-mode allows the underlying snapshotter to query for remote snapshots before fetching content. +Remote snapshotter needs to be plugged into containerd in [the same ways as normal snapshotters](/PLUGINS.md). + +```go +image, err := client.Pull(ctx, ref, + containerd.WithPullUnpack, + containerd.WithPullSnapshotter("my-remote-snapshotter"), +) +``` + +## Passing snapshotter-specific information + +Some remote snapshotters requires snapshotter-specific information through `Pull` API. +The information will be used in various ways including searching snapshot contents from a remote store. +One of the example snapshotters that requires snapshotter-specific information is stargz snapshotter. +It requires the image reference name and layer digests, etc. for searching layer contents from registries. + +Snapshotters receive the information through user-defined labels prefixed by `containerd.io/snapshot/`. +The containerd client supports two ways to pass these labels to the underlying snapshotter. + +### Using snapshotter's `WithLabels` option + +User-defined labels can be passed down to the underlying snapshotter using snapshotter option `WithLabels`. +Specified labels will be passed down every time the containerd client queries a remote snapshot. +This is useful if the values of these labels are determined statically regardless of the snapshots. +These user-defined labels must be prefixed by `containerd.io/snapshot/`. + +```go +import "github.com/containerd/containerd/snapshots" + +image, err := client.Pull(ctx, ref, + containerd.WithPullUnpack, + containerd.WithPullSnapshotter( + "my-remote-snapshotter", + snapshots.WithLabels(map[string]string{ + "containerd.io/snapshot/reference": ref, + }), + ), +) +``` + +### Using the containerd client's `WithImageHandlerWrapper` option + +User-defined labels can also be passed using an image handler wrapper. +This is useful when labels vary depending on the snapshot. + +Every time the containerd client queries remote snapshot, it passes `Annotations` appended to the targeting layer descriptor (means the layer descriptor that will be pulled and unpacked for preparing that snapshot) to the underlying snapshotter. +These annotations are passed to the snapshotter as user-defined labels. +The values of annotations can be dynamically added and modified in the handler wrapper. +Note that annotations must be prefixed by `containerd.io/snapshot/`. +[CRI plugin](https://github.com/containerd/cri/blob/09d6426f33cac217528158ddc6d254ca7d597a7b/pkg/server/image_pull.go#L127) and [stargz snapshotter](https://github.com/containerd/stargz-snapshotter/blob/875ec333403a885f5b6e5b64c94ec4dc713e0596/cmd/ctr-remote/commands/rpull.go#L97) leverage this method. + +```go +import "github.com/ktock/snapshotter/handler" + +if _, err := client.Pull(ctx, ref, + containerd.WithPullUnpack, + containerd.WithPullSnapshotter("my-remote-snapshotter"), + containerd.WithImageHandlerWrapper(handler.Wrapper(ref)), +) +``` + +## Snapshotter APIs for querying remote snapshots + +The containerd client queries remote snapshots to the underlying remote snapshotter using snapshotter APIs. +This section describes the high-level overview of how snapshotter APIs are used for remote snapshots functionality, with some piece of pseudo-codes that describe the simplified logic implemented in the containerd client. +For more details, see [`unpacker.go`](/unpacker.go) that implements this logic. + +During image pull, the containerd client calls `Prepare` API with the label `containerd.io/snapshot.ref`. +This is a containerd-defined label which contains ChainID that targets a committed snapshot that the client is trying to prepare. +At this moment, user-defined labels (prefixed by `containerd.io/snapshot/`) will also be merged into the labels option. + +```go +// Gets annotations appended to the targetting layer which would contain +// snapshotter-specific information passed by the user. +labels := snapshots.FilterInheritedLabels(desc.Annotations) +if labels == nil { + labels = make(map[string]string) +} + +// Specifies ChainID of the targeting committed snapshot. +labels["containerd.io/snapshot.ref"] = chainID + +// Merges snapshotter options specified by the user which would contain +// snapshotter-specific information passed by the user. +opts := append(rCtx.SnapshotterOpts, snapshots.WithLabels(labels)) + +// Calls `Prepare` API with target indentifier and snapshotter-specific +// information. +mounts, err = sn.Prepare(ctx, key, parent.String(), opts...) +``` + +If this snapshotter is a remote snapshotter, that committed snapshot hopefully exists, for example, in a shared remote store. +Remote snapshotter must define and enforce policies about whether it will use an existing snapshot. +When remote snapshotter allows the user to use that snapshot, it must return `ErrAlreadyExists`. + +If the containerd client gets `ErrAlreadyExists` by `Prepare`, it ensures the existence of that committed snapshot by calling `Stat` with the ChainID. +If this snapshot is available, the containerd client skips pulling and unpacking layer that would otherwise be needed for preparing and committing that snapshot. + +```go +mounts, err = sn.Prepare(ctx, key, parent.String(), opts...) +if err != nil { + if errdefs.IsAlreadyExists(err) { + // Ensures the layer existence + if _, err := sn.Stat(ctx, chainID); err != nil { + // Handling error + } else { + // snapshot found with ChainID + // pulling/unpacking will be skipped + continue + } + } else { + return err + } +} +```