Provide a flag which configures a lease to only hold
reference to its given references and ignore label references
during garbage collection rooted from the lease.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Enables showing debug logs in testing output.
For integration tests the client log output will show
in addition to daemon output, with timestamps for better
correlation.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Closes#3305
This adds an opt and a `--cgroup,-c` flag to `ctr namespaces rm` to
remove the cgroup that is commonly created with runtimes.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Add three methods for lease service so that the client can use it to
manage the resource by lease, not just gc.root label. With the following
methods, it is easy for client to maintain their own cache system.
```
- AddResource(context.Context, Lease, Resource) error
- RemoveResource(context.Context, Lease, Resource) error
- ListResources(context.Context, Lease) ([]Resource, error)
```
And the resource is to be
```golang
type Resource {
ID string
Type string
}
```
For the snapshots, the Type field will be formatted by
snapshots/%{type}, like snapshots/overlayfs.
fix: #3295
Signed-off-by: Wei Fu <fuweid89@gmail.com>
The namespaced writer Commit method must always have a
namespace in the context as checked by the removeIngestLease
function, resulting in a panic when not provided.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Extend the adaptImage function with a case for handling the annotations
so they can be used in the filter adaptors for fieldpaths.
Signed-off-by: Stefan Berger <stefanb@linux.ibm.com>
Currently the objects which can retain content from labels
are limited. This limitation has required clients to work
around this and and in some cases add outside reference
counting (e.g. buildkit keeping content for snapshots).
Updated the logic to treat content and snapshot labels equally
and simplified the code in the process.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Adds comment showing the structure of the buckets
as stored in boltdb. Includes all bucket names, key names
and descriptions of the stored values.
Moves the metadata comment to package level to be exposed
in the godoc overview.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
The noop-loop does nothing.
The containerd doesn't have any snapshotter buckets at the beginning.
If user uses specific dir as rootfs to create container, and sets
`snapshotter` key to the container, like `overlayfs` by mistake,
the gc scheduler will try to scan the snapshotter and panic.
In order to avoid this case, remove the noop-loop here.
Signed-off-by: Wei Fu <fuweid89@gmail.com>
megacheck, gosimple and unused has been deprecated and subsumed by
staticcheck. And staticcheck also has been upgraded. we need to update
code for the linter issue.
close: #2945
Signed-off-by: Wei Fu <fuweid89@gmail.com>
This changeset modifies the metadata store to allow one to set a
"content sharing policy" that defines how blobs are shared between
namespaces in the content store.
The default mode "shared" will make blobs available in all namespaces
once it is pulled into any namespace. The blob will be pulled into
the namespace if a writer is opened with the "Expected" digest that
is already present in the backend.
The alternative mode, "isolated" requires that clients prove they have
access to the content by providing all of the content to the ingest
before the blob is added to the namespace.
Both modes share backing data, while "shared" will reduce total
bandwidth across namespaces, at the cost of allowing access to any
blob just by knowing its digest.
Note: Most functional codes and changelog of this commit originate from
Stephen J Day <stephen.day@docker.com>, see
40455aade8Fixes#1713Fixes#2865
Signed-off-by: Eric Lin <linxiulei@gmail.com>
In many cases code is calling errors.Wrapf with an arbitrary string
instead of a format string. This causes confusing errors when the
wrapped error message contains '%' characters.
This change replaces such calls with calls to errors.Wrap.
Signed-off-by: John Starks <jostarks@microsoft.com>
The local store could end up in a state where the writer is
closed but the reference is locked after a commit on an
existing object.
Cleans up Commit logic to always close the writer even after
an error occurs, guaranteeing the reference is unlocked after commit.
Adds a test to the content test suite to verify this behavior.
Updates the content store interface definitions to clarify the behavior.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Allow content ingests to be cleaned up during gc.
Use a default expiration on content ingests or make
use of the lease expiration when provided.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Allow setting an expiration label to have the garbage
collector remove an item after the specified time.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
This change allows implementations to resolve the location of the actual data
using OCI descriptor fields such as MediaType.
No OCI descriptor field is written to the store.
No change on gRPC API.
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
Allows linking to multiple snapshots within the same snapshotter.
Adds support for using slash to separate content as well for consistency.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Allows the client to choose the context to finish the lease.
This allows the client to switch contexts when the main context
used to the create the lease may have been cancelled.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Update content ingests to use content from another namespace.
Ingests must be committed to make content available and the
client will see the sharing as an ingest which has already
been fully written to, but not completed.
Updated the database version to change the ingest record in
the database from a link key to an object with a link and
expected value. This expected value is used to indicate that
the content already exists and an underlying writer may
not yet exist.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Let the test runners choose the namespaces and
wrap the contexts. This allows the test suite to create
multiple contexts without worrying about namespacing
or leasing in the contexts.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Updates overlay remove to simply remove the reference, adds
a cleanup method for discarding the directory.
Updates snapshot create to setup the directory structure while
in the transaction, to prevent cleanup from removing directories
which are part of a create.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
This linter checks for unnecessary type convertions.
Some convertions are whitelisted because their type is different
on 32bit platforms
Signed-off-by: Daniel Nephin <dnephin@gmail.com>
When a writer is requested for an object that already
exists, add that object to the provided any lease to prevent
other operations from affecting the current lease's use of
that content.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
The boltdb image store now manages its own transactions when
one is not provided, but allows the caller to pass in a
transaction through the context. This makes the image store
more similar to the content and snapshot stores. Additionally,
use the reference to the metadata database to mark the content
store as dirty after an image has been deleted. The deletion
of an image means a reference to a piece of content is gone
and therefore garbage collection should be run to check if
any resources can be cleaned up as a result.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Directly get and check whether a bucket is empty.
This prevents unnecessarily loading all records of
the buckets into memory just to check existence.
Also added checks for content and snapshots.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Synchronous image delete provides an option image delete to wait
until the next garbage collection deletes after an image is removed
before returning success to the caller.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Add garbage collection as a background process and policy
configuration for configuring when to run garbage collection.
By default garbage collection will run when deletion occurs
and no more than 20ms out of every second.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Use single instance of content store instead of
creating new one for each collection. Using new
instance and wrapping causes failures.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Ensures that all callers and the garbage collector are using
the same lock instances to prevent cleanup of objects
during creation.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Marks and sweeps unreferenced objects.
Add snapshot cleanup to metadata.
Add content garbage collection
Add dirty flags for snapshotters and content store which
are set on deletion and used during the next garbage collection.
Cleanup content store backend when content metadata is removed.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Updates metadata plugin to require content and
snapshotter plugins be loaded and initializes with
those plugins, keeping the metadata database structure
static after initialization. Service plugins now only
require metadata plugin access snapshotter or content
stores through metadata, which was already required
behavior of the services.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Adds back links from parent to children in order to prevent
deletion of a referenced snapshot in a namespace.
Avoid removing snapshot during metadata delete to
prevent shared namespaces from being mistakenly deleted.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
