Commit Graph

115 Commits

Author SHA1 Message Date
Maksym Pavlenko
95f0a4903c
devmapper: rollback thin devices on error
Signed-off-by: Maksym Pavlenko <makpav@amazon.com>
2019-02-21 17:40:10 -08:00
Maksym Pavlenko
adf5c640f4
devmapper: don't create or reload thin-pool from snapshotter
Signed-off-by: Maksym Pavlenko <makpav@amazon.com>
2019-02-21 16:26:46 -08:00
Maksym Pavlenko
7efda48c53
devmapper: more precise way of checking if device is activated
Signed-off-by: Maksym Pavlenko <makpav@amazon.com>
2019-02-21 16:26:46 -08:00
Maksym Pavlenko
37cdedc61c
devmapper: add linux tags, fix build
Signed-off-by: Maksym Pavlenko <makpav@amazon.com>
2019-02-21 16:26:46 -08:00
Maksym Pavlenko
0c6d194cce
devmapper: add README and minor fixes
Signed-off-by: Maksym Pavlenko <makpav@amazon.com>
2019-02-21 16:25:55 -08:00
Maksym Pavlenko
2218275ec9
devmapper: register plugin
Signed-off-by: Maksym Pavlenko <makpav@amazon.com>
2019-02-21 16:25:55 -08:00
Maksym Pavlenko
cec72efc2a
devmapper: add snapshotter
Signed-off-by: Maksym Pavlenko <makpav@amazon.com>
2019-02-21 16:25:55 -08:00
Maksym Pavlenko
3a75882520
devmapper: add pool device manager
Signed-off-by: Maksym Pavlenko <makpav@amazon.com>
2019-02-21 16:25:55 -08:00
Maksym Pavlenko
6e0ae68e17
devmapper: add snapshotter config
Signed-off-by: Maksym Pavlenko <makpav@amazon.com>
2019-02-21 16:25:55 -08:00
Maksym Pavlenko
fcd9dc2749
devmapper: add pool metadata
Signed-off-by: Maksym Pavlenko <makpav@amazon.com>
2019-02-21 16:25:55 -08:00
Maksym Pavlenko
809e5fd3b8
devmapper: add dmsetup
Signed-off-by: Maksym Pavlenko <makpav@amazon.com>
2019-02-21 16:25:55 -08:00
Maksym Pavlenko
fe05e4d1a9
devmapper: add losetup
Signed-off-by: Maksym Pavlenko <makpav@amazon.com>
2019-02-21 16:25:55 -08:00
Oliver Stenbom
7f5d7ff6b8 Update snapshots docs with garbage collector label
Signed-off-by: Oliver Stenbom <oliver@stenbom.eu>
2019-01-29 15:37:56 +00:00
Ace-Tang
7faaa64cf9 fix: miss remove temp file in createSnapshot
```
func foo() error {

defer func() {
    if err != nil {
	...
    }
}()
...
}
```
use defer func to do something when err not nil, if foo() not use
named error, `err != nil` can not catch all errors, since when err
re-defined in if condition, it is a new variable.

Signed-off-by: Ace-Tang <aceapril@126.com>
2018-12-24 23:01:56 +08:00
zhangyue
7b1e6f323a fix: use func args instead of build new one
Signed-off-by: zhangyue <zy675793960@yeah.net>
2018-12-16 15:07:12 +08:00
Michael Crosby
16aaf6c065 Change unsupported snapshot warnings to INFO
Since there is no real action the user can do, these can safely be
informative that the underlying filesystem does not support a snapshot
plugin at boot.

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2018-10-26 12:44:14 -04:00
Justin Terry (VM)
3f1d9b2c4f Revendor github.com/Microsoft/hcsshim
Signed-off-by: Justin Terry (VM) <juterry@microsoft.com>
2018-10-15 13:38:24 -07:00
Justin Terry (VM)
00242e1668 Dismount sandbox VHD on snapshot remove
Signed-off-by: Justin Terry (VM) <juterry@microsoft.com>
2018-10-10 13:35:06 -07:00
Akihiro Suda
5349fa31df remove pkg/testutil/loopback_linux.go and use continuity/testutil/loopback
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-10-02 13:12:25 +09:00
Justin Terry (VM)
e373126bfb Fix race in lcow snapshot scratch.vhdx creation
Signed-off-by: Justin Terry (VM) <juterry@microsoft.com>
2018-09-25 09:35:46 -07:00
Justin Terry (VM)
7768ab1b5e Update runhcs-shim to use go-bindings
Signed-off-by: Justin Terry (VM) <juterry@microsoft.com>
2018-09-20 10:40:33 -07:00
John Howard
2586f3fbb9 boltdb/bolt --> go.etcd.io/bbolt
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-09-12 15:23:57 -07:00
Xiaodong Zhang
e6d787172c Fix some typo in runtime and snapshots
Signed-off-by: Xiaodong Zhang <a4012017@sina.com>
2018-09-08 08:31:42 +08:00
Michael Wan
92243ff72a bugfix: updatedAt timestamp file may be empty
Signed-off-by: Michael Wan <zirenwan@gmail.com>
2018-08-29 21:49:40 -04:00
Justin Terry (VM)
0110b3c0bc Introduce the Windows lcow diff/snaphotter
Implements the Windows lcow differ/snapshotter responsible for managing
the creation and lifetime of lcow containers on Windows.

Signed-off-by: Justin Terry (VM) <juterry@microsoft.com>
2018-08-28 13:51:04 -07:00
Wei Fu
67b54c6670 Support >= 128 layers in overlayfs snapshots
Auto-detect longest common dir in lowerdir option and compact it if the
option size is hitting one page size. If does, Use chdir + CLONE to do
mount thing to avoid hitting one page argument buffer in linux kernel
mount.

Signed-off-by: Wei Fu <fhfuwei@163.com>
2018-08-07 10:59:36 +08:00
Akihiro Suda
7461739b58 native: set '/' permission to 0755
Unlike other snapshotters, the permission was previously set to 0700
unexpectedly.

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-07-23 16:14:55 +09:00
Michael Crosby
e17969caad
Merge pull request #2436 from thaJeztah/fix_gofmt
Workaround for gofmt change in Go 1.11
2018-07-17 14:52:29 -04:00
Rolf Neugebauer
651920405e test: Increase btrfs loopback device size to 128MB
With btrfs-prog 4.17 as shipped with Alpine 3.8 the btrfs tests
fail with:

  ERROR: '/dev/loop1' is too small to make a usable filesystem
  ERROR: minimum size for each btrfs device is 114294784

btrfs-prog commit 997f9977c243 ("btrfs-progs: mkfs: Prevent
temporary system chunk to use space in reserved 1M range")
changed the code to compute the minimum size for a btrfs size
and is the likely reason for this error.

Increase the size of the loop back device to 128MB for
the btrfs test.

With this fix, the containerd tests pass on Alpine 3.8,
tested in LinuxKit with kernel 4.14.53.

fixes #2447

Signed-off-by: Rolf Neugebauer <rolf.neugebauer@gmail.com>
2018-07-08 10:53:03 +01:00
Sebastiaan van Stijn
383d750d4f
Workaround for gofmt change in Go 1.11
Go 1.11 uses a different formatting for maps, and now
aligns values; running `gofmt -s -w` on this file resulted
in this diff;

```patch
 content/testsuite/testsuite.go | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/content/testsuite/testsuite.go b/content/testsuite/testsuite.go
index 974c7cb8ed..d9ae9dc160 100644
--- a/content/testsuite/testsuite.go
+++ b/content/testsuite/testsuite.go
@@ -365,8 +365,8 @@ func checkLabels(ctx context.Context, t *testing.T, cs content.Store) {

 	rootTime := time.Now().UTC().Format(time.RFC3339)
 	labels := map[string]string{
-		"k1": "v1",
-		"k2": "v2",
+		"k1":                    "v1",
+		"k2":                    "v2",
 		"containerd.io/gc.root": rootTime,
 	}

@@ -402,7 +402,7 @@ func checkLabels(ctx context.Context, t *testing.T, cs content.Store) {
 	}

 	info.Labels = map[string]string{
-		"k1": "v1",
+		"k1":                    "v1",
 		"containerd.io/gc.root": rootTime,
 	}
 	preUpdate = time.Now()
```

Adding a whitespace before the long key to make it format the same
on both Go 1.11 and older versions of Go.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-07-05 11:05:30 +02:00
Vincent Demeester
832b05ae67
Update tests to use gotest.tools angel
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
2018-06-08 21:02:01 +02:00
Phil Estes
e9434a10bc
Merge pull request #2341 from dmcgowan/move-client-content-snapshot
Move client implementations for content store and snapshotter
2018-05-25 13:47:39 -04:00
Michael Crosby
009ba4d797 Move testutils to pkg
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2018-05-22 17:08:38 -04:00
Derek McGowan
48b0a022ef
Rename remote snapshotter to proxy
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-05-08 14:45:57 -07:00
Derek McGowan
0d1807a43d
Move client snapshotter to separate package
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-05-08 14:45:27 -07:00
Michael Wan
7fd6d5e2be fix words misspell
Signed-off-by: Michael Wan <zirenwan@gmail.com>
2018-04-23 00:09:42 -04:00
Christopher Jones
f1639bca4b
testing: cleanup loopback devices on test skip
Fixes a case where loopback devices wouldn't get cleaned up
when a test was being skipped

Signed-off-by: Christopher Jones <tophj@linux.vnet.ibm.com>
2018-04-18 12:39:21 -04:00
Akihiro Suda
83e35b3d3a enable native (formerly naive) snapshotter by default
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-04-02 13:29:19 +09:00
Derek McGowan
94cf25f7db
overlay: fix cleanup directory deletion race
Fixes a bug where a writable transaction may create or make changes to
a directory while the cleanup is running, leading to removal of a
directory which will be referenced.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-03-14 13:36:49 -07:00
Stephen Day
220a479292
Merge pull request #1833 from dmcgowan/snapshot-gc-3rd-phase
snapshots/overlay: add overlay cleanup
2018-03-12 17:49:56 -07:00
Akihiro Suda
5cc915c26c overlay: add Supported() checker
This function is not called during plugin initialization (#2140),
but should be useful for downstream projects that uses overlayfs
snapshotter as a Go library.

Benchmark result on Ubuntu 17.10, GCE n1-standard-4:

BenchmarkOverlaySupportedOnExt4-4                    100          20490598 ns/op
BenchmarkOverlayUnsupportedOnFType0XFS-4           30000             39316 ns/op
BenchmarkOverlaySupportedOnFType1XFS-4               100          19287083 ns/op
BenchmarkOverlayUnsupportedOnFAT-4                   100          14217772 ns/op

i.e. the overhead is typically about 20 msec on this machine.

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-03-06 16:53:21 +09:00
Kunal Kushwaha
b12c3215a0 Licence header added
Signed-off-by: Kunal Kushwaha <kushwaha_kunal_v7@lab.ntt.co.jp>
2018-02-19 10:32:26 +09:00
Christopher Jones
6e3701141b
[testing] use smaller filesize on 4KB pagesize systems
Use a smaller filesize on systems that have a default 4KB pagesize.

This is because mkfs.btrfs uses the default system pagesize as blocksize
when creating a device, so if the pagesize is larger, then the file needs
to be larger as well. This larger filesize is needed specifically for
systems where 64KB is default, such as ppc64le.

Signed-off-by: Christopher Jones <tophj@linux.vnet.ibm.com>
2018-02-15 15:33:46 -06:00
Daniel Nephin
3279acca82 Check timestamps in snapshot storage test suite
also use t.Helper()
convert assertions to canonical

Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2018-02-12 12:26:26 -05:00
Daniel Nephin
37aa41b164 Cleanup after migration
Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2018-02-12 12:26:26 -05:00
Daniel Nephin
ef48a0268e Migrate to gotestyourself/assert
Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2018-02-12 12:26:26 -05:00
Brian Goff
c776b6d8d9 Use continuity fs package
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-02-07 12:40:52 -05:00
Derek McGowan
1fd2b5783a
Add overlay options for making cleanup asynchronous
Allow configuring the overlay snapshotter to synchronously
or asynchronously do cleanup. When the driver is integrated
into a garbage collection system, the asynchronous cleanup
can reduce the time of removal and allow the longer disk
cleanup to be handled without locking the snapshotter.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-02-01 14:56:03 -08:00
Derek McGowan
14d402e289
Add overlay cleanup
Updates overlay remove to simply remove the reference, adds
a cleanup method for discarding the directory.
Updates snapshot create to setup the directory structure while
in the transaction, to prevent cleanup from removing directories
which are part of a create.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-01-30 16:17:41 -08:00
Christopher Jones
a8c5ff57e5
Cleanup loop devices after test failure
Cleans up loop devices if part of the test or mount process fails.

Also increases btrfs default file size to 650MB to accommodate
minimum btrfs size on ppc64le and s390x

Signed-off-by: Christopher Jones <tophj@linux.vnet.ibm.com>
2018-01-30 14:44:36 -05:00
Christopher Jones
051ac5dd63
running tests in a container
This provides a dockerfile for building a container to run the containerd tests

Signed-off-by: Christopher Jones <tophj@linux.vnet.ibm.com>
2018-01-30 14:44:30 -05:00
Darren Stahl
e981cc6289 Do not create new layer for View
Signed-off-by: Darren Stahl <darst@microsoft.com>
2018-01-23 14:40:24 -08:00
Darren Stahl
dcff993653 Update Windows runtime to use snapshotter and differ layers
This changes the Windows runtime to use the snapshotter and differ
created layers, and updates the ctr commands to use the snapshotter and differ.

Signed-off-by: Darren Stahl <darst@microsoft.com>
2018-01-23 14:40:24 -08:00
Darren Stahl
a5a9f91832 Implement Windows snapshotter and differ
This implements the Windows snapshotter and diff Apply function.
This allows for Windows layers to be created, and layers to be pulled
from the hub.

Signed-off-by: Darren Stahl <darst@microsoft.com>
2018-01-23 14:40:23 -08:00
Daniel Nephin
1cd32fa68d Cleanup extra returns in tests
Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2018-01-23 14:09:22 -05:00
Daniel Nephin
49fffe8ec7 Less verbose tests
Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2017-12-14 11:00:40 -05:00
Stephen Day
68d4dfe4ca
Merge pull request #1870 from dnephin/fix-error-messages
Update snapshot storage error message
2017-12-04 17:40:22 -08:00
Stephen Day
9570174ba9
Merge pull request #1871 from dmcgowan/btrfs-usage
Implement btrfs usage
2017-12-04 17:37:30 -08:00
Derek McGowan
3bc4e69ba1
Implement btrfs usage
Implements btrfs usage using a double walking diff and
counting the result. Walking gives the most accurate
count and includes inode usage.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-12-04 14:42:21 -08:00
Daniel Nephin
0e03d778f1 Update snapshot storage error message
Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2017-12-04 17:35:27 -05:00
yason
db742c90c0 get wrong parents bucket in withSnapshotBucket
improve error message for CommitActive
Signed-off-by: yason <yan.xuean@zte.com.cn>
2017-12-02 23:01:29 +08:00
Derek McGowan
b28d7cdf1b
Update snapshot and content related log messages
Make the tense and casing consistent.
Add useful log messages in image service.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-12-01 14:57:34 -08:00
yason
bb02302c9c improve rollback for overlay.prepare
improve err message
Signed-off-by: yason <yan.xuean@zte.com.cn>
2017-12-02 00:09:44 +08:00
Derek McGowan
25b5592eaf
Fix close twice test to retain snapshot
Update to set labels and use a unique key

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-11-30 14:43:33 -08:00
Jess Valarezo
9885edfc44 rename snapshot->snapshots pkg
Signed-off-by: Jess Valarezo <valarezo.jessica@gmail.com>
2017-11-29 14:55:02 -08:00