Commit Graph

291 Commits

Author SHA1 Message Date
Danny Canter
13ff185ba0 Blockfile: Enlighten blockfile copy on Darwin
The Go stdlib does not seem to have an efficient os.File.ReadFrom
routine for other platforms like it does on Linux with
copy_file_range. For Darwin at least we can use clonefile
in its place, otherwise if we have a sparse file we'd have
a fun surprise with the io.Copy approach..

We should see if there's other platforms that we can enhance here.
I've forgotten what's the right route on Windows.

Signed-off-by: Danny Canter <danny@dcantah.dev>
2023-08-31 20:18:48 -07:00
Danny Canter
34f32043bc Blockfile: Expose RecreateScratch option
This was the only option not configurable from the toml for the plugin.
This is useful if you want to restart containerd and try a different
blockfile/size for the snapshotter.

Signed-off-by: Danny Canter <danny@dcantah.dev>
2023-08-24 13:28:22 -07:00
Sebastiaan van Stijn
11a5dd8265
snapshots: replace some fmt.Sprintfs with strconv
Teeny-tiny optimizations:

    BenchmarkSprintf-10       37735996    32.31  ns/op  0 B/op  0 allocs/op
    BenchmarkItoa-10         591945836     2.031 ns/op  0 B/op  0 allocs/op
    BenchmarkFormatUint-10   593701444     2.014 ns/op  0 B/op  0 allocs/op

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-08-23 10:10:59 +02:00
Fu Wei
738c153573
Merge pull request #8992 from djdongjin/remove-hashicorp-multierror
Remove hashicorp/go-multierror dependency
2023-08-23 13:13:51 +08:00
Jin Dong
6d8329d7f5 Remove unnecessary joinError unwrap
Signed-off-by: Jin Dong <jin.dong@databricks.com>
2023-08-21 19:03:41 -07:00
Jin Dong
cd8c8ae4bc Remove hashicorp/go-multierror
Signed-off-by: Jin Dong <jin.dong@databricks.com>
2023-08-20 17:59:45 -07:00
Robbie Buxton
23c95359aa Add WithMetaStore to overlay snapshotter to allow bringing your own
Signed-off-by: Robbie Buxton <138501839+rbpdt@users.noreply.github.com>
2023-08-17 19:28:44 +01:00
Ben Foster
f3daf32c73 Fix ro mount option being passed
"ro" was not parsed out of the string, so it was passed as part of data
to mount().
This would lead to mount() returning an invalid argument code.
Separate out the "ro" option, much like "userxattr", which will allow
the MS-RDONLY mountflag to get set.

Signed-off-by: Ben Foster <bpfoster@gmail.com>
2023-07-20 17:08:01 -04:00
James Jenkins
5c480d9c88 Modify loopback size
Modify the loopback size in the blockfile snapshotter test setup.
Set the loopback size to 16MB when the page size is greater than 4096.

Signed-off-by: James Jenkins <James.Jenkins@ibm.com>
2023-07-10 11:06:24 -04:00
Danny Canter
b3ab1f26c4 Snapshots: Add From/ToProto helpers for types
Helpers to convert from snapshot types to their protobuf structures and
vice-versa appear three times. It seems sane to just expose this facility
in the snapshots pkg. From/ToKind weren't used anywhere but doesn't hurt to
round out the types by exposing them.

Signed-off-by: Danny Canter <danny@dcantah.dev>
2023-06-28 12:17:52 -07:00
Danny Canter
55a8102ec1 mount: Add From/ToProto helpers
Helpers to convert from containerd's [Mount] to its protobuf structure for
[Mount] and vice-versa appear three times. It seems sane to just expose
this facility in /mount.

Signed-off-by: Danny Canter <danny@dcantah.dev>
2023-06-28 04:03:18 -07:00
Wei Fu
a207b430e3 snapshots/blockfile: fix lint issue
Signed-off-by: Wei Fu <fuweid89@gmail.com>
2023-06-17 08:42:02 +08:00
Wei Fu
7de95cbc4c snapshots/blockfile: deflaky the testsuite
* Use direct-io mode to reduce IO.

* Add testViewHook helper to recovery the backing file since the ext4
  might need writable permission to handle recovery. If the backing file
  needs recovery and it's for View snapshot, the readonly mount will
  cause error.

* Use 8 MiB as capacity to reduce the IO.

Signed-off-by: Wei Fu <fuweid89@gmail.com>
2023-06-15 23:54:06 +08:00
Wei Fu
6dfb16f99a snapshots|pkg: umount without DETACH and nosync after umount
Signed-off-by: Wei Fu <fuweid89@gmail.com>
2023-06-15 23:53:47 +08:00
Derek McGowan
d115129d1e
Add configurable mount options to overlay
Allows default mount options to be provided through configuration.

Signed-off-by: Derek McGowan <derek@mcg.dev>
2023-06-12 09:17:07 -07:00
Derek McGowan
f49789fd54
Merge pull request #8594 from fuweid/fix-blockfile-mount
snapshots/blockfile: use loop if options is empty
2023-05-30 08:33:09 -07:00
Cardy.Tang
9702d4aeec snapshots/blockfile: use passed in dst in scratchGenerator
Signed-off-by: Cardy.Tang <zuniorone@gmail.com>
2023-05-30 11:42:19 +08:00
Wei Fu
bb82f3683e snapshots/blockfile: use loop if options is empty
Signed-off-by: Wei Fu <fuweid89@gmail.com>
2023-05-29 23:26:39 +08:00
Derek McGowan
7beaa5e859
Add mount options to blockfile snapshotter
Signed-off-by: Derek McGowan <derek@mcg.dev>
2023-05-23 10:56:20 -07:00
Cardy.Tang
b9f2e48e37 feat: make overlay sync removal configurable
Signed-off-by: Cardy.Tang <zuniorone@gmail.com>
2023-05-19 16:32:10 +08:00
Akihiro Suda
878132923d
Merge pull request #8511 from dmcgowan/blockfile-snapshotter
Add blockfile snapshotter
2023-05-18 11:52:37 +09:00
Derek McGowan
1be571b14f
Add sync before unmount on snapshotter layer test
Signed-off-by: Derek McGowan <derek@mcg.dev>
2023-05-17 14:25:27 -07:00
Derek McGowan
0c386e2ebd
Add blockfile snapshotter
Signed-off-by: Derek McGowan <derek@mcg.dev>
2023-05-17 14:25:27 -07:00
Akihiro Suda
5dbae38191
snapshots/testsuite: Rename: fix fuse-overlayfs incompatibility
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2023-05-11 16:59:53 +09:00
Derek McGowan
14d22efccc
Merge pull request #8254 from gabriel-samfira/fix-access-denied-on-vhdx-mount
Fix access denied on mounted vhdx root
2023-05-10 08:39:36 -07:00
Maksym Pavlenko
6f34da5f80 Cleanup logrus imports
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2023-05-05 11:54:14 -07:00
Gabriel Adrian Samfira
6f0714efcb Use RunWithPrivileges
RunWithPrivileges() will enable privileges will lock a thread, change
privileges, and run the function passed in, within that thread. This
allows us to limit the scope in which we enable privileges and avoids
accidentally enabling privileges in threads that should never have them.

Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2023-04-06 17:06:50 -07:00
Gabriel Adrian Samfira
b41ca11598 Fix access denied on mounted vhdx root
It seems that in certain situations, like having the containerd root
and state on a file system hosted on a mounted VHDX, we need
SeSecurityPrivilege when opening a file with winio.ACCESS_SYSTEM_SECURITY.
This happens in the base layer writer in hcsshim when adding a new file.

Enabling SeSecurityPrivilege allows the containerd root to be hosted on
a vhdx.

Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2023-04-06 17:06:50 -07:00
Gabriel Adrian Samfira
8538e7a2ac Improve error messages and remove check
* Improve error messages
  * remove a check for the existance of unmount target. We probably
    should not mask that the target was missing.

Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2023-04-04 12:07:34 -07:00
Gabriel Adrian Samfira
6a5b4c9c24 Remove "bind" code path from diff
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2023-04-03 08:11:35 -07:00
Gabriel Adrian Samfira
d373ebc4de Properly mount base layers
As opposed to a writable layer derived from a base layer, the volume
path of a base layer, once activated and prepared will not be a WCIFS
volume, but the actual path on disk to the snapshot. We cannot directly
mount this folder, as that would mean a client may gain access and
potentially damage important metadata files that would render the layer
unusabble.

For base layers we need to mount the Files folder which must exist in
any valid base windows-layer.

Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2023-04-02 08:35:34 -07:00
Gabriel Adrian Samfira
ca5605b4a2 Skip parent layer options on bind mounts
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2023-04-01 08:49:24 -07:00
Gabriel Adrian Samfira
4012c1b853 Remove escalated privileges
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2023-03-31 06:17:35 -07:00
Gabriel Adrian Samfira
95687a9324 Fix go.mod, simplify boolean logic, add logging
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2023-03-31 06:16:56 -07:00
Gabriel Adrian Samfira
feb637f92d Fix layer comparison and enable read-only checks
fstest.CheckDirectoryEqual checks if any files in the diff matches a
list of known metadataFiles. This only happens if we specify the initial
layer as the first parameter and the mutated layer as the second.

This also enables the read-only view checks, as the bind filter allows
us to mount a layer as ro.

Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2023-03-31 06:15:18 -07:00
Gabriel Adrian Samfira
36dc2782c4 Use bind filer for mounts
The bind filter supports bind-like mounts and volume mounts. It also
allows us to have read-only mounts.

Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2023-03-31 06:15:18 -07:00
Paul "TBBle" Hampson
7b36becd2d Run Windows snapshotter through the test suite
Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2023-03-31 06:15:18 -07:00
Paul "TBBle" Hampson
ac30eabbdc Fix misspelling of 'Native' as 'Naive'
Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2023-03-31 06:15:18 -07:00
Paul "TBBle" Hampson
639c5799a2 Add paired 'mount' log for 'unmount'
Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2023-03-31 06:15:18 -07:00
Paul "TBBle" Hampson
8395e3a89a Don't use all-upper-case filenames in snapshot tests
NTFS, when presented with an all-caps filename, assumes you are just
being loud for no reason, and instead stores an all-lower-case filename.

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2023-03-31 06:15:18 -07:00
Paul "TBBle" Hampson
909730decb Skip tests that do not apply to WCOW on Windows
Filesystem permissions and ownership are not modifiable via an image
mount.

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2023-03-31 06:15:18 -07:00
Paul "TBBle" Hampson
469c13997a Ensure mounts are unmounted before leaving the test
This is necessary on Windows, as it's not possible to delete a snapshot
while it is still mounted, even if the mount-point has been deleted.

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2023-03-31 06:15:17 -07:00
Paul "TBBle" Hampson
34b07d3e2d Implement WCOW parentless active snapshots and view snapshots
The WCOW layer support does not support creating sandboxes with no
parent.  Instead, parentless scratch layers must be layed out as a
directory containing only a directory named 'Files', and all data stored
inside 'Files'. At commit-time, this will be converted in-place into a
read-only layer suitable for use as a parent layer.

The WCOW layer support also does not deal with making read-only layers,
i.e. layers that are prepared to be parent layers, visible in a
read-only manner. A bind-mount or junction point cannot be made
read-only, so a view must instead be a small sandbox layer that we can
mount via WCOW, and discard later, to protect the layer against
accidental or deliberate modification.

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2023-03-31 06:15:17 -07:00
Brian Goff
c56f54aa1e devmapper plugin: skip plugin when not configured
This is not really an error in most cases as most people do not use
devmapper, however this shows up as an error in the logs and in the
plugin service when querying the plugins.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2023-03-28 00:00:55 +00:00
Akihiro Suda
86fc1ccab4
Remove aufs snapshotter (deprecated since v1.5)
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2023-03-14 14:37:13 +09:00
Fu Wei
8cb00f45c9
Merge pull request #8143 from mxpv/log
Add Fields type alias to log package
2023-02-21 10:22:23 +08:00
Maksym Pavlenko
06e085c8b5 Add Fields type alias to log package
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2023-02-20 17:29:08 -08:00
Akihiro Suda
d8b68e3ccc
Stop using math/rand.Read and rand.Seed (deprecated in Go 1.20)
From golangci-lint:

> SA1019: rand.Read has been deprecated since Go 1.20 because it
>shouldn't be used: For almost all use cases, crypto/rand.Read is more
>appropriate. (staticcheck)

> SA1019: rand.Seed has been deprecated since Go 1.20 and an alternative
>has been available since Go 1.0: Programs that call Seed and then expect
>a specific sequence of results from the global random source (using
>functions such as Int) can be broken when a dependency changes how
>much it consumes from the global random source. To avoid such breakages,
>programs that need a specific result sequence should use
>NewRand(NewSource(seed)) to obtain a random generator that other
>packages cannot access. (staticcheck)

See also:

- https://pkg.go.dev/math/rand@go1.20#Read
- https://pkg.go.dev/math/rand@go1.20#Seed

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2023-02-16 03:50:23 +09:00
Phil Estes
024a748c09
Merge pull request #7933 from AkihiroSuda/drop-libbtrfs
btrfs: depend on kernel UAPI instead of libbtrfs
2023-02-10 16:13:19 -05:00
Akihiro Suda
52f82acb7b
btrfs: depend on kernel UAPI instead of libbtrfs
See containerd/btrfs PR 40 and moby/moby PR 44761. (Thanks to [@]neersighted.)

The containerd/btrfs library now requires headers from kernel 4.12 or newer:
- https://github.com/torvalds/linux/blob/master/include/uapi/linux/btrfs.h
- https://github.com/torvalds/linux/blob/master/include/uapi/linux/btrfs_tree.h

These files are licensed under the GPL-2.0 WITH Linux-syscall-note, so it should be compatible with the Apache License 2.0.
https://spdx.org/licenses/Linux-syscall-note.html

The dependency on the kernel headers only affects users building from source.
Users on older kernels may opt to not compile this library (`BUILDTAGS=no_btfs`),
or to provide headers from a newer kernel.

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2023-02-10 10:07:34 +09:00