Commit Graph

60 Commits

Author SHA1 Message Date
zounengren
3a713811be run gofmt with Go 1.17
Signed-off-by: Zou Nengren <zouyee1989@gmail.com>
2021-10-07 20:16:59 +08:00
Eng Zer Jun
50da673592
refactor: move from io/ioutil to io and os package
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-09-21 09:50:38 +08:00
Sebastiaan van Stijn
2ac9968401
replace uses of os/exec with golang.org/x/sys/execabs
Go 1.15.7 contained a security fix for CVE-2021-3115, which allowed arbitrary
code to be executed at build time when using cgo on Windows. This issue also
affects Unix users who have “.” listed explicitly in their PATH and are running
“go get” outside of a module or with module mode disabled.

This issue is not limited to the go command itself, and can also affect binaries
that use `os.Command`, `os.LookPath`, etc.

From the related blogpost (ttps://blog.golang.org/path-security):

> Are your own programs affected?
>
> If you use exec.LookPath or exec.Command in your own programs, you only need to
> be concerned if you (or your users) run your program in a directory with untrusted
> contents. If so, then a subprocess could be started using an executable from dot
> instead of from a system directory. (Again, using an executable from dot happens
> always on Windows and only with uncommon PATH settings on Unix.)
>
> If you are concerned, then we’ve published the more restricted variant of os/exec
> as golang.org/x/sys/execabs. You can use it in your program by simply replacing

This patch replaces all uses of `os/exec` with `golang.org/x/sys/execabs`. While
some uses of `os/exec` should not be problematic (e.g. part of tests), it is
probably good to be consistent, in case code gets moved around.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-08-25 18:11:09 +02:00
Akihiro Suda
d3aa7ee9f0
Run go fmt with Go 1.17
The new `go fmt` adds `//go:build` lines (https://golang.org/doc/go1.17#tools).

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2021-08-22 09:31:50 +09:00
Derek McGowan
6f027e38a8
Remove redundant build tags
Remove build tags which are already implied by the name of the file.
Ensures build tags are used consistently

Signed-off-by: Derek McGowan <derek@mcg.dev>
2021-08-05 22:27:46 -07:00
ktock
b483177ee2 Support custom compressor for walking differ
Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>
2021-07-20 11:00:01 +09:00
Derek McGowan
69f43d4589
Revert diff/walking error change
Clarify error scope and create variable for deferring cleanup

Signed-off-by: Derek McGowan <derek@mcg.dev>
2021-06-04 08:44:24 -07:00
Iceber Gu
558fdc6808 diff/walking: fix defer cleanup
Signed-off-by: Iceber Gu <wei.cai-nat@daocloud.io>
2021-06-01 18:24:47 +08:00
Paul "TBBle" Hampson
c7504987e6 Implement windowsDiff.Compare via hcsshim/pkg/ociwclayer
This parallels the implementation of windowsDiff.Apply, including
bouncing very briefly though archive.WriteDiff and then straight back
out into Windows-specific code.

It's mostly pulling existing mechanisms from non-Windows Compare or
Windows Apply, and highlights that there's probably a lot of scope for
refactoring on top of this.

Now the export-related integration tests pass CI on Windows.

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2021-04-10 02:02:33 +10:00
Paul "TBBle" Hampson
149fa366f9 Don't tease the logger with a %-less format string
Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2021-04-10 02:02:33 +10:00
Paul "TBBle" Hampson
b399e2ef6d Don't lose Compare failure if aborting diff upload fails
Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2021-04-10 02:02:33 +10:00
Sebastiaan van Stijn
708299ca40
Move RunningInUserNS() to its own package
This allows using the utility without bringing whole of "sys" with it.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-03-23 11:29:53 +01:00
Michael Crosby
969b3d638b
Merge pull request #5202 from wzshiming/fix/dgst-debug
Change dgst to digest in debug
2021-03-18 11:54:44 -04:00
Shiming Zhang
766e7953ab Change dgst to digest in debug
Signed-off-by: Shiming Zhang <wzshiming@foxmail.com>
2021-03-15 16:46:19 +08:00
Derek McGowan
35eeb24a17
Fix exported comments enforcer in CI
Add comments where missing and fix incorrect comments

Signed-off-by: Derek McGowan <derek@mcg.dev>
2021-03-12 08:47:05 -08:00
Stefan Berger
1917ca5f79 Allow passing environent variables to StreamProcessors
Add support for an 'env' field to the StreamProcessor configuration
and append the environment variables found there to the os.Environ()
array.
The env field takes environment variables in the form of key=value.

Signed-off-by: Stefan Berger <stefanb@linux.ibm.com>
2021-01-11 13:15:12 -05:00
Kazuyoshi Kato
bb8aac38a0 Do not hardcode "amd64" on LCOW and Windows-related files
Fixes #3281.

Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2020-11-09 13:39:07 -08:00
Amit Barve
d3b817b95c Read trailing data from tar reader
Not reading all the data from the tar reader causes the
layer digest mismatch which causes failures during unpack
of certain images for lcow. This changes fixes that.

Signed-off-by: Amit Barve <ambarve@microsoft.com>
2020-10-13 22:38:47 -07:00
Paul "TBBle" Hampson
71f11db8a3 Annotate bare ErrNotImplemented returns
This makes it possible to see _what_ is not implemented from the caller.

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2020-07-16 20:16:11 +10:00
Sebastiaan van Stijn
6a9b94927f
Remove remaining uses of libcontainer/system package
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-05-04 18:56:57 +02:00
Akihiro Suda
c224edc5c6 apply: use naive applier when running in UserNS
`OverlayConvertWhiteout` calls `mknod c 0 0` which is not allowed when
running in a user namespace, even in Ubuntu kernel.

Although there is an alternative hacky way to create whiteouts without
calling mknod as Moby `overlay2` actually does(see #3762), let's use
naive applier when running in UserNS and call it a day.

Close #3762

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2019-10-24 13:33:09 +09:00
Derek McGowan
6f31417d49
Generalize media types
Avoid directly handling media types with "+" attributes,
instead handling the base and passing through the full
media type to the appropriate stream processor or decompression.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2019-09-19 16:09:19 -07:00
Derek McGowan
550518ab84
Read trailing data on apply
This was removed by mistake during the direct unpack
PR refactoring.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2019-08-21 11:41:56 -07:00
Derek McGowan
bcc4a146e4
Support applying with parent directories
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2019-08-16 15:49:47 -07:00
Lantao Liu
81386df917
Add direct unpack support for overlay and aufs
Signed-off-by: Lantao Liu <lantaol@google.com>
2019-08-15 12:09:32 -07:00
Michael Crosby
552a0b1be5 Handle stderr in stream processors
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2019-08-07 18:36:52 +00:00
Michael Crosby
3fded74bc7 Add unpack opts
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2019-08-07 18:35:55 +00:00
Michael Crosby
e1489f93c3 Use named pipes for windows processors
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2019-08-07 18:35:55 +00:00
Michael Crosby
134d3c8159 Add windows apply code
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2019-08-07 18:35:55 +00:00
Michael Crosby
f867401c69 Use fds and pass Payloads over diff api
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2019-08-07 18:35:55 +00:00
Michael Crosby
366823727f Add server config for stream processors
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2019-08-07 18:35:55 +00:00
Michael Crosby
97a98773cf Add StreamProcessor for apply
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2019-08-07 18:35:55 +00:00
Michael Crosby
052551573d Ensure labels is not nil in differ
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2019-06-14 15:20:32 +00:00
Michael Crosby
2ec2089b05
Merge pull request #3123 from stefanberger/extend_apply_call_with_options_parameter
Extend Applier's Apply() method with an optional options parameter
2019-06-03 11:13:40 -04:00
Justin Terry (VM)
e0c4fe4110 Forcibly sync writes on lcow layer.vhd after write
Signed-off-by: Justin Terry (VM) <juterry@microsoft.com>
2019-05-22 13:31:58 -07:00
Tonis Tiigi
5b9bd993a8 differ: fix deadlock on commit error
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-04-25 17:50:17 -07:00
Stefan Berger
baf3403439 Extend Applier's Apply() method with an optional options parameter
Extend the Applier interface's Apply method with an optional
options parameter.

For the container image encryption we intend to use the options
parameter to pass image decryption parameters ('dcparameters'),
which are primarily (privatte) keys, in form of a JSON document
under the map key '_dcparameters', and pass them to the Applier's
Apply() method. This helps us to access decryption keys and start
the pipeline with the layer decryption before the layer data are
unzipped and untarred.

Signed-off-by: Stefan Berger <stefanb@linux.ibm.com>
Signed-off-by: Harshal Patil <harshal.patil@in.ibm.com>
2019-04-02 18:19:48 -04:00
Justin Terry (VM)
7361b19875 Update Windows lcow differ to set NT VIRTUAL MACHINE\Virtual Machines SID
For LCOW using the Virtual Machines SID for the shared read-only layers
improves overall performance avoiding the need to set per VM access at runtime.

Signed-off-by: Justin Terry (VM) <juterry@microsoft.com>
2019-03-21 14:53:43 -07:00
Wei Fu
132ee9b826 fix: linter issue
megacheck, gosimple and unused has been deprecated and subsumed by
staticcheck. And staticcheck also has been upgraded. we need to update
code for the linter issue.

close: #2945

Signed-off-by: Wei Fu <fuweid89@gmail.com>
2019-01-23 22:54:51 +08:00
Derek McGowan
bb197ddc47
Set uncompressed label on diff when already exists
Diff expects to set the uncompressed label on Commit, however
if the blob already exists in the content store, no labels
will get set, requiring an Update. Check if the uncompressed
label is on the blob and set it if not.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-10-15 11:02:35 -07:00
Justin Terry (VM)
cce78d4b5d Remove dependency on tar2vhd for LCOW differ
This change no longer requires the use of a UtlityVM on Windows to convert the
layer tar to an ext4 vhd for LCOW. This has a significant performance boost
that makes linux/amd64 layer extraction comparable to native Linux performance.

Signed-off-by: Justin Terry (VM) <juterry@microsoft.com>
2018-10-10 11:40:10 -07:00
Derek McGowan
6875d3df3a
Always check exists on commit error
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-09-14 01:26:03 -07:00
Justin Terry (VM)
0110b3c0bc Introduce the Windows lcow diff/snaphotter
Implements the Windows lcow differ/snapshotter responsible for managing
the creation and lifetime of lcow containers on Windows.

Signed-off-by: Justin Terry (VM) <juterry@microsoft.com>
2018-08-28 13:51:04 -07:00
Dave Henderson
9a97ab34ce
Switching from crypto/rand to math/rand to avoid blocking
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
2018-07-11 16:24:51 -04:00
Akihiro Suda
d88de4a34f content: change Writer/ReaderAt to take OCI
This change allows implementations to resolve the location of the actual data
using OCI descriptor fields such as MediaType.

No OCI descriptor field is written to the store.

No change on gRPC API.

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-06-01 11:51:43 +09:00
Michael Wan
7fd6d5e2be fix words misspell
Signed-off-by: Michael Wan <zirenwan@gmail.com>
2018-04-23 00:09:42 -04:00
Kunal Kushwaha
b12c3215a0 Licence header added
Signed-off-by: Kunal Kushwaha <kushwaha_kunal_v7@lab.ntt.co.jp>
2018-02-19 10:32:26 +09:00
Akihiro Suda
ee70879202 *: use narrower content interfaces
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-02-09 19:52:52 +09:00
Derek McGowan
b763777288
diff: rename differ to comparer
Remove combined interface and split implementations.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-01-26 16:32:09 -08:00
Akihiro Suda
b580441f91
diff: resplit Applier from Differ
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-01-25 13:58:34 -08:00