The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Enables showing debug logs in testing output.
For integration tests the client log output will show
in addition to daemon output, with timestamps for better
correlation.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
This changeset modifies the metadata store to allow one to set a
"content sharing policy" that defines how blobs are shared between
namespaces in the content store.
The default mode "shared" will make blobs available in all namespaces
once it is pulled into any namespace. The blob will be pulled into
the namespace if a writer is opened with the "Expected" digest that
is already present in the backend.
The alternative mode, "isolated" requires that clients prove they have
access to the content by providing all of the content to the ingest
before the blob is added to the namespace.
Both modes share backing data, while "shared" will reduce total
bandwidth across namespaces, at the cost of allowing access to any
blob just by knowing its digest.
Note: Most functional codes and changelog of this commit originate from
Stephen J Day <stephen.day@docker.com>, see
40455aade8Fixes#1713Fixes#2865
Signed-off-by: Eric Lin <linxiulei@gmail.com>
The local store could end up in a state where the writer is
closed but the reference is locked after a commit on an
existing object.
Cleans up Commit logic to always close the writer even after
an error occurs, guaranteeing the reference is unlocked after commit.
Adds a test to the content test suite to verify this behavior.
Updates the content store interface definitions to clarify the behavior.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Go 1.11 uses a different formatting for maps, and now
aligns values; running `gofmt -s -w` on this file resulted
in this diff;
```patch
content/testsuite/testsuite.go | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/content/testsuite/testsuite.go b/content/testsuite/testsuite.go
index 974c7cb8ed..d9ae9dc160 100644
--- a/content/testsuite/testsuite.go
+++ b/content/testsuite/testsuite.go
@@ -365,8 +365,8 @@ func checkLabels(ctx context.Context, t *testing.T, cs content.Store) {
rootTime := time.Now().UTC().Format(time.RFC3339)
labels := map[string]string{
- "k1": "v1",
- "k2": "v2",
+ "k1": "v1",
+ "k2": "v2",
"containerd.io/gc.root": rootTime,
}
@@ -402,7 +402,7 @@ func checkLabels(ctx context.Context, t *testing.T, cs content.Store) {
}
info.Labels = map[string]string{
- "k1": "v1",
+ "k1": "v1",
"containerd.io/gc.root": rootTime,
}
preUpdate = time.Now()
```
Adding a whitespace before the long key to make it format the same
on both Go 1.11 and older versions of Go.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This change allows implementations to resolve the location of the actual data
using OCI descriptor fields such as MediaType.
No OCI descriptor field is written to the store.
No change on gRPC API.
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
Allows the client to choose the context to finish the lease.
This allows the client to switch contexts when the main context
used to the create the lease may have been cancelled.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Update content ingests to use content from another namespace.
Ingests must be committed to make content available and the
client will see the sharing as an ingest which has already
been fully written to, but not completed.
Updated the database version to change the ingest record in
the database from a link key to an object with a link and
expected value. This expected value is used to indicate that
the content already exists and an underlying writer may
not yet exist.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Let the test runners choose the namespaces and
wrap the contexts. This allows the test suite to create
multiple contexts without worrying about namespacing
or leasing in the contexts.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
checkUploadStatus actaully verifys contents on Updating already written buffers,
hence should be called as checkUpdateStatus
Signed-off-by: Kunal Kushwaha <kushwaha_kunal_v7@lab.ntt.co.jp>
Prevents the copy method from calling discard on the
writer when the reader is not seekable. Instead,
the copy method will discard up to the offset.
Truncate is a more expensive operation since any
bytes that are truncated already have their hash calculated
and are stored on disk in the backend. Re-writing bytes
which were truncated requires transfering the data over
GRPC again and re-computing the hash up to the point of
truncation.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
There is a bug in the windows CI that causes a time difference between
the host and the container.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
The locks now retry on the backend side to prevent clients from having
to round trip on locks that might be momentarily held. This exposed some
timing errors in the updated_at fields for content ingest, so we've had
to move that to a separate file to export the monotonic go runtime
timestamps.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
While early PoCs had download resumption working, we didn't have tests
and had not verified the behavior. With this test suite, we now are able
to show that download resumption is properly supported in the content
store. In particular, there was a bug where resuming a download would
not issue the writes to the correct offset in the file. A Seek was added
to ensure we are writing from the current ingest offset.
In this investigation, it was also discovered that using the OS/Disk
created time on files was skewed from the monotonic clock in Go's
runtime. The startedat values are now taken from the Go runtime and
written to a separate file.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Content commit is updated to take in a context, allowing
content to be committed within the same context the writer
was in. This is useful when commit may be able to use more
context to complete the action rather than creating its own.
An example of this being useful is for the metadata implementation
of content, having a context allows tests to fully create
content in one database transaction by making use of the context.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Ensure all writers are closed at end of test for content
test suite. Prevents test from leaving lingering connections
to the content store.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Add commit options which allow for setting labels on commit.
Prevents potential race between garbage collector reading labels
after commit and labels getting set.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Add content test suite with test for writer.
Update fs and metadata implementations to use test suite.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
