The goal of this change is to remove the registration of signal
handling from pkg/kubelet. We now pass in a stop channel.
If you register a signal handler in `main()` to aid in a controlled
and deliberate exit then the handler registered in `pkg/kubelet` often
wins and the process exits immediately. This means all other signal
handler registrations are currently racy if `DockerServer.Start()` is
directly or indirectly invoked.
This change also removes another signal handler registration from
`NewAPIServerCommand()`; a stop channel is now passed to this
function.
Windows supports both sandbox and non-sandbox cases. The non-sandbox
case is for Windows Server 2016 and for Windows Server version greater
than 1709 which use Hyper-V containers.
Currently, the CNI on Windows fetches the IP from the containers
within the pods regardless of the mode. This should be done only
in the non-sandbox mode where the IP of the actual container
will be different than the IP of the sandbox container.
In the case where the sandbox container is supported, all the containers
from the same pod will share the network details of the sandbox container.
This patch updates the CNI to fetch the IP from the sandbox container
when this mode is supported.
Signed-off-by: Alin Balutoiu <abalutoiu@cloudbasesolutions.com>
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix e2e "When checkpoint file is corrupted should complete pod sandbo…
…x clean up"
**What this PR does / why we need it**:
This PR fixes the e2e-node test, "When checkpoint file is corrupted should complete pod sandbox clean up"
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#62738
Related #62937
**Special notes for your reviewer**:
**Release note**:
```release-note
None
```
/cc @dashpole @derekwaynecarr
/sig node
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
should use time.Since instead of time.Now().Sub
**What this PR does / why we need it**:
should use time.Since instead of time.Now().Sub
**Special notes for your reviewer**:
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
kubelet - Remove unused code
**What this PR does / why we need it**:
Looks like we have a bunch of unused methods. Let's clean them up
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add support for CNI on Windows Server 2016 RTM
**What this PR does / why we need it**:
Windows Server 2016 RTM has limited CNI support. This PR makes it possible for the CNI plugin to be used to setup POD networking on Windows Server 2016 RTM (build number 14393).
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#61939
**Special notes for your reviewer**:
The old mode is not supported and tested on Windows Server 2016 RTM. This change allows the CNI plugin to be used on Windows Server 2016 RTM to retrieve the container IP instead of using workarounds (docker inspect).
CNI support has been added for Windows Server 2016 version 1709 (build number 16299), this patch will just allow the same support for older build numbers.
Windows Server 2016 RTM has a longer lifecycle (LTS) than Windows Server 2016 version 1709.
https://support.microsoft.com/en-us/lifecycle/search/19761 vs https://support.microsoft.com/en-us/lifecycle/search/20311
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Use a []string for CgroupName, which is a more accurate internal representation
**What this PR does / why we need it**:
This is purely a refactoring and should bring no essential change in behavior.
It does clarify the cgroup handling code quite a bit.
It is preparation for further changes we might want to do in the cgroup hierarchy. (But it's useful on its own, so even if we don't do any, it should still be considered.)
**Special notes for your reviewer**:
The slice of strings more precisely captures the hierarchic nature of the cgroup paths we use to represent pods and their groupings.
It also ensures we're reducing the chances of passing an incorrect path format to a cgroup driver that requires a different path naming, since now explicit conversions are always needed.
The new constructor `NewCgroupName` starts from an existing `CgroupName`, which enforces a hierarchy where a root is always needed. It also performs checking on the component names to ensure invalid characters ("/" and "_") are not in use.
A `RootCgroupName` for the top of the cgroup hierarchy tree is introduced.
This refactor results in a net reduction of around 30 lines of code,
mainly with the demise of ConvertCgroupNameToSystemd which had fairly
complicated logic in it and was doing just too many things.
There's a small TODO in a helper `updateSystemdCgroupInfo` that was introduced to make this commit possible. That logic really belongs in libcontainer, I'm planning to send a PR there to include it there. (The API already takes a field with that information, only that field is only processed in cgroupfs and not systemd driver, we should fix that.)
Tested: By running the e2e-node tests on both Ubuntu 16.04 (with cgroupfs driver) and CentOS 7 (with systemd driver.)
**NOTE**: I only tested this with dockershim, we should double-check that this works with the CRI endpoints too, both in cgroupfs and systemd modes.
/assign @derekwaynecarr
/assign @dashpole
/assign @Random-Liu
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Update all script shebangs to use /usr/bin/env interpreter instead of /bin/interpreter
This is required to support systems where bash doesn't reside in /bin (such as NixOS, or the *BSD family) and allow users to specify a different interpreter version through $PATH manipulation.
https://www.cyberciti.biz/tips/finding-bash-perl-python-portably-using-env.html
```release-note
Use /usr/bin/env in all script shebangs to increase portability.
```
The slice of strings more precisely captures the hierarchic nature of
the cgroup paths we use to represent pods and their groupings.
It also ensures we're reducing the chances of passing an incorrect path
format to a cgroup driver that requires a different path naming, since
now explicit conversions are always needed.
The new constructor NewCgroupName starts from an existing CgroupName,
which enforces a hierarchy where a root is always needed. It also
performs checking on the component names to ensure invalid characters
("/" and "_") are not in use.
A RootCgroupName for the top of the cgroup hierarchy tree is introduced.
This refactor results in a net reduction of around 30 lines of code,
mainly with the demise of ConvertCgroupNameToSystemd which had fairly
complicated logic in it and was doing just too many things.
There's a small TODO in a helper updateSystemdCgroupInfo that was
introduced to make this commit possible. That logic really belongs in
libcontainer, I'm planning to send a PR there to include it there.
(The API already takes a field with that information, only that field is
only processed in cgroupfs and not systemd driver, we should fix that.)
Tested by running the e2e-node tests on both Ubuntu 16.04 (with cgroupfs
driver) and CentOS 7 (with systemd driver.)
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix dockershim e2e
**What this PR does / why we need it**:
Delete checkpoint file when GetCheckpoint fails due to corrupt checkpoint. Earlier, before checkpointmanager, [`GetCheckpoint` in dockershim was deleting corrupt checkpoint file implicitly](https://github.com/kubernetes/kubernetes/pull/56040/files#diff-9a174fa21408b7faeed35309742cc631L116). In checkpointmanager's `GetCheckpoint` this implicit deletion of corrupt checkpoint is not happening. Because of this few e2e tests are failing because these tests are testing this deletion.
Changes are being added to delete checkpoint file if found corrupted.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#62738
**Special notes for your reviewer**:
No new behavior is being introduced. Implicit deletion of corrupt checkpoint is being done explicitly.
**Release note**:
```release-note
None
```
/cc @dashpole @sjenning @derekwaynecarr
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Change seccomp annotation from "docker/default" to "runtime/default"
**What this PR does / why we need it**:
This PR changes seccomp annotation from "docker/default" to "runtime/default", so that it is can be applied to all kinds of container runtimes. This PR is a followup of [#1963](https://github.com/kubernetes/community/pull/1963).
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#39845
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
If the CNI network plugin completes successfully, but something fails
between that success and dockerhsim's sandbox setup code, plugin resources
may not be cleaned up. A non-trivial amount of code runs after the
plugin itself exits and the CNI driver's SetUpPod() returns, and any error
condition recognized by that code would cause this leakage.
The Kubernetes CRI RunPodSandbox() request does not attempt to clean
up on errors, since it cannot know how much (if any) networking
was actually set up. It depends on the CRI implementation to do
that cleanup for it.
In the dockershim case, a SetUpPod() failure means networkReady is
FALSE for the sandbox, and TearDownPod() will not be called later by
garbage collection even though networking was configured, because
dockershim can't know how far SetUpPod() got.
Concrete examples include if the sandbox's container is somehow
removed during during that time, or another OS error is encountered,
or the plugin returns a malformed result to the CNI driver.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1532965
Automatic merge from submit-queue (batch tested with PRs 62448, 59317, 59947, 62418, 62352). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
fix assert.Equal argument order
Reference:
https://godoc.org/github.com/stretchr/testify/assert#Equal
**What this PR does / why we need it**:
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
With CRI, kubelet no longer sets up networking for the pods. The
dockershim package is the rightful owner and the only user of the
newtork package. This change moves the package into dockershim to make
the distinction obvious, and untangles the codebase.
The`network/dns`is kept in the original package since it is only used by
kubelet.
The code was added to support rktnetes and non-CRI docker integrations.
These legacy integrations have already been removed from the codebase.
This change removes the compatibility code existing soley for the
legacy integrations.
Currently the Windows Server 2016 RTM has no CNI support.
With this commit, the Windows Server 2016 RTM will be able
to use the CNI plugin for networking setup.
This commit also moves some comments to the right place.
Signed-off-by: Alin Balutoiu <abalutoiu@cloudbasesolutions.com>
Automatic merge from submit-queue (batch tested with PRs 60363, 59208, 59465, 60581, 60702). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
log an error message when imageToRuntimeAPIImage failed
**What this PR does / why we need it**:
fix todo: log an error message when imageToRuntimeAPIImage failed
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 59740, 59728, 60080, 60086, 58714). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
kubelet: make --cni-bin-dir accept a comma-separated list of CNI plugin directories
Allow CNI-related network plugin drivers (kubenet, cni) to search a list of
directories for plugin binaries instead of just one. This allows using an
administrator-provided path and fallbacks to others (like the previous default
of /opt/cni/bin) for backwards compatibility.
```release-note
kubelet's --cni-bin-dir option now accepts multiple comma-separated CNI binary directory paths, which are search for CNI plugins in the given order.
```
@kubernetes/rh-networking @kubernetes/sig-network-misc @freehan @pecameron @rajatchopra
Allow CNI-related network plugin drivers (kubenet, cni) to search a list of
directories for plugin binaries instead of just one. This allows using an
administrator-provided path and fallbacks to others (like the previous default
of /opt/cni/bin) for backwards compatibility.
It's only used for the test code and after talking with Rajat, the
vendor stuff was never really used anyway. So convert the vendor
code into a plain array of plugin binary search paths, which is all
the vendor code was doing anyway.
Automatic merge from submit-queue (batch tested with PRs 58171, 58036, 60540). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
dockershim: Return Labels as Info in ImageStatus.
c6ddc749e8 added an Info field to
ImageStatusResponse when Verbose is true. This makes the image's
Labels available in that field, rather than unconditionally returning
an empty map.
**What this PR does / why we need it**:
This PR exposes an image's `Labels` through the CRI. In particular, I want this so I can write an `ImageService` wrapper that delegates all operations to a real `ImageService` but also, when the right `Labels`, ensures any needed [nix store](https://nixos.org/nix/) paths are present on the system when an image is pulled, enabling users to use nix for package distribution while still using containers for isolation and kubernetes for orchestration. In general, though, this should be useful for anything that wants to know about an image's `Labels`
**Special notes for your reviewer**:
I'd prefer to put this change into the `Image` protobuf type instead of putting it into `Info` (gated by `Verbose` or not, available in other requests like `ListImages` or not), but that would be a change to the protocol and it seems `Info` was introduced exactly for this purpose. If it's acceptable to put this into `Image`, I'll rework this.
If this change is acceptable, I will also do the work for `cri-o`, `rktlet`, `frakti`, and `cri-containerd` where applicable.
I have started the process for my employer to sign on to the CLA. I don't have reason to expect it to take long, but because there is more work to do if this change is desired I'd prefer if we can start review before that is completed.
**Release note**:
```release-note
dockershim now makes an Image's Labels available in the Info field of ImageStatusResponse
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
kubelet: setup WindowsContainerResources for windows containers
**What this PR does / why we need it**:
This PR setups WindowsContainerResources for windows containers. It implements proposal here: https://github.com/kubernetes/community/pull/1510.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#56734
**Special notes for your reviewer**:
**Release note**:
```release-note
WindowsContainerResources is set now for windows containers
```
Automatic merge from submit-queue (batch tested with PRs 50724, 59025, 59710, 59404, 59958). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add detailed err in ensure docker process error
Signed-off-by: Peng Gao <peng.gao.dut@gmail.com>
**What this PR does / why we need it**:
Add detailed error.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 60435, 60334, 60458, 59301, 60125). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
dockershim: don't check pod IP in StopPodSandbox
We're about to tear the container down, there's no point. It also suppresses
an annoying error message due to kubelet stupidity that causes multiple
parallel calls to StopPodSandbox for the same sandbox.
docker_sandbox.go:355] failed to read pod IP from plugin/docker: NetworkPlugin cni failed on the status hook for pod "docker-registry-1-deploy_default": Unexpected command output nsenter: cannot open /proc/22646/ns/net: No such file or directory
1) A first StopPodSandbox() request triggered by SyncLoop(PLEG) for
a ContainerDied event calls into TearDownPod() and thus the network
plugin. Until this completes, networkReady=true for the
sandbox.
2) A second StopPodSandbox() request triggered by SyncLoop(REMOVE)
calls PodSandboxStatus() and calls into the network plugin to read
the IP address because networkReady=true
3) The first request exits the network plugin, sets networReady=false,
and calls StopContainer() on the sandbox. This destroys the network
namespace.
4) The second request finally gets around to running nsenter but
the network namespace is already destroyed. It returns an error
which is logged by getIP().
```release-note
NONE
```
@yujuhong @freehan
c6ddc749e8 added an Info field to
ImageStatusResponse when Verbose is true. This makes the image's
Labels available in that field, rather than unconditionally returning
an empty map.
According to docker docs, setting MemorySwap equals to Memory can
prevent docker containers from using any swap, instead of setting
MemorySwap to zero.
We're about to tear the container down, there's no point. It also suppresses
an annoying error message due to kubelet stupidity that causes multiple
parallel calls to StopPodSandbox for the same sandbox.
docker_sandbox.go:355] failed to read pod IP from plugin/docker: NetworkPlugin cni failed on the status hook for pod "docker-registry-1-deploy_default": Unexpected command output nsenter: cannot open /proc/22646/ns/net: No such file or directory
1) A first StopPodSandbox() request triggered by SyncLoop(PLEG) for
a ContainerDied event calls into TearDownPod() and thus the network
plugin. Until this completes, networkReady=true for the
sandbox.
2) A second StopPodSandbox() request triggered by SyncLoop(REMOVE)
calls PodSandboxStatus() and calls into the network plugin to read
the IP address because networkReady=true
3) The first request exits the network plugin, sets networReady=false,
and calls StopContainer() on the sandbox. This destroys the network
namespace.
4) The second request finally gets around to running nsenter but
the network namespace is already destroyed. It returns an error
which is logged by getIP().
Automatic merge from submit-queue (batch tested with PRs 57824, 58806, 59410, 59280). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
2nd try at using a vanity GCR name
The 2nd commit here is the changes relative to the reverted PR. Please focus review attention on that.
This is the 2nd attempt. The previous try (#57573) was reverted while we
figured out the regional mirrors (oops).
New plan: k8s.gcr.io is a read-only facade that auto-detects your source
region (us, eu, or asia for now) and pulls from the closest. To publish
an image, push k8s-staging.gcr.io and it will be synced to the regionals
automatically (similar to today). For now the staging is an alias to
gcr.io/google_containers (the legacy URL).
When we move off of google-owned projects (working on it), then we just
do a one-time sync, and change the google-internal config, and nobody
outside should notice.
We can, in parallel, change the auto-sync into a manual sync - send a PR
to "promote" something from staging, and a bot activates it. Nice and
visible, easy to keep track of.
xref https://github.com/kubernetes/release/issues/281
TL;DR:
* The new `staging-k8s.gcr.io` is where we push images. It is literally an alias to `gcr.io/google_containers` (the existing repo) and is hosted in the US.
* The contents of `staging-k8s.gcr.io` are automatically synced to `{asia,eu,us)-k8s.gcr.io`.
* The new `k8s.gcr.io` will be a read-only alias to whichever regional repo is closest to you.
* In the future, images will be promoted from `staging` to regional "prod" more explicitly and auditably.
```release-note
Use "k8s.gcr.io" for pulling container images rather than "gcr.io/google_containers". Images are already synced, so this should not impact anyone materially.
Documentation and tools should all convert to the new name. Users should take note of this in case they see this new name in the system.
```
This is the 2nd attempt. The previous was reverted while we figured out
the regional mirrors (oops).
New plan: k8s.gcr.io is a read-only facade that auto-detects your source
region (us, eu, or asia for now) and pulls from the closest. To publish
an image, push k8s-staging.gcr.io and it will be synced to the regionals
automatically (similar to today). For now the staging is an alias to
gcr.io/google_containers (the legacy URL).
When we move off of google-owned projects (working on it), then we just
do a one-time sync, and change the google-internal config, and nobody
outside should notice.
We can, in parallel, change the auto-sync into a manual sync - send a PR
to "promote" something from staging, and a bot activates it. Nice and
visible, easy to keep track of.
Automatic merge from submit-queue (batch tested with PRs 59276, 51042, 58973, 59377, 59472). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
clean up unused function GetKubeletDockerContainers
**What this PR does / why we need it**:
fix todo: function GetKubeletDockerContainers is not unused,it has been migrated off in test/e2e_node/garbage_collector_test.go in [#57976](https://github.com/kubernetes/kubernetes/pull/57976/files)
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
This also incorporates the version string into the package name so
that incompatibile versions will fail to connect.
Arbitrary choices:
- The proto3 package name is runtime.v1alpha2. The proto compiler
normally translates this to a go package of "runtime_v1alpha2", but
I renamed it to "v1alpha2" for consistency with existing packages.
- kubelet/apis/cri is used as "internalapi". I left it alone and put the
public "runtimeapi" in kubelet/apis/cri/runtime.
Automatic merge from submit-queue (batch tested with PRs 59106, 58985, 59068, 59120, 59126). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix cross-build breakage after #58174
**What this PR does / why we need it**:
Fix cross-build breakage after #58174
@cblecker
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#59121
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add support of hyperv isolation for windows containers
**What this PR does / why we need it**:
Add support of hyperv isolation for windows containers.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#58750
**Special notes for your reviewer**:
Only one container per pod is supported yet.
**Release note**:
```release-note
Windows containers now support experimental Hyper-V isolation by setting annotation `experimental.windows.kubernetes.io/isolation-type=hyperv` and feature gates HyperVContainer. Only one container per pod is supported yet.
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fixes for HostIPC tests to work when Docker has SELinux support enabled.
**What this PR does / why we need it**:
Fixes for HostIPC tests to work when Docker has SELinux support enabled.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
N/A
**Special notes for your reviewer**:
The core of the matter is to use `ipcs` from util-linux rather than the one from busybox. The typical SELinux policy has enough to allow Docker containers (running under svirt_lxc_net_t SELinux type) to access IPC information by reading the contents of the files under /proc/sysvipc/, but not by using the shmctl etc. syscalls.
The `ipcs` implementation in busybox will use `shmctl(0, SHM_INFO, ...)` to detect whether it can read IPC info (see source code [here](https://git.busybox.net/busybox/tree/util-linux/ipcs.c?h=1_28_0#n138)), while the one in util-linux will prefer to read from the /proc files directly if they are available (see source code [here](https://github.com/karelzak/util-linux/blob/v2.27.1/sys-utils/ipcutils.c#L108)).
It turns out the SELinux policy doesn't allow the shmctl syscalls in an unprivileged container, while access to it through the /proc interface is fine. (One could argue this is a bug in the SELinux policy, but getting it fixed on stable OSs is hard, and it's not that hard for us to test it with an util-linux `ipcs`, so I propose we do so.)
This PR also contains a refactor of the code setting IpcMode, since setting it in the "common options" function is misleading, as on containers other than the sandbox, it ends up always getting overwritten, so let's only set it to "host" in the Sandbox.
It also has a minor fix for the `ipcmk` call, since support for size suffix was only introduced in recent versions of it.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 58899, 58980). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
CRI: Add a call to reopen log file for a container
This allows a daemon external to the container runtime to rotate the log
file, and then ask the runtime to reopen the files.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#58823
**Release note**:
```release-note
CRI: Add a call to reopen log file for a container.
```
Having the field set in modifyCommonNamespaceOptions is misleading,
since for the actual container it is later unconditionally overwritten
to point to the sandbox container.
So let's move its setting to modifyHostOptionsForSandbox (renamed from
modifyHostNetworkOptionForSandbox as it's not about network only), since
that reflects what actually happens in practice.
This commit is purely a refactor, it doesn't change any behavior.
We let dockershim implement the kubelet's internal (CRI) API as an
intermediary step before transitioning fully to communicate using gRPC.
Now that kubelet has been communicating to the runtime over gRPC for
multiple releases, we can safely retire the extra interface in
dockershim.
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
dockershim: bump the minimum supported docker version to 1.11
Drop the 1.10 compatibilty code.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 50457, 55558, 53483, 55731, 52842). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
improve the logic setting cgroupparent in RunPodSandbox
Signed-off-by: yanxuean <yan.xuean@zte.com.cn>
**What this PR does / why we need it**:
The setting of cgroupparent is too confused!
The old logic is:
1. set CgroupParent correctly
2. reset CgroupParent incorrectly
3. set CgroupParent again (refer to #42055 )
The login is too confused, and It is sure that there are many people who drop in trap.
We only need to set it in one place.
kubernetes/pkg/kubelet/dockershim/docker_sandbox.go
```
func (ds *dockerService) makeSandboxDockerConfig(c *runtimeapi.PodSandboxConfig, image string) (*dockertypes.ContainerCreateConfig, error) {
....
// Apply linux-specific options.
if lc := c.GetLinux(); lc != nil {
if err := ds.applySandboxLinuxOptions(hc, lc, createConfig, image, securityOptSep); err != nil {
return nil, err
}
}
// Apply resource options.
setSandboxResources(hc) **<-- reset the CgroupParent incorrectly**
// Apply cgroupsParent derived from the sandbox config.
if lc := c.GetLinux(); lc != nil {
// Apply Cgroup options.
cgroupParent, err := ds.GenerateExpectedCgroupParent(lc.CgroupParent)
if err != nil {
return nil, fmt.Errorf("failed to generate cgroup parent in expected syntax for container %q: %v", c.Metadata.Name, err)
}
hc.CgroupParent = cgroupParent
}
```
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
[Part 1] Remove docker dep in kubelet startup
**What this PR does / why we need it**:
Remove dependency of docker during kubelet start up.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*:
Part 1 of #54090
**Special notes for your reviewer**:
Changes include:
1. Move docker client initialization into dockershim pkg.
2. Pass a docker `ClientConfig` from kubelet to dockershim
3. Pass parameters needed by `FakeDockerClient` thru `ClientConfig` to dockershim
(TODO, the second part) Make dockershim tolerate when dockerd is down, otherwise it will still fail kubelet
Please note after this PR, kubelet will still fail if dockerd is down, this will be fixed in the subsequent PR by making dockershim tolerate dockerd failure (initializing docker client in a separate goroutine), and refactoring cgroup and log driver detection.
**Release note**:
```release-note
Remove docker dependency during kubelet start up
```
Automatic merge from submit-queue (batch tested with PRs 53747, 54528, 55279, 55251, 55311). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
a typo in dockershim.cm.containerManager.doWork
Signed-off-by: yanxuean <yan.xuean@zte.com.cn>
**What this PR does / why we need it**:
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
/sig node
Automatic merge from submit-queue (batch tested with PRs 53679, 51063). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fixes to enable Windows CNI
**What this PR does / why we need it**:
This PR has fixed which enables Kubelet to use Windows CNI plugin.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
#49646
**Special notes for your reviewer**:
**Release note**:
```release-note
```
Automatic merge from submit-queue (batch tested with PRs 55050, 53464, 54936, 55028, 54928). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
kubelet: dockershim: remove orphaned checkpoint files
Fixes https://github.com/kubernetes/kubernetes/issues/55070
Currently, `ListPodSandbox()` returns a combined list of sandboxes populated from both the runtime and the dockershim checkpoint files. However the sandboxes in the checkpoint files might not exist anymore.
The kubelet sees the sandbox returned by `ListPodSandbox()` and determines it shouldn't be running and calls `StopPodSandbox()` on it. This generates an error when `StopContainer()` is called as the container does not exist. However the checkpoint file is not cleaned up. This leads to subsequent calls to `StopPodSandbox()` that fail in the same way each time.
This PR removes the checkpoint file if StopContainer fails due to container not found.
The only other place `RemoveCheckpoint()` is called, except if it is corrupt, is from `RemoveSandbox()`. If the container does not exist, what `RemoveSandbox()` would have done has been effectively been done already. So this is just clean up.
@derekwaynecarr @eparis @freehan @dcbw
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
StopPodSandbox should not log when container is already removed
**What this PR does / why we need it**:
StopPodSandbox should not log when a container is already gone. It should only log if it could not stop and the container was still present.
Fixes https://github.com/kubernetes/kubernetes/issues/55021
**Special notes for your reviewer**:
This was seen in our production logs, need to eliminate spam.
**Release note**:
```release-note
NONE
```
Following are part of this commit
+++++++++++++++++++++++++++++++++
* Windows CNI Support
(1) Support to use --network-plugin=cni
(2) Handled platform requirement of calling CNI ADD for all the containers.
(2.1) For POD Infra container, netNs has to be empty
(2.2) For all other containers, sharing the network namespace of POD container,
should pass netNS name as "container:<Pod Infra Container Id>", same as the
NetworkMode of the current container
(2.3) The Windows CNI plugin has to handle this to call into Platform.
Sample Windows CNI Plugin code to be shared soon.
* Sandbox support for Windows
(1) Sandbox support for Windows. Works only with Docker runtime.
(2) Retained CONTAINER_NETWORK as a backward compatibilty flag,
to not break existing deployments using it.
(3) Works only with CNI plugin enabled.
(*) Changes to reinvoke CNI ADD for every new container created. This is hooked up with PodStatus,
but would be ideal to move it outside of this, once we have CNI GET support
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add a file store utility package in kubelet
More and more components checkpoints (i.e., persist their states) in
kubelet. Refurbish and move the implementation in dockershim to a
utility package to improve code reusability.
Automatic merge from submit-queue (batch tested with PRs 54040, 52503). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Get fallback termination msg from docker when using journald log driver
**What this PR does / why we need it**:
When using the legacy docker container runtime and when a container has `terminationMessagePolicy=FallbackToLogsOnError` and when docker is configured with a log driver other than `json-log` (such as `journald`), the kubelet should not try to get the container's log from the json log file (since it's not there) but should instead ask docker for the logs.
**Which issue this PR fixes** fixes#52502
**Special notes for your reviewer**:
**Release note**:
```release-note
Fixed log fallback termination messages when using docker with journald log driver
```
Automatic merge from submit-queue (batch tested with PRs 53297, 53328). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
format some code in dockershim
**What this PR does / why we need it**:
format some code in dockershim
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
None
```
Automatic merge from submit-queue (batch tested with PRs 50555, 51152). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Implement CRI stats in Docker Shim
**What this PR does / why we need it**:
This PR implements CRI Stats in the Docker Shim. It is needed to enable CRI stats for Docker and ongoing /stats/summary API changes in moving to use CRI.
Related issues:
#46984 (CRI: instruct kubelet to (optionally) consume container stats from CRI)
#45614 (CRI: add methods for container stats)
This PR is also a followup to my original PR (https://github.com/kubernetes/kubernetes/pull/50396) to implement Windows Container Stats. The plan is that Windows Stats will use a hybrid model: pod and container level stats will come from CRI (via dockershim) and that node level stats will come from a "winstats" package that exports cadvisor like datastructures using windows specific perf counters from the node. I will update that PR to only export node level stats.
@yujuhong @yguo0905 @dchen1107 @jdumars @anhowe @michmike
**Special notes for your reviewer**:
**Release note**:
```release-note
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
enable to specific unconfined AppArmor profile
**What this PR does / why we need it**:
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#52370
**Special notes for your reviewer**:
/assign @tallclair @liggitt
**Release note**:
```release-note
enable to specific unconfined AppArmor profile
```
Implement CRI stats for dockershim using docker stats. This enables use
of the summary api to get container metrics on Windows where CRI stats
are enabled.
on projectatomic-based docker, we get "docker.io/library/busybox:latest"
when someone uses an unqualified name like "busybox". Though when we
inspect, the RepoTag will still say "docker.io/busybox:latest", So
we have reparse the tag, normalize it and try again. Please see the
additional test case.
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
Automatic merge from submit-queue (batch tested with PRs 50988, 50509, 52660, 52663, 52250). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Create container log symlink for all containers
**What this PR does / why we need it**:
dockershim only makes log symlink for running containers now, we should also create the log symlink for failed containers.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#50499
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
When using the legacy docker container runtime and when a container has
terminationMessagePolicy=FallbackToLogsOnError and when docker is
configured with a log driver other than json-log (such as journald),
the kubelet should not try to get the container's log from the
json log file (since it's not there) but should instead ask docker for
the logs.
Automatic merge from submit-queue (batch tested with PRs 51067, 52319, 52803, 52961, 51972). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..
Move prometheus metrics for docker operations into dockershim
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..
kubelet: remove the --docker-exec-handler flag
Stop supporting the "nsenter" exec handler. Only the Docker native exec
handler is supported.
The flag was deprecated in Kubernetes 1.6 and is safe to remove
in Kubernetes 1.9 according to the deprecation policy.
**What this PR does / why we need it**:
**Which issue this PR fixes** : fixes#40229
**Special notes for your reviewer**:
N/A
**Release note**:
```release-note
Remove the --docker-exec-handler flag. Only native exec handler is supported.
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..
dockershim: fine-tune network-ready handling on sandbox teardown and removal
If sandbox teardown results in an error, GC will periodically attempt
to again remove the sandbox. Until the sandbox is removed, pod
sandbox status calls will attempt to enter the pod's namespace and
retrieve the pod IP, but the first teardown attempt may have already
removed the network namespace, resulting in a pointless log error
message that the network namespace doesn't exist, or that nsenter
can't find eth0.
The network-ready mechanism originally attempted to suppress those
messages by ensuring that pod sandbox status skipped network checks
when networking was already torn down, but unfortunately the ready
value was cleared too early.
Also, don't tear down the pod network multiple times if the first
time we tore it down, it succeeded.
**What this PR does / why we need it**:
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
```
Automatic merge from submit-queue (batch tested with PRs 52240, 48145, 52220, 51698, 51777). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..
dockershim: remove support for legacy containers
The code was first introduced in 1.6 to help pre-CRI-kubelet upgrade to
using the CRI implementation. They can safely be removed now.
Stop supporting the "nsenter" exec handler. Only the Docker native exec
handler is supported.
The flag was deprecated in Kubernetes 1.6 and is safe to remove
in Kubernetes 1.9 according to the deprecation policy.
If sandbox teardown results in an error, GC will periodically attempt
to again remove the sandbox. Until the sandbox is removed, pod
sandbox status calls will attempt to enter the pod's namespace and
retrieve the pod IP, but the first teardown attempt may have already
removed the network namespace, resulting in a pointless log error
message that the network namespace doesn't exist, or that nsenter
can't find eth0.
The network-ready mechanism originally attempted to suppress those
messages by ensuring that pod sandbox status skipped network checks
when networking was already torn down, but unfortunately the ready
value was cleared too early.
Also, don't tear down the pod network multiple times if the first
time we tore it down, it succeeded.
Automatic merge from submit-queue (batch tested with PRs 52339, 52343, 52125, 52360, 52301)
dockershim: check if f.Sync() returns an error and surface it
```release-note
dockershim: check the error when syncing the checkpoint.
```
Automatic merge from submit-queue
Fix initial exec terminal dimensions
**What this PR does / why we need it**:
Delay attempting to send a terminal resize request to docker until after
the exec has started; otherwise, the initial resize request will fail.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#47990
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 50563, 50698, 50796)
Disable Docker's health check until we officially support it
Ref: https://github.com/kubernetes/kubernetes/issues/50703
Tested locally.
Without this PR:
```
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
afdd796ddddc gcr.io/ygg-gke-dev/nginx-healthcheck "nginx -g 'daemon ..." 5 minutes ago Up 5 minutes (healthy) k8s_test-container_test-pod_default_8a1ad225-82bf-11e7-becb-480fcf514648_0
```
With this PR:
```
e3fb2437555f gcr.io/ygg-gke-dev/nginx-healthcheck "nginx -g 'daemon ..." 10 seconds ago Up 9 seconds k8s_test-container_test-pod_default_de82e83c-82c0-11e7-b3fc-480fcf514648_0
```
**Release note**:
```
Disable Docker's health check until we officially support it.
```
/assign @yujuhong
/assign @dchen1107
Automatic merge from submit-queue (batch tested with PRs 50694, 50702)
Fix make cross build failure
**What this PR does / why we need it**:
had to fix the method getSecurityOpts in helpers_windows.go to
match the implementation in helpers_linux.go from commit:
bf01fa2f00
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
Fixes#50675
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 49488, 50407, 46105, 50456, 50258)
Add UpdateContainerResources method to CRI
This is first step toward support for opinionated cpu pinning for certain guaranteed pods.
In order to do this, the kubelet needs to be able to dynamically update the cpuset at the container level, which is managed by the container runtime. Thus the kubelet needs a method to communicate over the CRI so the runtime can then modify the container cgroup.
This is used in the situation where a core is added or removed from the shared pool to become a exclusive core for a new G pod. The cpuset for all containers in the shared pool will need to be updated to add or remove that core.
Opening this up now so we can start discussion. The need for a change to the CRI might be unexpected.
@derekwaynecarr @vishh @ConnorDoyle
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 49651, 49707, 49662, 47019, 49747)
Add support for `no_new_privs` via AllowPrivilegeEscalation
**What this PR does / why we need it**:
Implements kubernetes/community#639
Fixes#38417
Adds `AllowPrivilegeEscalation` and `DefaultAllowPrivilegeEscalation` to `PodSecurityPolicy`.
Adds `AllowPrivilegeEscalation` to container `SecurityContext`.
Adds the proposed behavior to `kuberuntime`, `dockershim`, and `rkt`. Adds a bunch of unit tests to ensure the desired default behavior and that when `DefaultAllowPrivilegeEscalation` is explicitly set.
Tests pass locally with docker and rkt runtimes. There are also a few integration tests with a `setuid` binary for sanity.
**Release note**:
```release-note
Adds AllowPrivilegeEscalation to control whether a process can gain more privileges than it's parent process
```
Automatic merge from submit-queue (batch tested with PRs 47738, 49196, 48907, 48533, 48822)
Fix TODO: rename podInfraContainerID to sandboxID
**What this PR does / why we need it**:
Code-cleanup in kubelet to use consistent naming for sandbox ID. Not super urgent, but thought it would be nice to knock off some TODOs.
**Which issue this PR fixes**
Fixes a TODO in the code, no associated issue.
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 49107, 47177, 49234, 49224, 49227)
Make sure the previous symlink file is deleted before trying to create a new one
**What this PR does / why we need it**:
It deletes possibly existing symlinks to container log files.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
fixes#49105
**Special notes for your reviewer**:
**Release note**:
```release-note
```
Automatic merge from submit-queue (batch tested with PRs 48405, 48742, 48748, 48571, 48482)
dockershim: clean up unused security context code
Most of the code in the `dockershim/securitycontext` package are
unused and can be removed. This PR migrates the rest of the code,
cleans it up (e.g., remove references to kubernetes API objects),
and removes the package entirely.
Automatic merge from submit-queue (batch tested with PRs 47850, 47835, 46197, 47250, 48284)
dockershim: checkpoint HostNetwork property
To ensure kubelet doesn't attempt network teardown on HostNetwork
containers that no longer exist but are still checkpointed, make
sure we preserve the HostNetwork property in checkpoints. If
the checkpoint indicates the container was a HostNetwork one,
don't tear down the network since that would fail anyway.
Related: https://github.com/kubernetes/kubernetes/issues/44307#issuecomment-299548609
@freehan @kubernetes/sig-network-misc
Automatic merge from submit-queue
Add type conversion judgment
If do not type conversion judgment, there may be panic.
**Release note**:
```release-note
NONE
```
To ensure kubelet doesn't attempt network teardown on HostNetwork
containers that no longer exist but are still checkpointed, make
sure we preserve the HostNetwork property in checkpoints. If
the checkpoint indicates the container was a HostNetwork one,
don't tear down the network since that would fail anyway.
Related: https://github.com/kubernetes/kubernetes/issues/44307#issuecomment-299548609
Automatic merge from submit-queue (batch tested with PRs 46441, 43987, 46921, 46823, 47276)
kubelet/network: report but tolerate errors returned from GetNetNS() v2
Runtimes should never return "" and nil errors, since network plugin
drivers need to treat netns differently in different cases. So return
errors when we can't get the netns, and fix up the plugins to do the
right thing.
Namely, we don't need a NetNS on pod network teardown. We do need
a netns for pod Status checks and for network setup.
V2: don't return errors from getIP(), since they will block pod status :( Just log them. But even so, this still fixes the original problem by ensuring we don't log errors when the network isn't ready.
@freehan @yujuhong
Fixes: https://github.com/kubernetes/kubernetes/issues/42735
Fixes: https://github.com/kubernetes/kubernetes/issues/44307
GenericPLEG's 1s relist() loop races against pod network setup. It
may be called after the infra container has started but before
network setup is done, since PLEG and the runtime's SyncPod() run
in different goroutines.
Track network setup status and don't bother trying to read the pod's
IP address if networking is not yet ready.
See also: https://bugzilla.redhat.com/show_bug.cgi?id=1434950
Mar 22 12:18:17 ip-172-31-43-89 atomic-openshift-node: E0322
12:18:17.651013 25624 docker_manager.go:378] NetworkPlugin
cni failed on the status hook for pod 'pausepods22' - Unexpected
command output Device "eth0" does not exist.
Runtimes should never return "" and nil errors, since network plugin
drivers need to treat netns differently in different cases. So return
errors when we can't get the netns, and fix up the plugins to do the
right thing.
Namely, we don't need a NetNS on pod network teardown. We do need
a netns for pod Status checks and for network setup.
This reverts commit fee4c9a7d9.
This is not the correct fix for the problem; and it causes other problems
like continuous:
docker_sandbox.go:234] NetworkPlugin cni failed on the status hook for pod
"someotherdc-1-deploy_default": Unexpected command output nsenter: cannot
open : No such file or directory with error: exit status 1
Because GetNetNS() is returning an empty network namespace. That is
not helpful nor should really be allowed; that's what the error return
from GetNetNS() is for.
Automatic merge from submit-queue (batch tested with PRs 46076, 43879, 44897, 46556, 46654)
kubelet/network: report but tolerate errors returned from GetNetNS()
Runtimes should never return "" and nil errors, since network plugin
drivers need to treat netns differently in different cases. So return
errors when we can't get the netns, and fix up the plugins to do the
right thing.
Namely, we don't need a NetNS on pod network teardown. We do need
a netns for pod Status checks and for network setup.
@kubernetes/rh-networking @kubernetes/sig-network-bugs @DirectXMan12
Automatic merge from submit-queue
use make slice to store objects to improve efficiency
Signed-off-by: allencloud <allen.sun@daocloud.io>
**What this PR does / why we need it**:
we we know the slice length in advance, I think we had better use make to create the specified length of slice. This will improve some kind of performance. Since if we create a slice with []type{}, we did not know how much space runtime should reserve, since slice implementation should be continuous in memory. While when we make a slice with specified length, runtime would reserve a continuous memory space which will not result in slice movement in case of current space is not enough.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
NONE
**Special notes for your reviewer**:
NONE
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 46252, 45524, 46236, 46277, 46522)
Support sandbox images from private registries
**What this PR does / why we need it**:
The --pod-infra-container-image parameter allows the user to specify
an arbitrary image to be used as the pod infra container (AKA
sandbox), an internal piece of the dockershim implementation of the
Container Runtime Interface.
The dockershim does not have access to any of the pod-level image pull
credentials configuration, so if the user specifies an image from a
private registry, the image pull will fail.
This change allows the dockershim to read local docker configuration
(e.g. /root/.docker/config.json) and use it when pulling the pod infra
container image.
**Which issue this PR fixes**: fixes#45738
**Special notes for your reviewer**:
The changes to fake_client for writing local config files deserve some
attention.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 45809, 46515, 46484, 46516, 45614)
CRI: add methods for container stats
**What this PR does / why we need it**:
Define methods in CRI to get container stats.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*:
Part of https://github.com/kubernetes/features/issues/290; addresses #27097
**Special notes for your reviewer**:
This PR defines the *minimum required* container metrics for the existing components to function, loosely based on the previous discussion on [core metrics](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/core-metrics-pipeline.md) as well as the existing cadvisor/summary APIs.
Two new RPC calls are added to the RuntimeService: `ContainerStats` and `ListContainerStats`. The former retrieves stats for a given container, while the latter gets stats for all containers in one call.
The stats gathering time of each subsystem can vary substantially (e.g., cpu vs. disk), so even though the on-demand model preferred due to its simplicity, we’d rather give the container runtime more flexibility to determine the collection frequency for each subsystem*. As a trade-off, each piece of stats for the subsystem must contain a timestamp to let kubelet know how fresh/recent the stats are. In the future, we should also recommend a guideline for how recent the stats should be in order to ensure the reliability (e.g., eviction) and the responsiveness (e.g., autoscaling) of the kubernetes cluster.
The next step is to plumb this through kubelet so that kubelet can choose consume container stats from CRI or cadvisor.
**Alternatively, we can add calls to get stats of individual subsystems. However, kubelet does not have the complete knowledge of the runtime environment, so this would only lead to unnecessary complexity in kubelet.*
**Release note**:
```release-note
Augment CRI to support retrieving container stats from the runtime.
```
Automatic merge from submit-queue (batch tested with PRs 42256, 46479, 45436, 46440, 46417)
Log out digest when digest is invalid
Notice this in frakti: missing image ref when logging it out.
**Release note**:
```release-note
NONE
```
**What this PR does / why we need it**:
The --pod-infra-container-image parameter allows the user to specify
an arbitrary image to be used as the pod infra container (AKA
sandbox), an internal piece of the dockershim implementation of the
Container Runtime Interface.
The dockershim does not have access to any of the pod-level image pull
credentials configuration, so if the user specifies an image from a
private registry, the image pull will fail.
This change allows the dockershim to read local docker configuration
(e.g. /root/.docker/config.json) and use it when pulling the pod infra
container image.
**Which issue this PR fixes**: fixes#45738
**Special notes for your reviewer**:
The changes to fake_client for writing local config files deserve some
attention.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 46124, 46434, 46089, 45589, 46045)
Support TCP type runtime endpoint for kubelet
**What this PR does / why we need it**:
Currently the grpc server for kubelet and dockershim has a hardcoded endpoint: unix socket '/var/run/dockershim.sock', which is not applicable on non-unix OS.
This PR is to support TCP endpoint type besides unix socket.
**Which issue this PR fixes**
This is a first attempt to address issue https://github.com/kubernetes/kubernetes/issues/45927
**Special notes for your reviewer**:
Before this change, running on Windows node results in:
```
Container Manager is unsupported in this build
```
After adding the cm stub, error becomes:
```
listen unix /var/run/dockershim.sock: socket: An address incompatible with the requested protocol was used.
```
This PR is to fix those two issues.
After this change, still meets 'seccomp' related issue when running on Windows node, needs more updates later.
**Release note**:
GenericPLEG's 1s relist() loop races against pod network setup. It
may be called after the infra container has started but before
network setup is done, since PLEG and the runtime's SyncPod() run
in different goroutines.
Track network setup status and don't bother trying to read the pod's
IP address if networking is not yet ready.
See also: https://bugzilla.redhat.com/show_bug.cgi?id=1434950
Mar 22 12:18:17 ip-172-31-43-89 atomic-openshift-node: E0322
12:18:17.651013 25624 docker_manager.go:378] NetworkPlugin
cni failed on the status hook for pod 'pausepods22' - Unexpected
command output Device "eth0" does not exist.
Runtimes should never return "" and nil errors, since network plugin
drivers need to treat netns differently in different cases. So return
errors when we can't get the netns, and fix up the plugins to do the
right thing.
Namely, we don't need a NetNS on pod network teardown. We do need
a netns for pod Status checks and for network setup.
This reverts commit fee4c9a7d9.
This is not the correct fix for the problem; and it causes other problems
like continuous:
docker_sandbox.go:234] NetworkPlugin cni failed on the status hook for pod
"someotherdc-1-deploy_default": Unexpected command output nsenter: cannot
open : No such file or directory with error: exit status 1
Because GetNetNS() is returning an empty network namespace. That is
not helpful nor should really be allowed; that's what the error return
from GetNetNS() is for.
Automatic merge from submit-queue (batch tested with PRs 44326, 45768)
[CRI] Forcibly remove container
Forcibly remove the running containers in `RemoveContainer`. Since we should forcibly remove the running containers in `RemovePodSandbox`. See [here](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/api/v1alpha1/runtime/api.proto#L35).
cc @feiskyer @Random-Liu
Signed-off-by: Xianglin Gao <xlgao@zju.edu.cn>
Automatic merge from submit-queue
fix typo in docker_container.go
Signed-off-by: Carlory <fanbaofa@gmail.com>
**What this PR does / why we need it**:
fix typo in docker_container.go
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
NONE
**Special notes for your reviewer**:
NONE
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 45171, 43947, 45788, 45822, 45808)
[CRI] Continue remove image when can't find image id with ImageRef
Signed-off-by: Crazykev <crazykev@zju.edu.cn>
**What this PR does / why we need it**:
Should try to remove imageRef as repo:tag when can't find it as imageID.
/cc @feiskyer @Random-Liu PTAL
also /cc @xlgao-zju @heartlock
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
None
```
Automatic merge from submit-queue (batch tested with PRs 41331, 45591, 45600, 45176, 45658)
Move client/unversioned/remotecommand to client-go
Module remotecommand originally part of kubernetes/pkg/client/unversioned was moved
to client-go/tools, and will be used as authoritative in kubectl, e2e and other places.
Module remotecommand relies on util/exec module which was copied to client-go/pkg/util
Module remotecommand originally part of kubernetes/pkg/client/unversioned was moved
to client-go/tools, and will be used as authoritative in kubectl, e2e and other places.
Module remotecommand relies on util/exec module which will be copied to client-go/pkg/util
Automatic merge from submit-queue (batch tested with PRs 45453, 45307, 44987)
Migrate the docker client code from dockertools to dockershim
Move docker client code from dockertools to dockershim/libdocker. This includes
DockerInterface (renamed to Interface), FakeDockerClient, etc.
This is part of #43234
Automatic merge from submit-queue (batch tested with PRs 45508, 44258, 44126, 45441, 45320)
Use existing global var criSupportedLogDrivers
**What this PR does / why we need it**:
Use existing global var `criSupportedLogDrivers` defined in docker_service.go. If CRI supports other log drivers in the future, we will only need to modify that global var.
cc @Random-Liu
Automatic merge from submit-queue
Use Docker API Version instead of docker version
**What this PR does / why we need it**:
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
Fixes#42492
**Special notes for your reviewer**:
**Release note**:
`Update cadvisor to latest head to use docker APIversion exposed by cadvisor`
Automatic merge from submit-queue (batch tested with PRs 45314, 45250, 41733)
CRI: add ImageFsInfo API
**What this PR does / why we need it**:
kubelet currently relies on cadvisor to get the ImageFS info for supported runtimes, i.e., docker and rkt. This PR adds ImageFsInfo API to CRI so kubelet could get the ImageFS correctly for all runtimes.
**Which issue this PR fixes**
First step for #33048 ~~also reverts temporary ImageStats in #33870~~.
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
The test was originally in docker_manager_test.go (now removed). I
copied and adapated the logic for the new test.
Also move the original test fixtures needed for the test.
Automatic merge from submit-queue (batch tested with PRs 45013, 45166)
CRI: remove PodSandboxStatus.Linux.Namespaces.Network
**What this PR does / why we need it**:
PodSandboxStatus.Linux.Namespaces.Network is not used, so this PR removes it from CRI.
**Which issue this PR fixes**
Closes: #44972
**Special notes for your reviewer**:
**Release note**:
```release-note
Remove PodSandboxStatus.Linux.Namespaces.Network from CRI.
```
/assign @Random-Liu @yujuhong
Previously we exported many constants and functions in dockertools to
share with the dockershim package. This change moves such
constants/functions to dockershim and unexport them.
This change involves only mechnical changes and should not have any
functional impact.
Automatic merge from submit-queue (batch tested with PRs 44406, 41543, 44071, 44374, 44299)
Decouple remotecommand
Refactored unversioned/remotecommand to decouple it from undesirable dependencies:
- term package now is not required, and functionality required to resize terminal size can be plugged in directly in kubectl
- in order to remove dependency on kubelet package - constants from kubelet/server/remotecommand were moved to separate util package (pkg/util/remotecommand)
- remotecommand_test.go moved to pkg/client/tests module
Automatic merge from submit-queue (batch tested with PRs 42025, 44169, 43940)
[CRI] Remove all containers in the sandbox
Remove all containers in the sandbox, when we remove the sandbox.
/cc @feiskyer @Random-Liu
Signed-off-by: Xianglin Gao <xlgao@zju.edu.cn>
Automatic merge from submit-queue
[CRI] Use DNSOptions passed by CRI in dockershim.
When @xlgao-zju is working on the CRI validation test, he found that dockershim is not using the DNSOptions passed in CRI. https://github.com/kubernetes-incubator/cri-tools/pull/30#issuecomment-290644357
This PR fixed the issue. I've manually tried, for `ClusterFirst` DNSPolicy, the resolv.conf will be:
```
nameserver 8.8.8.8
search corp.google.com prod.google.com prodz.google.com google.com
options ndots:5
```
For `Default` DNSPolicy, the resolv.conf will be:
```
nameserver 127.0.1.1
search corp.google.com prod.google.com prodz.google.com google.com
```
@xlgao-zju You should be able to test after this PR is merged.
/cc @yujuhong @feiskyer
Automatic merge from submit-queue
kubelet: check and enforce minimum docker api version
**What this PR does / why we need it**:
This PR adds enforcing a minimum docker api version (same with what we have do for dockertools).
**Which issue this PR fixes**
Fixes#42696.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 43378, 43216, 43384, 43083, 43428)
Fix tiny typo
**What this PR does / why we need it**:
**Which issue this PR fixes**
Fix type typo introduced by PR #43368.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 42522, 42545, 42556, 42006, 42631)
Use pod sandbox id in checkpoint
**What this PR does / why we need it**: we should log out sandbox id when checkpoint error
**Release note**:
```NONE
```
Automatic merge from submit-queue (batch tested with PRs 43398, 43368)
CRI: add support for dns cluster first policy
**What this PR does / why we need it**:
PR #29378 introduces ClusterFirstWithHostNet policy but only dockertools was updated to support the feature.
This PR updates kuberuntime to support it for all runtimes.
**Which issue this PR fixes**
fixes#43352
**Special notes for your reviewer**:
Candidate for v1.6.
**Release note**:
```release-note
NONE
```
cc @thockin @luxas @vefimova @Random-Liu
PR #29378 introduces ClusterFirstWithHostNet, but docker doesn't support
setting dns options togather with hostnetwork. This commit rewrites
resolv.conf same as dockertools.
Automatic merge from submit-queue (batch tested with PRs 42747, 43030)
dockershim: remove corrupted sandbox checkpoints
This is a workaround to ensure that kubelet doesn't block forever when
the checkpoint is corrupted.
This is a workaround for #43021
Automatic merge from submit-queue
dockershim: Fix the race condition in ListPodSandbox
In ListPodSandbox(), we
1. List all sandbox docker containers
2. List all sandbox checkpoints. If the checkpoint does not have a
corresponding container in (1), we return partial result based on
the checkpoint.
The problem is that new PodSandboxes can be created between step (1) and
(2). In those cases, we will see the checkpoints, but not the sandbox
containers. This leads to strange behavior because the partial result
from the checkpoint does not include some critical information. For
example, the creation timestamp'd be zero, and that would cause kubelet's
garbage collector to immediately remove the sandbox.
This change fixes that by getting the list of checkpoints before listing
all the containers (since in RunPodSandbox we create them in the reverse
order).
In ListPodSandbox(), we
1. List all sandbox docker containers
2. List all sandbox checkpoints. If the checkpoint does not have a
corresponding container in (1), we return partial result based on
the checkpoint.
The problem is that new PodSandboxes can be created between step (1) and
(2). In those cases, we will see the checkpoints, but not the sandbox
containers. This leads to strange behavior because the partial result
from the checkpoint does not include some critical information. For
example, the creation timestamp'd be zero, and that would cause kubelet's
garbage collector to immediately remove the sandbox.
This change fixes that by getting the list of checkpoints before listing
all the containers (since in RunPodSandbox we create them in the reverse
order).
* Properly return ImageNotFoundError
* Support inject "Images" or "ImageInspects" and keep both in sync.
* Remove the FakeDockerPuller and let FakeDockerClient subsumes its
functinality. This reduces the overhead to maintain both objects.
* Various small fixes and refactoring of the testing utils.
Automatic merge from submit-queue (batch tested with PRs 41980, 42192, 42223, 41822, 42048)
CRI: Make dockershim better implements CRI.
When thinking about CRI Validation test, I found that `PodSandboxStatus.Linux.Namespaces.Options.HostPid` and `PodSandboxStatus.Linux.Namespaces.Options.HostIpc` are not populated. Although they are not used by kuberuntime now, we should populate them to conform to CRI.
/cc @yujuhong @feiskyer
