github/kubernetes
4
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'upstream/master' into test-cmd-what

Browse Source
This commit is contained in:
Thomas Runyon
2019-02-01 11:40:18 -05:00
parent aeb92ee754 556c0b8593
commit 31aac97a85
484 changed files with 11235 additions and 4171 deletions
Download Patch File Download Diff File Expand all files Collapse all files

147
CHANGELOG-1.14.md
View File

@@ -1,17 +1,158 @@
<!-- BEGIN MUNGE: GENERATED_TOC -->
- [v1.14.0-alpha.1](#v1140-alpha1)
- [Downloads for v1.14.0-alpha.1](#downloads-for-v1140-alpha1)
- [v1.14.0-alpha.2](#v1140-alpha2)
- [Downloads for v1.14.0-alpha.2](#downloads-for-v1140-alpha2)
- [Client Binaries](#client-binaries)
- [Server Binaries](#server-binaries)
- [Node Binaries](#node-binaries)
- [Changelog since v1.13.0](#changelog-since-v1130)
- [Changelog since v1.14.0-alpha.1](#changelog-since-v1140-alpha1)
- [Action Required](#action-required)
- [Other notable changes](#other-notable-changes)
- [v1.14.0-alpha.1](#v1140-alpha1)
- [Downloads for v1.14.0-alpha.1](#downloads-for-v1140-alpha1)
- [Client Binaries](#client-binaries-1)
- [Server Binaries](#server-binaries-1)
- [Node Binaries](#node-binaries-1)
- [Changelog since v1.13.0](#changelog-since-v1130)
- [Action Required](#action-required-1)
- [Other notable changes](#other-notable-changes-1)
<!-- END MUNGE: GENERATED_TOC -->
<!-- NEW RELEASE NOTES ENTRY -->
# v1.14.0-alpha.2
[Documentation](https://docs.k8s.io)
## Downloads for v1.14.0-alpha.2
filename | sha512 hash
-------- | -----------
[kubernetes.tar.gz](https://dl.k8s.io/v1.14.0-alpha.2/kubernetes.tar.gz) | `1330e4421b61f6b1e6e4dee276d4742754bd3dd4493508d67ebb4445065277c619c4da8b4835febf0b2cdcf9e75fce96de1c1d99998904bae2bb794a453693f2`
[kubernetes-src.tar.gz](https://dl.k8s.io/v1.14.0-alpha.2/kubernetes-src.tar.gz) | `352c043bebf13a616441c920f3eec80d3f02f111d8488c31aa903e1483bce6d1fbe7472208f64730142960c8f778ab921ef7b654540a3ec09e53bd7e644521bd`
### Client Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-client-darwin-386.tar.gz](https://dl.k8s.io/v1.14.0-alpha.2/kubernetes-client-darwin-386.tar.gz) | `ee5aba4efce323167e6d897a2ff6962a240e466333bcae9390be2c8521c6da50ac2cb6139510b693aad49d6393b97a2118ed1fe4f999dd08bdca6d875d25f804`
[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.14.0-alpha.2/kubernetes-client-darwin-amd64.tar.gz) | `4b5c0b340322956a8d096c595124a765ac318d0eb460d6320218f2470e22d88221a0a9f1f93d5f3075f1c36b18c7041ee2fcb32e0f9c94d9f79bc3fd3005e68e`
[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.14.0-alpha.2/kubernetes-client-linux-386.tar.gz) | `7a5bfe68dd58c8478746a410872b615daf8abb9a78754140fb4d014a0c9177a87859ac046f56f5743fb97a9881abc2cf48c3e51aa02c8a86a754bf2cc59edb54`
[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.14.0-alpha.2/kubernetes-client-linux-amd64.tar.gz) | `c3139f58070241f2da815f701af3c0bd0ea4fdec1fe54bb859bd11237ac9b75ecb01b62ac1c7a459a4dd79696412c6d2f8cbd492fd062a790ceadd3dcc9b07fd`
[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.14.0-alpha.2/kubernetes-client-linux-arm.tar.gz) | `9d96d2e1e11aa61e2c3a5f4f27c18866feae9833b6ee70b15f5cdb5f992849dc1f79821af856b467487092a21a447231fb9c4de6ee6f17defed3cfa16d35b4c6`
[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.14.0-alpha.2/kubernetes-client-linux-arm64.tar.gz) | `7b4dd825cf9f217c18b28976a3faa94f0bd4868e541e5be7d57cd770e2b163c6daddf12e5f9ad51d92abde794a444f2a20bf582a30f03c39e60186d356030a2d`
[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.14.0-alpha.2/kubernetes-client-linux-ppc64le.tar.gz) | `490638e250c24b6bad8b67358fd7890f7a2f6456ae8ffe537c28bb5b3ce7abc591e6fecbddd6744f0f6c0e24b9f44c31f7ca1f7ebfc3c0d17a96fe8cf27b8548`
[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.14.0-alpha.2/kubernetes-client-linux-s390x.tar.gz) | `9dd8c3361eda15dd1594066c55b79cb9a34578c225b2b48647cd5b34619cf23106b845ee25b80d979f8b69e8733148842177500dc48989177b6944677f071f1c`
[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.14.0-alpha.2/kubernetes-client-windows-386.tar.gz) | `d624b8aead053201765b713d337528be82a71328ee3dd569f556868ceeb4904e64584892a016d247608fc4521c00ead7aed5d973b1206caa2d00406532d5b8b4`
[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.14.0-alpha.2/kubernetes-client-windows-amd64.tar.gz) | `a1cf8c67984dd4eb4610fa05d27fe9e9e4123159f933e3986e9db835b9cf136962168f0003071001e01e2c1831804ba0a366f2495741aa60a41587a69c09cb62`
### Server Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.14.0-alpha.2/kubernetes-server-linux-amd64.tar.gz) | `b93982b56371994c540cd11e6bc21808279340617164992c10f30d8e6ae4d5e270e41c1edc0625d3458a18944ec7aa8c273acbbcd718d60b6cacbc24220c42ac`
[kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.14.0-alpha.2/kubernetes-server-linux-arm.tar.gz) | `bfd76c6b26e5927166d776f6110b97ee36c1d63ad39e2d18899f3e428ebb0f9615bb677ac8e9bcc1864c72a40efd71e1314fe6d137f9c6e54f720270929e3f46`
[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.14.0-alpha.2/kubernetes-server-linux-arm64.tar.gz) | `6721dec0df9466cd6c056160c73d598296cebb0af9259eb21b693abb8708901bc8bc30e11815e14d00d6eb12b8bb90b699e3119b922da855e2c411bdf229d6e5`
[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.14.0-alpha.2/kubernetes-server-linux-ppc64le.tar.gz) | `f8cd307db8141d989ae1218dd2b438bc9cee017d533b1451d2345f9689c451fdb080acd1b9b2f535ed04017e44b81a0585072e7d58a9d201a0ec28fd09df0a6f`
[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.14.0-alpha.2/kubernetes-server-linux-s390x.tar.gz) | `de7514bbd87a1b363e1bc7787f37d5ea10faac4afe7c5163c23c4df16781aa77570ec553bc4f4b6094166c1fcfc3c431f13e51ffa32f7ea2849e76ec0151ea35`
### Node Binaries
filename | sha512 hash
-------- | -----------
[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.14.0-alpha.2/kubernetes-node-linux-amd64.tar.gz) | `8c37fd2fe6232d2c148e23df021b8b5347136263399932bcdff0c7a0186f3145de9ede4936b14de7484cc6db9241517d79b5306c380ed374396882900b63e912`
[kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.14.0-alpha.2/kubernetes-node-linux-arm.tar.gz) | `389e4e77ab9e62968a25b8f4e146a2c3fbb3db2e60e051922edf6395c26cc5380e5a77bf67022339d6ebfe9abd714636d77510bbc42924b4265fdb245fae08c9`
[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.14.0-alpha.2/kubernetes-node-linux-arm64.tar.gz) | `7efc32dfeefcef7f860913c25431bd891a435e92cb8d5a95f8deca1a82aa899a007d4b19134493694a4bccb5564867488634a780c128f0cf82c61d98afa889f5`
[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.14.0-alpha.2/kubernetes-node-linux-ppc64le.tar.gz) | `da30c03bca4b81d810a7df006db02333dea87e336d6cdca9c93392e01c7e43bf4902c969efa7fa53e8a70a0e863b403ec26b87bd38226b8b9f98777ddb0051a0`
[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.14.0-alpha.2/kubernetes-node-linux-s390x.tar.gz) | `cce43b7f0350b9e5a77ea703225adb9714ef022d176db5b99a0327937d19021d7a8e93ef1169389fd53b895bb98725d23c7565ef80afdd17596c26daf41eeeac`
[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.14.0-alpha.2/kubernetes-node-windows-amd64.tar.gz) | `d3accf522d80cbfb3d03e9eaa60a09767ba11e88a8a5b44a629192a7c6916b1fb3440f022a5ffc4ea78f3595f254a42f028dd428d117360091cd0c747ec39eb5`
## Changelog since v1.14.0-alpha.1
### Action Required
* Promote ValidateProxyRedirects to Beta, and enable by default. This feature restricts redirect following from the apiserver to same-host redirects. ([#72552](https://github.com/kubernetes/kubernetes/pull/72552), [@tallclair](https://github.com/tallclair))
* ACTION REQUIRED: If nodes are configured to respond to CRI streaming requests on a different host interface than what the apiserver makes requests on (only the case if not using the built-in dockershim & setting the kubelet flag `--redirect-container-streaming=true`), then these requests will be broken. In that case, the feature can be temporarily disabled until the node configuration is corrected. We suggest setting `--redirect-container-streaming=false` on the kubelet to avoid issues.
### Other notable changes
* Added alpha field storageVersionHash to the discovery document for each resource. Its value must be treated as opaque by clients. Only equality comparison on the value is valid. ([#73191](https://github.com/kubernetes/kubernetes/pull/73191), [@caesarxuchao](https://github.com/caesarxuchao))
* Fix admission metrics in seconds. ([#72343](https://github.com/kubernetes/kubernetes/pull/72343), [@danielqsj](https://github.com/danielqsj))
* Add metrics `*_admission_latencies_milliseconds` and `*_admission_latencies_milliseconds_summary` for backward compatible, but will be removed in a future release.
* Pod eviction now honors graceful deletion by default if no delete options are provided in the eviction request ([#72730](https://github.com/kubernetes/kubernetes/pull/72730), [@liggitt](https://github.com/liggitt))
* Update to go1.11.5 ([#73326](https://github.com/kubernetes/kubernetes/pull/73326), [@ixdy](https://github.com/ixdy))
* Change proxy metrics to conform metrics guidelines. ([#72334](https://github.com/kubernetes/kubernetes/pull/72334), [@danielqsj](https://github.com/danielqsj))
* The metrics `sync_proxy_rules_latency_microseconds` is deprecated, and will be removed in a future release, please convert to metrics`sync_proxy_rules_latency_seconds`.
* Add network stats for Windows nodes and pods. ([#70121](https://github.com/kubernetes/kubernetes/pull/70121), [@feiskyer](https://github.com/feiskyer))
* kubeadm: When certificates are present joining a new control plane make sure that they match at least the required SANs ([#73093](https://github.com/kubernetes/kubernetes/pull/73093), [@ereslibre](https://github.com/ereslibre))
* A new `TaintNodesByCondition` admission plugin taints newly created Node objects as "not ready", to fix a race condition that could cause pods to be scheduled on new nodes before their taints were updated to accurately reflect their reported conditions. This admission plugin is enabled by default if the `TaintNodesByCondition` feature is enabled. ([#73097](https://github.com/kubernetes/kubernetes/pull/73097), [@bsalamat](https://github.com/bsalamat))
* kube-addon-manager was updated to v9.0, and now uses kubectl v1.13.2 and prunes workload resources via the apps/v1 API ([#72978](https://github.com/kubernetes/kubernetes/pull/72978), [@liggitt](https://github.com/liggitt))
* When a watch is closed by an HTTP2 load balancer and we are told to go away, skip printing the message to stderr by default. ([#73277](https://github.com/kubernetes/kubernetes/pull/73277), [@smarterclayton](https://github.com/smarterclayton))
* If you are running the cloud-controller-manager and you have the `pvlabel.kubernetes.io` alpha Initializer enabled, you must now enable PersistentVolume labeling using the `PersistentVolumeLabel` admission controller instead. You can do this by adding `PersistentVolumeLabel` in the `--enable-admission-plugins` kube-apiserver flag. ([#73102](https://github.com/kubernetes/kubernetes/pull/73102), [@andrewsykim](https://github.com/andrewsykim))
* The alpha Initializers feature, `admissionregistration.k8s.io/v1alpha1` API version, `Initializers` admission plugin, and use of the `metadata.initializers` API field have been removed. Discontinue use of the alpha feature and delete any existing `InitializerConfiguration` API objects before upgrading. The `metadata.initializers` field will be removed in a future release. ([#72972](https://github.com/kubernetes/kubernetes/pull/72972), [@liggitt](https://github.com/liggitt))
* Scale max-inflight limits together with master VM sizes. ([#73268](https://github.com/kubernetes/kubernetes/pull/73268), [@wojtek-t](https://github.com/wojtek-t))
* kubectl supports copying files with wild card ([#72641](https://github.com/kubernetes/kubernetes/pull/72641), [@dixudx](https://github.com/dixudx))
* kubeadm: add back `--cert-dir` option for `kubeadm init phase certs sa` ([#73239](https://github.com/kubernetes/kubernetes/pull/73239), [@mattkelly](https://github.com/mattkelly))
* Remove deprecated args '--show-all' ([#69255](https://github.com/kubernetes/kubernetes/pull/69255), [@Pingan2017](https://github.com/Pingan2017))
* As per deprecation policy in https://kubernetes.io/docs/reference/using-api/deprecation-policy/ ([#73001](https://github.com/kubernetes/kubernetes/pull/73001), [@shivnagarajan](https://github.com/shivnagarajan))
* the taints "node.alpha.kubernetes.io/notReady" and "node.alpha.kubernetes.io/unreachable". are no
* longer supported or adjusted. These uses should be replaced with "node.kubernetes.io/not-ready"
* and "node.kubernetes.io/unreachable" respectively instead.
* The /swagger.json and /swagger-2.0.0.pb-v1 schema documents, deprecated since v1.10, have been removed in favor of `/openapi/v2` ([#73148](https://github.com/kubernetes/kubernetes/pull/73148), [@liggitt](https://github.com/liggitt))
* CoreDNS is only officially supported on Linux at this time. As such, when kubeadm is used to deploy this component into your kubernetes cluster, it will be restricted (using nodeSelectors) to run only on nodes with that operating system. This ensures that in clusters which include Windows nodes, the scheduler will not ever attempt to place CoreDNS pods on these machines, reducing setup latency and enhancing initial cluster stability. ([#69940](https://github.com/kubernetes/kubernetes/pull/69940), [@MarcPow](https://github.com/MarcPow))
* kubeadm now attempts to detect an installed CRI by its usual domain socket, so that --cri-socket can be omitted from the command line if Docker is not used and there is a single CRI installed. ([#69366](https://github.com/kubernetes/kubernetes/pull/69366), [@rosti](https://github.com/rosti))
* scheduler: makes pod less racing so as to be put back into activeQ properly ([#73078](https://github.com/kubernetes/kubernetes/pull/73078), [@Huang-Wei](https://github.com/Huang-Wei))
* jsonpath expressions containing `[start:end:step]` slice are now evaluated correctly ([#73149](https://github.com/kubernetes/kubernetes/pull/73149), [@liggitt](https://github.com/liggitt))
* metadata.deletionTimestamp is no longer moved into the future when issuing repeated DELETE requests against a resource containing a finalizer. ([#73138](https://github.com/kubernetes/kubernetes/pull/73138), [@liggitt](https://github.com/liggitt))
* The "kubectl api-resources" command will no longer fail to display any resources on a single failure ([#73035](https://github.com/kubernetes/kubernetes/pull/73035), [@juanvallejo](https://github.com/juanvallejo))
* e2e tests that require SSH may be used against clusters that have nodes without external IP addresses by setting the environment variable `KUBE_SSH_BASTION` to the `host:port` of a machine that is allowed to SSH to those nodes. The same private key that the test would use is used for the bastion host. The test connects to the bastion and then tunnels another SSH connection to the node. ([#72286](https://github.com/kubernetes/kubernetes/pull/72286), [@smarterclayton](https://github.com/smarterclayton))
* kubeadm: explicitly wait for `etcd` to have grown when joining a new control plane ([#72984](https://github.com/kubernetes/kubernetes/pull/72984), [@ereslibre](https://github.com/ereslibre))
* Install CSINodeInfo and CSIDriver CRDs in the local cluster. ([#72584](https://github.com/kubernetes/kubernetes/pull/72584), [@xing-yang](https://github.com/xing-yang))
* kubectl loads config file once and uses persistent client config ([#71117](https://github.com/kubernetes/kubernetes/pull/71117), [@dixudx](https://github.com/dixudx))
* remove stale OutOfDisk condition from kubelet side ([#72507](https://github.com/kubernetes/kubernetes/pull/72507), [@dixudx](https://github.com/dixudx))
* Node OS/arch labels are promoted to GA ([#73048](https://github.com/kubernetes/kubernetes/pull/73048), [@yujuhong](https://github.com/yujuhong))
* Fix graceful apiserver shutdown to not drop outgoing bytes before the process terminates. ([#72970](https://github.com/kubernetes/kubernetes/pull/72970), [@sttts](https://github.com/sttts))
* Change apiserver metrics to conform metrics guidelines. ([#72336](https://github.com/kubernetes/kubernetes/pull/72336), [@danielqsj](https://github.com/danielqsj))
* The following metrics are deprecated, and will be removed in a future release:
* `apiserver_request_count`
* `apiserver_request_latencies`
* `apiserver_request_latencies_summary`
* `apiserver_dropped_requests`
* `etcd_helper_cache_hit_count`
* `etcd_helper_cache_miss_count`
* `etcd_helper_cache_entry_count`
* `etcd_request_cache_get_latencies_summary`
* `etcd_request_cache_add_latencies_summary`
* `etcd_request_latencies_summary`
* `transformation_latencies_microseconds `
* `data_key_generation_latencies_microseconds`
* Please convert to the following metrics:
* `apiserver_request_total`
* `apiserver_request_latency_seconds`
* `apiserver_dropped_requests_total`
* `etcd_helper_cache_hit_total`
* `etcd_helper_cache_miss_total`
* `etcd_helper_cache_entry_total`
* `etcd_request_cache_get_latency_seconds`
* `etcd_request_cache_add_latency_seconds`
* `etcd_request_latency_seconds`
* `transformation_latencies_seconds`
* `data_key_generation_latencies_seconds`
* acquire lock before operating unschedulablepodsmap ([#73022](https://github.com/kubernetes/kubernetes/pull/73022), [@denkensk](https://github.com/denkensk))
* Print `SizeLimit` of `EmptyDir` in `kubectl describe pod` outputs. ([#69279](https://github.com/kubernetes/kubernetes/pull/69279), [@dtaniwaki](https://github.com/dtaniwaki))
* add goroutine to move unschedulable pods to activeq if they are not retried for more than 1 minute ([#72558](https://github.com/kubernetes/kubernetes/pull/72558), [@denkensk](https://github.com/denkensk))
* PidPressure evicts pods from lowest priority to highest priority ([#72844](https://github.com/kubernetes/kubernetes/pull/72844), [@dashpole](https://github.com/dashpole))
* Reduce GCE log rotation check from 1 hour to every 5 minutes. Rotation policy is unchanged (new day starts, log file size > 100MB). ([#72062](https://github.com/kubernetes/kubernetes/pull/72062), [@jpbetz](https://github.com/jpbetz))
* Add support for max attach limit for Cinder ([#72980](https://github.com/kubernetes/kubernetes/pull/72980), [@gnufied](https://github.com/gnufied))
* Fixes the setting of NodeAddresses when using the vSphere CloudProvider and nodes that have multiple IP addresses. ([#70805](https://github.com/kubernetes/kubernetes/pull/70805), [@danwinship](https://github.com/danwinship))
* kubeadm: pull images when joining a new control plane instance ([#72870](https://github.com/kubernetes/kubernetes/pull/72870), [@MalloZup](https://github.com/MalloZup))
* Enable mTLS encription between etcd and kube-apiserver in GCE ([#70144](https://github.com/kubernetes/kubernetes/pull/70144), [@wenjiaswe](https://github.com/wenjiaswe))
* The `/swaggerapi/*` schema docs, deprecated since 1.7, have been removed in favor of the /openapi/v2 schema docs. ([#72924](https://github.com/kubernetes/kubernetes/pull/72924), [@liggitt](https://github.com/liggitt))
* Allow users to use Docker 18.09 with kubeadm ([#72823](https://github.com/kubernetes/kubernetes/pull/72823), [@dims](https://github.com/dims))
# v1.14.0-alpha.1
[Documentation](https://docs.k8s.io)

98
Godeps/Godeps.json generated
View File

@@ -1555,8 +1555,8 @@
},
{
"ImportPath": "github.com/evanphx/json-patch",
"Comment": "v4.0.0-3-g36442dbdb58521",
"Rev": "36442dbdb585210f8d5a1b45e67aa323c197d5c4"
"Comment": "v4.1.0-11-gd4020504c68b6b",
"Rev": "d4020504c68b6bfa818032bedfb48e33e9638506"
},
{
"ImportPath": "github.com/exponent-io/jsonpath",
@@ -2101,123 +2101,127 @@
},
{
"ImportPath": "github.com/gophercloud/gophercloud",
"Rev": "781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d"
"Rev": "c818fa66e4c88b30db28038fe3f18f2f4a0db9a8"
},
{
"ImportPath": "github.com/gophercloud/gophercloud/openstack",
"Rev": "781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d"
"Rev": "c818fa66e4c88b30db28038fe3f18f2f4a0db9a8"
},
{
"ImportPath": "github.com/gophercloud/gophercloud/openstack/blockstorage/extensions/volumeactions",
"Rev": "781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d"
"Rev": "c818fa66e4c88b30db28038fe3f18f2f4a0db9a8"
},
{
"ImportPath": "github.com/gophercloud/gophercloud/openstack/blockstorage/v1/volumes",
"Rev": "781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d"
"Rev": "c818fa66e4c88b30db28038fe3f18f2f4a0db9a8"
},
{
"ImportPath": "github.com/gophercloud/gophercloud/openstack/blockstorage/v2/volumes",
"Rev": "781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d"
"Rev": "c818fa66e4c88b30db28038fe3f18f2f4a0db9a8"
},
{
"ImportPath": "github.com/gophercloud/gophercloud/openstack/blockstorage/v3/volumes",
"Rev": "781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d"
"Rev": "c818fa66e4c88b30db28038fe3f18f2f4a0db9a8"
},
{
"ImportPath": "github.com/gophercloud/gophercloud/openstack/common/extensions",
"Rev": "781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d"
"Rev": "c818fa66e4c88b30db28038fe3f18f2f4a0db9a8"
},
{
"ImportPath": "github.com/gophercloud/gophercloud/openstack/compute/v2/extensions/attachinterfaces",
"Rev": "781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d"
"Rev": "c818fa66e4c88b30db28038fe3f18f2f4a0db9a8"
},
{
"ImportPath": "github.com/gophercloud/gophercloud/openstack/compute/v2/extensions/volumeattach",
"Rev": "781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d"
"Rev": "c818fa66e4c88b30db28038fe3f18f2f4a0db9a8"
},
{
"ImportPath": "github.com/gophercloud/gophercloud/openstack/compute/v2/flavors",
"Rev": "781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d"
"Rev": "c818fa66e4c88b30db28038fe3f18f2f4a0db9a8"
},
{
"ImportPath": "github.com/gophercloud/gophercloud/openstack/compute/v2/images",
"Rev": "781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d"
"Rev": "c818fa66e4c88b30db28038fe3f18f2f4a0db9a8"
},
{
"ImportPath": "github.com/gophercloud/gophercloud/openstack/compute/v2/servers",
"Rev": "781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d"
"Rev": "c818fa66e4c88b30db28038fe3f18f2f4a0db9a8"
},
{
"ImportPath": "github.com/gophercloud/gophercloud/openstack/identity/v2/tenants",
"Rev": "781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d"
"Rev": "c818fa66e4c88b30db28038fe3f18f2f4a0db9a8"
},
{
"ImportPath": "github.com/gophercloud/gophercloud/openstack/identity/v2/tokens",
"Rev": "781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d"
"Rev": "c818fa66e4c88b30db28038fe3f18f2f4a0db9a8"
},
{
"ImportPath": "github.com/gophercloud/gophercloud/openstack/identity/v3/extensions/trusts",
"Rev": "781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d"
"Rev": "c818fa66e4c88b30db28038fe3f18f2f4a0db9a8"
},
{
"ImportPath": "github.com/gophercloud/gophercloud/openstack/identity/v3/tokens",
"Rev": "781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d"
"Rev": "c818fa66e4c88b30db28038fe3f18f2f4a0db9a8"
},
{
"ImportPath": "github.com/gophercloud/gophercloud/openstack/networking/v2/extensions",
"Rev": "781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d"
"Rev": "c818fa66e4c88b30db28038fe3f18f2f4a0db9a8"
},
{
"ImportPath": "github.com/gophercloud/gophercloud/openstack/networking/v2/extensions/external",
"Rev": "781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d"
"Rev": "c818fa66e4c88b30db28038fe3f18f2f4a0db9a8"
},
{
"ImportPath": "github.com/gophercloud/gophercloud/openstack/networking/v2/extensions/layer3/floatingips",
"Rev": "781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d"
"Rev": "c818fa66e4c88b30db28038fe3f18f2f4a0db9a8"
},
{
"ImportPath": "github.com/gophercloud/gophercloud/openstack/networking/v2/extensions/layer3/routers",
"Rev": "781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d"
"Rev": "c818fa66e4c88b30db28038fe3f18f2f4a0db9a8"
},
{
"ImportPath": "github.com/gophercloud/gophercloud/openstack/networking/v2/extensions/lbaas_v2/l7policies",
"Rev": "c818fa66e4c88b30db28038fe3f18f2f4a0db9a8"
},
{
"ImportPath": "github.com/gophercloud/gophercloud/openstack/networking/v2/extensions/lbaas_v2/listeners",
"Rev": "781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d"
"Rev": "c818fa66e4c88b30db28038fe3f18f2f4a0db9a8"
},
{
"ImportPath": "github.com/gophercloud/gophercloud/openstack/networking/v2/extensions/lbaas_v2/loadbalancers",
"Rev": "781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d"
"Rev": "c818fa66e4c88b30db28038fe3f18f2f4a0db9a8"
},
{
"ImportPath": "github.com/gophercloud/gophercloud/openstack/networking/v2/extensions/lbaas_v2/monitors",
"Rev": "781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d"
"Rev": "c818fa66e4c88b30db28038fe3f18f2f4a0db9a8"
},
{
"ImportPath": "github.com/gophercloud/gophercloud/openstack/networking/v2/extensions/lbaas_v2/pools",
"Rev": "781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d"
"Rev": "c818fa66e4c88b30db28038fe3f18f2f4a0db9a8"
},
{
"ImportPath": "github.com/gophercloud/gophercloud/openstack/networking/v2/extensions/security/groups",
"Rev": "781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d"
"Rev": "c818fa66e4c88b30db28038fe3f18f2f4a0db9a8"
},
{
"ImportPath": "github.com/gophercloud/gophercloud/openstack/networking/v2/extensions/security/rules",
"Rev": "781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d"
"Rev": "c818fa66e4c88b30db28038fe3f18f2f4a0db9a8"
},
{
"ImportPath": "github.com/gophercloud/gophercloud/openstack/networking/v2/networks",
"Rev": "781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d"
"Rev": "c818fa66e4c88b30db28038fe3f18f2f4a0db9a8"
},
{
"ImportPath": "github.com/gophercloud/gophercloud/openstack/networking/v2/ports",
"Rev": "781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d"
"Rev": "c818fa66e4c88b30db28038fe3f18f2f4a0db9a8"
},
{
"ImportPath": "github.com/gophercloud/gophercloud/openstack/utils",
"Rev": "781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d"
"Rev": "c818fa66e4c88b30db28038fe3f18f2f4a0db9a8"
},
{
"ImportPath": "github.com/gophercloud/gophercloud/pagination",
"Rev": "781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d"
"Rev": "c818fa66e4c88b30db28038fe3f18f2f4a0db9a8"
},
{
"ImportPath": "github.com/gorilla/websocket",
@@ -4090,39 +4094,51 @@
},
{
"ImportPath": "k8s.io/utils/buffer",
"Rev": "8a87304934321b4b0ad72a7cb3cbc715d67d38c7"
"Rev": "ed37f7428a91fc2a81070808937195dcd46d320e"
},
{
"ImportPath": "k8s.io/utils/clock",
"Rev": "8a87304934321b4b0ad72a7cb3cbc715d67d38c7"
"Rev": "ed37f7428a91fc2a81070808937195dcd46d320e"
},
{
"ImportPath": "k8s.io/utils/exec",
"Rev": "8a87304934321b4b0ad72a7cb3cbc715d67d38c7"
"Rev": "ed37f7428a91fc2a81070808937195dcd46d320e"
},
{
"ImportPath": "k8s.io/utils/exec/testing",
"Rev": "8a87304934321b4b0ad72a7cb3cbc715d67d38c7"
"Rev": "ed37f7428a91fc2a81070808937195dcd46d320e"
},
{
"ImportPath": "k8s.io/utils/integer",
"Rev": "8a87304934321b4b0ad72a7cb3cbc715d67d38c7"
"Rev": "ed37f7428a91fc2a81070808937195dcd46d320e"
},
{
"ImportPath": "k8s.io/utils/io",
"Rev": "8a87304934321b4b0ad72a7cb3cbc715d67d38c7"
"Rev": "ed37f7428a91fc2a81070808937195dcd46d320e"
},
{
"ImportPath": "k8s.io/utils/keymutex",
"Rev": "ed37f7428a91fc2a81070808937195dcd46d320e"
},
{
"ImportPath": "k8s.io/utils/nsenter",
"Rev": "8a87304934321b4b0ad72a7cb3cbc715d67d38c7"
"Rev": "ed37f7428a91fc2a81070808937195dcd46d320e"
},
{
"ImportPath": "k8s.io/utils/path",
"Rev": "ed37f7428a91fc2a81070808937195dcd46d320e"
},
{
"ImportPath": "k8s.io/utils/pointer",
"Rev": "8a87304934321b4b0ad72a7cb3cbc715d67d38c7"
"Rev": "ed37f7428a91fc2a81070808937195dcd46d320e"
},
{
"ImportPath": "k8s.io/utils/strings",
"Rev": "ed37f7428a91fc2a81070808937195dcd46d320e"
},
{
"ImportPath": "k8s.io/utils/trace",
"Rev": "8a87304934321b4b0ad72a7cb3cbc715d67d38c7"
"Rev": "ed37f7428a91fc2a81070808937195dcd46d320e"
},
{
"ImportPath": "sigs.k8s.io/yaml",

829
Godeps/LICENSES generated
View File

@@ -65927,6 +65927,205 @@ specific language governing permissions and limitations under the License.
================================================================================
================================================================================
= vendor/github.com/gophercloud/gophercloud/openstack/networking/v2/extensions/lbaas_v2/l7policies licensed under: =
Copyright 2012-2013 Rackspace, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use
this file except in compliance with the License. You may obtain a copy of the
License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed
under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.
------
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
= vendor/github.com/gophercloud/gophercloud/LICENSE dd19699707373c2ca31531a659130416
================================================================================
================================================================================
= vendor/github.com/gophercloud/gophercloud/openstack/networking/v2/extensions/lbaas_v2/listeners licensed under: =
@@ -116213,6 +116412,216 @@ third-party archives.
================================================================================
================================================================================
= vendor/k8s.io/utils/keymutex licensed under: =
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
= vendor/k8s.io/utils/LICENSE 3b83ef96387f14655fc854ddc3c6bd57
================================================================================
================================================================================
= vendor/k8s.io/utils/nsenter licensed under: =
@@ -116423,6 +116832,216 @@ third-party archives.
================================================================================
================================================================================
= vendor/k8s.io/utils/path licensed under: =
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
= vendor/k8s.io/utils/LICENSE 3b83ef96387f14655fc854ddc3c6bd57
================================================================================
================================================================================
= vendor/k8s.io/utils/pointer licensed under: =
@@ -116633,6 +117252,216 @@ third-party archives.
================================================================================
================================================================================
= vendor/k8s.io/utils/strings licensed under: =
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
= vendor/k8s.io/utils/LICENSE 3b83ef96387f14655fc854ddc3c6bd57
================================================================================
================================================================================
= vendor/k8s.io/utils/trace licensed under: =

4
api/openapi-spec/swagger.json generated
View File

@@ -92006,6 +92006,10 @@
"description": "singularName is the singular name of the resource. This allows clients to handle plural and singular opaquely. The singularName is more correct for reporting status on a single item and both singular and plural are allowed from the kubectl CLI interface.",
"type": "string"
},
"storageVersionHash": {
"description": "The hash value of the storage version, the version this resource is converted to when written to the data store. Value must be treated as opaque by clients. Only equality comparison on the value is valid. This is an alpha feature and may change or be removed in the future. The field is populated by the apiserver only if the StorageVersionHash feature gate is enabled. This field will remain optional even if it graduates.",
"type": "string"
},
"verbs": {
"description": "verbs is a list of supported kube verbs (this includes get, list, watch, create, update, patch, delete, deletecollection, and proxy)",
"type": "array",

40
build/common.sh
View File

@@ -26,13 +26,13 @@ USER_ID=$(id -u)
GROUP_ID=$(id -g)
DOCKER_OPTS=${DOCKER_OPTS:-""}
DOCKER=(docker ${DOCKER_OPTS})
IFS=" " read -r -a DOCKER <<< "docker ${DOCKER_OPTS}"
DOCKER_HOST=${DOCKER_HOST:-""}
DOCKER_MACHINE_NAME=${DOCKER_MACHINE_NAME:-"kube-dev"}
readonly DOCKER_MACHINE_DRIVER=${DOCKER_MACHINE_DRIVER:-"virtualbox --virtualbox-cpu-count -1"}
# This will canonicalize the path
KUBE_ROOT=$(cd "$(dirname "${BASH_SOURCE}")"/.. && pwd -P)
KUBE_ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")"/.. && pwd -P)
source "${KUBE_ROOT}/hack/lib/init.sh"
@@ -137,7 +137,7 @@ function kube::build::verify_prereqs() {
fi
kube::util::ensure_docker_daemon_connectivity || return 1
if (( ${KUBE_VERBOSE} > 6 )); then
if (( KUBE_VERBOSE > 6 )); then
kube::log::status "Docker Version:"
"${DOCKER[@]}" version | kube::log::info_from_stdin
fi
@@ -185,7 +185,8 @@ function kube::build::docker_available_on_osx() {
function kube::build::prepare_docker_machine() {
kube::log::status "docker-machine was found."
local available_memory_bytes=$(sysctl -n hw.memsize 2>/dev/null)
local available_memory_bytes
available_memory_bytes=$(sysctl -n hw.memsize 2>/dev/null)
local bytes_in_mb=1048576
@@ -193,7 +194,7 @@ function kube::build::prepare_docker_machine() {
# of multiple by .5, because bash can only multiply by ints.
local memory_divisor=2
local virtualbox_memory_mb=$(( ${available_memory_bytes} / (${bytes_in_mb} * ${memory_divisor}) ))
local virtualbox_memory_mb=$(( available_memory_bytes / (bytes_in_mb * memory_divisor) ))
docker-machine inspect "${DOCKER_MACHINE_NAME}" &> /dev/null || {
kube::log::status "Creating a machine to build Kubernetes"
@@ -215,7 +216,7 @@ function kube::build::prepare_docker_machine() {
while ! docker_machine_out=$(docker-machine env "${DOCKER_MACHINE_NAME}" 2>&1); do
if [[ ${docker_machine_out} =~ "Error checking TLS connection" ]]; then
echo "${docker_machine_out}"
docker-machine regenerate-certs ${DOCKER_MACHINE_NAME}
docker-machine regenerate-certs "${DOCKER_MACHINE_NAME}"
else
sleep 1
fi
@@ -368,7 +369,7 @@ function kube::build::short_hash() {
else
short_hash=$(echo -n "$1" | md5sum)
fi
echo ${short_hash:0:10}
echo "${short_hash:0:10}"
}
# Pedantically kill, wait-on and remove a container. The -f -v options
@@ -412,7 +413,7 @@ function kube::build::clean() {
function kube::build::build_image() {
mkdir -p "${LOCAL_OUTPUT_BUILD_CONTEXT}"
# Make sure the context directory owned by the right user for syncing sources to container.
chown -R ${USER_ID}:${GROUP_ID} "${LOCAL_OUTPUT_BUILD_CONTEXT}"
chown -R "${USER_ID}":"${GROUP_ID}" "${LOCAL_OUTPUT_BUILD_CONTEXT}"
cp /etc/localtime "${LOCAL_OUTPUT_BUILD_CONTEXT}/"
@@ -503,7 +504,7 @@ function kube::build::ensure_data_container() {
--name "${KUBE_DATA_CONTAINER_NAME}"
--hostname "${HOSTNAME}"
"${KUBE_BUILD_IMAGE}"
chown -R ${USER_ID}:${GROUP_ID}
chown -R "${USER_ID}":"${GROUP_ID}"
"${REMOTE_ROOT}"
/usr/local/go/pkg/
)
@@ -582,7 +583,7 @@ function kube::build::run_build_command_ex() {
if [[ -t 0 ]]; then
docker_run_opts+=(--interactive --tty)
elif [[ "${detach}" == false ]]; then
docker_run_opts+=(--attach=stdout --attach=stderr)
docker_run_opts+=("--attach=stdout" "--attach=stderr")
fi
local -ra docker_cmd=(
@@ -599,13 +600,13 @@ function kube::build::run_build_command_ex() {
function kube::build::rsync_probe {
# Wait unil rsync is up and running.
local tries=20
while (( ${tries} > 0 )) ; do
while (( tries > 0 )) ; do
if rsync "rsync://k8s@${1}:${2}/" \
--password-file="${LOCAL_OUTPUT_BUILD_CONTEXT}/rsyncd.password" \
&> /dev/null ; then
return 0
fi
tries=$(( ${tries} - 1))
tries=$(( tries - 1))
sleep 0.1
done
@@ -625,7 +626,7 @@ function kube::build::start_rsyncd_container() {
kube::build::stop_rsyncd_container
V=3 kube::log::status "Starting rsyncd container"
kube::build::run_build_command_ex \
"${KUBE_RSYNC_CONTAINER_NAME}" -p 127.0.0.1:${KUBE_RSYNC_PORT}:${KUBE_CONTAINER_RSYNC_PORT} -d \
"${KUBE_RSYNC_CONTAINER_NAME}" -p 127.0.0.1:"${KUBE_RSYNC_PORT}":"${KUBE_CONTAINER_RSYNC_PORT}" -d \
-e ALLOW_HOST="$(${IPTOOL} | grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*' | grep -Eo '([0-9]*\.){3}[0-9]*' | grep -v '127.0.0.1')" \
-- /rsyncd.sh >/dev/null
@@ -643,7 +644,7 @@ function kube::build::start_rsyncd_container() {
# machines) we have to talk directly to the container IP. There is no one
# strategy that works in all cases so we test to figure out which situation we
# are in.
if kube::build::rsync_probe 127.0.0.1 ${mapped_port}; then
if kube::build::rsync_probe 127.0.0.1 "${mapped_port}"; then
KUBE_RSYNC_ADDR="127.0.0.1:${mapped_port}"
return 0
elif kube::build::rsync_probe "${container_ip}" ${KUBE_CONTAINER_RSYNC_PORT}; then
@@ -664,12 +665,12 @@ function kube::build::stop_rsyncd_container() {
function kube::build::rsync {
local -a rsync_opts=(
--archive
--password-file="${LOCAL_OUTPUT_BUILD_CONTEXT}/rsyncd.password"
"--password-file=${LOCAL_OUTPUT_BUILD_CONTEXT}/rsyncd.password"
)
if (( ${KUBE_VERBOSE} >= 6 )); then
if (( KUBE_VERBOSE >= 6 )); then
rsync_opts+=("-iv")
fi
if (( ${KUBE_RSYNC_COMPRESS} > 0 )); then
if (( KUBE_RSYNC_COMPRESS > 0 )); then
rsync_opts+=("--compress-level=${KUBE_RSYNC_COMPRESS}")
fi
V=3 kube::log::status "Running rsync"
@@ -712,11 +713,6 @@ function kube::build::copy_output() {
kube::build::start_rsyncd_container
local rsync_extra=""
if (( ${KUBE_VERBOSE} >= 6 )); then
rsync_extra="-iv"
fi
# The filter syntax for rsync is a little obscure. It filters on files and
# directories. If you don't go in to a directory you won't find any files
# there. Rules are evaluated in order. The last two rules are a little

2
build/debian-base/Makefile
View File

@@ -18,7 +18,7 @@ REGISTRY ?= staging-k8s.gcr.io
IMAGE ?= $(REGISTRY)/debian-base
BUILD_IMAGE ?= debian-build
TAG ?= 0.4.0
TAG ?= 0.4.1
TAR_FILE ?= rootfs.tar
ARCH?=amd64

4
build/debian-hyperkube-base/Makefile
View File

@@ -19,12 +19,12 @@
REGISTRY?=staging-k8s.gcr.io
IMAGE?=$(REGISTRY)/debian-hyperkube-base
TAG=0.12.0
TAG=0.12.1
ARCH?=amd64
ALL_ARCH = amd64 arm arm64 ppc64le s390x
CACHEBUST?=1
BASEIMAGE=k8s.gcr.io/debian-base-$(ARCH):0.4.0
BASEIMAGE=k8s.gcr.io/debian-base-$(ARCH):0.4.1
CNI_VERSION=v0.6.0
TEMP_DIR:=$(shell mktemp -d)

4
build/debian-iptables/Makefile
View File

@@ -16,12 +16,12 @@
REGISTRY?="staging-k8s.gcr.io"
IMAGE=$(REGISTRY)/debian-iptables
TAG?=v11.0
TAG?=v11.0.1
ARCH?=amd64
ALL_ARCH = amd64 arm arm64 ppc64le s390x
TEMP_DIR:=$(shell mktemp -d)
BASEIMAGE?=k8s.gcr.io/debian-base-$(ARCH):0.4.0
BASEIMAGE?=k8s.gcr.io/debian-base-$(ARCH):0.4.1
# This option is for running docker manifest command
export DOCKER_CLI_EXPERIMENTAL := enabled

23
build/root/Makefile.generated_files
View File

@@ -296,16 +296,23 @@ $(DEFAULTER_GEN): $(k8s.io/kubernetes/vendor/k8s.io/code-generator/cmd/defaulter
# Conversion generation
# Any package that wants conversion functions generated into it must
# include one or more comment-tags in its `doc.go` file, of the form:
# // +k8s:conversion-gen=<INTERNAL_TYPES_DIR>
#
# Any package that wants conversion functions generated must include one or
# more comment-tags in any .go file, in column 0, of the form:
# // +k8s:conversion-gen=<CONVERSION_TARGET_DIR>
# The INTERNAL_TYPES_DIR is a project-local path to another directory
# which should be considered when evaluating peer types for
# conversions. An optional additional comment of the form
# // +k8s:conversion-gen-external-types=<EXTERNAL_TYPES_DIR>
#
# The CONVERSION_TARGET_DIR is a project-local path to another directory which
# should be considered when evaluating peer types for conversions. Types which
# are found in the source package (where conversions are being generated)
# but do not have a peer in one of the target directories will not have
# conversions generated.
# identifies where to find the external types; if there is no such
# comment then the external types are sought in the package where the
# `k8s:conversion` tag is found.
#
# Conversions, in both directions, are generated for every type name
# that is defined in both an internal types package and the external
# types package.
#
# TODO: it might be better in the long term to make peer-types explicit in the
# IDL.

4
cluster/addons/dns-horizontal-autoscaler/dns-horizontal-autoscaler.yaml
View File

@@ -33,7 +33,7 @@ rules:
- apiGroups: [""]
resources: ["replicationcontrollers/scale"]
verbs: ["get", "update"]
- apiGroups: ["extensions"]
- apiGroups: ["extensions","apps"]
resources: ["deployments/scale", "replicasets/scale"]
verbs: ["get", "update"]
# Remove the configmaps rule once below issue is fixed:
@@ -85,7 +85,7 @@ spec:
fsGroup: 65534
containers:
- name: autoscaler
image: k8s.gcr.io/cluster-proportional-autoscaler-amd64:1.3.0
image: k8s.gcr.io/cluster-proportional-autoscaler-amd64:1.4.0
resources:
requests:
cpu: "20m"

4
cluster/addons/dns/coredns/coredns.yaml.base
View File

@@ -70,7 +70,7 @@ data:
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
proxy . /etc/resolv.conf
forward . /etc/resolv.conf
cache 30
loop
reload
@@ -114,7 +114,7 @@ spec:
beta.kubernetes.io/os: linux
containers:
- name: coredns
image: k8s.gcr.io/coredns:1.2.6
image: k8s.gcr.io/coredns:1.3.1
imagePullPolicy: IfNotPresent
resources:
limits:

4
cluster/addons/dns/coredns/coredns.yaml.in
View File

@@ -70,7 +70,7 @@ data:
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
proxy . /etc/resolv.conf
forward . /etc/resolv.conf
cache 30
loop
reload
@@ -114,7 +114,7 @@ spec:
beta.kubernetes.io/os: linux
containers:
- name: coredns
image: k8s.gcr.io/coredns:1.2.6
image: k8s.gcr.io/coredns:1.3.1
imagePullPolicy: IfNotPresent
resources:
limits:

4
cluster/addons/dns/coredns/coredns.yaml.sed
View File

@@ -70,7 +70,7 @@ data:
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
proxy . /etc/resolv.conf
forward . /etc/resolv.conf
cache 30
loop
reload
@@ -114,7 +114,7 @@ spec:
beta.kubernetes.io/os: linux
containers:
- name: coredns
image: k8s.gcr.io/coredns:1.2.6
image: k8s.gcr.io/coredns:1.3.1
imagePullPolicy: IfNotPresent
resources:
limits:

1
cluster/addons/fluentd-gcp/fluentd-gcp-ds.yaml
View File

@@ -104,6 +104,7 @@ spec:
# END_PROMETHEUS_TO_SD
nodeSelector:
beta.kubernetes.io/fluentd-ds-ready: "true"
beta.kubernetes.io/os: linux
terminationGracePeriodSeconds: 60
tolerations:
- operator: "Exists"

1
cluster/addons/metadata-proxy/gce/metadata-proxy.yaml
View File

@@ -89,4 +89,5 @@ spec:
# END_PROMETHEUS_TO_SD
nodeSelector:
beta.kubernetes.io/metadata-proxy-ready: "true"
beta.kubernetes.io/os: linux
terminationGracePeriodSeconds: 30

3
cluster/common.sh
View File

@@ -321,6 +321,9 @@ function find-tar() {
# KUBE_MANIFESTS_TAR
function find-release-tars() {
SERVER_BINARY_TAR=$(find-tar kubernetes-server-linux-amd64.tar.gz)
if [[ "${NUM_WINDOWS_NODES}" -gt "0" && "${USE_RELEASE_NODE_BINARIES:-false}" == "false" ]]; then
NODE_BINARY_TAR=$(find-tar kubernetes-node-windows-amd64.tar.gz)
fi
# This tarball is used by GCI, Ubuntu Trusty, and Container Linux.
KUBE_MANIFESTS_TAR=

71
cluster/gce/config-common.sh
View File

@@ -14,26 +14,36 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# Returns the total number of Linux and Windows nodes in the cluster.
#
# Vars assumed:
# NUM_NODES
# NUM_WINDOWS_NODES
function get-num-nodes {
echo "$((${NUM_NODES} + ${NUM_WINDOWS_NODES}))"
}
# Vars assumed:
# NUM_NODES
# NUM_WINDOWS_NODES
function get-master-size {
local suggested_master_size=1
if [[ "${NUM_NODES}" -gt "5" ]]; then
if [[ "$(get-num-nodes)" -gt "5" ]]; then
suggested_master_size=2
fi
if [[ "${NUM_NODES}" -gt "10" ]]; then
if [[ "$(get-num-nodes)" -gt "10" ]]; then
suggested_master_size=4
fi
if [[ "${NUM_NODES}" -gt "100" ]]; then
if [[ "$(get-num-nodes)" -gt "100" ]]; then
suggested_master_size=8
fi
if [[ "${NUM_NODES}" -gt "250" ]]; then
if [[ "$(get-num-nodes)" -gt "250" ]]; then
suggested_master_size=16
fi
if [[ "${NUM_NODES}" -gt "500" ]]; then
if [[ "$(get-num-nodes)" -gt "500" ]]; then
suggested_master_size=32
fi
if [[ "${NUM_NODES}" -gt "3000" ]]; then
if [[ "$(get-num-nodes)" -gt "3000" ]]; then
suggested_master_size=64
fi
echo "${suggested_master_size}"
@@ -41,25 +51,27 @@ function get-master-size {
# Vars assumed:
# NUM_NODES
# NUM_WINDOWS_NODES
function get-master-root-disk-size() {
local suggested_master_root_disk_size="20GB"
if [[ "${NUM_NODES}" -gt "1000" ]]; then
suggested_master_root_disk_size="50GB"
fi
if [[ "${NUM_NODES}" -gt "2000" ]]; then
if [[ "$(get-num-nodes)" -gt "500" ]]; then
suggested_master_root_disk_size="100GB"
fi
if [[ "$(get-num-nodes)" -gt "3000" ]]; then
suggested_master_root_disk_size="500GB"
fi
echo "${suggested_master_root_disk_size}"
}
# Vars assumed:
# NUM_NODES
# NUM_WINDOWS_NODES
function get-master-disk-size() {
local suggested_master_disk_size="20GB"
if [[ "${NUM_NODES}" -gt "1000" ]]; then
if [[ "$(get-num-nodes)" -gt "500" ]]; then
suggested_master_disk_size="100GB"
fi
if [[ "${NUM_NODES}" -gt "2000" ]]; then
if [[ "$(get-num-nodes)" -gt "3000" ]]; then
suggested_master_disk_size="200GB"
fi
echo "${suggested_master_disk_size}"
@@ -72,13 +84,13 @@ function get-node-ip-range {
return
fi
local suggested_range="10.40.0.0/22"
if [[ "${NUM_NODES}" -gt 1000 ]]; then
if [[ "$(get-num-nodes)" -gt 1000 ]]; then
suggested_range="10.40.0.0/21"
fi
if [[ "${NUM_NODES}" -gt 2000 ]]; then
if [[ "$(get-num-nodes)" -gt 2000 ]]; then
suggested_range="10.40.0.0/20"
fi
if [[ "${NUM_NODES}" -gt 4000 ]]; then
if [[ "$(get-num-nodes)" -gt 4000 ]]; then
suggested_range="10.40.0.0/19"
fi
echo "${suggested_range}"
@@ -86,13 +98,13 @@ function get-node-ip-range {
function get-cluster-ip-range {
local suggested_range="10.64.0.0/14"
if [[ "${NUM_NODES}" -gt 1000 ]]; then
if [[ "$(get-num-nodes)" -gt 1000 ]]; then
suggested_range="10.64.0.0/13"
fi
if [[ "${NUM_NODES}" -gt 2000 ]]; then
if [[ "$(get-num-nodes)" -gt 2000 ]]; then
suggested_range="10.64.0.0/12"
fi
if [[ "${NUM_NODES}" -gt 4000 ]]; then
if [[ "$(get-num-nodes)" -gt 4000 ]]; then
suggested_range="10.64.0.0/11"
fi
echo "${suggested_range}"
@@ -114,3 +126,26 @@ function get-alias-range-size() {
# NOTE: Avoid giving nodes empty scopes, because kubelet needs a service account
# in order to initialize properly.
NODE_SCOPES="${NODE_SCOPES:-monitoring,logging-write,storage-ro}"
# Root directory for Kubernetes files on Windows nodes.
WINDOWS_K8S_DIR="C:\etc\kubernetes"
# Directory where Kubernetes binaries will be installed on Windows nodes.
WINDOWS_NODE_DIR="${WINDOWS_K8S_DIR}\node\bin"
# Directory where Kubernetes log files will be stored on Windows nodes.
WINDOWS_LOGS_DIR="${WINDOWS_K8S_DIR}\logs"
# Directory where CNI binaries will be stored on Windows nodes.
WINDOWS_CNI_DIR="${WINDOWS_K8S_DIR}\cni"
# Directory where CNI config files will be stored on Windows nodes.
WINDOWS_CNI_CONFIG_DIR="${WINDOWS_K8S_DIR}\cni\config"
# Pod manifests directory for Windows nodes on Windows nodes.
WINDOWS_MANIFESTS_DIR="${WINDOWS_K8S_DIR}\manifests"
# Directory where cert/key files will be stores on Windows nodes.
WINDOWS_PKI_DIR="${WINDOWS_K8S_DIR}\pki"
# Path for kubelet config file on Windows nodes.
WINDOWS_KUBELET_CONFIG_FILE="${WINDOWS_K8S_DIR}\kubelet-config.yaml"
# Path for kubeconfig file on Windows nodes.
WINDOWS_KUBECONFIG_FILE="${WINDOWS_K8S_DIR}\kubelet.kubeconfig"
# Path for bootstrap kubeconfig file on Windows nodes.
WINDOWS_BOOTSTRAP_KUBECONFIG_FILE="${WINDOWS_K8S_DIR}\kubelet.bootstrap-kubeconfig"
# Path for kube-proxy kubeconfig file on Windows nodes.
WINDOWS_KUBEPROXY_KUBECONFIG_FILE="${WINDOWS_K8S_DIR}\kubeproxy.kubeconfig"

12
cluster/gce/config-default.sh
View File

@@ -29,6 +29,7 @@ RELEASE_REGION_FALLBACK=${RELEASE_REGION_FALLBACK:-false}
REGIONAL_KUBE_ADDONS=${REGIONAL_KUBE_ADDONS:-true}
NODE_SIZE=${NODE_SIZE:-n1-standard-2}
NUM_NODES=${NUM_NODES:-3}
NUM_WINDOWS_NODES=${NUM_WINDOWS_NODES:-0}
MASTER_SIZE=${MASTER_SIZE:-n1-standard-$(get-master-size)}
MASTER_MIN_CPU_ARCHITECTURE=${MASTER_MIN_CPU_ARCHITECTURE:-} # To allow choosing better architectures.
MASTER_DISK_TYPE=pd-ssd
@@ -44,6 +45,7 @@ NODE_LOCAL_SSDS=${NODE_LOCAL_SSDS:-0}
# fluentd is not running as a manifest pod with appropriate label.
# TODO(piosz): remove this in 1.8
NODE_LABELS="${KUBE_NODE_LABELS:-beta.kubernetes.io/fluentd-ds-ready=true}"
WINDOWS_NODE_LABELS="${WINDOWS_NODE_LABELS:-}"
# An extension to local SSDs allowing users to specify block/fs and SCSI/NVMe devices
# Format of this variable will be "#,scsi/nvme,block/fs" you can specify multiple
@@ -63,6 +65,7 @@ MIG_WAIT_UNTIL_STABLE_TIMEOUT=${MIG_WAIT_UNTIL_STABLE_TIMEOUT:-1800}
MASTER_OS_DISTRIBUTION=${KUBE_MASTER_OS_DISTRIBUTION:-${KUBE_OS_DISTRIBUTION:-gci}}
NODE_OS_DISTRIBUTION=${KUBE_NODE_OS_DISTRIBUTION:-${KUBE_OS_DISTRIBUTION:-gci}}
WINDOWS_NODE_OS_DISTRIBUTION=${WINDOWS_NODE_OS_DISTRIBUTION:-win1803}
if [[ "${MASTER_OS_DISTRIBUTION}" == "cos" ]]; then
MASTER_OS_DISTRIBUTION="gci"
@@ -173,15 +176,19 @@ HEAPSTER_MACHINE_TYPE="${HEAPSTER_MACHINE_TYPE:-}"
# NON_MASTER_NODE_LABELS are labels will only be applied on non-master nodes.
NON_MASTER_NODE_LABELS="${KUBE_NON_MASTER_NODE_LABELS:-}"
WINDOWS_NON_MASTER_NODE_LABELS="${WINDOWS_NON_MASTER_NODE_LABELS:-}"
if [[ "${PREEMPTIBLE_MASTER}" == "true" ]]; then
NODE_LABELS="${NODE_LABELS},cloud.google.com/gke-preemptible=true"
WINDOWS_NODE_LABELS="${WINDOWS_NODE_LABELS},cloud.google.com/gke-preemptible=true"
elif [[ "${PREEMPTIBLE_NODE}" == "true" ]]; then
NON_MASTER_NODE_LABELS="${NON_MASTER_NODE_LABELS},cloud.google.com/gke-preemptible=true"
WINDOWS_NON_MASTER_NODE_LABELS="${WINDOWS_NON_MASTER_NODE_LABELS},cloud.google.com/gke-preemptible=true"
fi
# To avoid running Calico on a node that is not configured appropriately,
# label each Node so that the DaemonSet can run the Pods only on ready Nodes.
# Windows nodes do not support Calico.
if [[ ${NETWORK_POLICY_PROVIDER:-} == "calico" ]]; then
NON_MASTER_NODE_LABELS="${NON_MASTER_NODE_LABELS:+${NON_MASTER_NODE_LABELS},}projectcalico.org/ds-ready=true"
fi
@@ -194,6 +201,7 @@ CUSTOM_TYPHA_DEPLOYMENT_YAML="${KUBE_CUSTOM_TYPHA_DEPLOYMENT_YAML:-}"
# To avoid running netd on a node that is not configured appropriately,
# label each Node so that the DaemonSet can run the Pods only on ready Nodes.
# Windows nodes do not support netd.
if [[ ${ENABLE_NETD:-} == "true" ]]; then
NON_MASTER_NODE_LABELS="${NON_MASTER_NODE_LABELS:+${NON_MASTER_NODE_LABELS},}cloud.google.com/gke-netd-ready=true"
fi
@@ -467,3 +475,7 @@ ENABLE_NODE_TERMINATION_HANDLER="${ENABLE_NODE_TERMINATION_HANDLER:-false}"
if [[ "${NODE_TERMINATION_HANDLER_IMAGE:-}" ]]; then
PROVIDER_VARS="${PROVIDER_VARS:-} NODE_TERMINATION_HANDLER_IMAGE"
fi
# Taint Windows nodes by default to prevent Linux workloads from being
# scheduled onto them.
WINDOWS_NODE_TAINTS="${WINDOWS_NODE_TAINTS:-node.kubernetes.io/os=windows:NoSchedule}"

17
cluster/gce/config-test.sh
View File

@@ -29,6 +29,7 @@ RELEASE_REGION_FALLBACK=${RELEASE_REGION_FALLBACK:-false}
REGIONAL_KUBE_ADDONS=${REGIONAL_KUBE_ADDONS:-true}
NODE_SIZE=${NODE_SIZE:-n1-standard-2}
NUM_NODES=${NUM_NODES:-3}
NUM_WINDOWS_NODES=${NUM_WINDOWS_NODES:-0}
MASTER_SIZE=${MASTER_SIZE:-n1-standard-$(get-master-size)}
MASTER_MIN_CPU_ARCHITECTURE=${MASTER_MIN_CPU_ARCHITECTURE:-} # To allow choosing better architectures.
MASTER_DISK_TYPE=pd-ssd
@@ -44,6 +45,7 @@ NODE_LOCAL_SSDS=${NODE_LOCAL_SSDS:-0}
# fluentd is not running as a manifest pod with appropriate label.
# TODO(piosz): remove this in 1.8
NODE_LABELS="${KUBE_NODE_LABELS:-beta.kubernetes.io/fluentd-ds-ready=true}"
WINDOWS_NODE_LABELS="${WINDOWS_NODE_LABELS:-}"
# An extension to local SSDs allowing users to specify block/fs and SCSI/NVMe devices
# Format of this variable will be "#,scsi/nvme,block/fs" you can specify multiple
@@ -66,6 +68,8 @@ MIG_WAIT_UNTIL_STABLE_TIMEOUT=${MIG_WAIT_UNTIL_STABLE_TIMEOUT:-1800}
MASTER_OS_DISTRIBUTION=${KUBE_MASTER_OS_DISTRIBUTION:-${KUBE_OS_DISTRIBUTION:-gci}}
NODE_OS_DISTRIBUTION=${KUBE_NODE_OS_DISTRIBUTION:-${KUBE_OS_DISTRIBUTION:-gci}}
WINDOWS_NODE_OS_DISTRIBUTION=${WINDOWS_NODE_OS_DISTRIBUTION:-win1803}
if [[ "${MASTER_OS_DISTRIBUTION}" == "cos" ]]; then
MASTER_OS_DISTRIBUTION="gci"
fi
@@ -81,7 +85,7 @@ fi
# To avoid failing large tests due to some flakes in starting nodes, allow
# for a small percentage of nodes to not start during cluster startup.
ALLOWED_NOTREADY_NODES="${ALLOWED_NOTREADY_NODES:-$((NUM_NODES / 100))}"
ALLOWED_NOTREADY_NODES="${ALLOWED_NOTREADY_NODES:-$(($(get-num-nodes) / 100))}"
# By default a cluster will be started with the master and nodes
# on Container-optimized OS (cos, previously known as gci). If
@@ -208,18 +212,21 @@ fi
if [[ "${MASTER_OS_DISTRIBUTION}" == "gci" ]] || [[ "${MASTER_OS_DISTRIBUTION}" == "ubuntu" ]]; then
MASTER_KUBELET_TEST_ARGS="${MASTER_KUBELET_TEST_ARGS:-} --experimental-kernel-memcg-notification=true"
fi
APISERVER_TEST_ARGS="${APISERVER_TEST_ARGS:-} --vmodule=httplog=3 --runtime-config=extensions/v1beta1,scheduling.k8s.io/v1alpha1,settings.k8s.io/v1alpha1 ${TEST_CLUSTER_DELETE_COLLECTION_WORKERS} ${TEST_CLUSTER_MAX_REQUESTS_INFLIGHT}"
APISERVER_TEST_ARGS="${APISERVER_TEST_ARGS:-} --runtime-config=extensions/v1beta1,scheduling.k8s.io/v1alpha1,settings.k8s.io/v1alpha1 ${TEST_CLUSTER_DELETE_COLLECTION_WORKERS} ${TEST_CLUSTER_MAX_REQUESTS_INFLIGHT}"
CONTROLLER_MANAGER_TEST_ARGS="${CONTROLLER_MANAGER_TEST_ARGS:-} ${TEST_CLUSTER_RESYNC_PERIOD} ${TEST_CLUSTER_API_CONTENT_TYPE}"
SCHEDULER_TEST_ARGS="${SCHEDULER_TEST_ARGS:-} ${TEST_CLUSTER_API_CONTENT_TYPE}"
KUBEPROXY_TEST_ARGS="${KUBEPROXY_TEST_ARGS:-} ${TEST_CLUSTER_API_CONTENT_TYPE}"
# NON_MASTER_NODE_LABELS are labels will only be applied on non-master nodes.
NON_MASTER_NODE_LABELS="${KUBE_NON_MASTER_NODE_LABELS:-}"
WINDOWS_NON_MASTER_NODE_LABELS="${WINDOWS_NON_MASTER_NODE_LABELS:-}"
if [[ "${PREEMPTIBLE_MASTER}" == "true" ]]; then
NODE_LABELS="${NODE_LABELS},cloud.google.com/gke-preemptible=true"
WINDOWS_NODE_LABELS="${WINDOWS_NODE_LABELS},cloud.google.com/gke-preemptible=true"
elif [[ "${PREEMPTIBLE_NODE}" == "true" ]]; then
NON_MASTER_NODE_LABELS="${NON_MASTER_NODE_LABELS},cloud.google.com/gke-preemptible=true"
WINDOWS_NON_MASTER_NODE_LABELS="${WINDOWS_NON_MASTER_NODE_LABELS},cloud.google.com/gke-preemptible=true"
fi
# Optional: Enable netd.
@@ -230,6 +237,7 @@ CUSTOM_TYPHA_DEPLOYMENT_YAML="${KUBE_CUSTOM_TYPHA_DEPLOYMENT_YAML:-}"
# To avoid running netd on a node that is not configured appropriately,
# label each Node so that the DaemonSet can run the Pods only on ready Nodes.
# Windows nodes do not support netd.
if [[ ${ENABLE_NETD:-} == "true" ]]; then
NON_MASTER_NODE_LABELS="${NON_MASTER_NODE_LABELS:+${NON_MASTER_NODE_LABELS},}cloud.google.com/gke-netd-ready=true"
fi
@@ -238,6 +246,7 @@ ENABLE_NODELOCAL_DNS="${KUBE_ENABLE_NODELOCAL_DNS:-false}"
# To avoid running Calico on a node that is not configured appropriately,
# label each Node so that the DaemonSet can run the Pods only on ready Nodes.
# Windows nodes do not support Calico.
if [[ ${NETWORK_POLICY_PROVIDER:-} == "calico" ]]; then
NON_MASTER_NODE_LABELS="${NON_MASTER_NODE_LABELS:+${NON_MASTER_NODE_LABELS},}projectcalico.org/ds-ready=true"
fi
@@ -486,3 +495,7 @@ ENABLE_NODE_TERMINATION_HANDLER="${ENABLE_NODE_TERMINATION_HANDLER:-false}"
if [[ "${NODE_TERMINATION_HANDLER_IMAGE:-}" ]]; then
PROVIDER_VARS="${PROVIDER_VARS:-} NODE_TERMINATION_HANDLER_IMAGE"
fi
# Taint Windows nodes by default to prevent Linux workloads from being
# scheduled onto them.
WINDOWS_NODE_TAINTS="${WINDOWS_NODE_TAINTS:-node.kubernetes.io/os=windows:NoSchedule}"

6
cluster/gce/gci/node-helper.sh
View File

@@ -17,7 +17,7 @@
# A library of helper functions and constant for GCI distro
source "${KUBE_ROOT}/cluster/gce/gci/helper.sh"
function get-node-instance-metadata {
function get-node-instance-metadata-from-file {
local metadata=""
metadata+="kube-env=${KUBE_TEMP}/node-kube-env.yaml,"
metadata+="kubelet-config=${KUBE_TEMP}/node-kubelet-config.yaml,"
@@ -34,8 +34,8 @@ function get-node-instance-metadata {
}
# $1: template name (required).
function create-node-instance-template {
function create-linux-node-instance-template {
local template_name="$1"
ensure-gci-metadata-files
create-node-template "$template_name" "${scope_flags[*]}" "$(get-node-instance-metadata)"
create-node-template "${template_name}" "${scope_flags[*]}" "$(get-node-instance-metadata-from-file)" "" "linux"
}

511
cluster/gce/util.sh
View File

@@ -32,6 +32,8 @@ else
exit 1
fi
source "${KUBE_ROOT}/cluster/gce/${WINDOWS_NODE_OS_DISTRIBUTION}/node-helper.sh"
if [[ "${MASTER_OS_DISTRIBUTION}" == "trusty" || "${MASTER_OS_DISTRIBUTION}" == "gci" || "${MASTER_OS_DISTRIBUTION}" == "ubuntu" ]]; then
source "${KUBE_ROOT}/cluster/gce/${MASTER_OS_DISTRIBUTION}/master-helper.sh"
else
@@ -57,23 +59,50 @@ fi
# Sets node image based on the specified os distro. Currently this function only
# supports gci and debian.
function set-node-image() {
if [[ "${NODE_OS_DISTRIBUTION}" == "gci" ]]; then
DEFAULT_GCI_PROJECT=google-containers
if [[ "${GCI_VERSION}" == "cos"* ]]; then
DEFAULT_GCI_PROJECT=cos-cloud
fi
# If the node image is not set, we use the latest GCI image.
# Otherwise, we respect whatever is set by the user.
NODE_IMAGE=${KUBE_GCE_NODE_IMAGE:-${GCI_VERSION}}
NODE_IMAGE_PROJECT=${KUBE_GCE_NODE_PROJECT:-${DEFAULT_GCI_PROJECT}}
#
# Requires:
# NODE_OS_DISTRIBUTION
# Sets:
# DEFAULT_GCI_PROJECT
# NODE_IMAGE
# NODE_IMAGE_PROJECT
function set-linux-node-image() {
if [[ "${NODE_OS_DISTRIBUTION}" == "gci" ]]; then
DEFAULT_GCI_PROJECT=google-containers
if [[ "${GCI_VERSION}" == "cos"* ]]; then
DEFAULT_GCI_PROJECT=cos-cloud
fi
# If the node image is not set, we use the latest GCI image.
# Otherwise, we respect whatever is set by the user.
NODE_IMAGE=${KUBE_GCE_NODE_IMAGE:-${GCI_VERSION}}
NODE_IMAGE_PROJECT=${KUBE_GCE_NODE_PROJECT:-${DEFAULT_GCI_PROJECT}}
fi
}
set-node-image
# Requires:
# WINDOWS_NODE_OS_DISTRIBUTION
# Sets:
# WINDOWS_NODE_IMAGE_FAMILY
# WINDOWS_NODE_IMAGE_PROJECT
function set-windows-node-image() {
WINDOWS_NODE_IMAGE_PROJECT="windows-cloud"
if [[ "${WINDOWS_NODE_OS_DISTRIBUTION}" == "win1803" ]]; then
WINDOWS_NODE_IMAGE_FAMILY="windows-1803-core-for-containers"
elif [[ "${WINDOWS_NODE_OS_DISTRIBUTION}" == "win2019" ]]; then
WINDOWS_NODE_IMAGE_FAMILY="windows-2019-core-for-containers"
elif [[ "${WINDOWS_NODE_OS_DISTRIBUTION}" == "win1809" ]]; then
WINDOWS_NODE_IMAGE_FAMILY="windows-1809-core-for-containers"
else
echo "Unknown WINDOWS_NODE_OS_DISTRIBUTION ${WINDOWS_NODE_OS_DISTRIBUTION}" >&2
exit 1
fi
}
# Verfiy cluster autoscaler configuration.
set-linux-node-image
set-windows-node-image
# Verify cluster autoscaler configuration.
if [[ "${ENABLE_CLUSTER_AUTOSCALER}" == "true" ]]; then
if [[ -z $AUTOSCALER_MIN_NODES ]]; then
echo "AUTOSCALER_MIN_NODES not set."
@@ -342,7 +371,7 @@ function upload-tars() {
fi
}
# Detect minions created in the minion group
# Detect Linux and Windows nodes created in the instance group.
#
# Assumed vars:
# NODE_INSTANCE_PREFIX
@@ -535,23 +564,29 @@ function write-master-env {
KUBERNETES_MASTER_NAME="${MASTER_NAME}"
fi
construct-kubelet-flags true
build-kube-env true "${KUBE_TEMP}/master-kube-env.yaml"
build-kubelet-config true "${KUBE_TEMP}/master-kubelet-config.yaml"
construct-linux-kubelet-flags true
build-linux-kube-env true "${KUBE_TEMP}/master-kube-env.yaml"
build-kubelet-config true "linux" "${KUBE_TEMP}/master-kubelet-config.yaml"
build-kube-master-certs "${KUBE_TEMP}/kube-master-certs.yaml"
}
function write-node-env {
function write-linux-node-env {
if [[ -z "${KUBERNETES_MASTER_NAME:-}" ]]; then
KUBERNETES_MASTER_NAME="${MASTER_NAME}"
fi
construct-kubelet-flags false
build-kube-env false "${KUBE_TEMP}/node-kube-env.yaml"
build-kubelet-config false "${KUBE_TEMP}/node-kubelet-config.yaml"
construct-linux-kubelet-flags false
build-linux-kube-env false "${KUBE_TEMP}/node-kube-env.yaml"
build-kubelet-config false "linux" "${KUBE_TEMP}/node-kubelet-config.yaml"
}
function build-node-labels {
function write-windows-node-env {
construct-windows-kubelet-flags
build-windows-kube-env "${KUBE_TEMP}/windows-node-kube-env.yaml"
build-kubelet-config false "windows" "${KUBE_TEMP}/windows-node-kubelet-config.yaml"
}
function build-linux-node-labels {
local master=$1
local node_labels=""
if [[ "${KUBE_PROXY_DAEMONSET:-}" == "true" && "${master}" != "true" ]]; then
@@ -568,6 +603,17 @@ function build-node-labels {
echo $node_labels
}
function build-windows-node-labels {
local node_labels=""
if [[ -n "${WINDOWS_NODE_LABELS:-}" ]]; then
node_labels="${node_labels:+${node_labels},}${WINDOWS_NODE_LABELS}"
fi
if [[ -n "${WINDOWS_NON_MASTER_NODE_LABELS:-}" ]]; then
node_labels="${node_labels:+${node_labels},}${WINDOWS_NON_MASTER_NODE_LABELS}"
fi
echo $node_labels
}
# yaml-map-string-stringarray converts the encoded structure to yaml format, and echoes the result
# under the provided name. If the encoded structure is empty, echoes nothing.
# 1: name to be output in yaml
@@ -645,12 +691,26 @@ function yaml-map-string-string {
fi
}
# $1: if 'true', we're rendering flags for a master, else a node
function construct-kubelet-flags {
local master=$1
# Returns kubelet flags used on both Linux and Windows nodes.
function construct-common-kubelet-flags {
local flags="${KUBELET_TEST_LOG_LEVEL:-"--v=2"} ${KUBELET_TEST_ARGS:-}"
flags+=" --allow-privileged=true"
flags+=" --cloud-provider=gce"
# TODO(mtaufen): ROTATE_CERTIFICATES seems unused; delete it?
if [[ -n "${ROTATE_CERTIFICATES:-}" ]]; then
flags+=" --rotate-certificates=true"
fi
if [[ -n "${MAX_PODS_PER_NODE:-}" ]]; then
flags+=" --max-pods=${MAX_PODS_PER_NODE}"
fi
echo $flags
}
# Sets KUBELET_ARGS with the kubelet flags for Linux nodes.
# $1: if 'true', we're rendering flags for a master, else a node
function construct-linux-kubelet-flags {
local master="$1"
local flags="$(construct-common-kubelet-flags)"
flags+=" --allow-privileged=true"
# Keep in sync with CONTAINERIZED_MOUNTER_HOME in configure-helper.sh
flags+=" --experimental-mounter-path=/home/kubernetes/containerized_mounter/mounter"
flags+=" --experimental-check-node-capabilities-before-mount=true"
@@ -695,60 +755,175 @@ function construct-kubelet-flags {
flags+=" --non-masquerade-cidr=${NON_MASQUERADE_CIDR}"
fi
flags+=" --volume-plugin-dir=${VOLUME_PLUGIN_DIR}"
local node_labels=$(build-node-labels ${master})
local node_labels="$(build-linux-node-labels ${master})"
if [[ -n "${node_labels:-}" ]]; then
flags+=" --node-labels=${node_labels}"
fi
if [[ -n "${NODE_TAINTS:-}" ]]; then
flags+=" --register-with-taints=${NODE_TAINTS}"
fi
# TODO(mtaufen): ROTATE_CERTIFICATES seems unused; delete it?
if [[ -n "${ROTATE_CERTIFICATES:-}" ]]; then
flags+=" --rotate-certificates=true"
fi
if [[ -n "${CONTAINER_RUNTIME:-}" ]]; then
flags+=" --container-runtime=${CONTAINER_RUNTIME}"
fi
if [[ -n "${CONTAINER_RUNTIME_ENDPOINT:-}" ]]; then
flags+=" --container-runtime-endpoint=${CONTAINER_RUNTIME_ENDPOINT}"
fi
if [[ -n "${MAX_PODS_PER_NODE:-}" ]]; then
flags+=" --max-pods=${MAX_PODS_PER_NODE}"
KUBELET_ARGS="${flags}"
}
# Sets KUBELET_ARGS with the kubelet flags for Windows nodes.
function construct-windows-kubelet-flags {
local flags="$(construct-common-kubelet-flags)"
# Note: NODE_KUBELET_TEST_ARGS is empty in typical kube-up runs.
flags+=" ${NODE_KUBELET_TEST_ARGS:-}"
local node_labels="$(build-windows-node-labels)"
if [[ -n "${node_labels:-}" ]]; then
flags+=" --node-labels=${node_labels}"
fi
# Concatenate common and windows-only node taints and apply them.
local node_taints="${NODE_TAINTS:-}"
if [[ -n "${node_taints}" && -n "${WINDOWS_NODE_TAINTS:-}" ]]; then
node_taints+=":${WINDOWS_NODE_TAINTS}"
else
node_taints="${WINDOWS_NODE_TAINTS:-}"
fi
if [[ -n "${node_taints}" ]]; then
flags+=" --register-with-taints=${node_taints}"
fi
# Many of these flags were adapted from
# https://github.com/Microsoft/SDN/blob/master/Kubernetes/windows/start-kubelet.ps1.
flags+=" --config=${WINDOWS_KUBELET_CONFIG_FILE}"
# Path to a kubeconfig file that will be used to get client certificate for
# kubelet. If the file specified by --kubeconfig does not exist, the bootstrap
# kubeconfig is used to request a client certificate from the API server. On
# success, a kubeconfig file referencing the generated client certificate and
# key is written to the path specified by --kubeconfig. The client certificate
# and key file will be stored in the directory pointed by --cert-dir.
#
# See also:
# https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/
flags+=" --bootstrap-kubeconfig=${WINDOWS_BOOTSTRAP_KUBECONFIG_FILE}"
flags+=" --kubeconfig=${WINDOWS_KUBECONFIG_FILE}"
# The directory where the TLS certs are located.
flags+=" --cert-dir=${WINDOWS_PKI_DIR}"
flags+=" --network-plugin=cni"
flags+=" --cni-bin-dir=${WINDOWS_CNI_DIR}"
flags+=" --cni-conf-dir=${WINDOWS_CNI_CONFIG_DIR}"
flags+=" --pod-manifest-path=${WINDOWS_MANIFESTS_DIR}"
# Windows images are large and we don't have gcr mirrors yet. Allow longer
# pull progress deadline.
flags+=" --image-pull-progress-deadline=5m"
flags+=" --enable-debugging-handlers=true"
# Configure kubelet to run as a windows service.
flags+=" --windows-service=true"
# TODO(mtaufen): Configure logging for kubelet running as a service. I haven't
# been able to figure out how to direct stdout/stderr into log files when
# configuring it to run via sc.exe, so we just manually override logging
# config here.
flags+=" --log-file=${WINDOWS_LOGS_DIR}\kubelet.log"
# klog sets this to true internally, so need to override to false so we
# actually log to the file
flags+=" --logtostderr=false"
# Configure flags with explicit empty string values. We can't escape
# double-quotes, because they still break sc.exe after expansion in the
# binPath parameter, and single-quotes get parsed as characters instead of
# string delimiters.
flags+=" --resolv-conf="
# Both --cgroups-per-qos and --enforce-node-allocatable should be disabled on
# windows; the latter requires the former to be enabled to work.
flags+=" --cgroups-per-qos=false --enforce-node-allocatable="
# Turn off kernel memory cgroup notification.
flags+=" --experimental-kernel-memcg-notification=false"
KUBELET_ARGS="${flags}"
}
# $1: if 'true', we're rendering config for a master, else a node
function build-kubelet-config {
local master=$1
local file=$2
local master="$1"
local os="$2"
local file="$3"
rm -f "${file}"
{
declare quoted_dns_server_ip
declare quoted_dns_domain
quoted_dns_server_ip=$(yaml-quote "${DNS_SERVER_IP}")
if [[ "${ENABLE_NODELOCAL_DNS:-}" == "true" ]]; then
quoted_dns_server_ip=$(yaml-quote "${LOCAL_DNS_IP}")
print-common-kubelet-config
if [[ "${master}" == "true" ]]; then
print-master-kubelet-config
else
print-common-node-kubelet-config
if [[ "${os}" == "linux" ]]; then
print-linux-node-kubelet-config
elif [[ "${os}" == "windows" ]]; then
print-windows-node-kubelet-config
else
echo "Unknown OS ${os}" >&2
exit 1
fi
fi
quoted_dns_domain=$(yaml-quote "${DNS_DOMAIN}")
cat <<EOF
} > "${file}"
}
# cat the Kubelet config yaml in common between masters, linux nodes, and
# windows nodes
function print-common-kubelet-config {
declare quoted_dns_server_ip
declare quoted_dns_domain
quoted_dns_server_ip=$(yaml-quote "${DNS_SERVER_IP}")
if [[ "${ENABLE_NODELOCAL_DNS:-}" == "true" ]]; then
quoted_dns_server_ip=$(yaml-quote "${LOCAL_DNS_IP}")
fi
quoted_dns_domain=$(yaml-quote "${DNS_DOMAIN}")
cat <<EOF
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupRoot: /
clusterDNS:
- ${quoted_dns_server_ip}
clusterDomain: ${quoted_dns_domain}
staticPodPath: /etc/kubernetes/manifests
readOnlyPort: 10255
EOF
# --- begin master-specific config ---
if [[ "${master}" == "true" ]]; then
cat <<EOF
# Note: ENABLE_MANIFEST_URL is used by GKE.
# TODO(mtaufen): remove this since it's not used in kubernetes/kubernetes nor
# kubernetes/test-infra.
if [[ "${ENABLE_MANIFEST_URL:-}" == "true" ]]; then
declare quoted_manifest_url
quoted_manifest_url=$(yaml-quote "${MANIFEST_URL}")
cat <<EOF
staticPodURL: ${quoted_manifest_url}
EOF
yaml-map-string-stringarray 'staticPodURLHeader' "${MANIFEST_URL_HEADER}"
fi
if [[ -n "${EVICTION_HARD:-}" ]]; then
yaml-map-string-string 'evictionHard' "${EVICTION_HARD}" true '<'
fi
if [[ -n "${FEATURE_GATES:-}" ]]; then
yaml-map-string-string 'featureGates' "${FEATURE_GATES}" false '='
fi
}
# cat the Kubelet config yaml for masters
function print-master-kubelet-config {
cat <<EOF
enableDebuggingHandlers: false
hairpinMode: none
staticPodPath: /etc/kubernetes/manifests
authentication:
webhook:
enabled: false
@@ -757,54 +932,65 @@ authentication:
authorization:
mode: AlwaysAllow
EOF
if [[ "${REGISTER_MASTER_KUBELET:-false}" == "false" ]]; then
# Note: Standalone mode is used by GKE
declare quoted_master_ip_range
quoted_master_ip_range=$(yaml-quote "${MASTER_IP_RANGE}")
cat <<EOF
if [[ "${REGISTER_MASTER_KUBELET:-false}" == "false" ]]; then
# Note: Standalone mode is used by GKE
declare quoted_master_ip_range
quoted_master_ip_range=$(yaml-quote "${MASTER_IP_RANGE}")
cat <<EOF
podCidr: ${quoted_master_ip_range}
EOF
fi
# --- end master-specific config ---
else
# --- begin node-specific config ---
# Keep authentication.x509.clientCAFile in sync with CA_CERT_BUNDLE_PATH in configure-helper.sh
cat <<EOF
fi
}
# cat the Kubelet config yaml in common between linux nodes and windows nodes
function print-common-node-kubelet-config {
cat <<EOF
enableDebuggingHandlers: true
EOF
if [[ "${HAIRPIN_MODE:-}" == "promiscuous-bridge" ]] || \
[[ "${HAIRPIN_MODE:-}" == "hairpin-veth" ]] || \
[[ "${HAIRPIN_MODE:-}" == "none" ]]; then
declare quoted_hairpin_mode
quoted_hairpin_mode=$(yaml-quote "${HAIRPIN_MODE}")
cat <<EOF
hairpinMode: ${quoted_hairpin_mode}
EOF
fi
}
# cat the Kubelet config yaml for linux nodes
function print-linux-node-kubelet-config {
# Keep authentication.x509.clientCAFile in sync with CA_CERT_BUNDLE_PATH in configure-helper.sh
cat <<EOF
staticPodPath: /etc/kubernetes/manifests
authentication:
x509:
clientCAFile: /etc/srv/kubernetes/pki/ca-certificates.crt
EOF
if [[ "${HAIRPIN_MODE:-}" == "promiscuous-bridge" ]] || \
[[ "${HAIRPIN_MODE:-}" == "hairpin-veth" ]] || \
[[ "${HAIRPIN_MODE:-}" == "none" ]]; then
declare quoted_hairpin_mode
quoted_hairpin_mode=$(yaml-quote "${HAIRPIN_MODE}")
cat <<EOF
hairpinMode: ${quoted_hairpin_mode}
}
# cat the Kubelet config yaml for windows nodes
function print-windows-node-kubelet-config {
# Notes:
# - We don't run any static pods on Windows nodes yet.
# TODO(mtaufen): Does it make any sense to set eviction thresholds for inodes
# on Windows?
# TODO(pjh, mtaufen): It may make sense to use a different hairpin mode on
# Windows. We're currently using hairpin-veth, but
# https://github.com/Microsoft/SDN/blob/master/Kubernetes/windows/start-kubelet.ps1#L121
# uses promiscuous-bridge.
# TODO(pjh, mtaufen): Does cgroupRoot make sense for Windows?
# Keep authentication.x509.clientCAFile in sync with CA_CERT_BUNDLE_PATH in
# k8s-node-setup.psm1.
cat <<EOF
authentication:
x509:
clientCAFile: '${WINDOWS_PKI_DIR}\ca-certificates.crt'
EOF
fi
# --- end node-specific config ---
fi
# Note: ENABLE_MANIFEST_URL is used by GKE
if [[ "${ENABLE_MANIFEST_URL:-}" == "true" ]]; then
declare quoted_manifest_url
quoted_manifest_url=$(yaml-quote "${MANIFEST_URL}")
cat <<EOF
staticPodURL: ${quoted_manifest_url}
EOF
yaml-map-string-stringarray 'staticPodURLHeader' "${MANIFEST_URL_HEADER}"
fi
if [[ -n "${EVICTION_HARD:-}" ]]; then
yaml-map-string-string 'evictionHard' "${EVICTION_HARD}" true '<'
fi
if [[ -n "${FEATURE_GATES:-}" ]]; then
yaml-map-string-string 'featureGates' "${FEATURE_GATES}" false '='
fi
} > "${file}"
}
function build-kube-master-certs {
@@ -828,9 +1014,9 @@ EOF
}
# $1: if 'true', we're building a master yaml, else a node
function build-kube-env {
local master=$1
local file=$2
function build-linux-kube-env {
local master="$1"
local file="$2"
local server_binary_tar_url=$SERVER_BINARY_TAR_URL
local kube_manifests_tar_url="${KUBE_MANIFESTS_TAR_URL:-}"
@@ -1187,7 +1373,7 @@ EOF
# TODO(kubernetes/autoscaler#718): AUTOSCALER_ENV_VARS is a hotfix for cluster autoscaler,
# which reads the kube-env to determine the shape of a node and was broken by #60020.
# This should be removed as soon as a more reliable source of information is available!
local node_labels=$(build-node-labels false)
local node_labels="$(build-linux-node-labels false)"
local node_taints="${NODE_TAINTS:-}"
local autoscaler_env_vars="node_labels=${node_labels};node_taints=${node_taints}"
cat >>$file <<EOF
@@ -1207,6 +1393,29 @@ EOF
fi
}
function build-windows-kube-env {
local file="$1"
# For now the Windows kube-env is a superset of the Linux kube-env.
build-linux-kube-env false $file
cat >>$file <<EOF
NODE_BINARY_TAR_URL: $(yaml-quote ${NODE_BINARY_TAR_URL})
NODE_BINARY_TAR_HASH: $(yaml-quote ${NODE_BINARY_TAR_HASH})
K8S_DIR: $(yaml-quote ${WINDOWS_K8S_DIR})
NODE_DIR: $(yaml-quote ${WINDOWS_NODE_DIR})
LOGS_DIR: $(yaml-quote ${WINDOWS_LOGS_DIR})
CNI_DIR: $(yaml-quote ${WINDOWS_CNI_DIR})
CNI_CONFIG_DIR: $(yaml-quote ${WINDOWS_CNI_CONFIG_DIR})
MANIFESTS_DIR: $(yaml-quote ${WINDOWS_MANIFESTS_DIR})
PKI_DIR: $(yaml-quote ${WINDOWS_PKI_DIR})
KUBELET_CONFIG_FILE: $(yaml-quote ${WINDOWS_KUBELET_CONFIG_FILE})
KUBECONFIG_FILE: $(yaml-quote ${WINDOWS_KUBECONFIG_FILE})
BOOTSTRAP_KUBECONFIG_FILE: $(yaml-quote ${WINDOWS_BOOTSTRAP_KUBECONFIG_FILE})
KUBEPROXY_KUBECONFIG_FILE: $(yaml-quote ${WINDOWS_KUBEPROXY_KUBECONFIG_FILE})
EOF
}
function sha1sum-file() {
if which sha1sum >/dev/null 2>&1; then
sha1sum "$1" | awk '{ print $1 }'
@@ -1521,6 +1730,7 @@ for c in required:
if missing:
for c in missing:
print ("missing required gcloud component \"{0}\"".format(c))
print ("Try running `gcloud components install {0}`".format(c))
exit(1)
' """${version}"""
fi
@@ -1670,19 +1880,23 @@ function validate-node-local-ssds-ext(){
# Robustly try to create an instance template.
# $1: The name of the instance template.
# $2: The scopes flag.
# $3: String of comma-separated metadata entries (must all be from a file).
# $3: String of comma-separated metadata-from-file entries.
# $4: String of comma-separated metadata (key=value) entries.
# $5: the node OS ("linux" or "windows").
function create-node-template() {
detect-project
detect-subnetworks
local template_name="$1"
local metadata_values="$4"
local os="$5"
# First, ensure the template doesn't exist.
# TODO(zmerlynn): To make this really robust, we need to parse the output and
# add retries. Just relying on a non-zero exit code doesn't
# distinguish an ephemeral failed call from a "not-exists".
if gcloud compute instance-templates describe "$template_name" --project "${PROJECT}" &>/dev/null; then
if gcloud compute instance-templates describe "${template_name}" --project "${PROJECT}" &>/dev/null; then
echo "Instance template ${1} already exists; deleting." >&2
if ! gcloud compute instance-templates delete "$template_name" --project "${PROJECT}" --quiet &>/dev/null; then
if ! gcloud compute instance-templates delete "${template_name}" --project "${PROJECT}" --quiet &>/dev/null; then
echo -e "${color_yellow}Failed to delete existing instance template${color_norm}" >&2
exit 2
fi
@@ -1737,17 +1951,28 @@ function create-node-template() {
"${ENABLE_IP_ALIASES:-}" \
"${IP_ALIAS_SIZE:-}")
local node_image_flags=""
if [[ "${os}" == 'linux' ]]; then
node_image_flags="--image-project ${NODE_IMAGE_PROJECT} --image ${NODE_IMAGE}"
elif [[ "${os}" == 'windows' ]]; then
node_image_flags="--image-project ${WINDOWS_NODE_IMAGE_PROJECT} --image-family ${WINDOWS_NODE_IMAGE_FAMILY}"
else
echo "Unknown OS ${os}" >&2
exit 1
fi
local metadata_flag="${metadata_values:+--metadata ${metadata_values}}"
local attempt=1
while true; do
echo "Attempt ${attempt} to create ${1}" >&2
if ! ${gcloud} compute instance-templates create \
"$template_name" \
"${template_name}" \
--project "${PROJECT}" \
--machine-type "${NODE_SIZE}" \
--boot-disk-type "${NODE_DISK_TYPE}" \
--boot-disk-size "${NODE_DISK_SIZE}" \
--image-project="${NODE_IMAGE_PROJECT}" \
--image "${NODE_IMAGE}" \
${node_image_flags} \
--service-account "${NODE_SERVICE_ACCOUNT}" \
--tags "${NODE_TAG}" \
${accelerator_args} \
@@ -1756,19 +1981,20 @@ function create-node-template() {
${network} \
${preemptible_minions} \
$2 \
--metadata-from-file $3 >&2; then
--metadata-from-file $3 \
${metadata_flag} >&2; then
if (( attempt > 5 )); then
echo -e "${color_red}Failed to create instance template $template_name ${color_norm}" >&2
echo -e "${color_red}Failed to create instance template ${template_name} ${color_norm}" >&2
exit 2
fi
echo -e "${color_yellow}Attempt ${attempt} failed to create instance template $template_name. Retrying.${color_norm}" >&2
echo -e "${color_yellow}Attempt ${attempt} failed to create instance template ${template_name}. Retrying.${color_norm}" >&2
attempt=$(($attempt+1))
sleep $(($attempt * 5))
# In case the previous attempt failed with something like a
# Backend Error and left the entry laying around, delete it
# before we try again.
gcloud compute instance-templates delete "$template_name" --project "${PROJECT}" &>/dev/null || true
gcloud compute instance-templates delete "${template_name}" --project "${PROJECT}" &>/dev/null || true
else
break
fi
@@ -1799,7 +2025,9 @@ function kube-up() {
parse-master-env
create-subnetworks
detect-subnetworks
create-nodes
# Windows nodes take longer to boot and setup so create them first.
create-windows-nodes
create-linux-nodes
elif [[ ${KUBE_REPLICATE_EXISTING_MASTER:-} == "true" ]]; then
if [[ "${MASTER_OS_DISTRIBUTION}" != "gci" && "${MASTER_OS_DISTRIBUTION}" != "ubuntu" ]]; then
echo "Master replication supported only for gci and ubuntu"
@@ -1822,7 +2050,9 @@ function kube-up() {
create-master
create-nodes-firewall
create-nodes-template
create-nodes
# Windows nodes take longer to boot and setup so create them first.
create-windows-nodes
create-linux-nodes
check-cluster
fi
}
@@ -1897,6 +2127,17 @@ function create-network() {
--source-ranges "0.0.0.0/0" \
--allow "tcp:22" &
fi
# Open up TCP 3389 to allow RDP connections.
if [[ ${NUM_WINDOWS_NODES} -gt 0 ]]; then
if ! gcloud compute firewall-rules describe --project "${NETWORK_PROJECT}" "${NETWORK}-default-rdp" &>/dev/null; then
gcloud compute firewall-rules create "${NETWORK}-default-rdp" \
--project "${NETWORK_PROJECT}" \
--network "${NETWORK}" \
--source-ranges "0.0.0.0/0" \
--allow "tcp:3389" &
fi
fi
}
function expand-default-subnetwork() {
@@ -2187,7 +2428,7 @@ function create-master() {
create-etcd-certs ${MASTER_NAME}
create-etcd-apiserver-certs "etcd-${MASTER_NAME}" ${MASTER_NAME}
if [[ "${NUM_NODES}" -ge "50" ]]; then
if [[ "$(get-num-nodes)" -ge "50" ]]; then
# We block on master creation for large clusters to avoid doing too much
# unnecessary work in case master start-up fails (like creation of nodes).
create-master-instance "${MASTER_RESERVED_IP}"
@@ -2377,17 +2618,25 @@ function create-nodes-template() {
local scope_flags=$(get-scope-flags)
write-node-env
write-linux-node-env
write-windows-node-env
local template_name="${NODE_INSTANCE_PREFIX}-template"
create-node-instance-template $template_name
# NOTE: these template names and their format must match
# create-[linux,windows]-nodes() as well as get-template()!
# TODO(pjh): find a better way to manage these (get-template() is annoying).
local linux_template_name="${NODE_INSTANCE_PREFIX}-template"
local windows_template_name="${NODE_INSTANCE_PREFIX}-template-windows"
create-linux-node-instance-template $linux_template_name
create-windows-node-instance-template $windows_template_name "${scope_flags[*]}"
}
# Assumes:
# - MAX_INSTANCES_PER_MIG
# - NUM_NODES
# - NUM_WINDOWS_NODES
# exports:
# - NUM_MIGS
# - NUM_WINDOWS_MIGS
function set_num_migs() {
local defaulted_max_instances_per_mig=${MAX_INSTANCES_PER_MIG:-1000}
@@ -2396,6 +2645,7 @@ function set_num_migs() {
defaulted_max_instances_per_mig=1000
fi
export NUM_MIGS=$(((${NUM_NODES} + ${defaulted_max_instances_per_mig} - 1) / ${defaulted_max_instances_per_mig}))
export NUM_WINDOWS_MIGS=$(((${NUM_WINDOWS_NODES} + ${defaulted_max_instances_per_mig} - 1) / ${defaulted_max_instances_per_mig}))
}
# Assumes:
@@ -2404,7 +2654,7 @@ function set_num_migs() {
# - NUM_NODES
# - PROJECT
# - ZONE
function create-nodes() {
function create-linux-nodes() {
local template_name="${NODE_INSTANCE_PREFIX}-template"
if [[ -z "${HEAPSTER_MACHINE_TYPE:-}" ]]; then
@@ -2434,7 +2684,7 @@ function create-nodes() {
--zone "${ZONE}" \
--base-instance-name "${group_name}" \
--size "${this_mig_size}" \
--template "$template_name" || true;
--template "${template_name}" || true;
gcloud compute instance-groups managed wait-until-stable \
"${group_name}" \
--zone "${ZONE}" \
@@ -2444,6 +2694,44 @@ function create-nodes() {
wait
}
# Assumes:
# - NUM_WINDOWS_MIGS
# - NODE_INSTANCE_PREFIX
# - NUM_WINDOWS_NODES
# - PROJECT
# - ZONE
function create-windows-nodes() {
local template_name="${NODE_INSTANCE_PREFIX}-template-windows"
local -r nodes="${NUM_WINDOWS_NODES}"
local instances_left=${nodes}
for ((i=1; i<=${NUM_WINDOWS_MIGS}; i++)); do
local group_name="${NODE_INSTANCE_PREFIX}-windows-group-$i"
if [[ $i == ${NUM_WINDOWS_MIGS} ]]; then
# TODO: We don't add a suffix for the last group to keep backward compatibility when there's only one MIG.
# We should change it at some point, but note #18545 when changing this.
group_name="${NODE_INSTANCE_PREFIX}-windows-group"
fi
# Spread the remaining number of nodes evenly
this_mig_size=$((${instances_left} / (${NUM_WINDOWS_MIGS}-${i}+1)))
instances_left=$((instances_left-${this_mig_size}))
gcloud compute instance-groups managed \
create "${group_name}" \
--project "${PROJECT}" \
--zone "${ZONE}" \
--base-instance-name "${group_name}" \
--size "${this_mig_size}" \
--template "${template_name}" || true;
gcloud compute instance-groups managed wait-until-stable \
"${group_name}" \
--zone "${ZONE}" \
--project "${PROJECT}" \
--timeout "${MIG_WAIT_UNTIL_STABLE_TIMEOUT}" || true;
done
}
# Assumes:
# - NODE_INSTANCE_PREFIX
# - PROJECT
@@ -2486,7 +2774,7 @@ function create-heapster-node() {
--tags "${NODE_TAG}" \
${network} \
$(get-scope-flags) \
--metadata-from-file "$(get-node-instance-metadata)"
--metadata-from-file "$(get-node-instance-metadata-from-file)"
}
# Assumes:
@@ -2513,6 +2801,11 @@ function create-cluster-autoscaler-mig-config() {
echo "AUTOSCALER_MAX_NODES must be greater or equal ${NUM_MIGS}"
exit 2
fi
if [[ ${NUM_WINDOWS_MIGS} -gt 0 ]]; then
# TODO(pjh): implement Windows support in this function.
echo "Not implemented yet: autoscaler config for Windows MIGs"
exit 2
fi
# The code assumes that the migs were created with create-nodes
# function which tries to evenly spread nodes across the migs.

2
cluster/gce/win1803/OWNERS Normal file
View File

@@ -0,0 +1,2 @@
approvers:
- yujuhong

187
cluster/gce/win1803/README-GCE-Windows-kube-up.md Normal file
View File

@@ -0,0 +1,187 @@
# Starting a Windows Kubernetes cluster on GCE using kube-up
## Bring up the cluster
Prerequisites: a Google Cloud Platform project.
### 0. Prepare your environment
Clone this repository under your `$GOPATH/src` directory on a Linux machine.
Then, optionally clean/prepare your environment using these commands:
```
# Remove files that interfere with get-kube / kube-up:
rm -rf ./kubernetes/; rm -f kubernetes.tar.gz; rm -f ~/.kube/config
# Set the default gcloud project for this shell. This is optional but convenient
# if you're working with multiple projects and don't want to repeatedly switch
# between gcloud config configurations.
export CLOUDSDK_CORE_PROJECT=<your_project_name>
```
### 1. Build Kubernetes
The most straightforward approach to build those binaries is to run `make
release`. However, that builds binaries for all supported platforms, and can be
slow. You can speed up the process by following the instructions below to only
build the necessary binaries.
```
# Fetch the PR: https://github.com/pjh/kubernetes/pull/43
git remote add pjh https://github.com/pjh/kubernetes
git fetch pjh pull/43/head
# Get the commit hash and cherry-pick the commit to your current branch
BUILD_WIN_COMMIT=$(git ls-remote pjh | grep refs/pull/43/head | cut -f 1)
git cherry-pick $BUILD_WIN_COMMIT
# Build binaries for both Linux and Windows
make quick-release
```
### 2 Create a Kubernetes cluster
You can create a regular Kubernetes cluster or an end-to-end test cluster.
Please make sure you set the environment variables properly following the
instructions in the previous section.
First, set the following environment variables which are required for
controlling the number of Linux and Windows nodes in the cluster and for
enabling IP aliases (which are required for Windows pod routing):
```
export NUM_NODES=2 # number of Linux nodes
export NUM_WINDOWS_NODES=2
export KUBE_GCE_ENABLE_IP_ALIASES=true
```
If you wish to use `netd` as the CNI plugin for Linux nodes, set these
variables:
```
export KUBE_ENABLE_NETD=true
export KUBE_CUSTOM_NETD_YAML=$(curl -s \
https://raw.githubusercontent.com/GoogleCloudPlatform/netd/master/netd.yaml \
| sed -e 's/^/ /')
```
Now bring up a cluster using one of the following two methods:
#### 2.a Create a regular Kubernetes cluster
```
# Invoke kube-up.sh with these environment variables:
# PROJECT: text name of your GCP project.
# KUBERNETES_SKIP_CONFIRM: skips any kube-up prompts.
PROJECT=${CLOUDSDK_CORE_PROJECT} KUBERNETES_SKIP_CONFIRM=y ./cluster/kube-up.sh
```
To teardown the cluster run:
```
PROJECT=${CLOUDSDK_CORE_PROJECT} KUBERNETES_SKIP_CONFIRM=y ./cluster/kube-down.sh
```
#### 2.b Create a Kubernetes end-to-end (E2E) test cluster
```
PROJECT=${CLOUDSDK_CORE_PROJECT} go run ./hack/e2e.go -- --up
```
This command, by default, tears down the existing E2E cluster and create a new
one.
No matter what type of cluster you chose to create, the result should be a
Kubernetes cluster with one Linux master node, `NUM_NODES` Linux worker nodes
and `NUM_WINDOWS_NODES` Windows worker nodes.
## Validating the cluster
Invoke this script to run a smoke test that verifies that the cluster has been
brought up correctly:
```
cluster/gce/win1803/smoke-test.sh
```
## Running tests against the cluster
These steps are based on
[kubernetes-sigs/windows-testing](https://github.com/kubernetes-sigs/windows-testing).
* TODO(pjh): use patched `cluster/local/util.sh` from
https://github.com/pjh/kubernetes/blob/windows-up/cluster/local/util.sh.
* If necessary run `alias kubectl=client/bin/kubectl` .
* Set the following environment variables (these values should make sense if
you built your cluster using the kube-up steps above):
```
export KUBE_HOME=$(pwd)
export KUBECONFIG=~/.kube/config
export KUBE_MASTER=local
export KUBE_MASTER_NAME=kubernetes-master
export KUBE_MASTER_IP=$(kubectl get node ${KUBE_MASTER_NAME} -o jsonpath='{.status.addresses[?(@.type=="ExternalIP")].address}')
export KUBE_MASTER_URL=https://${KUBE_MASTER_IP}
export KUBE_MASTER_PORT=443
```
* Download the list of Windows e2e tests:
```
curl https://raw.githubusercontent.com/e2e-win/e2e-win-prow-deployment/master/repo-list.txt -o ${KUBE_HOME}/repo-list.yaml
export KUBE_TEST_REPO_LIST=${KUBE_HOME}/repo-list.yaml
```
* Download and configure the list of tests to exclude:
```
curl https://raw.githubusercontent.com/e2e-win/e2e-win-prow-deployment/master/exclude_conformance_test.txt -o ${KUBE_HOME}/exclude_conformance_test.txt
export EXCLUDED_TESTS=$(cat exclude_conformance_test.txt |
tr -d '\r' | # remove Windows carriage returns
tr -s '\n' '|' | # coalesce newlines into |
tr -s ' ' '.' | # coalesce spaces into .
sed -e 's/[]\[()]/\\&/g' | # escape brackets and parentheses
sed -e 's/.$//g') # remove final | added by tr
```
* Taint the Linux nodes so that test pods will not land on them:
```
export LINUX_NODES=$(kubectl get nodes -l beta.kubernetes.io/os=linux,kubernetes.io/hostname!=${KUBE_MASTER_NAME} -o name)
export LINUX_NODE_COUNT=$(echo ${LINUX_NODES} | wc -w)
for node in $LINUX_NODES; do
kubectl taint node $node node-under-test=false:NoSchedule
done
```
* Build necessary test binaries:
```
make WHAT=test/e2e/e2e.test
```
* Run the tests with flags that point at the "local" (already-running) cluster
and that permit the `NoSchedule` Linux nodes:
```
export KUBETEST_ARGS="--ginkgo.noColor=true "\
"--report-dir=${KUBE_HOME}/e2e-reports "\
"--allowed-not-ready-nodes=${LINUX_NODE_COUNT} "\
"--ginkgo.dryRun=false "\
"--ginkgo.focus=\[Conformance\] "\
"--ginkgo.skip=${EXCLUDED_TESTS}"
go run ${KUBE_HOME}/hack/e2e.go -- --verbose-commands \
--ginkgo-parallel=4 \
--check-version-skew=false --test --provider=local \
--test_args="${KUBETEST_ARGS}" &> ${KUBE_HOME}/conformance.out
```
TODO: copy log files from Windows nodes using some command like:
```
scp -r -o PreferredAuthentications=keyboard-interactive,password \
-o PubkeyAuthentication=no \
user@kubernetes-minion-windows-group-mk0p:C:\\etc\\kubernetes\\logs \
kubetest-logs/
```

90
cluster/gce/win1803/common.psm1 Normal file
View File

@@ -0,0 +1,90 @@
# Copyright 2019 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
<#
.SYNOPSIS
Library containing common variables and code used by other PowerShell modules
and scripts for configuring Windows nodes.
#>
# REDO_STEPS affects the behavior of a node that is rebooted after initial
# bringup. When true, on a reboot the scripts will redo steps that were
# determined to have already been completed once (e.g. to overwrite
# already-existing config files). When false the scripts will perform the
# minimum required steps to re-join this node to the cluster.
$REDO_STEPS = $false
Export-ModuleMember -Variable REDO_STEPS
# Writes $Message to the console. Terminates the script if $Fatal is set.
function Log-Output {
param (
[parameter(Mandatory=$true)] [string]$Message,
[switch]$Fatal
)
Write-Host "${Message}"
if (${Fatal}) {
Exit 1
}
}
# Checks if a file should be written or overwritten by testing if it already
# exists and checking the value of the global $REDO_STEPS variable. Emits an
# informative message if the file already exists.
#
# Returns $true if the file does not exist, or if it does but the global
# $REDO_STEPS variable is set to $true. Returns $false if the file exists and
# the caller should not overwrite it.
function ShouldWrite-File {
param (
[parameter(Mandatory=$true)] [string]$Filename
)
if (Test-Path $Filename) {
if ($REDO_STEPS) {
Log-Output "Warning: $Filename already exists, will overwrite it"
return $true
}
Log-Output "Skip: $Filename already exists, not overwriting it"
return $false
}
return $true
}
# Returns the GCE instance metadata value for $Key. If the key is not present
# in the instance metadata returns $Default if set, otherwise returns $null.
function Get-InstanceMetadataValue {
param (
[parameter(Mandatory=$true)] [string]$Key,
[parameter(Mandatory=$false)] [string]$Default
)
$url = ("http://metadata.google.internal/computeMetadata/v1/instance/" +
"attributes/$Key")
try {
$client = New-Object Net.WebClient
$client.Headers.Add('Metadata-Flavor', 'Google')
return ($client.DownloadString($url)).Trim()
}
catch [System.Net.WebException] {
if ($Default) {
return $Default
}
else {
Log-Output "Failed to retrieve value for $Key."
return $null
}
}
}
# Export all public functions:
Export-ModuleMember -Function *-*

119
cluster/gce/win1803/configure.ps1 Normal file
View File

@@ -0,0 +1,119 @@
# Copyright 2019 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
<#
.SYNOPSIS
Top-level script that runs on Windows nodes to join them to the K8s cluster.
#>
$ErrorActionPreference = 'Stop'
# Turn on tracing to debug
# Set-PSDebug -Trace 1
# Update TLS setting to enable Github downloads and disable progress bar to
# increase download speed.
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$ProgressPreference = 'SilentlyContinue'
# Returns the GCE instance metadata value for $Key. If the key is not present
# in the instance metadata returns $Default if set, otherwise returns $null.
function Get-InstanceMetadataValue {
param (
[parameter(Mandatory=$true)] [string]$Key,
[parameter(Mandatory=$false)] [string]$Default
)
$url = ("http://metadata.google.internal/computeMetadata/v1/instance/" +
"attributes/$Key")
try {
$client = New-Object Net.WebClient
$client.Headers.Add('Metadata-Flavor', 'Google')
return ($client.DownloadString($url)).Trim()
}
catch [System.Net.WebException] {
if ($Default) {
return $Default
}
else {
Write-Host "Failed to retrieve value for $Key."
return $null
}
}
}
# Fetches the value of $MetadataKey, saves it to C:\$Filename and imports it as
# a PowerShell module.
#
# Note: this function depends on common.psm1.
function FetchAndImport-ModuleFromMetadata {
param (
[parameter(Mandatory=$true)] [string]$MetadataKey,
[parameter(Mandatory=$true)] [string]$Filename
)
$module = Get-InstanceMetadataValue $MetadataKey
if (Test-Path C:\$Filename) {
if (-not $REDO_STEPS) {
Log-Output "Skip: C:\$Filename already exists, not overwriting"
Import-Module -Force C:\$Filename
return
}
Log-Output "Warning: C:\$Filename already exists, will overwrite it."
}
New-Item -ItemType file -Force C:\$Filename | Out-Null
Set-Content C:\$Filename $module
Import-Module -Force C:\$Filename
}
try {
# Don't use FetchAndImport-ModuleFromMetadata for common.psm1 - the common
# module includes variables and functions that any other function may depend
# on.
$module = Get-InstanceMetadataValue 'common-psm1'
New-Item -ItemType file -Force C:\common.psm1 | Out-Null
Set-Content C:\common.psm1 $module
Import-Module -Force C:\common.psm1
# TODO(pjh): update the function to set $Filename automatically from the key,
# then put these calls into a loop over a list of XYZ-psm1 keys.
FetchAndImport-ModuleFromMetadata 'k8s-node-setup-psm1' 'k8s-node-setup.psm1'
Set-PrerequisiteOptions
$kube_env = Fetch-KubeEnv
Set-EnvironmentVars
Create-Directories
Download-HelperScripts
Create-PauseImage
DownloadAndInstall-KubernetesBinaries
Create-NodePki
Create-KubeletKubeconfig
Create-KubeproxyKubeconfig
Set-PodCidr
Configure-HostNetworkingService
Configure-CniNetworking
Configure-Kubelet
Start-WorkerServices
Log-Output 'Waiting 15 seconds for node to join cluster.'
Start-Sleep 15
Verify-WorkerServices
}
catch {
Write-Host 'Exception caught in script:'
Write-Host $_.InvocationInfo.PositionMessage
Write-Host "Kubernetes Windows node setup failed: $($_.Exception.Message)"
exit 1
}

1028
cluster/gce/win1803/k8s-node-setup.psm1 Normal file
View File

File diff suppressed because it is too large Load Diff

52
cluster/gce/win1803/node-helper.sh Executable file
View File

@@ -0,0 +1,52 @@
#!/usr/bin/env bash
# Copyright 2019 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# A library of helper functions and constants for Windows nodes.
function get-windows-node-instance-metadata-from-file {
local metadata=""
metadata+="cluster-name=${KUBE_TEMP}/cluster-name.txt,"
metadata+="kube-env=${KUBE_TEMP}/windows-node-kube-env.yaml,"
metadata+="kubelet-config=${KUBE_TEMP}/windows-node-kubelet-config.yaml,"
# To get startup script output run "gcloud compute instances
# get-serial-port-output <instance>" from the location where you're running
# kube-up.
metadata+="windows-startup-script-ps1=${KUBE_ROOT}/cluster/gce/${WINDOWS_NODE_OS_DISTRIBUTION}/configure.ps1,"
metadata+="common-psm1=${KUBE_ROOT}/cluster/gce/${WINDOWS_NODE_OS_DISTRIBUTION}/common.psm1,"
metadata+="k8s-node-setup-psm1=${KUBE_ROOT}/cluster/gce/${WINDOWS_NODE_OS_DISTRIBUTION}/k8s-node-setup.psm1,"
metadata+="user-profile-psm1=${KUBE_ROOT}/cluster/gce/${WINDOWS_NODE_OS_DISTRIBUTION}/user-profile.psm1,"
metadata+="${NODE_EXTRA_METADATA}"
echo "${metadata}"
}
function get-windows-node-instance-metadata {
local metadata=""
metadata+="k8s-version=${KUBE_VERSION:-v1.13.2},"
metadata+="serial-port-enable=1,"
# This enables logging the serial port output.
# https://cloud.google.com/compute/docs/instances/viewing-serial-port-output
metadata+="serial-port-logging-enable=true,"
metadata+="win-version=${WINDOWS_NODE_OS_DISTRIBUTION}"
echo "${metadata}"
}
# $1: template name (required).
# $2: scopes flag.
function create-windows-node-instance-template {
local template_name="$1"
local scopes_flag="$2"
create-node-template "${template_name}" "${scopes_flag}" "$(get-windows-node-instance-metadata-from-file)" "$(get-windows-node-instance-metadata)" "windows"
}

672
cluster/gce/win1803/smoke-test.sh Executable file
View File

@@ -0,0 +1,672 @@
#!/bin/bash
# Copyright 2019 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# A small smoke test to run against a just-deployed kube-up cluster with Windows
# nodes. Performs checks such as:
# 1) Verifying that all Windows nodes have status Ready.
# 2) Verifying that no system pods are attempting to run on Windows nodes.
# 3) Verifying pairwise connectivity between most of the following: Linux
# pods, Windows pods, K8s services, and the Internet.
# 4) Verifying that basic DNS resolution works in Windows pods.
#
# This script assumes that it is run from the root of the kubernetes repository
# and that kubectl is present at client/bin/kubectl.
#
# TODOs:
# - Implement the node-to-pod checks.
# - Capture stdout for each command to a file and only print it when the test
# fails.
# - Move copy-pasted code into reusable functions.
# - Continue running all checks after one fails.
# - Test service connectivity by running a test pod with an http server and
# exposing it as a service (rather than curl-ing from existing system
# services that don't serve http requests).
# - Add test retries for transient errors, such as:
# "error: unable to upgrade connection: Authorization error
# (user=kube-apiserver, verb=create, resource=nodes, subresource=proxy)"
# Override this to use a different kubectl binary.
kubectl=kubectl
linux_deployment_timeout=60
windows_deployment_timeout=240
output_file=/tmp/k8s-smoke-test.out
function check_windows_nodes_are_ready {
# kubectl filtering is the worst.
statuses=$(${kubectl} get nodes -l beta.kubernetes.io/os=windows \
-o jsonpath='{.items[*].status.conditions[?(@.type=="Ready")].status}')
for status in $statuses; do
if [[ $status == "False" ]]; then
echo "ERROR: some Windows node has status != Ready"
echo "kubectl get nodes -l beta.kubernetes.io/os=windows"
${kubectl} get nodes -l beta.kubernetes.io/os=windows
exit 1
fi
done
echo "Verified that all Windows nodes have status Ready"
}
function check_no_system_pods_on_windows_nodes {
windows_system_pods=$(${kubectl} get pods --namespace kube-system \
-o wide | grep -E "Pending|windows" | wc -w)
if [[ $windows_system_pods -ne 0 ]]; then
echo "ERROR: there are kube-system pods trying to run on Windows nodes"
echo "kubectl get pods --namespace kube-system -o wide"
${kubectl} get pods --namespace kube-system -o wide
exit 1
fi
echo "Verified that all system pods are running on Linux nodes"
}
linux_webserver_deployment=linux-nginx
linux_webserver_pod_label=nginx
function deploy_linux_webserver_pod {
echo "Writing example deployment to $linux_webserver_deployment.yaml"
cat <<EOF > $linux_webserver_deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: $linux_webserver_deployment
labels:
app: $linux_webserver_pod_label
spec:
replicas: 1
selector:
matchLabels:
app: $linux_webserver_pod_label
template:
metadata:
labels:
app: $linux_webserver_pod_label
spec:
containers:
- name: nginx
image: nginx:1.7.9
nodeSelector:
beta.kubernetes.io/os: linux
EOF
if ! ${kubectl} create -f $linux_webserver_deployment.yaml; then
echo "kubectl create -f $linux_webserver_deployment.yaml failed"
exit 1
fi
timeout=$linux_deployment_timeout
while [[ $timeout -gt 0 ]]; do
echo "Waiting for Linux $linux_webserver_pod_label pods to become Ready"
statuses=$(${kubectl} get pods -l app=$linux_webserver_pod_label \
-o jsonpath='{.items[*].status.conditions[?(@.type=="Ready")].status}' \
| grep "False" | wc -w)
if [[ $statuses -eq 0 ]]; then
break
else
sleep 10
(( timeout=timeout-10 ))
fi
done
if [[ $timeout -gt 0 ]]; then
echo "All $linux_webserver_pod_label pods became Ready"
else
echo "ERROR: Not all $linux_webserver_pod_label pods became Ready"
echo "kubectl get pods -l app=$linux_webserver_pod_label"
${kubectl} get pods -l app=$linux_webserver_pod_label
cleanup_deployments
exit 1
fi
}
# Returns the name of an arbitrary Linux webserver pod.
function get_linux_webserver_pod_name {
$kubectl get pods -l app=$linux_webserver_pod_label \
-o jsonpath='{.items[0].metadata.name}'
}
# Returns the IP address of an arbitrary Linux webserver pod.
function get_linux_webserver_pod_ip {
$kubectl get pods -l app=$linux_webserver_pod_label \
-o jsonpath='{.items[0].status.podIP}'
}
function undeploy_linux_webserver_pod {
${kubectl} delete deployment $linux_webserver_deployment
}
linux_command_deployment=linux-ubuntu
linux_command_pod_label=ubuntu
function deploy_linux_command_pod {
echo "Writing example deployment to $linux_command_deployment.yaml"
cat <<EOF > $linux_command_deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: $linux_command_deployment
labels:
app: $linux_command_pod_label
spec:
replicas: 1
selector:
matchLabels:
app: $linux_command_pod_label
template:
metadata:
labels:
app: $linux_command_pod_label
spec:
containers:
- name: ubuntu
image: ubuntu
command: ["sleep", "123456"]
nodeSelector:
beta.kubernetes.io/os: linux
EOF
if ! ${kubectl} create -f $linux_command_deployment.yaml; then
echo "kubectl create -f $linux_command_deployment.yaml failed"
exit 1
fi
timeout=$linux_deployment_timeout
while [[ $timeout -gt 0 ]]; do
echo "Waiting for Linux $linux_command_pod_label pods to become Ready"
statuses=$(${kubectl} get pods -l app=$linux_command_pod_label \
-o jsonpath='{.items[*].status.conditions[?(@.type=="Ready")].status}' \
| grep "False" | wc -w)
if [[ $statuses -eq 0 ]]; then
break
else
sleep 10
(( timeout=timeout-10 ))
fi
done
if [[ $timeout -gt 0 ]]; then
echo "All $linux_command_pod_label pods became Ready"
else
echo "ERROR: Not all $linux_command_pod_label pods became Ready"
echo "kubectl get pods -l app=$linux_command_pod_label"
${kubectl} get pods -l app=$linux_command_pod_label
cleanup_deployments
exit 1
fi
}
# Returns the name of an arbitrary Linux command pod.
function get_linux_command_pod_name {
$kubectl get pods -l app=$linux_command_pod_label \
-o jsonpath='{.items[0].metadata.name}'
}
# Returns the IP address of an arbitrary Linux command pod.
function get_linux_command_pod_ip {
$kubectl get pods -l app=$linux_command_pod_label \
-o jsonpath='{.items[0].status.podIP}'
}
# Installs test executables (ping, curl) in the Linux command pod.
# NOTE: this assumes that there is only one Linux "command pod".
# TODO(pjh): fix this.
function prepare_linux_command_pod {
local linux_command_pod
linux_command_pod="$(get_linux_command_pod_name)"
echo "Installing test utilities in Linux command pod, may take a minute"
$kubectl exec "$linux_command_pod" -- apt-get update > /dev/null
$kubectl exec "$linux_command_pod" -- \
apt-get install -y iputils-ping curl > /dev/null
}
function undeploy_linux_command_pod {
${kubectl} delete deployment $linux_command_deployment
}
windows_webserver_deployment=windows-nettest
windows_webserver_pod_label=nettest
function deploy_windows_webserver_pod {
echo "Writing example deployment to $windows_webserver_deployment.yaml"
cat <<EOF > $windows_webserver_deployment.yaml
# You can run a pod with the e2eteam/nettest:1.0 image (which should listen on
# <podIP>:8080) and create another pod on a different node (linux would be
# easier) to curl the http server:
# curl http://<pod_ip>:8080/read
apiVersion: apps/v1
kind: Deployment
metadata:
name: $windows_webserver_deployment
labels:
app: $windows_webserver_pod_label
spec:
replicas: 1
selector:
matchLabels:
app: $windows_webserver_pod_label
template:
metadata:
labels:
app: $windows_webserver_pod_label
spec:
containers:
- name: nettest
image: e2eteam/nettest:1.0
nodeSelector:
beta.kubernetes.io/os: windows
tolerations:
- effect: NoSchedule
key: node.kubernetes.io/os
operator: Equal
value: windows
EOF
if ! ${kubectl} create -f $windows_webserver_deployment.yaml; then
echo "kubectl create -f $windows_webserver_deployment.yaml failed"
exit 1
fi
timeout=$windows_deployment_timeout
while [[ $timeout -gt 0 ]]; do
echo "Waiting for Windows $windows_webserver_pod_label pods to become Ready"
statuses=$(${kubectl} get pods -l app=$windows_webserver_pod_label \
-o jsonpath='{.items[*].status.conditions[?(@.type=="Ready")].status}' \
| grep "False" | wc -w)
if [[ $statuses -eq 0 ]]; then
break
else
sleep 10
(( timeout=timeout-10 ))
fi
done
if [[ $timeout -gt 0 ]]; then
echo "All $windows_webserver_pod_label pods became Ready"
else
echo "ERROR: Not all $windows_webserver_pod_label pods became Ready"
echo "kubectl get pods -l app=$windows_webserver_pod_label"
${kubectl} get pods -l app=$windows_webserver_pod_label
cleanup_deployments
exit 1
fi
}
function get_windows_webserver_pod_name {
$kubectl get pods -l app=$windows_webserver_pod_label \
-o jsonpath='{.items[0].metadata.name}'
}
function get_windows_webserver_pod_ip {
$kubectl get pods -l app=$windows_webserver_pod_label \
-o jsonpath='{.items[0].status.podIP}'
}
function undeploy_windows_webserver_pod {
${kubectl} delete deployment $windows_webserver_deployment
}
windows_command_deployment=windows-powershell
windows_command_pod_label=powershell
function deploy_windows_command_pod {
echo "Writing example deployment to $windows_command_deployment.yaml"
cat <<EOF > $windows_command_deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: $windows_command_deployment
labels:
app: $windows_command_pod_label
spec:
replicas: 1
selector:
matchLabels:
app: $windows_command_pod_label
template:
metadata:
labels:
app: $windows_command_pod_label
spec:
containers:
- name: nettest
image: e2eteam/nettest:1.0
nodeSelector:
beta.kubernetes.io/os: windows
tolerations:
- effect: NoSchedule
key: node.kubernetes.io/os
operator: Equal
value: windows
EOF
if ! ${kubectl} create -f $windows_command_deployment.yaml; then
echo "kubectl create -f $windows_command_deployment.yaml failed"
exit 1
fi
timeout=$windows_deployment_timeout
while [[ $timeout -gt 0 ]]; do
echo "Waiting for Windows $windows_command_pod_label pods to become Ready"
statuses=$(${kubectl} get pods -l app=$windows_command_pod_label \
-o jsonpath='{.items[*].status.conditions[?(@.type=="Ready")].status}' \
| grep "False" | wc -w)
if [[ $statuses -eq 0 ]]; then
break
else
sleep 10
(( timeout=timeout-10 ))
fi
done
if [[ $timeout -gt 0 ]]; then
echo "All $windows_command_pod_label pods became Ready"
else
echo "ERROR: Not all $windows_command_pod_label pods became Ready"
echo "kubectl get pods -l app=$windows_command_pod_label"
${kubectl} get pods -l app=$windows_command_pod_label
cleanup_deployments
exit 1
fi
}
function get_windows_command_pod_name {
$kubectl get pods -l app=$windows_command_pod_label \
-o jsonpath='{.items[0].metadata.name}'
}
function get_windows_command_pod_ip {
$kubectl get pods -l app=$windows_command_pod_label \
-o jsonpath='{.items[0].status.podIP}'
}
function undeploy_windows_command_pod {
${kubectl} delete deployment $windows_command_deployment
}
function test_linux_node_to_linux_pod {
echo "TODO: ${FUNCNAME[0]}"
}
function test_linux_node_to_windows_pod {
echo "TODO: ${FUNCNAME[0]}"
}
function test_linux_pod_to_linux_pod {
echo "TEST: ${FUNCNAME[0]}"
local linux_command_pod
linux_command_pod="$(get_linux_command_pod_name)"
local linux_webserver_pod_ip
linux_webserver_pod_ip="$(get_linux_webserver_pod_ip)"
if ! $kubectl exec "$linux_command_pod" -- curl -m 20 \
"http://$linux_webserver_pod_ip" &> $output_file; then
cleanup_deployments
echo "Failing output: $(cat $output_file)"
echo "FAILED: ${FUNCNAME[0]}"
exit 1
fi
}
# TODO(pjh): this test flakily fails on brand-new clusters, not sure why.
# % Total % Received % Xferd Average Speed Time Time Time Current
# Dload Upload Total Spent Left Speed
# 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
# curl: (6) Could not resolve host:
# command terminated with exit code 6
function test_linux_pod_to_windows_pod {
echo "TEST: ${FUNCNAME[0]}"
local linux_command_pod
linux_command_pod="$(get_linux_command_pod_name)"
local windows_webserver_pod_ip
windows_webserver_pod_ip="$(get_windows_webserver_pod_ip)"
if ! $kubectl exec "$linux_command_pod" -- curl -m 20 \
"http://$windows_webserver_pod_ip:8080/read" &> $output_file; then
cleanup_deployments
echo "Failing output: $(cat $output_file)"
echo "FAILED: ${FUNCNAME[0]}"
echo "This test seems to be flaky. TODO(pjh): investigate."
exit 1
fi
}
function test_linux_pod_to_internet {
echo "TEST: ${FUNCNAME[0]}"
local linux_command_pod
linux_command_pod="$(get_linux_command_pod_name)"
local internet_ip="8.8.8.8" # Google DNS
# This is expected to return 404 (not found).
if ! $kubectl exec "$linux_command_pod" -- curl -m 20 \
"http://$internet_ip" > $output_file; then
cleanup_deployments
echo "Failing output: $(cat $output_file)"
echo "FAILED: ${FUNCNAME[0]}"
exit 1
fi
}
function test_linux_pod_to_k8s_service {
echo "TEST: ${FUNCNAME[0]}"
local linux_command_pod
linux_command_pod="$(get_linux_command_pod_name)"
local service="heapster"
local service_ip
service_ip=$($kubectl get service --namespace kube-system $service \
-o jsonpath='{.spec.clusterIP}')
local service_port
service_port=$($kubectl get service --namespace kube-system $service \
-o jsonpath='{.spec.ports[?(@.protocol=="TCP")].port}')
echo "curl-ing $service address from Linux pod: $service_ip:$service_port"
# curl-ing the heapster service results in an expected 404 response code. The
# curl command does not set a failure return code in this case.
if ! $kubectl exec "$linux_command_pod" -- \
curl -m 20 "http://$service_ip:$service_port" &> $output_file; then
cleanup_deployments
echo "Failing output: $(cat $output_file)"
echo "FAILED: ${FUNCNAME[0]}"
exit 1
fi
}
function test_windows_node_to_linux_pod {
echo "TODO: ${FUNCNAME[0]}"
}
function test_windows_node_to_windows_pod {
echo "TODO: ${FUNCNAME[0]}"
}
# TODO(pjh): this test failed for me once with
# error: unable to upgrade connection: container not found ("nettest")
# Maybe the container crashed for some reason? Investigate if it happens more.
#
# TODO(pjh): another one-time failure:
# error: unable to upgrade connection: Authorization error
# (user=kube-apiserver, verb=create, resource=nodes, subresource=proxy)
function test_windows_pod_to_linux_pod {
echo "TEST: ${FUNCNAME[0]}"
local windows_command_pod
windows_command_pod="$(get_windows_command_pod_name)"
local linux_webserver_pod_ip
linux_webserver_pod_ip="$(get_linux_webserver_pod_ip)"
if ! $kubectl exec "$windows_command_pod" -- powershell.exe \
"curl -UseBasicParsing http://$linux_webserver_pod_ip" > \
$output_file; then
cleanup_deployments
echo "Failing output: $(cat $output_file)"
echo "FAILED: ${FUNCNAME[0]}"
exit 1
fi
}
function test_windows_pod_to_windows_pod {
echo "TEST: ${FUNCNAME[0]}"
local windows_command_pod
windows_command_pod="$(get_windows_command_pod_name)"
local windows_webserver_pod_ip
windows_webserver_pod_ip="$(get_windows_webserver_pod_ip)"
if ! $kubectl exec "$windows_command_pod" -- powershell.exe \
"curl -UseBasicParsing http://$windows_webserver_pod_ip:8080/read" \
> $output_file; then
cleanup_deployments
echo "Failing output: $(cat $output_file)"
echo "FAILED: ${FUNCNAME[0]}"
exit 1
fi
}
function test_windows_pod_to_internet {
echo "TEST: ${FUNCNAME[0]}"
local windows_command_pod
windows_command_pod="$(get_windows_command_pod_name)"
local internet_ip="8.8.8.8"
# This snippet tests Internet connectivity without depending on DNS by
# attempting to curl Google's well-known DNS IP, 8.8.8.8. On success we expect
# to get back a 404 status code; on failure the response object will have a
# status code of 0 or some other HTTP code.
if ! $kubectl exec "$windows_command_pod" -- powershell.exe \
"\$response = try { \`
(curl -UseBasicParsing http://$internet_ip \`
-ErrorAction Stop).BaseResponse \`
} catch [System.Net.WebException] { \`
\$_.Exception.Response \`
}; \`
\$statusCodeInt = [int]\$response.StatusCode; \`
if (\$statusCodeInt -eq 404) { \`
exit 0 \`
} else { \`
Write-Host \"curl $internet_ip got unexpected status code \$statusCodeInt\"
exit 1 \`
}" > $output_file; then
cleanup_deployments
echo "Failing output: $(cat $output_file)"
echo "FAILED: ${FUNCNAME[0]}"
exit 1
fi
}
function test_windows_pod_to_k8s_service {
echo "TEST: ${FUNCNAME[0]}"
local windows_command_pod
windows_command_pod="$(get_windows_command_pod_name)"
local service="heapster"
local service_ip
service_ip=$($kubectl get service --namespace kube-system $service \
-o jsonpath='{.spec.clusterIP}')
local service_port
service_port=$($kubectl get service --namespace kube-system $service \
-o jsonpath='{.spec.ports[?(@.protocol=="TCP")].port}')
local service_address="$service_ip:$service_port"
echo "curl-ing $service address from Windows pod: $service_address"
# Performing a web request to the heapster service results in an expected 404
# response; this code snippet filters out the expected 404 from other status
# codes that indicate failure.
if ! $kubectl exec "$windows_command_pod" -- powershell.exe \
"\$response = try { \`
(curl -UseBasicParsing http://$service_address \`
-ErrorAction Stop).BaseResponse \`
} catch [System.Net.WebException] { \`
\$_.Exception.Response \`
}; \`
\$statusCodeInt = [int]\$response.StatusCode; \`
if (\$statusCodeInt -eq 404) { \`
exit 0 \`
} else { \`
Write-Host \"curl $service_address got unexpected status code \$statusCodeInt\"
exit 1 \`
}" > $output_file; then
cleanup_deployments
echo "Failing output: $(cat $output_file)"
echo "FAILED: ${FUNCNAME[0]}"
exit 1
fi
}
function test_kube_dns_in_windows_pod {
echo "TEST: ${FUNCNAME[0]}"
local windows_command_pod
windows_command_pod="$(get_windows_command_pod_name)"
local service="kube-dns"
local service_ip
service_ip=$($kubectl get service --namespace kube-system $service \
-o jsonpath='{.spec.clusterIP}')
if ! $kubectl exec "$windows_command_pod" -- powershell.exe \
"Resolve-DnsName www.bing.com -server $service_ip" > $output_file; then
cleanup_deployments
echo "Failing output: $(cat $output_file)"
echo "FAILED: ${FUNCNAME[0]}"
exit 1
fi
}
function test_dns_just_works_in_windows_pod {
echo "TEST: ${FUNCNAME[0]}"
local windows_command_pod
windows_command_pod="$(get_windows_command_pod_name)"
if ! $kubectl exec "$windows_command_pod" -- powershell.exe \
"curl -UseBasicParsing http://www.bing.com" > $output_file; then
cleanup_deployments
echo "Failing output: $(cat $output_file)"
echo "FAILED: ${FUNCNAME[0]}"
exit 1
fi
}
function cleanup_deployments {
undeploy_linux_webserver_pod
undeploy_linux_command_pod
undeploy_windows_webserver_pod
undeploy_windows_command_pod
}
check_windows_nodes_are_ready
check_no_system_pods_on_windows_nodes
deploy_linux_webserver_pod
deploy_linux_command_pod
deploy_windows_webserver_pod
deploy_windows_command_pod
prepare_linux_command_pod
echo ""
test_linux_node_to_linux_pod
test_linux_node_to_windows_pod
test_linux_pod_to_linux_pod
test_linux_pod_to_windows_pod
test_linux_pod_to_k8s_service
# Note: test_windows_node_to_k8s_service is not supported at this time.
# https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/common-problems#my-windows-node-cannot-access-my-services-using-the-service-ip
test_windows_node_to_linux_pod
test_windows_node_to_windows_pod
test_windows_pod_to_linux_pod
test_windows_pod_to_windows_pod
test_windows_pod_to_internet
test_windows_pod_to_k8s_service
test_kube_dns_in_windows_pod
test_dns_just_works_in_windows_pod
echo ""
cleanup_deployments
echo "All tests passed!"
exit 0

337
cluster/gce/win1803/user-profile.psm1 Normal file
View File

@@ -0,0 +1,337 @@
<#
.Synopsis
Rough PS functions to create new user profiles
.DESCRIPTION
Call the Create-NewProfile function directly to create a new profile
.EXAMPLE
Create-NewProfile -Username 'testUser1' -Password 'testUser1'
.NOTES
Created by: Josh Rickard (@MS_dministrator) and Thom Schumacher (@driberif)
Forked by: @crshnbrn66, then @pjh (2018-11-08). See
https://gist.github.com/pjh/9753cd14400f4e3d4567f4553ba75f1d/revisions
Date: 24MAR2017
Location: https://gist.github.com/crshnbrn66/7e81bf20408c05ddb2b4fdf4498477d8
Contact: https://github.com/MSAdministrator
MSAdministrator.com
https://github.com/crshnbrn66
powershellposse.com
#>
#Function to create the new local user first
function New-LocalUser
{
[CmdletBinding()]
[Alias()]
[OutputType([int])]
Param
(
# Param1 help description
[Parameter(Mandatory=$true,
ValueFromPipelineByPropertyName=$true,
Position=0)]
$userName,
# Param2 help description
[string]
$password
)
$system = [ADSI]"WinNT://$env:COMPUTERNAME";
$user = $system.Create("user",$userName);
$user.SetPassword($password);
$user.SetInfo();
$flag=$user.UserFlags.value -bor 0x10000;
$user.put("userflags",$flag);
$user.SetInfo();
$group = [ADSI]("WinNT://$env:COMPUTERNAME/Users");
$group.PSBase.Invoke("Add", $user.PSBase.Path);
}
#function to register a native method
function Register-NativeMethod
{
[CmdletBinding()]
[Alias()]
[OutputType([int])]
Param
(
# Param1 help description
[Parameter(Mandatory=$true,
ValueFromPipelineByPropertyName=$true,
Position=0)]
[string]$dll,
# Param2 help description
[Parameter(Mandatory=$true,
ValueFromPipelineByPropertyName=$true,
Position=1)]
[string]
$methodSignature
)
$script:nativeMethods += [PSCustomObject]@{ Dll = $dll; Signature = $methodSignature; }
}
function Get-Win32LastError
{
[CmdletBinding()]
[Alias()]
[OutputType([int])]
Param($typeName = 'LastError')
if (-not ([System.Management.Automation.PSTypeName]$typeName).Type)
{
$lasterrorCode = $script:lasterror | ForEach-Object{
'[DllImport("kernel32.dll", SetLastError = true)]
public static extern uint GetLastError();'
}
Add-Type @"
using System;
using System.Text;
using System.Runtime.InteropServices;
public static class $typeName {
$lasterrorCode
}
"@
}
}
#function to add native method
function Add-NativeMethods
{
[CmdletBinding()]
[Alias()]
[OutputType([int])]
Param($typeName = 'NativeMethods')
$nativeMethodsCode = $script:nativeMethods | ForEach-Object { "
[DllImport(`"$($_.Dll)`")]
public static extern $($_.Signature);
" }
Add-Type @"
using System;
using System.Text;
using System.Runtime.InteropServices;
public static class $typeName {
$nativeMethodsCode
}
"@
}
#Main function to create the new user profile
function Create-NewProfile {
[CmdletBinding()]
[Alias()]
[OutputType([int])]
Param
(
# Param1 help description
[Parameter(Mandatory=$true,
ValueFromPipelineByPropertyName=$true,
Position=0)]
[string]$UserName,
# Param2 help description
[Parameter(Mandatory=$true,
ValueFromPipelineByPropertyName=$true,
Position=1)]
[string]
$Password
)
Write-Verbose "Creating local user $Username";
try
{
New-LocalUser -username $UserName -password $Password;
}
catch
{
Write-Error $_.Exception.Message;
break;
}
$methodName = 'UserEnvCP'
$script:nativeMethods = @();
if (-not ([System.Management.Automation.PSTypeName]$MethodName).Type)
{
Register-NativeMethod "userenv.dll" "int CreateProfile([MarshalAs(UnmanagedType.LPWStr)] string pszUserSid,`
[MarshalAs(UnmanagedType.LPWStr)] string pszUserName,`
[Out][MarshalAs(UnmanagedType.LPWStr)] StringBuilder pszProfilePath, uint cchProfilePath)";
Add-NativeMethods -typeName $MethodName;
}
$localUser = New-Object System.Security.Principal.NTAccount("$UserName");
$userSID = $localUser.Translate([System.Security.Principal.SecurityIdentifier]);
$sb = new-object System.Text.StringBuilder(260);
$pathLen = $sb.Capacity;
Write-Verbose "Creating user profile for $Username";
try
{
[UserEnvCP]::CreateProfile($userSID.Value, $Username, $sb, $pathLen) | Out-Null;
}
catch
{
Write-Error $_.Exception.Message;
break;
}
}
function New-ProfileFromSID {
[CmdletBinding()]
[Alias()]
[OutputType([int])]
Param
(
# Param1 help description
[Parameter(Mandatory=$true,
ValueFromPipelineByPropertyName=$true,
Position=0)]
[string]$UserName,
[string]$domain = 'PHCORP'
)
$methodname = 'UserEnvCP2'
$script:nativeMethods = @();
if (-not ([System.Management.Automation.PSTypeName]$methodname).Type)
{
Register-NativeMethod "userenv.dll" "int CreateProfile([MarshalAs(UnmanagedType.LPWStr)] string pszUserSid,`
[MarshalAs(UnmanagedType.LPWStr)] string pszUserName,`
[Out][MarshalAs(UnmanagedType.LPWStr)] StringBuilder pszProfilePath, uint cchProfilePath)";
Add-NativeMethods -typeName $methodname;
}
$sb = new-object System.Text.StringBuilder(260);
$pathLen = $sb.Capacity;
Write-Verbose "Creating user profile for $Username";
#$SID= ((get-aduser -id $UserName -ErrorAction Stop).sid.value)
if($domain)
{
$objUser = New-Object System.Security.Principal.NTAccount($domain, $UserName)
$strSID = $objUser.Translate([System.Security.Principal.SecurityIdentifier])
$SID = $strSID.Value
}
else
{
$objUser = New-Object System.Security.Principal.NTAccount($UserName)
$strSID = $objUser.Translate([System.Security.Principal.SecurityIdentifier])
$SID = $strSID.Value
}
Write-Verbose "$UserName SID: $SID"
try
{
$result = [UserEnvCP2]::CreateProfile($SID, $Username, $sb, $pathLen)
if($result -eq '-2147024713')
{
$status = "$userName already exists"
write-verbose "$username Creation Result: $result"
}
elseif($result -eq '-2147024809')
{
$staus = "$username Not Found"
write-verbose "$username creation result: $result"
}
elseif($result -eq 0)
{
$status = "$username Profile has been created"
write-verbose "$username Creation Result: $result"
}
else
{
$status = "$UserName unknown return result: $result"
}
}
catch
{
Write-Error $_.Exception.Message;
break;
}
$status
}
Function Remove-Profile {
[CmdletBinding()]
[Alias()]
[OutputType([int])]
Param
(
# Param1 help description
[Parameter(Mandatory=$true,
ValueFromPipelineByPropertyName=$true,
Position=0)]
[string]$UserName,
[string]$ProfilePath,
[string]$domain = 'PHCORP'
)
$methodname = 'userenvDP'
$script:nativeMethods = @();
if (-not ([System.Management.Automation.PSTypeName]"$methodname.profile").Type)
{
add-type @"
using System.Runtime.InteropServices;
namespace $typename
{
public static class UserEnv
{
[DllImport("userenv.dll", CharSet = CharSet.Unicode, ExactSpelling = false, SetLastError = true)]
public static extern bool DeleteProfile(string sidString, string profilePath, string computerName);
[DllImport("kernel32.dll")]
public static extern uint GetLastError();
}
public static class Profile
{
public static uint Delete(string sidString)
{ //Profile path and computer name are optional
if (!UserEnv.DeleteProfile(sidString, null, null))
{
return UserEnv.GetLastError();
}
return 0;
}
}
}
"@
}
#$SID= ((get-aduser -id $UserName -ErrorAction Stop).sid.value)
if($domain)
{
$objUser = New-Object System.Security.Principal.NTAccount($domain, $UserName)
$strSID = $objUser.Translate([System.Security.Principal.SecurityIdentifier])
$SID = $strSID.Value
}
else
{
$objUser = New-Object System.Security.Principal.NTAccount($UserName)
$strSID = $objUser.Translate([System.Security.Principal.SecurityIdentifier])
$SID = $strSID.Value
}
Write-Verbose "$UserName SID: $SID"
try
{
#http://stackoverflow.com/questions/31949002/c-sharp-delete-user-profile
$result = [userenvDP.Profile]::Delete($SID)
}
catch
{
Write-Error $_.Exception.Message;
break;
}
$LastError
}
Export-ModuleMember Create-NewProfile

2
cluster/kubemark/gce/config-default.sh
View File

@@ -95,7 +95,7 @@ KUBEMARK_MASTER_COMPONENTS_QPS_LIMITS="${KUBEMARK_MASTER_COMPONENTS_QPS_LIMITS:-
CUSTOM_ADMISSION_PLUGINS="${CUSTOM_ADMISSION_PLUGINS:-NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,Priority,StorageObjectInUseProtection,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota}"
# Master components' test arguments.
APISERVER_TEST_ARGS="${KUBEMARK_APISERVER_TEST_ARGS:-} --vmodule=httplog=3 --runtime-config=extensions/v1beta1,scheduling.k8s.io/v1alpha1 ${API_SERVER_TEST_LOG_LEVEL} ${TEST_CLUSTER_MAX_REQUESTS_INFLIGHT} ${TEST_CLUSTER_DELETE_COLLECTION_WORKERS}"
APISERVER_TEST_ARGS="${KUBEMARK_APISERVER_TEST_ARGS:-} --runtime-config=extensions/v1beta1,scheduling.k8s.io/v1alpha1 ${API_SERVER_TEST_LOG_LEVEL} ${TEST_CLUSTER_MAX_REQUESTS_INFLIGHT} ${TEST_CLUSTER_DELETE_COLLECTION_WORKERS}"
CONTROLLER_MANAGER_TEST_ARGS="${KUBEMARK_CONTROLLER_MANAGER_TEST_ARGS:-} ${CONTROLLER_MANAGER_TEST_LOG_LEVEL} ${TEST_CLUSTER_RESYNC_PERIOD} ${TEST_CLUSTER_API_CONTENT_TYPE} ${KUBEMARK_MASTER_COMPONENTS_QPS_LIMITS}"
SCHEDULER_TEST_ARGS="${KUBEMARK_SCHEDULER_TEST_ARGS:-} ${SCHEDULER_TEST_LOG_LEVEL} ${TEST_CLUSTER_API_CONTENT_TYPE} ${KUBEMARK_MASTER_COMPONENTS_QPS_LIMITS}"

5
cluster/validate-cluster.sh
View File

@@ -50,9 +50,8 @@ function kubectl_retry() {
ALLOWED_NOTREADY_NODES="${ALLOWED_NOTREADY_NODES:-0}"
CLUSTER_READY_ADDITIONAL_TIME_SECONDS="${CLUSTER_READY_ADDITIONAL_TIME_SECONDS:-30}"
EXPECTED_NUM_NODES="${NUM_NODES}"
if [[ "${KUBERNETES_PROVIDER:-}" == "gce" ]]; then
EXPECTED_NUM_NODES="$(get-num-nodes)"
echo "Validating gce cluster, MULTIZONE=${MULTIZONE:-}"
# In multizone mode we need to add instances for all nodes in the region.
if [[ "${MULTIZONE:-}" == "true" ]]; then
@@ -60,6 +59,8 @@ if [[ "${KUBERNETES_PROVIDER:-}" == "gce" ]]; then
--filter="name ~ '${NODE_INSTANCE_PREFIX}.*' AND zone:($(gcloud -q compute zones list --project="${PROJECT}" --filter=region=${REGION} --format=csv[no-heading]\(name\) | tr "\n" "," | sed "s/,$//"))" | wc -l)
echo "Computing number of nodes, NODE_INSTANCE_PREFIX=${NODE_INSTANCE_PREFIX}, REGION=${REGION}, EXPECTED_NUM_NODES=${EXPECTED_NUM_NODES}"
fi
else
EXPECTED_NUM_NODES="${NUM_NODES}"
fi
if [[ "${REGISTER_MASTER_KUBELET:-}" == "true" ]]; then

2
cmd/hyperkube/main.go
View File

@@ -170,7 +170,7 @@ func makeSymlinks(targetName string, commandFns []func() *cobra.Command) error {
}
if errs {
return errors.New("Error creating one or more symlinks.")
return errors.New("Error creating one or more symlinks")
}
return nil
}

4
cmd/kube-controller-manager/app/core.go
View File

@@ -82,8 +82,8 @@ func startServiceController(ctx ControllerContext) (http.Handler, bool, error) {
}
func startNodeIpamController(ctx ControllerContext) (http.Handler, bool, error) {
var clusterCIDR *net.IPNet = nil
var serviceCIDR *net.IPNet = nil
var clusterCIDR *net.IPNet
var serviceCIDR *net.IPNet
if !ctx.ComponentConfig.KubeCloudShared.AllocateNodeCIDRs {
return nil, false, nil

1
cmd/kube-proxy/app/BUILD
View File

@@ -22,6 +22,7 @@ go_library(
"//pkg/kubelet/qos:go_default_library",
"//pkg/master/ports:go_default_library",
"//pkg/proxy:go_default_library",
"//pkg/proxy/apis:go_default_library",
"//pkg/proxy/apis/config:go_default_library",
"//pkg/proxy/apis/config/scheme:go_default_library",
"//pkg/proxy/apis/config/validation:go_default_library",

3
cmd/kube-proxy/app/server.go
View File

@@ -53,6 +53,7 @@ import (
"k8s.io/kubernetes/pkg/kubelet/qos"
"k8s.io/kubernetes/pkg/master/ports"
"k8s.io/kubernetes/pkg/proxy"
"k8s.io/kubernetes/pkg/proxy/apis"
kubeproxyconfig "k8s.io/kubernetes/pkg/proxy/apis/config"
"k8s.io/kubernetes/pkg/proxy/apis/config/scheme"
"k8s.io/kubernetes/pkg/proxy/apis/config/validation"
@@ -583,7 +584,7 @@ func (s *ProxyServer) Run() error {
informerFactory := informers.NewSharedInformerFactoryWithOptions(s.Client, s.ConfigSyncPeriod,
informers.WithTweakListOptions(func(options *v1meta.ListOptions) {
options.LabelSelector = "!service.kubernetes.io/service-proxy-name"
options.LabelSelector = "!" + apis.LabelServiceProxyName
}))
// Create configs (i.e. Watches for Services and Endpoints)

2
cmd/kubeadm/.import-restrictions
View File

@@ -41,6 +41,7 @@
"AllowedPrefixes": [
"k8s.io/utils/exec",
"k8s.io/utils/integer",
"k8s.io/utils/path",
"k8s.io/utils/pointer"
]
},
@@ -77,7 +78,6 @@
"k8s.io/kubernetes/pkg/scheduler/util",
"k8s.io/kubernetes/pkg/security/apparmor",
"k8s.io/kubernetes/pkg/serviceaccount",
"k8s.io/kubernetes/pkg/util/file",
"k8s.io/kubernetes/pkg/util/hash",
"k8s.io/kubernetes/pkg/util/initsystem",
"k8s.io/kubernetes/pkg/util/ipvs",

2
cmd/kubeadm/app/cmd/alpha/selfhosting.go
View File

@@ -125,7 +125,7 @@ func getSelfhostingSubCommand(in io.Reader) *cobra.Command {
// KubernetesVersion is not used, but we set it explicitly to avoid the lookup
// of the version from the internet when executing ConfigFileAndDefaultsToInternalConfig
phases.SetKubernetesVersion(cfg)
phases.SetKubernetesVersion(&cfg.ClusterConfiguration)
// This call returns the ready-to-use configuration based on the configuration file that might or might not exist and the default cfg populated by flags
internalcfg, err := configutil.ConfigFileAndDefaultsToInternalConfig(cfgPath, cfg)

2
cmd/kubeadm/app/cmd/config.go
View File

@@ -388,7 +388,7 @@ func RunConfigView(out io.Writer, client clientset.Interface) error {
func uploadConfiguration(client clientset.Interface, cfgPath string, defaultcfg *kubeadmapiv1beta1.InitConfiguration) error {
// KubernetesVersion is not used, but we set it explicitly to avoid the lookup
// of the version from the internet when executing ConfigFileAndDefaultsToInternalConfig
phaseutil.SetKubernetesVersion(defaultcfg)
phaseutil.SetKubernetesVersion(&defaultcfg.ClusterConfiguration)
// Default both statically and dynamically, convert to internal API type, and validate everything
// First argument is unset here as we shouldn't load a config file from disk

21
cmd/kubeadm/app/cmd/init.go
View File

@@ -148,6 +148,7 @@ func NewCmdInit(out io.Writer, initOptions *initOptions) *cobra.Command {
err = showJoinCommand(data, out)
kubeadmutil.CheckErr(err)
},
Args: cobra.NoArgs,
}
// adds flags to the init command
@@ -286,27 +287,27 @@ func newInitData(cmd *cobra.Command, args []string, options *initOptions, out io
// validated values to the public kubeadm config API when applicable
var err error
if options.externalcfg.FeatureGates, err = features.NewFeatureGate(&features.InitFeatureGates, options.featureGatesString); err != nil {
return &initData{}, err
return nil, err
}
ignorePreflightErrorsSet, err := validation.ValidateIgnorePreflightErrors(options.ignorePreflightErrors)
if err != nil {
return &initData{}, err
return nil, err
}
if err = validation.ValidateMixedArguments(cmd.Flags()); err != nil {
return &initData{}, err
return nil, err
}
if err = options.bto.ApplyTo(options.externalcfg); err != nil {
return &initData{}, err
return nil, err
}
// Either use the config file if specified, or convert public kubeadm API to the internal InitConfiguration
// and validates InitConfiguration
cfg, err := configutil.ConfigFileAndDefaultsToInternalConfig(options.cfgPath, options.externalcfg)
if err != nil {
return &initData{}, err
return nil, err
}
// override node name and CRI socket from the command line options
@@ -318,29 +319,29 @@ func newInitData(cmd *cobra.Command, args []string, options *initOptions, out io
}
if err := configutil.VerifyAPIServerBindAddress(cfg.LocalAPIEndpoint.AdvertiseAddress); err != nil {
return &initData{}, err
return nil, err
}
if err := features.ValidateVersion(features.InitFeatureGates, cfg.FeatureGates, cfg.KubernetesVersion); err != nil {
return &initData{}, err
return nil, err
}
// if dry running creates a temporary folder for saving kubeadm generated files
dryRunDir := ""
if options.dryRun {
if dryRunDir, err = ioutil.TempDir("", "kubeadm-init-dryrun"); err != nil {
return &initData{}, errors.Wrap(err, "couldn't create a temporary directory")
return nil, errors.Wrap(err, "couldn't create a temporary directory")
}
}
// Checks if an external CA is provided by the user.
externalCA, _ := certsphase.UsingExternalCA(cfg)
externalCA, _ := certsphase.UsingExternalCA(&cfg.ClusterConfiguration)
if externalCA {
kubeconfigDir := kubeadmconstants.KubernetesDir
if options.dryRun {
kubeconfigDir = dryRunDir
}
if err := kubeconfigphase.ValidateKubeconfigsForExternalCA(kubeconfigDir, cfg); err != nil {
return &initData{}, err
return nil, err
}
}

163
cmd/kubeadm/app/cmd/join.go
View File

@@ -17,7 +17,6 @@ limitations under the License.
package cmd
import (
"bytes"
"fmt"
"io"
"os"
@@ -38,6 +37,7 @@ import (
kubeadmapiv1beta1 "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1"
"k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/validation"
"k8s.io/kubernetes/cmd/kubeadm/app/cmd/options"
"k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases"
"k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow"
cmdutil "k8s.io/kubernetes/cmd/kubeadm/app/cmd/util"
kubeadmconstants "k8s.io/kubernetes/cmd/kubeadm/app/constants"
@@ -68,17 +68,6 @@ var (
`)
notReadyToJoinControPlaneTemp = template.Must(template.New("join").Parse(dedent.Dedent(`
One or more conditions for hosting a new control plane instance is not satisfied.
{{.Error}}
Please ensure that:
* The cluster has a stable controlPlaneEndpoint address.
* The certificates that must be shared among control plane instances are provided.
`)))
joinControPlaneDoneTemp = template.Must(template.New("join").Parse(dedent.Dedent(`
This node has joined the cluster and a new control plane instance was created:
@@ -203,14 +192,14 @@ func NewCmdJoin(out io.Writer, joinOptions *joinOptions) *cobra.Command {
err = data.Run()
kubeadmutil.CheckErr(err)
},
// We accept the master location as an optional positional argument
Args: cobra.MaximumNArgs(1),
}
AddJoinConfigFlags(cmd.Flags(), joinOptions.externalcfg)
AddJoinOtherFlags(cmd.Flags(), &joinOptions.cfgPath, &joinOptions.ignorePreflightErrors, &joinOptions.controlPlane, &joinOptions.token)
addJoinConfigFlags(cmd.Flags(), joinOptions.externalcfg)
addJoinOtherFlags(cmd.Flags(), &joinOptions.cfgPath, &joinOptions.ignorePreflightErrors, &joinOptions.controlPlane, &joinOptions.token)
// initialize the workflow runner with the list of phases
// TODO: append phases here like so:
// joinRunner.AppendPhase(phases.NewPreflightMasterPhase())
joinRunner.AppendPhase(phases.NewPreflightJoinPhase())
// sets the data builder function, that will be used by the runner
// both when running the entire workflow or single phases
@@ -225,60 +214,48 @@ func NewCmdJoin(out io.Writer, joinOptions *joinOptions) *cobra.Command {
return cmd
}
// AddJoinConfigFlags adds join flags bound to the config to the specified flagset
func AddJoinConfigFlags(flagSet *flag.FlagSet, cfg *kubeadmapiv1beta1.JoinConfiguration) {
// addJoinConfigFlags adds join flags bound to the config to the specified flagset
func addJoinConfigFlags(flagSet *flag.FlagSet, cfg *kubeadmapiv1beta1.JoinConfiguration) {
flagSet.StringVar(
&cfg.NodeRegistration.Name, options.NodeName, cfg.NodeRegistration.Name,
`Specify the node name.`,
)
// add control plane endpoint flags to the specified flagset
flagSet.StringVar(
&cfg.ControlPlane.LocalAPIEndpoint.AdvertiseAddress, options.APIServerAdvertiseAddress, cfg.ControlPlane.LocalAPIEndpoint.AdvertiseAddress,
"If the node should host a new control plane instance, the IP address the API Server will advertise it's listening on. If not set the default network interface will be used.",
)
flagSet.Int32Var(
&cfg.ControlPlane.LocalAPIEndpoint.BindPort, options.APIServerBindPort, cfg.ControlPlane.LocalAPIEndpoint.BindPort,
"If the node should host a new control plane instance, the port for the API Server to bind to.",
)
// adds bootstrap token specific discovery flags to the specified flagset
flagSet.StringVar(
&cfg.Discovery.BootstrapToken.Token, options.TokenDiscovery, "",
"For token-based discovery, the token used to validate cluster information fetched from the API server.",
)
flagSet.StringSliceVar(
&cfg.Discovery.BootstrapToken.CACertHashes, options.TokenDiscoveryCAHash, []string{},
"For token-based discovery, validate that the root CA public key matches this hash (format: \"<type>:<value>\").",
)
flagSet.BoolVar(
&cfg.Discovery.BootstrapToken.UnsafeSkipCAVerification, options.TokenDiscoverySkipCAHash, false,
"For token-based discovery, allow joining without --discovery-token-ca-cert-hash pinning.",
)
// discovery via kube config file flag
flagSet.StringVar(
&cfg.Discovery.File.KubeConfigPath, options.FileDiscovery, "",
"For file-based discovery, a file or URL from which to load cluster information.",
)
flagSet.StringVar(
&cfg.Discovery.TLSBootstrapToken, options.TLSBootstrapToken, cfg.Discovery.TLSBootstrapToken,
`Specify the token used to temporarily authenticate with the Kubernetes Master while joining the node.`,
)
AddControlPlaneFlags(flagSet, cfg.ControlPlane)
AddJoinBootstrapTokenDiscoveryFlags(flagSet, cfg.Discovery.BootstrapToken)
AddJoinFileDiscoveryFlags(flagSet, cfg.Discovery.File)
cmdutil.AddCRISocketFlag(flagSet, &cfg.NodeRegistration.CRISocket)
}
// AddJoinBootstrapTokenDiscoveryFlags adds bootstrap token specific discovery flags to the specified flagset
func AddJoinBootstrapTokenDiscoveryFlags(flagSet *flag.FlagSet, btd *kubeadmapiv1beta1.BootstrapTokenDiscovery) {
flagSet.StringVar(
&btd.Token, options.TokenDiscovery, "",
"For token-based discovery, the token used to validate cluster information fetched from the API server.",
)
flagSet.StringSliceVar(
&btd.CACertHashes, options.TokenDiscoveryCAHash, []string{},
"For token-based discovery, validate that the root CA public key matches this hash (format: \"<type>:<value>\").",
)
flagSet.BoolVar(
&btd.UnsafeSkipCAVerification, options.TokenDiscoverySkipCAHash, false,
"For token-based discovery, allow joining without --discovery-token-ca-cert-hash pinning.",
)
}
// AddJoinFileDiscoveryFlags adds file discovery flags to the specified flagset
func AddJoinFileDiscoveryFlags(flagSet *flag.FlagSet, fd *kubeadmapiv1beta1.FileDiscovery) {
flagSet.StringVar(
&fd.KubeConfigPath, options.FileDiscovery, "",
"For file-based discovery, a file or URL from which to load cluster information.",
)
}
// AddControlPlaneFlags adds file control plane flags to the specified flagset
func AddControlPlaneFlags(flagSet *flag.FlagSet, cp *kubeadmapiv1beta1.JoinControlPlane) {
flagSet.StringVar(
&cp.LocalAPIEndpoint.AdvertiseAddress, options.APIServerAdvertiseAddress, cp.LocalAPIEndpoint.AdvertiseAddress,
"If the node should host a new control plane instance, the IP address the API Server will advertise it's listening on. If not set the default network interface will be used.",
)
flagSet.Int32Var(
&cp.LocalAPIEndpoint.BindPort, options.APIServerBindPort, cp.LocalAPIEndpoint.BindPort,
"If the node should host a new control plane instance, the port for the API Server to bind to.",
)
}
// AddJoinOtherFlags adds join flags that are not bound to a configuration file to the given flagset
func AddJoinOtherFlags(flagSet *flag.FlagSet, cfgPath *string, ignorePreflightErrors *[]string, controlPlane *bool, token *string) {
// addJoinOtherFlags adds join flags that are not bound to a configuration file to the given flagset
func addJoinOtherFlags(flagSet *flag.FlagSet, cfgPath *string, ignorePreflightErrors *[]string, controlPlane *bool, token *string) {
flagSet.StringVar(
cfgPath, options.CfgPath, *cfgPath,
"Path to kubeadm config file.",
@@ -358,11 +335,11 @@ func newJoinData(cmd *cobra.Command, args []string, options *joinOptions, out io
ignorePreflightErrorsSet, err := validation.ValidateIgnorePreflightErrors(options.ignorePreflightErrors)
if err != nil {
return &joinData{}, err
return nil, err
}
if err = validation.ValidateMixedArguments(cmd.Flags()); err != nil {
return &joinData{}, err
return nil, err
}
// Either use the config file if specified, or convert public kubeadm API to the internal JoinConfiguration
@@ -377,7 +354,7 @@ func newJoinData(cmd *cobra.Command, args []string, options *joinOptions, out io
cfg, err := configutil.JoinConfigFileAndDefaultsToInternalConfig(options.cfgPath, options.externalcfg)
if err != nil {
return &joinData{}, err
return nil, err
}
// override node name and CRI socket from the command line options
@@ -390,7 +367,7 @@ func newJoinData(cmd *cobra.Command, args []string, options *joinOptions, out io
if cfg.ControlPlane != nil {
if err := configutil.VerifyAPIServerBindAddress(cfg.ControlPlane.LocalAPIEndpoint.AdvertiseAddress); err != nil {
return &joinData{}, err
return nil, err
}
}
@@ -443,14 +420,6 @@ func (j *joinData) OutputWriter() io.Writer {
// Run executes worker node provisioning and tries to join an existing cluster.
func (j *joinData) Run() error {
fmt.Println("[preflight] Running pre-flight checks")
// Start with general checks
klog.V(1).Infoln("[preflight] Running general checks")
if err := preflight.RunJoinNodeChecks(utilsexec.New(), j.cfg, j.ignorePreflightErrors); err != nil {
return err
}
// Fetch the init configuration based on the join configuration.
// TODO: individual phases should call these:
// - phases that need initCfg should call joinData.InitCfg().
@@ -463,35 +432,7 @@ func (j *joinData) Run() error {
if err != nil {
return err
}
// Continue with more specific checks based on the init configuration
klog.V(1).Infoln("[preflight] Running configuration dependant checks")
if err := preflight.RunOptionalJoinNodeChecks(utilsexec.New(), initCfg, j.ignorePreflightErrors); err != nil {
return err
}
if j.cfg.ControlPlane != nil {
// Checks if the cluster configuration supports
// joining a new control plane instance and if all the necessary certificates are provided
if err := j.CheckIfReadyForAdditionalControlPlane(initCfg); err != nil {
// outputs the not ready for hosting a new control plane instance message
ctx := map[string]string{
"Error": err.Error(),
}
var msg bytes.Buffer
notReadyToJoinControPlaneTemp.Execute(&msg, ctx)
return errors.New(msg.String())
}
// run kubeadm init preflight checks for checking all the prequisites
fmt.Println("[join] Running pre-flight checks before initializing the new control plane instance")
preflight.RunInitMasterChecks(utilsexec.New(), initCfg, j.ignorePreflightErrors)
fmt.Println("[join] Pulling control-plane images")
if err := preflight.RunPullImagesCheck(utilsexec.New(), initCfg, j.ignorePreflightErrors); err != nil {
return err
}
// Prepares the node for hosting a new control plane instance by writing necessary
// kubeconfig files, and static pod manifests
if err := j.PrepareForHostingControlPlane(initCfg); err != nil {
@@ -537,22 +478,6 @@ func (j *joinData) Run() error {
return nil
}
// CheckIfReadyForAdditionalControlPlane ensures that the cluster is in a state that supports
// joining an additional control plane instance and if the node is ready to join
func (j *joinData) CheckIfReadyForAdditionalControlPlane(initConfiguration *kubeadmapi.InitConfiguration) error {
// blocks if the cluster was created without a stable control plane endpoint
if initConfiguration.ControlPlaneEndpoint == "" {
return errors.New("unable to add a new control plane instance a cluster that doesn't have a stable controlPlaneEndpoint address")
}
// checks if the certificates that must be equal across contolplane instances are provided
if ret, err := certsphase.SharedCertificateExists(initConfiguration); !ret {
return err
}
return nil
}
// PrepareForHostingControlPlane makes all preparation activities require for a node hosting a new control plane instance
func (j *joinData) PrepareForHostingControlPlane(initConfiguration *kubeadmapi.InitConfiguration) error {
@@ -586,7 +511,7 @@ func (j *joinData) PrepareForHostingControlPlane(initConfiguration *kubeadmapi.I
return errors.Wrap(err, "couldn't create Kubernetes client")
}
if err := etcdphase.CheckLocalEtcdClusterStatus(client, initConfiguration); err != nil {
if err := etcdphase.CheckLocalEtcdClusterStatus(client, &initConfiguration.ClusterConfiguration); err != nil {
return err
}
}
@@ -638,7 +563,7 @@ func (j *joinData) BootstrapKubelet(tlsBootstrapCfg *clientcmdapi.Config, initCo
// register the joining node with the specified taints if the node
// is not a master. The markmaster phase will register the taints otherwise.
registerTaintsUsingFlags := j.cfg.ControlPlane == nil
if err := kubeletphase.WriteKubeletDynamicEnvFile(initConfiguration, registerTaintsUsingFlags, kubeadmconstants.KubeletRunDirectory); err != nil {
if err := kubeletphase.WriteKubeletDynamicEnvFile(&initConfiguration.ClusterConfiguration, &initConfiguration.NodeRegistration, registerTaintsUsingFlags, kubeadmconstants.KubeletRunDirectory); err != nil {
return err
}
@@ -690,7 +615,7 @@ func (j *joinData) PostInstallControlPlane(initConfiguration *kubeadmapi.InitCon
// because it needs two members as majority to agree on the consensus. You will only see this behavior between the time
// etcdctl member add informs the cluster about the new member and the new member successfully establishing a connection to the existing one."
klog.V(1).Info("[join] adding etcd")
if err := etcdphase.CreateStackedEtcdStaticPodManifestFile(client, kubeadmconstants.GetStaticPodDirectory(), initConfiguration); err != nil {
if err := etcdphase.CreateStackedEtcdStaticPodManifestFile(client, kubeadmconstants.GetStaticPodDirectory(), initConfiguration.NodeRegistration.Name, &initConfiguration.ClusterConfiguration, &initConfiguration.LocalAPIEndpoint); err != nil {
return errors.Wrap(err, "error creating local etcd static pod manifest file")
}
}

4
cmd/kubeadm/app/cmd/phases/addons.go
View File

@@ -95,7 +95,7 @@ func runCoreDNSAddon(c workflow.RunData) error {
if err != nil {
return err
}
return dnsaddon.EnsureDNSAddon(cfg, client)
return dnsaddon.EnsureDNSAddon(&cfg.ClusterConfiguration, client)
}
// runKubeProxyAddon installs KubeProxy addon to a Kubernetes cluster
@@ -104,7 +104,7 @@ func runKubeProxyAddon(c workflow.RunData) error {
if err != nil {
return err
}
return proxyaddon.EnsureProxyAddon(cfg, client)
return proxyaddon.EnsureProxyAddon(&cfg.ClusterConfiguration, &cfg.LocalAPIEndpoint, client)
}
func getAddonPhaseFlags(name string) []string {

7
cmd/kubeadm/app/cmd/phases/certs.go
View File

@@ -204,13 +204,8 @@ func runCertsSa(c workflow.RunData) error {
return nil
}
// if dryrunning, write certificates to a temporary folder (and defer restore to the path originally specified by the user)
cfg := data.Cfg()
cfg.CertificatesDir = data.CertificateWriteDir()
defer func() { cfg.CertificatesDir = data.CertificateDir() }()
// create the new service account key (or use existing)
return certsphase.CreateServiceAccountKeyAndPublicKeyFiles(cfg)
return certsphase.CreateServiceAccountKeyAndPublicKeyFiles(data.CertificateWriteDir())
}
func runCerts(c workflow.RunData) error {

7
cmd/kubeadm/app/cmd/phases/controlplane.go
View File

@@ -34,10 +34,10 @@ var (
controlPlaneExample = normalizer.Examples(`
# Generates all static Pod manifest files for control plane components,
# functionally equivalent to what is generated by kubeadm init.
kubeadm init phase control-plane
kubeadm init phase control-plane all
# Generates all static Pod manifest files using options read from a configuration file.
kubeadm init phase control-plane --config config.yaml
kubeadm init phase control-plane all --config config.yaml
`)
controlPlanePhaseProperties = map[string]struct {
@@ -80,6 +80,7 @@ func NewControlPlanePhase() workflow.Phase {
Name: "all",
Short: "Generates all static Pod manifest files",
InheritFlags: getControlPlanePhaseFlags("all"),
Example: controlPlaneExample,
RunAllSiblings: true,
},
newControlPlaneSubPhase(kubeadmconstants.KubeAPIServer),
@@ -150,6 +151,6 @@ func runControlPlaneSubPhase(component string) func(c workflow.RunData) error {
cfg := data.Cfg()
fmt.Printf("[control-plane] Creating static Pod manifest for %q\n", component)
return controlplane.CreateStaticPodFiles(data.ManifestDir(), cfg, component)
return controlplane.CreateStaticPodFiles(data.ManifestDir(), &cfg.ClusterConfiguration, &cfg.LocalAPIEndpoint, component)
}
}

2
cmd/kubeadm/app/cmd/phases/etcd.go
View File

@@ -90,7 +90,7 @@ func runEtcdPhaseLocal() func(c workflow.RunData) error {
// Add etcd static pod spec only if external etcd is not configured
if cfg.Etcd.External == nil {
fmt.Printf("[etcd] Creating static Pod manifest for local etcd in %q\n", data.ManifestDir())
if err := etcdphase.CreateLocalEtcdStaticPodManifestFile(data.ManifestDir(), cfg); err != nil {
if err := etcdphase.CreateLocalEtcdStaticPodManifestFile(data.ManifestDir(), cfg.NodeRegistration.Name, &cfg.ClusterConfiguration, &cfg.LocalAPIEndpoint); err != nil {
return errors.Wrap(err, "error creating local etcd static pod manifest file")
}
} else {

2
cmd/kubeadm/app/cmd/phases/kubelet.go
View File

@@ -75,7 +75,7 @@ func runKubeletStart(c workflow.RunData) error {
// Write env file with flags for the kubelet to use. We do not need to write the --register-with-taints for the master,
// as we handle that ourselves in the markmaster phase
// TODO: Maybe we want to do that some time in the future, in order to remove some logic from the markmaster phase?
if err := kubeletphase.WriteKubeletDynamicEnvFile(data.Cfg(), false, data.KubeletDir()); err != nil {
if err := kubeletphase.WriteKubeletDynamicEnvFile(&data.Cfg().ClusterConfiguration, &data.Cfg().NodeRegistration, false, data.KubeletDir()); err != nil {
return errors.Wrap(err, "error writing a dynamic environment file for the kubelet")
}

124
cmd/kubeadm/app/cmd/phases/preflight.go
View File

@@ -17,23 +17,43 @@ limitations under the License.
package phases
import (
"bytes"
"fmt"
"text/template"
"github.com/lithammer/dedent"
"github.com/pkg/errors"
"k8s.io/apimachinery/pkg/util/sets"
"k8s.io/klog"
kubeadmapi "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm"
"k8s.io/kubernetes/cmd/kubeadm/app/cmd/options"
"k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow"
"k8s.io/kubernetes/cmd/kubeadm/app/phases/certs"
"k8s.io/kubernetes/cmd/kubeadm/app/preflight"
"k8s.io/kubernetes/pkg/util/normalizer"
utilsexec "k8s.io/utils/exec"
)
var (
masterPreflightExample = normalizer.Examples(`
initPreflightExample = normalizer.Examples(`
# Run master pre-flight checks using a config file.
kubeadm init phase preflight --config kubeadm-config.yml
`)
joinPreflightExample = normalizer.Examples(`
# Run join pre-flight checks using a config file.
kubeadm join phase preflight --config kubeadm-config.yml
`)
notReadyToJoinControPlaneTemp = template.Must(template.New("join").Parse(dedent.Dedent(`
One or more conditions for hosting a new control plane instance is not satisfied.
{{.Error}}
Please ensure that:
* The cluster has a stable controlPlaneEndpoint address.
* The certificates that must be shared among control plane instances are provided.
`)))
)
// preflightMasterData defines the behavior that a runtime data struct passed to the PreflightMaster master phase
@@ -45,13 +65,19 @@ type preflightMasterData interface {
IgnorePreflightErrors() sets.String
}
type preflightJoinData interface {
Cfg() *kubeadmapi.JoinConfiguration
InitCfg() (*kubeadmapi.InitConfiguration, error)
IgnorePreflightErrors() sets.String
}
// NewPreflightMasterPhase creates a kubeadm workflow phase that implements preflight checks for a new master node.
func NewPreflightMasterPhase() workflow.Phase {
return workflow.Phase{
Name: "preflight",
Short: "Run master pre-flight checks",
Long: "Run master pre-flight checks, functionally equivalent to what implemented by kubeadm init.",
Example: masterPreflightExample,
Example: initPreflightExample,
Run: runPreflightMaster,
InheritFlags: []string{
options.CfgPath,
@@ -60,6 +86,7 @@ func NewPreflightMasterPhase() workflow.Phase {
}
}
// TODO(dmaiocchi): rename all instances of master to controlPlane in this file.
// runPreflightMaster executes preflight checks logic.
func runPreflightMaster(c workflow.RunData) error {
data, ok := c.(preflightMasterData)
@@ -85,3 +112,96 @@ func runPreflightMaster(c workflow.RunData) error {
return nil
}
// NewPreflightJoinPhase creates a kubeadm workflow phase that implements preflight checks for a new node join
func NewPreflightJoinPhase() workflow.Phase {
return workflow.Phase{
Name: "preflight",
Short: "Run join pre-flight checks",
Long: "Run join pre-flight checks, functionally equivalent to what is implemented by kubeadm join.",
Example: joinPreflightExample,
Run: runPreflightJoin,
InheritFlags: []string{
options.CfgPath,
options.IgnorePreflightErrors,
options.TLSBootstrapToken,
options.TokenStr,
options.ControlPlane,
options.APIServerAdvertiseAddress,
options.APIServerBindPort,
options.NodeCRISocket,
options.NodeName,
options.FileDiscovery,
options.TokenDiscovery,
options.TokenDiscoveryCAHash,
options.TokenDiscoverySkipCAHash,
},
}
}
// runPreflightJoin executes preflight checks logic.
func runPreflightJoin(c workflow.RunData) error {
j, ok := c.(preflightJoinData)
if !ok {
return errors.New("preflight phase invoked with an invalid data struct")
}
// Start with general checks
klog.V(1).Infoln("[preflight] Running general checks")
if err := preflight.RunJoinNodeChecks(utilsexec.New(), j.Cfg(), j.IgnorePreflightErrors()); err != nil {
return err
}
initCfg, err := j.InitCfg()
if err != nil {
return err
}
// Continue with more specific checks based on the init configuration
klog.V(1).Infoln("[preflight] Running configuration dependant checks")
if err := preflight.RunOptionalJoinNodeChecks(utilsexec.New(), &initCfg.ClusterConfiguration, j.IgnorePreflightErrors()); err != nil {
return err
}
if j.Cfg().ControlPlane != nil {
// Checks if the cluster configuration supports
// joining a new control plane instance and if all the necessary certificates are provided
if err := checkIfReadyForAdditionalControlPlane(&initCfg.ClusterConfiguration); err != nil {
// outputs the not ready for hosting a new control plane instance message
ctx := map[string]string{
"Error": err.Error(),
}
var msg bytes.Buffer
notReadyToJoinControPlaneTemp.Execute(&msg, ctx)
return errors.New(msg.String())
}
// run kubeadm init preflight checks for checking all the prequisites
fmt.Println("[preflight] Running pre-flight checks before initializing the new control plane instance")
if err := preflight.RunInitMasterChecks(utilsexec.New(), initCfg, j.IgnorePreflightErrors()); err != nil {
return err
}
fmt.Println("[preflight] Pulling control-plane images")
if err := preflight.RunPullImagesCheck(utilsexec.New(), initCfg, j.IgnorePreflightErrors()); err != nil {
return err
}
}
return nil
}
// checkIfReadyForAdditionalControlPlane ensures that the cluster is in a state that supports
// joining an additional control plane instance and if the node is ready to preflight
func checkIfReadyForAdditionalControlPlane(initConfiguration *kubeadmapi.ClusterConfiguration) error {
// blocks if the cluster was created without a stable control plane endpoint
if initConfiguration.ControlPlaneEndpoint == "" {
return errors.New("unable to add a new control plane instance a cluster that doesn't have a stable controlPlaneEndpoint address")
}
// checks if the certificates that must be equal across contolplane instances are provided
if ret, err := certs.SharedCertificateExists(initConfiguration); !ret {
return err
}
return nil
}

2
cmd/kubeadm/app/cmd/phases/uploadconfig.go
View File

@@ -126,7 +126,7 @@ func runUploadKubeletConfig(c workflow.RunData) error {
}
klog.V(1).Infof("[upload-config] Uploading the kubelet component config to a ConfigMap")
if err = kubeletphase.CreateConfigMap(cfg, client); err != nil {
if err = kubeletphase.CreateConfigMap(cfg.ClusterConfiguration.ComponentConfigs.Kubelet, cfg.KubernetesVersion, client); err != nil {
return errors.Wrap(err, "error creating kubelet configuration ConfigMap")
}

2
cmd/kubeadm/app/cmd/phases/util.go
View File

@@ -23,7 +23,7 @@ import (
// SetKubernetesVersion gets the current Kubeadm version and sets it as KubeadmVersion in the config,
// unless it's already set to a value different from the default.
func SetKubernetesVersion(cfg *kubeadmapiv1beta1.InitConfiguration) {
func SetKubernetesVersion(cfg *kubeadmapiv1beta1.ClusterConfiguration) {
if cfg.KubernetesVersion != kubeadmapiv1beta1.DefaultKubernetesVersion && cfg.KubernetesVersion != "" {
return

2
cmd/kubeadm/app/cmd/phases/util_test.go
View File

@@ -51,7 +51,7 @@ func TestSetKubernetesVersion(t *testing.T) {
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
cfg := &kubeadmapiv1beta1.InitConfiguration{ClusterConfiguration: kubeadmapiv1beta1.ClusterConfiguration{KubernetesVersion: test.input}}
cfg := &kubeadmapiv1beta1.ClusterConfiguration{KubernetesVersion: test.input}
SetKubernetesVersion(cfg)
if cfg.KubernetesVersion != test.output {
t.Fatalf("expected %q, got %q", test.output, cfg.KubernetesVersion)

7
cmd/kubeadm/app/cmd/phases/workflow/runner.go
View File

@@ -309,7 +309,6 @@ func (e *Runner) BindToCommand(cmd *cobra.Command) {
phaseCommand := &cobra.Command{
Use: "phase",
Short: fmt.Sprintf("use this command to invoke single phase of the %s workflow", cmd.Name()),
// TODO: this logic is currently lacking verification if a suphase name is valid!
}
cmd.AddCommand(phaseCommand)
@@ -352,7 +351,6 @@ func (e *Runner) BindToCommand(cmd *cobra.Command) {
os.Exit(1)
}
},
Args: cobra.NoArgs, // this forces cobra to fail if a wrong phase name is passed
}
// makes the new command inherits local flags from the parent command
@@ -371,6 +369,11 @@ func (e *Runner) BindToCommand(cmd *cobra.Command) {
})
}
// if this phase has children (not a leaf) it doesn't accept any args
if len(p.Phases) > 0 {
phaseCmd.Args = cobra.NoArgs
}
// adds the command to parent
if p.level == 0 {
phaseCommand.AddCommand(phaseCmd)

2
cmd/kubeadm/app/cmd/token.go
View File

@@ -212,7 +212,7 @@ func NewCmdTokenGenerate(out io.Writer) *cobra.Command {
func RunCreateToken(out io.Writer, client clientset.Interface, cfgPath string, cfg *kubeadmapiv1beta1.InitConfiguration, printJoinCommand bool, kubeConfigFile string) error {
// KubernetesVersion is not used, but we set it explicitly to avoid the lookup
// of the version from the internet when executing ConfigFileAndDefaultsToInternalConfig
phaseutil.SetKubernetesVersion(cfg)
phaseutil.SetKubernetesVersion(&cfg.ClusterConfiguration)
// This call returns the ready-to-use configuration based on the configuration file that might or might not exist and the default cfg populated by flags
klog.V(1).Infoln("[token] loading configurations")

2
cmd/kubeadm/app/cmd/upgrade/diff.go
View File

@@ -105,7 +105,7 @@ func runDiff(flags *diffFlags, args []string) error {
return err
}
specs := controlplane.GetStaticPodSpecs(cfg, k8sVer)
specs := controlplane.GetStaticPodSpecs(&cfg.ClusterConfiguration, &cfg.LocalAPIEndpoint, k8sVer)
for spec, pod := range specs {
var path string
switch spec {

2
cmd/kubeadm/app/constants/constants.go
View File

@@ -332,7 +332,7 @@ const (
KubeDNSVersion = "1.14.13"
// CoreDNSVersion is the version of CoreDNS to be deployed if it is used
CoreDNSVersion = "1.2.6"
CoreDNSVersion = "1.3.1"
// ClusterConfigurationKind is the string kind value for the ClusterConfiguration struct
ClusterConfigurationKind = "ClusterConfiguration"

22
cmd/kubeadm/app/phases/addons/dns/dns.go
View File

@@ -74,14 +74,14 @@ func DeployedDNSAddon(client clientset.Interface) (kubeadmapi.DNSAddOnType, stri
}
// EnsureDNSAddon creates the kube-dns or CoreDNS addon
func EnsureDNSAddon(cfg *kubeadmapi.InitConfiguration, client clientset.Interface) error {
func EnsureDNSAddon(cfg *kubeadmapi.ClusterConfiguration, client clientset.Interface) error {
if cfg.DNS.Type == kubeadmapi.CoreDNS {
return coreDNSAddon(cfg, client)
}
return kubeDNSAddon(cfg, client)
}
func kubeDNSAddon(cfg *kubeadmapi.InitConfiguration, client clientset.Interface) error {
func kubeDNSAddon(cfg *kubeadmapi.ClusterConfiguration, client clientset.Interface) error {
if err := CreateServiceAccount(client); err != nil {
return err
}
@@ -103,9 +103,9 @@ func kubeDNSAddon(cfg *kubeadmapi.InitConfiguration, client clientset.Interface)
dnsDeploymentBytes, err := kubeadmutil.ParseTemplate(KubeDNSDeployment,
struct{ DeploymentName, KubeDNSImage, DNSMasqImage, SidecarImage, DNSBindAddr, DNSProbeAddr, DNSDomain, MasterTaintKey string }{
DeploymentName: kubeadmconstants.KubeDNSDeploymentName,
KubeDNSImage: images.GetDNSImage(&cfg.ClusterConfiguration, kubeadmconstants.KubeDNSKubeDNSImageName),
DNSMasqImage: images.GetDNSImage(&cfg.ClusterConfiguration, kubeadmconstants.KubeDNSDnsMasqNannyImageName),
SidecarImage: images.GetDNSImage(&cfg.ClusterConfiguration, kubeadmconstants.KubeDNSSidecarImageName),
KubeDNSImage: images.GetDNSImage(cfg, kubeadmconstants.KubeDNSKubeDNSImageName),
DNSMasqImage: images.GetDNSImage(cfg, kubeadmconstants.KubeDNSDnsMasqNannyImageName),
SidecarImage: images.GetDNSImage(cfg, kubeadmconstants.KubeDNSSidecarImageName),
DNSBindAddr: dnsBindAddr,
DNSProbeAddr: dnsProbeAddr,
DNSDomain: cfg.Networking.DNSDomain,
@@ -155,11 +155,11 @@ func createKubeDNSAddon(deploymentBytes, serviceBytes []byte, client clientset.I
return createDNSService(kubednsService, serviceBytes, client)
}
func coreDNSAddon(cfg *kubeadmapi.InitConfiguration, client clientset.Interface) error {
func coreDNSAddon(cfg *kubeadmapi.ClusterConfiguration, client clientset.Interface) error {
// Get the YAML manifest
coreDNSDeploymentBytes, err := kubeadmutil.ParseTemplate(CoreDNSDeployment, struct{ DeploymentName, Image, MasterTaintKey string }{
DeploymentName: kubeadmconstants.CoreDNSDeploymentName,
Image: images.GetDNSImage(&cfg.ClusterConfiguration, kubeadmconstants.CoreDNSImageName),
Image: images.GetDNSImage(cfg, kubeadmconstants.CoreDNSImageName),
MasterTaintKey: kubeadmconstants.LabelNodeRoleMaster,
})
if err != nil {
@@ -172,7 +172,7 @@ func coreDNSAddon(cfg *kubeadmapi.InitConfiguration, client clientset.Interface)
return err
}
stubDomain, err := translateStubDomainOfKubeDNSToProxyCoreDNS(kubeDNSStubDomain, kubeDNSConfigMap)
stubDomain, err := translateStubDomainOfKubeDNSToForwardCoreDNS(kubeDNSStubDomain, kubeDNSConfigMap)
if err != nil {
return err
}
@@ -294,9 +294,9 @@ func createDNSService(dnsService *v1.Service, serviceBytes []byte, client client
return nil
}
// translateStubDomainOfKubeDNSToProxyCoreDNS translates StubDomain Data in kube-dns ConfigMap
// translateStubDomainOfKubeDNSToForwardCoreDNS translates StubDomain Data in kube-dns ConfigMap
// in the form of Proxy for the CoreDNS Corefile.
func translateStubDomainOfKubeDNSToProxyCoreDNS(dataField string, kubeDNSConfigMap *v1.ConfigMap) (string, error) {
func translateStubDomainOfKubeDNSToForwardCoreDNS(dataField string, kubeDNSConfigMap *v1.ConfigMap) (string, error) {
if kubeDNSConfigMap == nil {
return "", nil
}
@@ -316,7 +316,7 @@ func translateStubDomainOfKubeDNSToProxyCoreDNS(dataField string, kubeDNSConfigM
{"errors"},
{"cache", "30"},
{"loop"},
append([]string{"proxy", "."}, proxyIP...),
append([]string{"forward", "."}, proxyIP...),
}
proxyStanza = append(proxyStanza, pStanza)
}

18
cmd/kubeadm/app/phases/addons/dns/dns_test.go
View File

@@ -206,28 +206,28 @@ func TestTranslateStubDomainKubeDNSToCoreDNS(t *testing.T) {
errors
cache 30
loop
proxy . 1.2.3.4:5300 3.3.3.3
forward . 1.2.3.4:5300 3.3.3.3
}
my.cluster.local:53 {
errors
cache 30
loop
proxy . 2.3.4.5
forward . 2.3.4.5
}`,
expectTwo: `
my.cluster.local:53 {
errors
cache 30
loop
proxy . 2.3.4.5
forward . 2.3.4.5
}
foo.com:53 {
errors
cache 30
loop
proxy . 1.2.3.4:5300 3.3.3.3
forward . 1.2.3.4:5300 3.3.3.3
}`,
},
{
@@ -257,28 +257,28 @@ func TestTranslateStubDomainKubeDNSToCoreDNS(t *testing.T) {
errors
cache 30
loop
proxy . 1.2.3.4:5300
forward . 1.2.3.4:5300
}
my.cluster.local:53 {
errors
cache 30
loop
proxy . 2.3.4.5
forward . 2.3.4.5
}`,
expectTwo: `
my.cluster.local:53 {
errors
cache 30
loop
proxy . 2.3.4.5
forward . 2.3.4.5
}
foo.com:53 {
errors
cache 30
loop
proxy . 1.2.3.4:5300
forward . 1.2.3.4:5300
}`,
},
{
@@ -296,7 +296,7 @@ func TestTranslateStubDomainKubeDNSToCoreDNS(t *testing.T) {
},
}
for _, testCase := range testCases {
out, err := translateStubDomainOfKubeDNSToProxyCoreDNS(kubeDNSStubDomain, testCase.configMap)
out, err := translateStubDomainOfKubeDNSToForwardCoreDNS(kubeDNSStubDomain, testCase.configMap)
if err != nil {
t.Errorf("unexpected error: %v", err)
}

2
cmd/kubeadm/app/phases/addons/dns/manifests.go
View File

@@ -313,7 +313,7 @@ data:
fallthrough in-addr.arpa ip6.arpa
}{{ .Federation }}
prometheus :9153
proxy . {{ .UpstreamNameserver }}
forward . {{ .UpstreamNameserver }}
cache 30
loop
reload

6
cmd/kubeadm/app/phases/addons/proxy/proxy.go
View File

@@ -47,13 +47,13 @@ const (
)
// EnsureProxyAddon creates the kube-proxy addons
func EnsureProxyAddon(cfg *kubeadmapi.InitConfiguration, client clientset.Interface) error {
func EnsureProxyAddon(cfg *kubeadmapi.ClusterConfiguration, localEndpoint *kubeadmapi.APIEndpoint, client clientset.Interface) error {
if err := CreateServiceAccount(client); err != nil {
return errors.Wrap(err, "error when creating kube-proxy service account")
}
// Generate Master Enpoint kubeconfig file
masterEndpoint, err := kubeadmutil.GetMasterEndpoint(cfg)
masterEndpoint, err := kubeadmutil.GetMasterEndpoint(cfg.ControlPlaneEndpoint, localEndpoint)
if err != nil {
return err
}
@@ -81,7 +81,7 @@ func EnsureProxyAddon(cfg *kubeadmapi.InitConfiguration, client clientset.Interf
return errors.Wrap(err, "error when parsing kube-proxy configmap template")
}
proxyDaemonSetBytes, err = kubeadmutil.ParseTemplate(KubeProxyDaemonSet19, struct{ Image, ProxyConfigMap, ProxyConfigMapKey string }{
Image: images.GetKubernetesImage(constants.KubeProxy, &cfg.ClusterConfiguration),
Image: images.GetKubernetesImage(constants.KubeProxy, cfg),
ProxyConfigMap: constants.KubeProxyConfigMap,
ProxyConfigMapKey: constants.KubeProxyConfigMapKey,
})

2
cmd/kubeadm/app/phases/addons/proxy/proxy_test.go
View File

@@ -223,7 +223,7 @@ func TestEnsureProxyAddon(t *testing.T) {
t.Errorf("test failed to set dynamic defaults: %v", err)
break
}
err = EnsureProxyAddon(intMaster, client)
err = EnsureProxyAddon(&intMaster.ClusterConfiguration, &intMaster.LocalAPIEndpoint, client)
// Compare actual to expected errors
actErr := "No error"

15
cmd/kubeadm/app/phases/certs/certs.go
View File

@@ -27,8 +27,9 @@ import (
certutil "k8s.io/client-go/util/cert"
"k8s.io/klog"
kubeadmapi "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm"
pkiutil "k8s.io/kubernetes/cmd/kubeadm/app/util/pkiutil"
kubeadmconstants "k8s.io/kubernetes/cmd/kubeadm/app/constants"
"k8s.io/kubernetes/cmd/kubeadm/app/util/pkiutil"
)
// CreatePKIAssets will create and write to disk all PKI assets necessary to establish the control plane.
@@ -59,12 +60,12 @@ func CreatePKIAssets(cfg *kubeadmapi.InitConfiguration) error {
fmt.Printf("[certs] valid certificates and keys now exist in %q\n", cfg.CertificatesDir)
// Service accounts are not x509 certs, so handled separately
return CreateServiceAccountKeyAndPublicKeyFiles(cfg)
return CreateServiceAccountKeyAndPublicKeyFiles(cfg.CertificatesDir)
}
// CreateServiceAccountKeyAndPublicKeyFiles create a new public/private key files for signing service account users.
// If the sa public/private key files already exists in the target folder, they are used only if evaluated equals; otherwise an error is returned.
func CreateServiceAccountKeyAndPublicKeyFiles(cfg *kubeadmapi.InitConfiguration) error {
func CreateServiceAccountKeyAndPublicKeyFiles(certsDir string) error {
klog.V(1).Infoln("creating a new public/private key files for signing service account users")
saSigningKey, err := NewServiceAccountSigningKey()
if err != nil {
@@ -72,7 +73,7 @@ func CreateServiceAccountKeyAndPublicKeyFiles(cfg *kubeadmapi.InitConfiguration)
}
return writeKeyFilesIfNotExist(
cfg.CertificatesDir,
certsDir,
kubeadmconstants.ServiceAccountKeyBaseName,
saSigningKey,
)
@@ -81,7 +82,7 @@ func CreateServiceAccountKeyAndPublicKeyFiles(cfg *kubeadmapi.InitConfiguration)
// NewServiceAccountSigningKey generate public/private key pairs for signing service account tokens.
func NewServiceAccountSigningKey() (*rsa.PrivateKey, error) {
// The key does NOT exist, let's generate it now
saSigningKey, err := certutil.NewPrivateKey()
saSigningKey, err := pkiutil.NewPrivateKey()
if err != nil {
return nil, errors.Wrap(err, "failure while creating service account token signing key")
}
@@ -328,7 +329,7 @@ type certKeyLocation struct {
// SharedCertificateExists verifies if the shared certificates - the certificates that must be
// equal across masters: ca.key, ca.crt, sa.key, sa.pub + etcd/ca.key, etcd/ca.crt if local/stacked etcd
func SharedCertificateExists(cfg *kubeadmapi.InitConfiguration) (bool, error) {
func SharedCertificateExists(cfg *kubeadmapi.ClusterConfiguration) (bool, error) {
if err := validateCACertAndKey(certKeyLocation{cfg.CertificatesDir, kubeadmconstants.CACertAndKeyBaseName, "", "CA"}); err != nil {
return false, err
@@ -355,7 +356,7 @@ func SharedCertificateExists(cfg *kubeadmapi.InitConfiguration) (bool, error) {
// UsingExternalCA determines whether the user is relying on an external CA. We currently implicitly determine this is the case
// when both the CA Cert and the front proxy CA Cert are present but the CA Key and front proxy CA Key are not.
// This allows us to, e.g., skip generating certs or not start the csr signing controller.
func UsingExternalCA(cfg *kubeadmapi.InitConfiguration) (bool, error) {
func UsingExternalCA(cfg *kubeadmapi.ClusterConfiguration) (bool, error) {
if err := validateCACert(certKeyLocation{cfg.CertificatesDir, kubeadmconstants.CACertAndKeyBaseName, "", "CA"}); err != nil {
return false, err

8
cmd/kubeadm/app/phases/certs/certs_test.go
View File

@@ -463,10 +463,8 @@ func TestSharedCertificateExists(t *testing.T) {
os.MkdirAll(tmpdir+"/etcd", os.ModePerm)
defer os.RemoveAll(tmpdir)
cfg := &kubeadmapi.InitConfiguration{
ClusterConfiguration: kubeadmapi.ClusterConfiguration{
CertificatesDir: tmpdir,
},
cfg := &kubeadmapi.ClusterConfiguration{
CertificatesDir: tmpdir,
}
// created expected keys
@@ -554,7 +552,7 @@ func TestUsingExternalCA(t *testing.T) {
}
}
if val, _ := UsingExternalCA(cfg); val != test.expected {
if val, _ := UsingExternalCA(&cfg.ClusterConfiguration); val != test.expected {
t.Errorf("UsingExternalCA did not match expected: %v", test.expected)
}
}

3
cmd/kubeadm/app/phases/certs/renewal/certsapi.go
View File

@@ -31,6 +31,7 @@ import (
certstype "k8s.io/client-go/kubernetes/typed/certificates/v1beta1"
certutil "k8s.io/client-go/util/cert"
csrutil "k8s.io/client-go/util/certificate/csr"
pkiutil "k8s.io/kubernetes/cmd/kubeadm/app/util/pkiutil"
)
const certAPIPrefixName = "kubeadm-cert"
@@ -60,7 +61,7 @@ func (r *CertsAPIRenewal) Renew(cfg *certutil.Config) (*x509.Certificate, *rsa.P
IPAddresses: cfg.AltNames.IPs,
}
key, err := certutil.NewPrivateKey()
key, err := pkiutil.NewPrivateKey()
if err != nil {
return nil, nil, errors.Wrap(err, "couldn't create new private key")
}

2
cmd/kubeadm/app/phases/certs/renewal/renewal_test.go
View File

@@ -136,7 +136,7 @@ func getCertReq(t *testing.T, caCert *x509.Certificate, caKey *rsa.PrivateKey) *
Type: certsapi.CertificateApproved,
},
},
Certificate: certutil.EncodeCertPEM(cert),
Certificate: pkiutil.EncodeCertPEM(cert),
},
}
}

1
cmd/kubeadm/app/phases/controlplane/BUILD
View File

@@ -41,6 +41,7 @@ go_library(
"//cmd/kubeadm/app/util/staticpod:go_default_library",
"//pkg/kubeapiserver/authorizer/modes:go_default_library",
"//staging/src/k8s.io/api/core/v1:go_default_library",
"//staging/src/k8s.io/apimachinery/pkg/util/intstr:go_default_library",
"//staging/src/k8s.io/apimachinery/pkg/util/sets:go_default_library",
"//staging/src/k8s.io/apimachinery/pkg/util/version:go_default_library",
"//vendor/github.com/pkg/errors:go_default_library",

53
cmd/kubeadm/app/phases/controlplane/manifests.go
View File

@@ -26,6 +26,7 @@ import (
"github.com/pkg/errors"
"k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/util/intstr"
"k8s.io/apimachinery/pkg/util/version"
"k8s.io/klog"
kubeadmapi "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm"
@@ -40,12 +41,12 @@ import (
// CreateInitStaticPodManifestFiles will write all static pod manifest files needed to bring up the control plane.
func CreateInitStaticPodManifestFiles(manifestDir string, cfg *kubeadmapi.InitConfiguration) error {
klog.V(1).Infoln("[control-plane] creating static Pod files")
return CreateStaticPodFiles(manifestDir, cfg, kubeadmconstants.KubeAPIServer, kubeadmconstants.KubeControllerManager, kubeadmconstants.KubeScheduler)
return CreateStaticPodFiles(manifestDir, &cfg.ClusterConfiguration, &cfg.LocalAPIEndpoint, kubeadmconstants.KubeAPIServer, kubeadmconstants.KubeControllerManager, kubeadmconstants.KubeScheduler)
}
// GetStaticPodSpecs returns all staticPodSpecs actualized to the context of the current InitConfiguration
// GetStaticPodSpecs returns all staticPodSpecs actualized to the context of the current configuration
// NB. this methods holds the information about how kubeadm creates static pod manifests.
func GetStaticPodSpecs(cfg *kubeadmapi.InitConfiguration, k8sVersion *version.Version) map[string]v1.Pod {
func GetStaticPodSpecs(cfg *kubeadmapi.ClusterConfiguration, endpoint *kubeadmapi.APIEndpoint, k8sVersion *version.Version) map[string]v1.Pod {
// Get the required hostpath mounts
mounts := getHostPathVolumesForTheControlPlane(cfg)
@@ -53,31 +54,31 @@ func GetStaticPodSpecs(cfg *kubeadmapi.InitConfiguration, k8sVersion *version.Ve
staticPodSpecs := map[string]v1.Pod{
kubeadmconstants.KubeAPIServer: staticpodutil.ComponentPod(v1.Container{
Name: kubeadmconstants.KubeAPIServer,
Image: images.GetKubernetesImage(kubeadmconstants.KubeAPIServer, &cfg.ClusterConfiguration),
Image: images.GetKubernetesImage(kubeadmconstants.KubeAPIServer, cfg),
ImagePullPolicy: v1.PullIfNotPresent,
Command: getAPIServerCommand(cfg),
Command: getAPIServerCommand(cfg, endpoint),
VolumeMounts: staticpodutil.VolumeMountMapToSlice(mounts.GetVolumeMounts(kubeadmconstants.KubeAPIServer)),
LivenessProbe: staticpodutil.ComponentProbe(cfg, kubeadmconstants.KubeAPIServer, int(cfg.LocalAPIEndpoint.BindPort), "/healthz", v1.URISchemeHTTPS),
LivenessProbe: livenessProbe(staticpodutil.GetAPIServerProbeAddress(endpoint), int(endpoint.BindPort), v1.URISchemeHTTPS),
Resources: staticpodutil.ComponentResources("250m"),
Env: getProxyEnvVars(),
}, mounts.GetVolumes(kubeadmconstants.KubeAPIServer)),
kubeadmconstants.KubeControllerManager: staticpodutil.ComponentPod(v1.Container{
Name: kubeadmconstants.KubeControllerManager,
Image: images.GetKubernetesImage(kubeadmconstants.KubeControllerManager, &cfg.ClusterConfiguration),
Image: images.GetKubernetesImage(kubeadmconstants.KubeControllerManager, cfg),
ImagePullPolicy: v1.PullIfNotPresent,
Command: getControllerManagerCommand(cfg, k8sVersion),
VolumeMounts: staticpodutil.VolumeMountMapToSlice(mounts.GetVolumeMounts(kubeadmconstants.KubeControllerManager)),
LivenessProbe: staticpodutil.ComponentProbe(cfg, kubeadmconstants.KubeControllerManager, 10252, "/healthz", v1.URISchemeHTTP),
LivenessProbe: livenessProbe(staticpodutil.GetControllerManagerProbeAddress(cfg), 10252, v1.URISchemeHTTP),
Resources: staticpodutil.ComponentResources("200m"),
Env: getProxyEnvVars(),
}, mounts.GetVolumes(kubeadmconstants.KubeControllerManager)),
kubeadmconstants.KubeScheduler: staticpodutil.ComponentPod(v1.Container{
Name: kubeadmconstants.KubeScheduler,
Image: images.GetKubernetesImage(kubeadmconstants.KubeScheduler, &cfg.ClusterConfiguration),
Image: images.GetKubernetesImage(kubeadmconstants.KubeScheduler, cfg),
ImagePullPolicy: v1.PullIfNotPresent,
Command: getSchedulerCommand(cfg),
VolumeMounts: staticpodutil.VolumeMountMapToSlice(mounts.GetVolumeMounts(kubeadmconstants.KubeScheduler)),
LivenessProbe: staticpodutil.ComponentProbe(cfg, kubeadmconstants.KubeScheduler, 10251, "/healthz", v1.URISchemeHTTP),
LivenessProbe: livenessProbe(staticpodutil.GetSchedulerProbeAddress(cfg), 10251, v1.URISchemeHTTP),
Resources: staticpodutil.ComponentResources("100m"),
Env: getProxyEnvVars(),
}, mounts.GetVolumes(kubeadmconstants.KubeScheduler)),
@@ -85,17 +86,33 @@ func GetStaticPodSpecs(cfg *kubeadmapi.InitConfiguration, k8sVersion *version.Ve
return staticPodSpecs
}
func livenessProbe(host string, port int, scheme v1.URIScheme) *v1.Probe {
return &v1.Probe{
Handler: v1.Handler{
HTTPGet: &v1.HTTPGetAction{
Host: host,
Path: "/healthz",
Port: intstr.FromInt(port),
Scheme: scheme,
},
},
InitialDelaySeconds: 15,
TimeoutSeconds: 15,
FailureThreshold: 8,
}
}
// CreateStaticPodFiles creates all the requested static pod files.
func CreateStaticPodFiles(manifestDir string, cfg *kubeadmapi.InitConfiguration, componentNames ...string) error {
func CreateStaticPodFiles(manifestDir string, cfg *kubeadmapi.ClusterConfiguration, endpoint *kubeadmapi.APIEndpoint, componentNames ...string) error {
// TODO: Move the "pkg/util/version".Version object into the internal API instead of always parsing the string
k8sVersion, err := version.ParseSemantic(cfg.KubernetesVersion)
if err != nil {
return err
}
// gets the StaticPodSpecs, actualized for the current InitConfiguration
// gets the StaticPodSpecs, actualized for the current ClusterConfiguration
klog.V(1).Infoln("[control-plane] getting StaticPodSpecs")
specs := GetStaticPodSpecs(cfg, k8sVersion)
specs := GetStaticPodSpecs(cfg, endpoint, k8sVersion)
// creates required static pod specs
for _, componentName := range componentNames {
@@ -117,9 +134,9 @@ func CreateStaticPodFiles(manifestDir string, cfg *kubeadmapi.InitConfiguration,
}
// getAPIServerCommand builds the right API server command from the given config object and version
func getAPIServerCommand(cfg *kubeadmapi.InitConfiguration) []string {
func getAPIServerCommand(cfg *kubeadmapi.ClusterConfiguration, localAPIEndpoint *kubeadmapi.APIEndpoint) []string {
defaultArguments := map[string]string{
"advertise-address": cfg.LocalAPIEndpoint.AdvertiseAddress,
"advertise-address": localAPIEndpoint.AdvertiseAddress,
"enable-admission-plugins": "NodeRestriction",
"service-cluster-ip-range": cfg.Networking.ServiceSubnet,
"service-account-key-file": filepath.Join(cfg.CertificatesDir, kubeadmconstants.ServiceAccountPublicKeyName),
@@ -129,7 +146,7 @@ func getAPIServerCommand(cfg *kubeadmapi.InitConfiguration) []string {
"kubelet-client-certificate": filepath.Join(cfg.CertificatesDir, kubeadmconstants.APIServerKubeletClientCertName),
"kubelet-client-key": filepath.Join(cfg.CertificatesDir, kubeadmconstants.APIServerKubeletClientKeyName),
"enable-bootstrap-token-auth": "true",
"secure-port": fmt.Sprintf("%d", cfg.LocalAPIEndpoint.BindPort),
"secure-port": fmt.Sprintf("%d", localAPIEndpoint.BindPort),
"allow-privileged": "true",
"kubelet-preferred-address-types": "InternalIP,ExternalIP,Hostname",
// add options to configure the front proxy. Without the generated client cert, this will never be useable
@@ -243,7 +260,7 @@ func calcNodeCidrSize(podSubnet string) string {
}
// getControllerManagerCommand builds the right controller manager command from the given config object and version
func getControllerManagerCommand(cfg *kubeadmapi.InitConfiguration, k8sVersion *version.Version) []string {
func getControllerManagerCommand(cfg *kubeadmapi.ClusterConfiguration, k8sVersion *version.Version) []string {
kubeconfigFile := filepath.Join(kubeadmconstants.KubernetesDir, kubeadmconstants.ControllerManagerKubeConfigFileName)
caFile := filepath.Join(cfg.CertificatesDir, kubeadmconstants.CACertName)
@@ -287,7 +304,7 @@ func getControllerManagerCommand(cfg *kubeadmapi.InitConfiguration, k8sVersion *
}
// getSchedulerCommand builds the right scheduler command from the given config object and version
func getSchedulerCommand(cfg *kubeadmapi.InitConfiguration) []string {
func getSchedulerCommand(cfg *kubeadmapi.ClusterConfiguration) []string {
defaultArguments := map[string]string{
"bind-address": "127.0.0.1",
"leader-elect": "true",

175
cmd/kubeadm/app/phases/controlplane/manifests_test.go
View File

@@ -44,11 +44,9 @@ var cpVersion = kubeadmconstants.MinimumControlPlaneVersion.WithPreRelease("beta
func TestGetStaticPodSpecs(t *testing.T) {
// Creates a Master Configuration
cfg := &kubeadmapi.InitConfiguration{
ClusterConfiguration: kubeadmapi.ClusterConfiguration{
KubernetesVersion: "v1.9.0",
},
// Creates a Cluster Configuration
cfg := &kubeadmapi.ClusterConfiguration{
KubernetesVersion: "v1.9.0",
}
// Executes GetStaticPodSpecs
@@ -56,7 +54,7 @@ func TestGetStaticPodSpecs(t *testing.T) {
// TODO: Move the "pkg/util/version".Version object into the internal API instead of always parsing the string
k8sVersion, _ := version.ParseSemantic(cfg.KubernetesVersion)
specs := GetStaticPodSpecs(cfg, k8sVersion)
specs := GetStaticPodSpecs(cfg, &kubeadmapi.APIEndpoint{}, k8sVersion)
var assertions = []struct {
staticPodName string
@@ -113,16 +111,14 @@ func TestCreateStaticPodFilesAndWrappers(t *testing.T) {
tmpdir := testutil.SetupTempDir(t)
defer os.RemoveAll(tmpdir)
// Creates a Master Configuration
cfg := &kubeadmapi.InitConfiguration{
ClusterConfiguration: kubeadmapi.ClusterConfiguration{
KubernetesVersion: "v1.9.0",
},
// Creates a Cluster Configuration
cfg := &kubeadmapi.ClusterConfiguration{
KubernetesVersion: "v1.9.0",
}
// Execute createStaticPodFunction
manifestPath := filepath.Join(tmpdir, kubeadmconstants.ManifestsSubDirName)
err := CreateStaticPodFiles(manifestPath, cfg, test.components...)
err := CreateStaticPodFiles(manifestPath, cfg, &kubeadmapi.APIEndpoint{}, test.components...)
if err != nil {
t.Errorf("Error executing createStaticPodFunction: %v", err)
continue
@@ -140,18 +136,17 @@ func TestCreateStaticPodFilesAndWrappers(t *testing.T) {
func TestGetAPIServerCommand(t *testing.T) {
var tests = []struct {
name string
cfg *kubeadmapi.InitConfiguration
cfg *kubeadmapi.ClusterConfiguration
endpoint *kubeadmapi.APIEndpoint
expected []string
}{
{
name: "testing defaults",
cfg: &kubeadmapi.InitConfiguration{
LocalAPIEndpoint: kubeadmapi.APIEndpoint{BindPort: 123, AdvertiseAddress: "1.2.3.4"},
ClusterConfiguration: kubeadmapi.ClusterConfiguration{
Networking: kubeadmapi.Networking{ServiceSubnet: "bar"},
CertificatesDir: testCertsDir,
},
cfg: &kubeadmapi.ClusterConfiguration{
Networking: kubeadmapi.Networking{ServiceSubnet: "bar"},
CertificatesDir: testCertsDir,
},
endpoint: &kubeadmapi.APIEndpoint{BindPort: 123, AdvertiseAddress: "1.2.3.4"},
expected: []string{
"kube-apiserver",
"--enable-admission-plugins=NodeRestriction",
@@ -183,13 +178,11 @@ func TestGetAPIServerCommand(t *testing.T) {
},
{
name: "ipv6 advertise address",
cfg: &kubeadmapi.InitConfiguration{
LocalAPIEndpoint: kubeadmapi.APIEndpoint{BindPort: 123, AdvertiseAddress: "2001:db8::1"},
ClusterConfiguration: kubeadmapi.ClusterConfiguration{
Networking: kubeadmapi.Networking{ServiceSubnet: "bar"},
CertificatesDir: testCertsDir,
},
cfg: &kubeadmapi.ClusterConfiguration{
Networking: kubeadmapi.Networking{ServiceSubnet: "bar"},
CertificatesDir: testCertsDir,
},
endpoint: &kubeadmapi.APIEndpoint{BindPort: 123, AdvertiseAddress: "2001:db8::1"},
expected: []string{
"kube-apiserver",
"--enable-admission-plugins=NodeRestriction",
@@ -221,21 +214,19 @@ func TestGetAPIServerCommand(t *testing.T) {
},
{
name: "an external etcd with custom ca, certs and keys",
cfg: &kubeadmapi.InitConfiguration{
LocalAPIEndpoint: kubeadmapi.APIEndpoint{BindPort: 123, AdvertiseAddress: "2001:db8::1"},
ClusterConfiguration: kubeadmapi.ClusterConfiguration{
Networking: kubeadmapi.Networking{ServiceSubnet: "bar"},
Etcd: kubeadmapi.Etcd{
External: &kubeadmapi.ExternalEtcd{
Endpoints: []string{"https://8.6.4.1:2379", "https://8.6.4.2:2379"},
CAFile: "fuz",
CertFile: "fiz",
KeyFile: "faz",
},
cfg: &kubeadmapi.ClusterConfiguration{
Networking: kubeadmapi.Networking{ServiceSubnet: "bar"},
Etcd: kubeadmapi.Etcd{
External: &kubeadmapi.ExternalEtcd{
Endpoints: []string{"https://8.6.4.1:2379", "https://8.6.4.2:2379"},
CAFile: "fuz",
CertFile: "fiz",
KeyFile: "faz",
},
CertificatesDir: testCertsDir,
},
CertificatesDir: testCertsDir,
},
endpoint: &kubeadmapi.APIEndpoint{BindPort: 123, AdvertiseAddress: "2001:db8::1"},
expected: []string{
"kube-apiserver",
"--enable-admission-plugins=NodeRestriction",
@@ -267,18 +258,16 @@ func TestGetAPIServerCommand(t *testing.T) {
},
{
name: "an insecure etcd",
cfg: &kubeadmapi.InitConfiguration{
LocalAPIEndpoint: kubeadmapi.APIEndpoint{BindPort: 123, AdvertiseAddress: "2001:db8::1"},
ClusterConfiguration: kubeadmapi.ClusterConfiguration{
Networking: kubeadmapi.Networking{ServiceSubnet: "bar"},
Etcd: kubeadmapi.Etcd{
External: &kubeadmapi.ExternalEtcd{
Endpoints: []string{"http://127.0.0.1:2379", "http://127.0.0.1:2380"},
},
cfg: &kubeadmapi.ClusterConfiguration{
Networking: kubeadmapi.Networking{ServiceSubnet: "bar"},
Etcd: kubeadmapi.Etcd{
External: &kubeadmapi.ExternalEtcd{
Endpoints: []string{"http://127.0.0.1:2379", "http://127.0.0.1:2380"},
},
CertificatesDir: testCertsDir,
},
CertificatesDir: testCertsDir,
},
endpoint: &kubeadmapi.APIEndpoint{BindPort: 123, AdvertiseAddress: "2001:db8::1"},
expected: []string{
"kube-apiserver",
"--enable-admission-plugins=NodeRestriction",
@@ -307,23 +296,21 @@ func TestGetAPIServerCommand(t *testing.T) {
},
{
name: "test APIServer.ExtraArgs works as expected",
cfg: &kubeadmapi.InitConfiguration{
LocalAPIEndpoint: kubeadmapi.APIEndpoint{BindPort: 123, AdvertiseAddress: "1.2.3.4"},
ClusterConfiguration: kubeadmapi.ClusterConfiguration{
Networking: kubeadmapi.Networking{ServiceSubnet: "bar"},
CertificatesDir: testCertsDir,
APIServer: kubeadmapi.APIServer{
ControlPlaneComponent: kubeadmapi.ControlPlaneComponent{
ExtraArgs: map[string]string{
"service-cluster-ip-range": "baz",
"advertise-address": "9.9.9.9",
"audit-policy-file": "/etc/config/audit.yaml",
"audit-log-path": "/var/log/kubernetes",
},
cfg: &kubeadmapi.ClusterConfiguration{
Networking: kubeadmapi.Networking{ServiceSubnet: "bar"},
CertificatesDir: testCertsDir,
APIServer: kubeadmapi.APIServer{
ControlPlaneComponent: kubeadmapi.ControlPlaneComponent{
ExtraArgs: map[string]string{
"service-cluster-ip-range": "baz",
"advertise-address": "9.9.9.9",
"audit-policy-file": "/etc/config/audit.yaml",
"audit-log-path": "/var/log/kubernetes",
},
},
},
},
endpoint: &kubeadmapi.APIEndpoint{BindPort: 123, AdvertiseAddress: "1.2.3.4"},
expected: []string{
"kube-apiserver",
"--enable-admission-plugins=NodeRestriction",
@@ -357,20 +344,18 @@ func TestGetAPIServerCommand(t *testing.T) {
},
{
name: "authorization-mode extra-args ABAC",
cfg: &kubeadmapi.InitConfiguration{
LocalAPIEndpoint: kubeadmapi.APIEndpoint{BindPort: 123, AdvertiseAddress: "1.2.3.4"},
ClusterConfiguration: kubeadmapi.ClusterConfiguration{
Networking: kubeadmapi.Networking{ServiceSubnet: "bar"},
CertificatesDir: testCertsDir,
APIServer: kubeadmapi.APIServer{
ControlPlaneComponent: kubeadmapi.ControlPlaneComponent{
ExtraArgs: map[string]string{
"authorization-mode": authzmodes.ModeABAC,
},
cfg: &kubeadmapi.ClusterConfiguration{
Networking: kubeadmapi.Networking{ServiceSubnet: "bar"},
CertificatesDir: testCertsDir,
APIServer: kubeadmapi.APIServer{
ControlPlaneComponent: kubeadmapi.ControlPlaneComponent{
ExtraArgs: map[string]string{
"authorization-mode": authzmodes.ModeABAC,
},
},
},
},
endpoint: &kubeadmapi.APIEndpoint{BindPort: 123, AdvertiseAddress: "1.2.3.4"},
expected: []string{
"kube-apiserver",
"--enable-admission-plugins=NodeRestriction",
@@ -402,20 +387,18 @@ func TestGetAPIServerCommand(t *testing.T) {
},
{
name: "secure-port extra-args",
cfg: &kubeadmapi.InitConfiguration{
LocalAPIEndpoint: kubeadmapi.APIEndpoint{BindPort: 123, AdvertiseAddress: "1.2.3.4"},
ClusterConfiguration: kubeadmapi.ClusterConfiguration{
Networking: kubeadmapi.Networking{ServiceSubnet: "bar"},
CertificatesDir: testCertsDir,
APIServer: kubeadmapi.APIServer{
ControlPlaneComponent: kubeadmapi.ControlPlaneComponent{
ExtraArgs: map[string]string{
"secure-port": "123",
},
cfg: &kubeadmapi.ClusterConfiguration{
Networking: kubeadmapi.Networking{ServiceSubnet: "bar"},
CertificatesDir: testCertsDir,
APIServer: kubeadmapi.APIServer{
ControlPlaneComponent: kubeadmapi.ControlPlaneComponent{
ExtraArgs: map[string]string{
"secure-port": "123",
},
},
},
},
endpoint: &kubeadmapi.APIEndpoint{BindPort: 123, AdvertiseAddress: "1.2.3.4"},
expected: []string{
"kube-apiserver",
"--enable-admission-plugins=NodeRestriction",
@@ -447,20 +430,18 @@ func TestGetAPIServerCommand(t *testing.T) {
},
{
name: "authorization-mode extra-args Webhook",
cfg: &kubeadmapi.InitConfiguration{
LocalAPIEndpoint: kubeadmapi.APIEndpoint{BindPort: 123, AdvertiseAddress: "1.2.3.4"},
ClusterConfiguration: kubeadmapi.ClusterConfiguration{
Networking: kubeadmapi.Networking{ServiceSubnet: "bar"},
CertificatesDir: testCertsDir,
APIServer: kubeadmapi.APIServer{
ControlPlaneComponent: kubeadmapi.ControlPlaneComponent{
ExtraArgs: map[string]string{
"authorization-mode": authzmodes.ModeWebhook,
},
cfg: &kubeadmapi.ClusterConfiguration{
Networking: kubeadmapi.Networking{ServiceSubnet: "bar"},
CertificatesDir: testCertsDir,
APIServer: kubeadmapi.APIServer{
ControlPlaneComponent: kubeadmapi.ControlPlaneComponent{
ExtraArgs: map[string]string{
"authorization-mode": authzmodes.ModeWebhook,
},
},
},
},
endpoint: &kubeadmapi.APIEndpoint{BindPort: 123, AdvertiseAddress: "1.2.3.4"},
expected: []string{
"kube-apiserver",
"--enable-admission-plugins=NodeRestriction",
@@ -494,7 +475,7 @@ func TestGetAPIServerCommand(t *testing.T) {
for _, rt := range tests {
t.Run(rt.name, func(t *testing.T) {
actual := getAPIServerCommand(rt.cfg)
actual := getAPIServerCommand(rt.cfg, rt.endpoint)
sort.Strings(actual)
sort.Strings(rt.expected)
if !reflect.DeepEqual(actual, rt.expected) {
@@ -638,11 +619,7 @@ func TestGetControllerManagerCommand(t *testing.T) {
}
for _, rt := range tests {
// TODO: Make getControllerManagerCommand accept a ClusterConfiguration object instead of InitConfiguration
initcfg := &kubeadmapi.InitConfiguration{
ClusterConfiguration: *rt.cfg,
}
actual := getControllerManagerCommand(initcfg, version.MustParseSemantic(rt.cfg.KubernetesVersion))
actual := getControllerManagerCommand(rt.cfg, version.MustParseSemantic(rt.cfg.KubernetesVersion))
sort.Strings(actual)
sort.Strings(rt.expected)
if !reflect.DeepEqual(actual, rt.expected) {
@@ -815,7 +792,7 @@ func TestGetControllerManagerCommandExternalCA(t *testing.T) {
}
}
actual := getControllerManagerCommand(test.cfg, version.MustParseSemantic(test.cfg.KubernetesVersion))
actual := getControllerManagerCommand(&test.cfg.ClusterConfiguration, version.MustParseSemantic(test.cfg.KubernetesVersion))
expected := test.expectedArgFunc(tmpdir)
sort.Strings(actual)
sort.Strings(expected)
@@ -844,11 +821,7 @@ func TestGetSchedulerCommand(t *testing.T) {
}
for _, rt := range tests {
// TODO: Make getSchedulerCommand accept a ClusterConfiguration object instead of InitConfiguration
initcfg := &kubeadmapi.InitConfiguration{
ClusterConfiguration: *rt.cfg,
}
actual := getSchedulerCommand(initcfg)
actual := getSchedulerCommand(rt.cfg)
sort.Strings(actual)
sort.Strings(rt.expected)
if !reflect.DeepEqual(actual, rt.expected) {

2
cmd/kubeadm/app/phases/controlplane/volumes.go
View File

@@ -42,7 +42,7 @@ const (
var caCertsExtraVolumePaths = []string{"/etc/pki", "/usr/share/ca-certificates", "/usr/local/share/ca-certificates", "/etc/ca-certificates"}
// getHostPathVolumesForTheControlPlane gets the required hostPath volumes and mounts for the control plane
func getHostPathVolumesForTheControlPlane(cfg *kubeadmapi.InitConfiguration) controlPlaneHostPathMounts {
func getHostPathVolumesForTheControlPlane(cfg *kubeadmapi.ClusterConfiguration) controlPlaneHostPathMounts {
hostPathDirectoryOrCreate := v1.HostPathDirectoryOrCreate
hostPathFileOrCreate := v1.HostPathFileOrCreate
mounts := newControlPlaneHostPathMounts()

6
cmd/kubeadm/app/phases/controlplane/volumes_test.go
View File

@@ -514,11 +514,7 @@ func TestGetHostPathVolumesForTheControlPlane(t *testing.T) {
defer func() { caCertsExtraVolumePaths = []string{"/etc/pki", "/usr/share/ca-certificates"} }()
for _, rt := range tests {
// TODO: Make getHostPathVolumesForTheControlPlane accept a ClusterConfiguration object instead of InitConfiguration
initcfg := &kubeadmapi.InitConfiguration{
ClusterConfiguration: *rt.cfg,
}
mounts := getHostPathVolumesForTheControlPlane(initcfg)
mounts := getHostPathVolumesForTheControlPlane(rt.cfg)
// Avoid unit test errors when the flexvolume is mounted
if _, ok := mounts.volumes[kubeadmconstants.KubeControllerManager][flexvolumeDirVolumeName]; ok {

40
cmd/kubeadm/app/phases/etcd/local.go
View File

@@ -46,8 +46,8 @@ const (
// CreateLocalEtcdStaticPodManifestFile will write local etcd static pod manifest file.
// This function is used by init - when the etcd cluster is empty - or by kubeadm
// upgrade - when the etcd cluster is already up and running (and the --initial-cluster flag have no impact)
func CreateLocalEtcdStaticPodManifestFile(manifestDir string, cfg *kubeadmapi.InitConfiguration) error {
if cfg.ClusterConfiguration.Etcd.External != nil {
func CreateLocalEtcdStaticPodManifestFile(manifestDir string, nodeName string, cfg *kubeadmapi.ClusterConfiguration, endpoint *kubeadmapi.APIEndpoint) error {
if cfg.Etcd.External != nil {
return errors.New("etcd static pod manifest cannot be generated for cluster using external etcd")
}
// gets etcd StaticPodSpec
@@ -58,7 +58,7 @@ func CreateLocalEtcdStaticPodManifestFile(manifestDir string, cfg *kubeadmapi.In
return errors.Wrapf(err, "failed to create etcd directory %q", cfg.Etcd.Local.DataDir)
}
spec := GetEtcdPodSpec(cfg, emptyInitialCluster)
spec := GetEtcdPodSpec(cfg, endpoint, nodeName, emptyInitialCluster)
// writes etcd StaticPod to disk
if err := staticpodutil.WriteStaticPodToDisk(kubeadmconstants.Etcd, manifestDir, spec); err != nil {
return err
@@ -69,7 +69,7 @@ func CreateLocalEtcdStaticPodManifestFile(manifestDir string, cfg *kubeadmapi.In
}
// CheckLocalEtcdClusterStatus verifies health state of local/stacked etcd cluster before installing a new etcd member
func CheckLocalEtcdClusterStatus(client clientset.Interface, cfg *kubeadmapi.InitConfiguration) error {
func CheckLocalEtcdClusterStatus(client clientset.Interface, cfg *kubeadmapi.ClusterConfiguration) error {
fmt.Println("[etcd] Checking etcd cluster health")
// creates an etcd client that connects to all the local/stacked etcd members
@@ -91,7 +91,7 @@ func CheckLocalEtcdClusterStatus(client clientset.Interface, cfg *kubeadmapi.Ini
// CreateStackedEtcdStaticPodManifestFile will write local etcd static pod manifest file
// for an additional etcd member that is joining an existing local/stacked etcd cluster.
// Other members of the etcd cluster will be notified of the joining node in beforehand as well.
func CreateStackedEtcdStaticPodManifestFile(client clientset.Interface, manifestDir string, cfg *kubeadmapi.InitConfiguration) error {
func CreateStackedEtcdStaticPodManifestFile(client clientset.Interface, manifestDir string, nodeName string, cfg *kubeadmapi.ClusterConfiguration, endpoint *kubeadmapi.APIEndpoint) error {
// creates an etcd client that connects to all the local/stacked etcd members
klog.V(1).Info("creating etcd client that connects to etcd pods")
etcdClient, err := etcdutil.NewFromCluster(client, cfg.CertificatesDir)
@@ -100,10 +100,10 @@ func CreateStackedEtcdStaticPodManifestFile(client clientset.Interface, manifest
}
// notifies the other members of the etcd cluster about the joining member
etcdPeerAddress := etcdutil.GetPeerURL(cfg)
etcdPeerAddress := etcdutil.GetPeerURL(endpoint)
klog.V(1).Infof("Adding etcd member: %s", etcdPeerAddress)
initialCluster, err := etcdClient.AddMember(cfg.NodeRegistration.Name, etcdPeerAddress)
initialCluster, err := etcdClient.AddMember(nodeName, etcdPeerAddress)
if err != nil {
return err
}
@@ -117,7 +117,7 @@ func CreateStackedEtcdStaticPodManifestFile(client clientset.Interface, manifest
klog.V(1).Info("Creating local etcd static pod manifest file")
// gets etcd StaticPodSpec, actualized for the current InitConfiguration and the new list of etcd members
spec := GetEtcdPodSpec(cfg, initialCluster)
spec := GetEtcdPodSpec(cfg, endpoint, nodeName, initialCluster)
// writes etcd StaticPod to disk
if err := staticpodutil.WriteStaticPodToDisk(kubeadmconstants.Etcd, manifestDir, spec); err != nil {
return err
@@ -133,9 +133,9 @@ func CreateStackedEtcdStaticPodManifestFile(client clientset.Interface, manifest
return nil
}
// GetEtcdPodSpec returns the etcd static Pod actualized to the context of the current InitConfiguration
// GetEtcdPodSpec returns the etcd static Pod actualized to the context of the current configuration
// NB. GetEtcdPodSpec methods holds the information about how kubeadm creates etcd static pod manifests.
func GetEtcdPodSpec(cfg *kubeadmapi.InitConfiguration, initialCluster []etcdutil.Member) v1.Pod {
func GetEtcdPodSpec(cfg *kubeadmapi.ClusterConfiguration, endpoint *kubeadmapi.APIEndpoint, nodeName string, initialCluster []etcdutil.Member) v1.Pod {
pathType := v1.HostPathDirectoryOrCreate
etcdMounts := map[string]v1.Volume{
etcdVolumeName: staticpodutil.NewVolume(etcdVolumeName, cfg.Etcd.Local.DataDir, &pathType),
@@ -143,8 +143,8 @@ func GetEtcdPodSpec(cfg *kubeadmapi.InitConfiguration, initialCluster []etcdutil
}
return staticpodutil.ComponentPod(v1.Container{
Name: kubeadmconstants.Etcd,
Command: getEtcdCommand(cfg, initialCluster),
Image: images.GetEtcdImage(&cfg.ClusterConfiguration),
Command: getEtcdCommand(cfg, endpoint, nodeName, initialCluster),
Image: images.GetEtcdImage(cfg),
ImagePullPolicy: v1.PullIfNotPresent,
// Mount the etcd datadir path read-write so etcd can store data in a more persistent manner
VolumeMounts: []v1.VolumeMount{
@@ -152,20 +152,20 @@ func GetEtcdPodSpec(cfg *kubeadmapi.InitConfiguration, initialCluster []etcdutil
staticpodutil.NewVolumeMount(certsVolumeName, cfg.CertificatesDir+"/etcd", false),
},
LivenessProbe: staticpodutil.EtcdProbe(
cfg, kubeadmconstants.Etcd, kubeadmconstants.EtcdListenClientPort, cfg.CertificatesDir,
&cfg.Etcd, kubeadmconstants.EtcdListenClientPort, cfg.CertificatesDir,
kubeadmconstants.EtcdCACertName, kubeadmconstants.EtcdHealthcheckClientCertName, kubeadmconstants.EtcdHealthcheckClientKeyName,
),
}, etcdMounts)
}
// getEtcdCommand builds the right etcd command from the given config object
func getEtcdCommand(cfg *kubeadmapi.InitConfiguration, initialCluster []etcdutil.Member) []string {
func getEtcdCommand(cfg *kubeadmapi.ClusterConfiguration, endpoint *kubeadmapi.APIEndpoint, nodeName string, initialCluster []etcdutil.Member) []string {
defaultArguments := map[string]string{
"name": cfg.GetNodeName(),
"listen-client-urls": fmt.Sprintf("%s,%s", etcdutil.GetClientURLByIP("127.0.0.1"), etcdutil.GetClientURL(cfg)),
"advertise-client-urls": etcdutil.GetClientURL(cfg),
"listen-peer-urls": etcdutil.GetPeerURL(cfg),
"initial-advertise-peer-urls": etcdutil.GetPeerURL(cfg),
"name": nodeName,
"listen-client-urls": fmt.Sprintf("%s,%s", etcdutil.GetClientURLByIP("127.0.0.1"), etcdutil.GetClientURL(endpoint)),
"advertise-client-urls": etcdutil.GetClientURL(endpoint),
"listen-peer-urls": etcdutil.GetPeerURL(endpoint),
"initial-advertise-peer-urls": etcdutil.GetPeerURL(endpoint),
"data-dir": cfg.Etcd.Local.DataDir,
"cert-file": filepath.Join(cfg.CertificatesDir, kubeadmconstants.EtcdServerCertName),
"key-file": filepath.Join(cfg.CertificatesDir, kubeadmconstants.EtcdServerKeyName),
@@ -179,7 +179,7 @@ func getEtcdCommand(cfg *kubeadmapi.InitConfiguration, initialCluster []etcdutil
}
if len(initialCluster) == 0 {
defaultArguments["initial-cluster"] = fmt.Sprintf("%s=%s", cfg.GetNodeName(), etcdutil.GetPeerURL(cfg))
defaultArguments["initial-cluster"] = fmt.Sprintf("%s=%s", nodeName, etcdutil.GetPeerURL(endpoint))
} else {
// NB. the joining etcd member should be part of the initialCluster list
endpoints := []string{}

74
cmd/kubeadm/app/phases/etcd/local_test.go
View File

@@ -32,19 +32,18 @@ import (
func TestGetEtcdPodSpec(t *testing.T) {
// Creates a Master Configuration
cfg := &kubeadmapi.InitConfiguration{
ClusterConfiguration: kubeadmapi.ClusterConfiguration{
KubernetesVersion: "v1.7.0",
Etcd: kubeadmapi.Etcd{
Local: &kubeadmapi.LocalEtcd{
DataDir: "/var/lib/etcd",
},
cfg := &kubeadmapi.ClusterConfiguration{
KubernetesVersion: "v1.7.0",
Etcd: kubeadmapi.Etcd{
Local: &kubeadmapi.LocalEtcd{
DataDir: "/var/lib/etcd",
},
},
}
endpoint := &kubeadmapi.APIEndpoint{}
// Executes GetEtcdPodSpec
spec := GetEtcdPodSpec(cfg, []etcdutil.Member{})
spec := GetEtcdPodSpec(cfg, endpoint, "", []etcdutil.Member{})
// Assert each specs refers to the right pod
if spec.Spec.Containers[0].Name != kubeadmconstants.Etcd {
@@ -58,35 +57,31 @@ func TestCreateLocalEtcdStaticPodManifestFile(t *testing.T) {
defer os.RemoveAll(tmpdir)
var tests = []struct {
cfg *kubeadmapi.InitConfiguration
cfg *kubeadmapi.ClusterConfiguration
expectedError bool
}{
{
cfg: &kubeadmapi.InitConfiguration{
ClusterConfiguration: kubeadmapi.ClusterConfiguration{
KubernetesVersion: "v1.7.0",
Etcd: kubeadmapi.Etcd{
Local: &kubeadmapi.LocalEtcd{
DataDir: tmpdir + "/etcd",
},
cfg: &kubeadmapi.ClusterConfiguration{
KubernetesVersion: "v1.7.0",
Etcd: kubeadmapi.Etcd{
Local: &kubeadmapi.LocalEtcd{
DataDir: tmpdir + "/etcd",
},
},
},
expectedError: false,
},
{
cfg: &kubeadmapi.InitConfiguration{
ClusterConfiguration: kubeadmapi.ClusterConfiguration{
KubernetesVersion: "v1.7.0",
Etcd: kubeadmapi.Etcd{
External: &kubeadmapi.ExternalEtcd{
Endpoints: []string{
"https://etcd-instance:2379",
},
CAFile: "/etc/kubernetes/pki/etcd/ca.crt",
CertFile: "/etc/kubernetes/pki/etcd/apiserver-etcd-client.crt",
KeyFile: "/etc/kubernetes/pki/etcd/apiserver-etcd-client.key",
cfg: &kubeadmapi.ClusterConfiguration{
KubernetesVersion: "v1.7.0",
Etcd: kubeadmapi.Etcd{
External: &kubeadmapi.ExternalEtcd{
Endpoints: []string{
"https://etcd-instance:2379",
},
CAFile: "/etc/kubernetes/pki/etcd/ca.crt",
CertFile: "/etc/kubernetes/pki/etcd/apiserver-etcd-client.crt",
KeyFile: "/etc/kubernetes/pki/etcd/apiserver-etcd-client.key",
},
},
},
@@ -97,7 +92,7 @@ func TestCreateLocalEtcdStaticPodManifestFile(t *testing.T) {
for _, test := range tests {
// Execute createStaticPodFunction
manifestPath := filepath.Join(tmpdir, kubeadmconstants.ManifestsSubDirName)
err := CreateLocalEtcdStaticPodManifestFile(manifestPath, test.cfg)
err := CreateLocalEtcdStaticPodManifestFile(manifestPath, "", test.cfg, &kubeadmapi.APIEndpoint{})
if !test.expectedError {
if err != nil {
@@ -230,23 +225,18 @@ func TestGetEtcdCommand(t *testing.T) {
for _, rt := range tests {
t.Run(rt.name, func(t *testing.T) {
cfg := &kubeadmapi.InitConfiguration{
LocalAPIEndpoint: kubeadmapi.APIEndpoint{
AdvertiseAddress: rt.advertiseAddress,
},
NodeRegistration: kubeadmapi.NodeRegistrationOptions{
Name: rt.nodeName,
},
ClusterConfiguration: kubeadmapi.ClusterConfiguration{
Etcd: kubeadmapi.Etcd{
Local: &kubeadmapi.LocalEtcd{
DataDir: "/var/lib/etcd",
ExtraArgs: rt.extraArgs,
},
endpoint := &kubeadmapi.APIEndpoint{
AdvertiseAddress: rt.advertiseAddress,
}
cfg := &kubeadmapi.ClusterConfiguration{
Etcd: kubeadmapi.Etcd{
Local: &kubeadmapi.LocalEtcd{
DataDir: "/var/lib/etcd",
ExtraArgs: rt.extraArgs,
},
},
}
actual := getEtcdCommand(cfg, rt.initialCluster)
actual := getEtcdCommand(cfg, endpoint, rt.nodeName, rt.initialCluster)
sort.Strings(actual)
sort.Strings(rt.expected)
if !reflect.DeepEqual(actual, rt.expected) {

15
cmd/kubeadm/app/phases/kubeconfig/kubeconfig.go
View File

@@ -33,8 +33,9 @@ import (
kubeadmapi "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm"
kubeadmconstants "k8s.io/kubernetes/cmd/kubeadm/app/constants"
kubeadmutil "k8s.io/kubernetes/cmd/kubeadm/app/util"
pkiutil "k8s.io/kubernetes/cmd/kubeadm/app/util/pkiutil"
kubeconfigutil "k8s.io/kubernetes/cmd/kubeadm/app/util/kubeconfig"
"k8s.io/kubernetes/cmd/kubeadm/app/util/pkiutil"
)
// clientCertAuth struct holds info required to build a client certificate to provide authentication info in a kubeconfig object
@@ -134,7 +135,7 @@ func getKubeConfigSpecs(cfg *kubeadmapi.InitConfiguration) (map[string]*kubeConf
return nil, errors.Wrap(err, "couldn't create a kubeconfig; the CA files couldn't be loaded")
}
masterEndpoint, err := kubeadmutil.GetMasterEndpoint(cfg)
masterEndpoint, err := kubeadmutil.GetMasterEndpoint(cfg.ControlPlaneEndpoint, &cfg.LocalAPIEndpoint)
if err != nil {
return nil, err
}
@@ -189,7 +190,7 @@ func buildKubeConfigFromSpec(spec *kubeConfigSpec, clustername string) (*clientc
spec.APIServer,
clustername,
spec.ClientName,
certutil.EncodeCertPEM(spec.CACert),
pkiutil.EncodeCertPEM(spec.CACert),
spec.TokenAuth.Token,
), nil
}
@@ -210,9 +211,9 @@ func buildKubeConfigFromSpec(spec *kubeConfigSpec, clustername string) (*clientc
spec.APIServer,
clustername,
spec.ClientName,
certutil.EncodeCertPEM(spec.CACert),
pkiutil.EncodeCertPEM(spec.CACert),
certutil.EncodePrivateKeyPEM(clientKey),
certutil.EncodeCertPEM(clientCert),
pkiutil.EncodeCertPEM(clientCert),
), nil
}
@@ -284,7 +285,7 @@ func WriteKubeConfigWithClientCert(out io.Writer, cfg *kubeadmapi.InitConfigurat
return errors.Wrap(err, "couldn't create a kubeconfig; the CA files couldn't be loaded")
}
masterEndpoint, err := kubeadmutil.GetMasterEndpoint(cfg)
masterEndpoint, err := kubeadmutil.GetMasterEndpoint(cfg.ControlPlaneEndpoint, &cfg.LocalAPIEndpoint)
if err != nil {
return err
}
@@ -311,7 +312,7 @@ func WriteKubeConfigWithToken(out io.Writer, cfg *kubeadmapi.InitConfiguration,
return errors.Wrap(err, "couldn't create a kubeconfig; the CA files couldn't be loaded")
}
masterEndpoint, err := kubeadmutil.GetMasterEndpoint(cfg)
masterEndpoint, err := kubeadmutil.GetMasterEndpoint(cfg.ControlPlaneEndpoint, &cfg.LocalAPIEndpoint)
if err != nil {
return err
}

2
cmd/kubeadm/app/phases/kubeconfig/kubeconfig_test.go
View File

@@ -162,7 +162,7 @@ func TestGetKubeConfigSpecs(t *testing.T) {
}
// Asserts InitConfiguration values injected into spec
masterEndpoint, err := kubeadmutil.GetMasterEndpoint(cfg)
masterEndpoint, err := kubeadmutil.GetMasterEndpoint(cfg.ControlPlaneEndpoint, &cfg.LocalAPIEndpoint)
if err != nil {
t.Error(err)
}

7
cmd/kubeadm/app/phases/kubelet/config.go
View File

@@ -30,7 +30,6 @@ import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/util/version"
clientset "k8s.io/client-go/kubernetes"
kubeadmapi "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm"
"k8s.io/kubernetes/cmd/kubeadm/app/componentconfigs"
kubeadmconstants "k8s.io/kubernetes/cmd/kubeadm/app/constants"
"k8s.io/kubernetes/cmd/kubeadm/app/util/apiclient"
@@ -51,9 +50,9 @@ func WriteConfigToDisk(kubeletConfig *kubeletconfig.KubeletConfiguration, kubele
// CreateConfigMap creates a ConfigMap with the generic kubelet configuration.
// Used at "kubeadm init" and "kubeadm upgrade" time
func CreateConfigMap(cfg *kubeadmapi.InitConfiguration, client clientset.Interface) error {
func CreateConfigMap(cfg *kubeletconfig.KubeletConfiguration, k8sVersionStr string, client clientset.Interface) error {
k8sVersion, err := version.ParseSemantic(cfg.KubernetesVersion)
k8sVersion, err := version.ParseSemantic(k8sVersionStr)
if err != nil {
return err
}
@@ -61,7 +60,7 @@ func CreateConfigMap(cfg *kubeadmapi.InitConfiguration, client clientset.Interfa
configMapName := kubeadmconstants.GetKubeletConfigMapName(k8sVersion)
fmt.Printf("[kubelet] Creating a ConfigMap %q in namespace %s with the configuration for the kubelets in the cluster\n", configMapName, metav1.NamespaceSystem)
kubeletBytes, err := getConfigBytes(cfg.ComponentConfigs.Kubelet)
kubeletBytes, err := getConfigBytes(cfg)
if err != nil {
return err
}

14
cmd/kubeadm/app/phases/kubelet/config_test.go
View File

@@ -25,7 +25,6 @@ import (
"k8s.io/apimachinery/pkg/util/version"
"k8s.io/client-go/kubernetes/fake"
core "k8s.io/client-go/testing"
kubeadmapi "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm"
"k8s.io/kubernetes/cmd/kubeadm/app/constants"
kubeletconfig "k8s.io/kubernetes/pkg/kubelet/apis/config"
)
@@ -33,15 +32,8 @@ import (
func TestCreateConfigMap(t *testing.T) {
nodeName := "fake-node"
client := fake.NewSimpleClientset()
cfg := &kubeadmapi.InitConfiguration{
NodeRegistration: kubeadmapi.NodeRegistrationOptions{Name: nodeName},
ClusterConfiguration: kubeadmapi.ClusterConfiguration{
KubernetesVersion: constants.CurrentKubernetesVersion.String(),
ComponentConfigs: kubeadmapi.ComponentConfigs{
Kubelet: &kubeletconfig.KubeletConfiguration{},
},
},
}
k8sVersionStr := constants.CurrentKubernetesVersion.String()
cfg := &kubeletconfig.KubeletConfiguration{}
client.PrependReactor("get", "nodes", func(action core.Action) (bool, runtime.Object, error) {
return true, &v1.Node{
@@ -61,7 +53,7 @@ func TestCreateConfigMap(t *testing.T) {
return true, nil, nil
})
if err := CreateConfigMap(cfg, client); err != nil {
if err := CreateConfigMap(cfg, k8sVersionStr, client); err != nil {
t.Errorf("CreateConfigMap: unexpected error %v", err)
}
}

10
cmd/kubeadm/app/phases/kubelet/flags.go
View File

@@ -46,29 +46,29 @@ type kubeletFlagsOpts struct {
// WriteKubeletDynamicEnvFile writes an environment file with dynamic flags to the kubelet.
// Used at "kubeadm init" and "kubeadm join" time.
func WriteKubeletDynamicEnvFile(cfg *kubeadmapi.InitConfiguration, registerTaintsUsingFlags bool, kubeletDir string) error {
func WriteKubeletDynamicEnvFile(cfg *kubeadmapi.ClusterConfiguration, nodeReg *kubeadmapi.NodeRegistrationOptions, registerTaintsUsingFlags bool, kubeletDir string) error {
hostName, err := nodeutil.GetHostname("")
if err != nil {
return err
}
flagOpts := kubeletFlagsOpts{
nodeRegOpts: &cfg.NodeRegistration,
nodeRegOpts: nodeReg,
featureGates: cfg.FeatureGates,
pauseImage: images.GetPauseImage(&cfg.ClusterConfiguration),
pauseImage: images.GetPauseImage(cfg),
registerTaintsUsingFlags: registerTaintsUsingFlags,
execer: utilsexec.New(),
pidOfFunc: procfs.PidOf,
defaultHostname: hostName,
}
stringMap := buildKubeletArgMap(flagOpts)
argList := kubeadmutil.BuildArgumentListFromMap(stringMap, cfg.NodeRegistration.KubeletExtraArgs)
argList := kubeadmutil.BuildArgumentListFromMap(stringMap, nodeReg.KubeletExtraArgs)
envFileContent := fmt.Sprintf("%s=%s\n", constants.KubeletEnvFileVariableName, strings.Join(argList, " "))
return writeKubeletFlagBytesToDisk([]byte(envFileContent), kubeletDir)
}
// buildKubeletArgMap takes a InitConfiguration object and builds based on that a string-string map with flags
// buildKubeletArgMap takes a kubeletFlagsOpts object and builds based on that a string-string map with flags
// that should be given to the local kubelet daemon.
func buildKubeletArgMap(opts kubeletFlagsOpts) map[string]string {
kubeletFlags := map[string]string{}

28
cmd/kubeadm/app/phases/upgrade/compute_test.go
View File

@@ -174,7 +174,7 @@ func TestGetAvailableUpgrades(t *testing.T) {
KubeVersion: "v1.10.3",
KubeadmVersion: "v1.10.3",
DNSType: kubeadmapi.CoreDNS,
DNSVersion: "1.2.6",
DNSVersion: "1.3.1",
EtcdVersion: "3.1.12",
},
},
@@ -213,7 +213,7 @@ func TestGetAvailableUpgrades(t *testing.T) {
KubeVersion: "v1.10.3",
KubeadmVersion: "v1.10.3",
DNSType: kubeadmapi.CoreDNS,
DNSVersion: "1.2.6",
DNSVersion: "1.3.1",
EtcdVersion: "3.1.12",
},
},
@@ -252,7 +252,7 @@ func TestGetAvailableUpgrades(t *testing.T) {
KubeVersion: "v1.11.0",
KubeadmVersion: "v1.11.0",
DNSType: kubeadmapi.CoreDNS,
DNSVersion: "1.2.6",
DNSVersion: "1.3.1",
EtcdVersion: "3.2.18",
},
},
@@ -291,7 +291,7 @@ func TestGetAvailableUpgrades(t *testing.T) {
KubeVersion: "v1.10.5",
KubeadmVersion: "v1.10.5", // Note: The kubeadm version mustn't be "downgraded" here
DNSType: kubeadmapi.CoreDNS,
DNSVersion: "1.2.6",
DNSVersion: "1.3.1",
EtcdVersion: "3.1.12",
},
},
@@ -311,7 +311,7 @@ func TestGetAvailableUpgrades(t *testing.T) {
KubeVersion: "v1.11.1",
KubeadmVersion: "v1.11.1",
DNSType: kubeadmapi.CoreDNS,
DNSVersion: "1.2.6",
DNSVersion: "1.3.1",
EtcdVersion: "3.2.18",
},
},
@@ -370,7 +370,7 @@ func TestGetAvailableUpgrades(t *testing.T) {
KubeVersion: "v1.11.0-alpha.2",
KubeadmVersion: "v1.11.0-alpha.2",
DNSType: kubeadmapi.CoreDNS,
DNSVersion: "1.2.6",
DNSVersion: "1.3.1",
EtcdVersion: "3.2.18",
},
},
@@ -410,7 +410,7 @@ func TestGetAvailableUpgrades(t *testing.T) {
KubeVersion: "v1.11.0-alpha.2",
KubeadmVersion: "v1.11.0-alpha.2",
DNSType: kubeadmapi.CoreDNS,
DNSVersion: "1.2.6",
DNSVersion: "1.3.1",
EtcdVersion: "3.2.18",
},
},
@@ -451,7 +451,7 @@ func TestGetAvailableUpgrades(t *testing.T) {
KubeVersion: "v1.11.0-beta.1",
KubeadmVersion: "v1.11.0-beta.1",
DNSType: kubeadmapi.CoreDNS,
DNSVersion: "1.2.6",
DNSVersion: "1.3.1",
EtcdVersion: "3.2.18",
},
},
@@ -492,7 +492,7 @@ func TestGetAvailableUpgrades(t *testing.T) {
KubeVersion: "v1.11.0-rc.1",
KubeadmVersion: "v1.11.0-rc.1",
DNSType: kubeadmapi.CoreDNS,
DNSVersion: "1.2.6",
DNSVersion: "1.3.1",
EtcdVersion: "3.2.18",
},
},
@@ -533,7 +533,7 @@ func TestGetAvailableUpgrades(t *testing.T) {
KubeVersion: "v1.11.6-rc.1",
KubeadmVersion: "v1.11.6-rc.1",
DNSType: kubeadmapi.CoreDNS,
DNSVersion: "1.2.6",
DNSVersion: "1.3.1",
EtcdVersion: "3.2.18",
},
},
@@ -574,7 +574,7 @@ func TestGetAvailableUpgrades(t *testing.T) {
KubeVersion: "v1.11.0-rc.1",
KubeadmVersion: "v1.11.0-rc.1",
DNSType: kubeadmapi.CoreDNS,
DNSVersion: "1.2.6",
DNSVersion: "1.3.1",
EtcdVersion: "3.2.18",
},
},
@@ -594,7 +594,7 @@ func TestGetAvailableUpgrades(t *testing.T) {
KubeVersion: constants.MinimumControlPlaneVersion.WithPreRelease("alpha.2").String(),
KubeadmVersion: constants.MinimumControlPlaneVersion.WithPreRelease("alpha.2").String(),
DNSType: kubeadmapi.CoreDNS,
DNSVersion: "1.2.6",
DNSVersion: "1.3.1",
EtcdVersion: "3.2.24",
},
},
@@ -647,7 +647,7 @@ func TestGetAvailableUpgrades(t *testing.T) {
KubeVersion: constants.MinimumControlPlaneVersion.WithPatch(1).String(),
KubeadmVersion: constants.MinimumControlPlaneVersion.WithPatch(1).String(),
DNSType: kubeadmapi.CoreDNS,
DNSVersion: "1.2.6",
DNSVersion: "1.3.1",
EtcdVersion: "3.2.24",
},
},
@@ -684,7 +684,7 @@ func TestGetAvailableUpgrades(t *testing.T) {
KubeVersion: constants.MinimumControlPlaneVersion.String(),
KubeadmVersion: constants.MinimumControlPlaneVersion.String(),
DNSType: kubeadmapi.CoreDNS,
DNSVersion: "1.2.6",
DNSVersion: "1.3.1",
EtcdVersion: "3.2.24",
},
},

12
cmd/kubeadm/app/phases/upgrade/postupgrade.go
View File

@@ -61,7 +61,7 @@ func PerformPostUpgradeTasks(client clientset.Interface, cfg *kubeadmapi.InitCon
}
// Create the new, version-branched kubelet ComponentConfig ConfigMap
if err := kubeletphase.CreateConfigMap(cfg, client); err != nil {
if err := kubeletphase.CreateConfigMap(cfg.ClusterConfiguration.ComponentConfigs.Kubelet, cfg.KubernetesVersion, client); err != nil {
errs = append(errs, errors.Wrap(err, "error creating kubelet configuration ConfigMap"))
}
@@ -108,21 +108,21 @@ func PerformPostUpgradeTasks(client clientset.Interface, cfg *kubeadmapi.InitCon
}
// Upgrade kube-dns/CoreDNS and kube-proxy
if err := dns.EnsureDNSAddon(cfg, client); err != nil {
if err := dns.EnsureDNSAddon(&cfg.ClusterConfiguration, client); err != nil {
errs = append(errs, err)
}
// Remove the old DNS deployment if a new DNS service is now used (kube-dns to CoreDNS or vice versa)
if err := removeOldDNSDeploymentIfAnotherDNSIsUsed(cfg, client, dryRun); err != nil {
if err := removeOldDNSDeploymentIfAnotherDNSIsUsed(&cfg.ClusterConfiguration, client, dryRun); err != nil {
errs = append(errs, err)
}
if err := proxy.EnsureProxyAddon(cfg, client); err != nil {
if err := proxy.EnsureProxyAddon(&cfg.ClusterConfiguration, &cfg.LocalAPIEndpoint, client); err != nil {
errs = append(errs, err)
}
return errorsutil.NewAggregate(errs)
}
func removeOldDNSDeploymentIfAnotherDNSIsUsed(cfg *kubeadmapi.InitConfiguration, client clientset.Interface, dryRun bool) error {
func removeOldDNSDeploymentIfAnotherDNSIsUsed(cfg *kubeadmapi.ClusterConfiguration, client clientset.Interface, dryRun bool) error {
return apiclient.TryRunCommand(func() error {
installedDeploymentName := kubeadmconstants.KubeDNSDeploymentName
deploymentToDelete := kubeadmconstants.CoreDNSDeploymentName
@@ -210,7 +210,7 @@ func writeKubeletConfigFiles(client clientset.Interface, cfg *kubeadmapi.InitCon
// Write env file with flags for the kubelet to use. We do not need to write the --register-with-taints for the master,
// as we handle that ourselves in the markmaster phase
// TODO: Maybe we want to do that some time in the future, in order to remove some logic from the markmaster phase?
if err := kubeletphase.WriteKubeletDynamicEnvFile(cfg, false, kubeletDir); err != nil {
if err := kubeletphase.WriteKubeletDynamicEnvFile(&cfg.ClusterConfiguration, &cfg.NodeRegistration, false, kubeletDir); err != nil {
errs = append(errs, errors.Wrap(err, "error writing a dynamic environment file for the kubelet"))
}

4
cmd/kubeadm/app/phases/upgrade/staticpods.go
View File

@@ -267,7 +267,7 @@ func performEtcdStaticPodUpgrade(client clientset.Interface, waiter apiclient.Wa
if err != nil {
return true, errors.Wrap(err, "failed to retrieve the current etcd version")
}
currentEtcdVersionStr, ok := currentEtcdVersions[etcdutil.GetClientURL(cfg)]
currentEtcdVersionStr, ok := currentEtcdVersions[etcdutil.GetClientURL(&cfg.LocalAPIEndpoint)]
if !ok {
return true, errors.Wrap(err, "failed to retrieve the current etcd version")
}
@@ -293,7 +293,7 @@ func performEtcdStaticPodUpgrade(client clientset.Interface, waiter apiclient.Wa
// Write the updated etcd static Pod manifest into the temporary directory, at this point no etcd change
// has occurred in any aspects.
if err := etcdphase.CreateLocalEtcdStaticPodManifestFile(pathMgr.TempManifestDir(), cfg); err != nil {
if err := etcdphase.CreateLocalEtcdStaticPodManifestFile(pathMgr.TempManifestDir(), cfg.NodeRegistration.Name, &cfg.ClusterConfiguration, &cfg.LocalAPIEndpoint); err != nil {
return true, errors.Wrap(err, "error creating local etcd static pod manifest file")
}

2
cmd/kubeadm/app/phases/upgrade/staticpods_test.go
View File

@@ -453,7 +453,7 @@ func TestStaticPodControlPlane(t *testing.T) {
if err != nil {
t.Fatalf("couldn't run CreateInitStaticPodManifestFiles: %v", err)
}
err = etcdphase.CreateLocalEtcdStaticPodManifestFile(pathMgr.RealManifestDir(), oldcfg)
err = etcdphase.CreateLocalEtcdStaticPodManifestFile(pathMgr.RealManifestDir(), oldcfg.NodeRegistration.Name, &oldcfg.ClusterConfiguration, &oldcfg.LocalAPIEndpoint)
if err != nil {
t.Fatalf("couldn't run CreateLocalEtcdStaticPodManifestFile: %v", err)
}

4
cmd/kubeadm/app/preflight/checks.go
View File

@@ -973,11 +973,11 @@ func RunJoinNodeChecks(execer utilsexec.Interface, cfg *kubeadmapi.JoinConfigura
}
// RunOptionalJoinNodeChecks executes all individual, applicable to node configuration dependant checks
func RunOptionalJoinNodeChecks(execer utilsexec.Interface, initCfg *kubeadmapi.InitConfiguration, ignorePreflightErrors sets.String) error {
func RunOptionalJoinNodeChecks(execer utilsexec.Interface, cfg *kubeadmapi.ClusterConfiguration, ignorePreflightErrors sets.String) error {
checks := []Checker{}
// Check ipvs required kernel module if we use ipvs kube-proxy mode
if initCfg.ComponentConfigs.KubeProxy != nil && initCfg.ComponentConfigs.KubeProxy.Mode == ipvsutil.IPVSProxyMode {
if cfg.ComponentConfigs.KubeProxy != nil && cfg.ComponentConfigs.KubeProxy.Mode == ipvsutil.IPVSProxyMode {
checks = append(checks,
ipvsutil.RequiredIPVSKernelModulesAvailableCheck{Executor: execer},
)

2
cmd/kubeadm/app/preflight/utils.go
View File

@@ -33,7 +33,7 @@ func GetKubeletVersion(execer utilsexec.Interface) (*version.Version, error) {
command := execer.Command("kubelet", "--version")
out, err := command.CombinedOutput()
if err != nil {
return nil, err
return nil, errors.Wrap(err, "cannot execute 'kubelet --version'")
}
cleanOutput := strings.TrimSpace(string(out))

6
cmd/kubeadm/app/util/certs/util.go
View File

@@ -24,7 +24,7 @@ import (
"testing"
certutil "k8s.io/client-go/util/cert"
"k8s.io/kubernetes/cmd/kubeadm/app/util/pkiutil"
pkiutil "k8s.io/kubernetes/cmd/kubeadm/app/util/pkiutil"
)
// SetupCertificateAuthorithy is a utility function for kubeadm testing that creates a
@@ -230,11 +230,11 @@ func WritePKIFiles(t *testing.T, dir string, files PKIFiles) {
for filename, body := range files {
switch body := body.(type) {
case *x509.Certificate:
if err := certutil.WriteCert(path.Join(dir, filename), certutil.EncodeCertPEM(body)); err != nil {
if err := certutil.WriteCert(path.Join(dir, filename), pkiutil.EncodeCertPEM(body)); err != nil {
t.Errorf("unable to write certificate to file %q: [%v]", dir, err)
}
case *rsa.PublicKey:
publicKeyBytes, err := certutil.EncodePublicKeyPEM(body)
publicKeyBytes, err := pkiutil.EncodePublicKeyPEM(body)
if err != nil {
t.Errorf("unable to write public key to file %q: [%v]", filename, err)
}

24
cmd/kubeadm/app/util/endpoint.go
View File

@@ -29,35 +29,35 @@ import (
)
// GetMasterEndpoint returns a properly formatted endpoint for the control plane built according following rules:
// - If the ControlPlaneEndpoint is defined, use it.
// - if the ControlPlaneEndpoint is defined but without a port number, use the ControlPlaneEndpoint + api.BindPort is used.
// - Otherwise, in case the ControlPlaneEndpoint is not defined, use the api.AdvertiseAddress + the api.BindPort.
func GetMasterEndpoint(cfg *kubeadmapi.InitConfiguration) (string, error) {
// - If the controlPlaneEndpoint is defined, use it.
// - if the controlPlaneEndpoint is defined but without a port number, use the controlPlaneEndpoint + localEndpoint.BindPort is used.
// - Otherwise, in case the controlPlaneEndpoint is not defined, use the localEndpoint.AdvertiseAddress + the localEndpoint.BindPort.
func GetMasterEndpoint(controlPlaneEndpoint string, localEndpoint *kubeadmapi.APIEndpoint) (string, error) {
// parse the bind port
bindPortString := strconv.Itoa(int(cfg.LocalAPIEndpoint.BindPort))
bindPortString := strconv.Itoa(int(localEndpoint.BindPort))
if _, err := ParsePort(bindPortString); err != nil {
return "", errors.Wrapf(err, "invalid value %q given for api.bindPort", cfg.LocalAPIEndpoint.BindPort)
return "", errors.Wrapf(err, "invalid value %q given for api.bindPort", localEndpoint.BindPort)
}
// parse the AdvertiseAddress
var ip = net.ParseIP(cfg.LocalAPIEndpoint.AdvertiseAddress)
var ip = net.ParseIP(localEndpoint.AdvertiseAddress)
if ip == nil {
return "", errors.Errorf("invalid value `%s` given for api.advertiseAddress", cfg.LocalAPIEndpoint.AdvertiseAddress)
return "", errors.Errorf("invalid value `%s` given for api.advertiseAddress", localEndpoint.AdvertiseAddress)
}
// set the master url using cfg.API.AdvertiseAddress + the cfg.API.BindPort
// set the master url using localEndpoint.AdvertiseAddress + the localEndpoint.BindPort
masterURL := &url.URL{
Scheme: "https",
Host: net.JoinHostPort(ip.String(), bindPortString),
}
// if the controlplane endpoint is defined
if len(cfg.ControlPlaneEndpoint) > 0 {
if len(controlPlaneEndpoint) > 0 {
// parse the controlplane endpoint
var host, port string
var err error
if host, port, err = ParseHostPort(cfg.ControlPlaneEndpoint); err != nil {
return "", errors.Wrapf(err, "invalid value %q given for controlPlaneEndpoint", cfg.ControlPlaneEndpoint)
if host, port, err = ParseHostPort(controlPlaneEndpoint); err != nil {
return "", errors.Wrapf(err, "invalid value %q given for controlPlaneEndpoint", controlPlaneEndpoint)
}
// if a port is provided within the controlPlaneAddress warn the users we are using it, else use the bindport

2
cmd/kubeadm/app/util/endpoint_test.go
View File

@@ -198,7 +198,7 @@ func TestGetMasterEndpoint(t *testing.T) {
for _, rt := range tests {
t.Run(rt.name, func(t *testing.T) {
actualEndpoint, actualError := GetMasterEndpoint(rt.cfg)
actualEndpoint, actualError := GetMasterEndpoint(rt.cfg.ControlPlaneEndpoint, &rt.cfg.LocalAPIEndpoint)
if (actualError != nil) && !rt.expectedError {
t.Errorf("%s unexpected failure: %v", rt.name, actualError)

8
cmd/kubeadm/app/util/etcd/etcd.go
View File

@@ -290,14 +290,14 @@ func CheckConfigurationIsHA(cfg *kubeadmapi.Etcd) bool {
// GetClientURL creates an HTTPS URL that uses the configured advertise
// address and client port for the API controller
func GetClientURL(cfg *kubeadmapi.InitConfiguration) string {
return "https://" + net.JoinHostPort(cfg.LocalAPIEndpoint.AdvertiseAddress, strconv.Itoa(constants.EtcdListenClientPort))
func GetClientURL(localEndpoint *kubeadmapi.APIEndpoint) string {
return "https://" + net.JoinHostPort(localEndpoint.AdvertiseAddress, strconv.Itoa(constants.EtcdListenClientPort))
}
// GetPeerURL creates an HTTPS URL that uses the configured advertise
// address and peer port for the API controller
func GetPeerURL(cfg *kubeadmapi.InitConfiguration) string {
return "https://" + net.JoinHostPort(cfg.LocalAPIEndpoint.AdvertiseAddress, strconv.Itoa(constants.EtcdListenPeerPort))
func GetPeerURL(localEndpoint *kubeadmapi.APIEndpoint) string {
return "https://" + net.JoinHostPort(localEndpoint.AdvertiseAddress, strconv.Itoa(constants.EtcdListenPeerPort))
}
// GetClientURLByIP creates an HTTPS URL based on an IP address

9
cmd/kubeadm/app/util/etcd/etcd_test.go
View File

@@ -72,7 +72,7 @@ func TestCheckConfigurationIsHA(t *testing.T) {
}
}
func testGetURL(t *testing.T, getURLFunc func(*kubeadmapi.InitConfiguration) string, port int) {
func testGetURL(t *testing.T, getURLFunc func(*kubeadmapi.APIEndpoint) string, port int) {
portStr := strconv.Itoa(port)
var tests = []struct {
name string
@@ -102,12 +102,7 @@ func testGetURL(t *testing.T, getURLFunc func(*kubeadmapi.InitConfiguration) str
}
for _, test := range tests {
cfg := &kubeadmapi.InitConfiguration{
LocalAPIEndpoint: kubeadmapi.APIEndpoint{
AdvertiseAddress: test.advertiseAddress,
},
}
url := getURLFunc(cfg)
url := getURLFunc(&kubeadmapi.APIEndpoint{AdvertiseAddress: test.advertiseAddress})
if url != test.expectedURL {
t.Errorf("expected %s, got %s", test.expectedURL, url)
}

88
cmd/kubeadm/app/util/pkiutil/pki_helpers.go
View File

@@ -18,6 +18,7 @@ package pkiutil
import (
"crypto"
"crypto/rand"
cryptorand "crypto/rand"
"crypto/rsa"
"crypto/x509"
@@ -25,6 +26,8 @@ import (
"encoding/pem"
"fmt"
"io/ioutil"
"math"
"math/big"
"net"
"os"
"path/filepath"
@@ -40,9 +43,22 @@ import (
"k8s.io/kubernetes/pkg/registry/core/service/ipallocator"
)
const (
// PrivateKeyBlockType is a possible value for pem.Block.Type.
PrivateKeyBlockType = "PRIVATE KEY"
// PublicKeyBlockType is a possible value for pem.Block.Type.
PublicKeyBlockType = "PUBLIC KEY"
// CertificateBlockType is a possible value for pem.Block.Type.
CertificateBlockType = "CERTIFICATE"
// RSAPrivateKeyBlockType is a possible value for pem.Block.Type.
RSAPrivateKeyBlockType = "RSA PRIVATE KEY"
rsaKeySize = 2048
duration365d = time.Hour * 24 * 365
)
// NewCertificateAuthority creates new certificate and private key for the certificate authority
func NewCertificateAuthority(config *certutil.Config) (*x509.Certificate, *rsa.PrivateKey, error) {
key, err := certutil.NewPrivateKey()
key, err := NewPrivateKey()
if err != nil {
return nil, nil, errors.Wrap(err, "unable to create private key")
}
@@ -57,12 +73,12 @@ func NewCertificateAuthority(config *certutil.Config) (*x509.Certificate, *rsa.P
// NewCertAndKey creates new certificate and key by passing the certificate authority certificate and key
func NewCertAndKey(caCert *x509.Certificate, caKey *rsa.PrivateKey, config *certutil.Config) (*x509.Certificate, *rsa.PrivateKey, error) {
key, err := certutil.NewPrivateKey()
key, err := NewPrivateKey()
if err != nil {
return nil, nil, errors.Wrap(err, "unable to create private key")
}
cert, err := certutil.NewSignedCert(*config, key, caCert, caKey)
cert, err := NewSignedCert(config, key, caCert, caKey)
if err != nil {
return nil, nil, errors.Wrap(err, "unable to sign certificate")
}
@@ -72,7 +88,7 @@ func NewCertAndKey(caCert *x509.Certificate, caKey *rsa.PrivateKey, config *cert
// NewCSRAndKey generates a new key and CSR and that could be signed to create the given certificate
func NewCSRAndKey(config *certutil.Config) (*x509.CertificateRequest, *rsa.PrivateKey, error) {
key, err := certutil.NewPrivateKey()
key, err := NewPrivateKey()
if err != nil {
return nil, nil, errors.Wrap(err, "unable to create private key")
}
@@ -111,7 +127,7 @@ func WriteCert(pkiPath, name string, cert *x509.Certificate) error {
}
certificatePath := pathForCert(pkiPath, name)
if err := certutil.WriteCert(certificatePath, certutil.EncodeCertPEM(cert)); err != nil {
if err := certutil.WriteCert(certificatePath, EncodeCertPEM(cert)); err != nil {
return errors.Wrapf(err, "unable to write certificate to file %s", certificatePath)
}
@@ -159,7 +175,7 @@ func WritePublicKey(pkiPath, name string, key *rsa.PublicKey) error {
return errors.New("public key cannot be nil when writing to file")
}
publicKeyBytes, err := certutil.EncodePublicKeyPEM(key)
publicKeyBytes, err := EncodePublicKeyPEM(key)
if err != nil {
return err
}
@@ -504,3 +520,63 @@ func NewCSR(cfg certutil.Config, key crypto.Signer) (*x509.CertificateRequest, e
return x509.ParseCertificateRequest(csrBytes)
}
// EncodeCertPEM returns PEM-endcoded certificate data
func EncodeCertPEM(cert *x509.Certificate) []byte {
block := pem.Block{
Type: CertificateBlockType,
Bytes: cert.Raw,
}
return pem.EncodeToMemory(&block)
}
// EncodePublicKeyPEM returns PEM-encoded public data
func EncodePublicKeyPEM(key *rsa.PublicKey) ([]byte, error) {
der, err := x509.MarshalPKIXPublicKey(key)
if err != nil {
return []byte{}, err
}
block := pem.Block{
Type: PublicKeyBlockType,
Bytes: der,
}
return pem.EncodeToMemory(&block), nil
}
// NewPrivateKey creates an RSA private key
func NewPrivateKey() (*rsa.PrivateKey, error) {
return rsa.GenerateKey(cryptorand.Reader, rsaKeySize)
}
// NewSignedCert creates a signed certificate using the given CA certificate and key
func NewSignedCert(cfg *certutil.Config, key crypto.Signer, caCert *x509.Certificate, caKey crypto.Signer) (*x509.Certificate, error) {
serial, err := rand.Int(rand.Reader, new(big.Int).SetInt64(math.MaxInt64))
if err != nil {
return nil, err
}
if len(cfg.CommonName) == 0 {
return nil, errors.New("must specify a CommonName")
}
if len(cfg.Usages) == 0 {
return nil, errors.New("must specify at least one ExtKeyUsage")
}
certTmpl := x509.Certificate{
Subject: pkix.Name{
CommonName: cfg.CommonName,
Organization: cfg.Organization,
},
DNSNames: cfg.AltNames.DNSNames,
IPAddresses: cfg.AltNames.IPs,
SerialNumber: serial,
NotBefore: caCert.NotBefore,
NotAfter: time.Now().Add(duration365d).UTC(),
KeyUsage: x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature,
ExtKeyUsage: cfg.Usages,
}
certDERBytes, err := x509.CreateCertificate(cryptorand.Reader, &certTmpl, caCert, key.Public(), caKey)
if err != nil {
return nil, err
}
return x509.ParseCertificate(certDERBytes)
}

3
cmd/kubeadm/app/util/staticpod/BUILD
View File

@@ -12,11 +12,9 @@ go_test(
embed = [":go_default_library"],
deps = [
"//cmd/kubeadm/app/apis/kubeadm:go_default_library",
"//cmd/kubeadm/app/constants:go_default_library",
"//cmd/kubeadm/test:go_default_library",
"//staging/src/k8s.io/api/core/v1:go_default_library",
"//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library",
"//staging/src/k8s.io/apimachinery/pkg/util/intstr:go_default_library",
],
)
@@ -32,7 +30,6 @@ go_library(
"//staging/src/k8s.io/api/core/v1:go_default_library",
"//staging/src/k8s.io/apimachinery/pkg/api/resource:go_default_library",
"//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library",
"//staging/src/k8s.io/apimachinery/pkg/util/intstr:go_default_library",
"//vendor/github.com/pkg/errors:go_default_library",
],
)

153
cmd/kubeadm/app/util/staticpod/utils.go
View File

@@ -31,7 +31,6 @@ import (
"k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/resource"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/util/intstr"
kubeadmapi "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm"
kubeadmconstants "k8s.io/kubernetes/cmd/kubeadm/app/constants"
"k8s.io/kubernetes/cmd/kubeadm/app/util"
@@ -82,28 +81,11 @@ func ComponentResources(cpu string) v1.ResourceRequirements {
}
}
// ComponentProbe is a helper function building a ready v1.Probe object from some simple parameters
func ComponentProbe(cfg *kubeadmapi.InitConfiguration, componentName string, port int, path string, scheme v1.URIScheme) *v1.Probe {
return &v1.Probe{
Handler: v1.Handler{
HTTPGet: &v1.HTTPGetAction{
Host: GetProbeAddress(cfg, componentName),
Path: path,
Port: intstr.FromInt(port),
Scheme: scheme,
},
},
InitialDelaySeconds: 15,
TimeoutSeconds: 15,
FailureThreshold: 8,
}
}
// EtcdProbe is a helper function for building a shell-based, etcdctl v1.Probe object to healthcheck etcd
func EtcdProbe(cfg *kubeadmapi.InitConfiguration, componentName string, port int, certsDir string, CACertName string, CertName string, KeyName string) *v1.Probe {
func EtcdProbe(cfg *kubeadmapi.Etcd, port int, certsDir string, CACertName string, CertName string, KeyName string) *v1.Probe {
tlsFlags := fmt.Sprintf("--cacert=%[1]s/%[2]s --cert=%[1]s/%[3]s --key=%[1]s/%[4]s", certsDir, CACertName, CertName, KeyName)
// etcd pod is alive if a linearizable get succeeds.
cmd := fmt.Sprintf("ETCDCTL_API=3 etcdctl --endpoints=https://[%s]:%d %s get foo", GetProbeAddress(cfg, componentName), port, tlsFlags)
cmd := fmt.Sprintf("ETCDCTL_API=3 etcdctl --endpoints=https://[%s]:%d %s get foo", GetEtcdProbeAddress(cfg), port, tlsFlags)
return &v1.Probe{
Handler: v1.Handler{
@@ -225,71 +207,80 @@ func ReadStaticPodFromDisk(manifestPath string) (*v1.Pod, error) {
return pod, nil
}
// GetProbeAddress returns an IP address or 127.0.0.1 to use for liveness probes
// in static pod manifests.
func GetProbeAddress(cfg *kubeadmapi.InitConfiguration, componentName string) string {
switch {
case componentName == kubeadmconstants.KubeAPIServer:
// In the case of a self-hosted deployment, the initial host on which kubeadm --init is run,
// will generate a DaemonSet with a nodeSelector such that all nodes with the label
// node-role.kubernetes.io/master='' will have the API server deployed to it. Since the init
// is run only once on an initial host, the API advertise address will be invalid for any
// future hosts that do not have the same address. Furthermore, since liveness and readiness
// probes do not support the Downward API we cannot dynamically set the advertise address to
// the node's IP. The only option then is to use localhost.
if cfg.LocalAPIEndpoint.AdvertiseAddress != "" {
return cfg.LocalAPIEndpoint.AdvertiseAddress
}
case componentName == kubeadmconstants.KubeControllerManager:
if addr, exists := cfg.ControllerManager.ExtraArgs[kubeControllerManagerAddressArg]; exists {
return addr
}
case componentName == kubeadmconstants.KubeScheduler:
if addr, exists := cfg.Scheduler.ExtraArgs[kubeSchedulerAddressArg]; exists {
return addr
}
case componentName == kubeadmconstants.Etcd:
if cfg.Etcd.Local != nil && cfg.Etcd.Local.ExtraArgs != nil {
if arg, exists := cfg.Etcd.Local.ExtraArgs[etcdListenClientURLsArg]; exists {
// Use the first url in the listen-client-urls if multiple url's are specified.
if strings.ContainsAny(arg, ",") {
arg = strings.Split(arg, ",")[0]
// GetAPIServerProbeAddress returns the probe address for the API server
func GetAPIServerProbeAddress(endpoint *kubeadmapi.APIEndpoint) string {
// In the case of a self-hosted deployment, the initial host on which kubeadm --init is run,
// will generate a DaemonSet with a nodeSelector such that all nodes with the label
// node-role.kubernetes.io/master='' will have the API server deployed to it. Since the init
// is run only once on an initial host, the API advertise address will be invalid for any
// future hosts that do not have the same address. Furthermore, since liveness and readiness
// probes do not support the Downward API we cannot dynamically set the advertise address to
// the node's IP. The only option then is to use localhost.
if endpoint != nil && endpoint.AdvertiseAddress != "" {
return endpoint.AdvertiseAddress
}
return "127.0.0.1"
}
// GetControllerManagerProbeAddress returns the kubernetes controller manager probe address
func GetControllerManagerProbeAddress(cfg *kubeadmapi.ClusterConfiguration) string {
if addr, exists := cfg.ControllerManager.ExtraArgs[kubeControllerManagerAddressArg]; exists {
return addr
}
return "127.0.0.1"
}
// GetSchedulerProbeAddress returns the kubernetes scheduler probe address
func GetSchedulerProbeAddress(cfg *kubeadmapi.ClusterConfiguration) string {
if addr, exists := cfg.Scheduler.ExtraArgs[kubeSchedulerAddressArg]; exists {
return addr
}
return "127.0.0.1"
}
// GetEtcdProbeAddress returns the etcd probe address
func GetEtcdProbeAddress(cfg *kubeadmapi.Etcd) string {
if cfg.Local != nil && cfg.Local.ExtraArgs != nil {
if arg, exists := cfg.Local.ExtraArgs[etcdListenClientURLsArg]; exists {
// Use the first url in the listen-client-urls if multiple url's are specified.
if strings.ContainsAny(arg, ",") {
arg = strings.Split(arg, ",")[0]
}
parsedURL, err := url.Parse(arg)
if err != nil || parsedURL.Hostname() == "" {
return "127.0.0.1"
}
// Return the IP if the URL contains an address instead of a name.
if ip := net.ParseIP(parsedURL.Hostname()); ip != nil {
// etcdctl doesn't support auto-converting zero addresses into loopback addresses
if ip.Equal(net.IPv4zero) {
return "127.0.0.1"
}
parsedURL, err := url.Parse(arg)
if err != nil || parsedURL.Hostname() == "" {
break
}
// Return the IP if the URL contains an address instead of a name.
if ip := net.ParseIP(parsedURL.Hostname()); ip != nil {
// etcdctl doesn't support auto-converting zero addresses into loopback addresses
if ip.Equal(net.IPv4zero) {
return "127.0.0.1"
}
if ip.Equal(net.IPv6zero) {
return net.IPv6loopback.String()
}
return ip.String()
}
// Use the local resolver to try resolving the name within the URL.
// If the name can not be resolved, return an IPv4 loopback address.
// Otherwise, select the first valid IPv4 address.
// If the name does not resolve to an IPv4 address, select the first valid IPv6 address.
addrs, err := net.LookupIP(parsedURL.Hostname())
if err != nil {
break
}
var ip net.IP
for _, addr := range addrs {
if addr.To4() != nil {
ip = addr
break
}
if addr.To16() != nil && ip == nil {
ip = addr
}
if ip.Equal(net.IPv6zero) {
return net.IPv6loopback.String()
}
return ip.String()
}
// Use the local resolver to try resolving the name within the URL.
// If the name can not be resolved, return an IPv4 loopback address.
// Otherwise, select the first valid IPv4 address.
// If the name does not resolve to an IPv4 address, select the first valid IPv6 address.
addrs, err := net.LookupIP(parsedURL.Hostname())
if err != nil {
return "127.0.0.1"
}
var ip net.IP
for _, addr := range addrs {
if addr.To4() != nil {
ip = addr
break
}
if addr.To16() != nil && ip == nil {
ip = addr
}
}
return ip.String()
}
}
return "127.0.0.1"

378
cmd/kubeadm/app/util/staticpod/utils_test.go
View File

@@ -27,9 +27,7 @@ import (
"k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/util/intstr"
kubeadmapi "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm"
kubeadmconstants "k8s.io/kubernetes/cmd/kubeadm/app/constants"
testutil "k8s.io/kubernetes/cmd/kubeadm/test"
)
@@ -42,151 +40,73 @@ func TestComponentResources(t *testing.T) {
}
}
func TestComponentProbe(t *testing.T) {
var tests = []struct {
name string
cfg *kubeadmapi.InitConfiguration
component string
port int
path string
scheme v1.URIScheme
expected string
func TestGetAPIServerProbeAddress(t *testing.T) {
tests := []struct {
desc string
endpoint *kubeadmapi.APIEndpoint
expected string
}{
{
name: "default apiserver advertise address with http",
cfg: &kubeadmapi.InitConfiguration{
LocalAPIEndpoint: kubeadmapi.APIEndpoint{
AdvertiseAddress: "",
},
},
component: kubeadmconstants.KubeAPIServer,
port: 1,
path: "foo",
scheme: v1.URISchemeHTTP,
expected: "127.0.0.1",
desc: "nil endpoint returns 127.0.0.1",
expected: "127.0.0.1",
},
{
name: "default apiserver advertise address with https",
cfg: &kubeadmapi.InitConfiguration{
LocalAPIEndpoint: kubeadmapi.APIEndpoint{
AdvertiseAddress: "",
},
},
component: kubeadmconstants.KubeAPIServer,
port: 2,
path: "bar",
scheme: v1.URISchemeHTTPS,
expected: "127.0.0.1",
desc: "empty AdvertiseAddress endpoint returns 127.0.0.1",
endpoint: &kubeadmapi.APIEndpoint{},
expected: "127.0.0.1",
},
{
name: "valid ipv4 apiserver advertise address with http",
cfg: &kubeadmapi.InitConfiguration{
LocalAPIEndpoint: kubeadmapi.APIEndpoint{
AdvertiseAddress: "1.2.3.4",
},
desc: "filled in AdvertiseAddress endpoint returns it",
endpoint: &kubeadmapi.APIEndpoint{
AdvertiseAddress: "10.10.10.10",
},
component: kubeadmconstants.KubeAPIServer,
port: 1,
path: "foo",
scheme: v1.URISchemeHTTP,
expected: "1.2.3.4",
},
{
name: "valid ipv6 apiserver advertise address with http",
cfg: &kubeadmapi.InitConfiguration{
LocalAPIEndpoint: kubeadmapi.APIEndpoint{
AdvertiseAddress: "2001:db8::1",
},
},
component: kubeadmconstants.KubeAPIServer,
port: 1,
path: "foo",
scheme: v1.URISchemeHTTP,
expected: "2001:db8::1",
},
{
name: "valid IPv4 controller-manager probe",
cfg: &kubeadmapi.InitConfiguration{
ClusterConfiguration: kubeadmapi.ClusterConfiguration{
ControllerManager: kubeadmapi.ControlPlaneComponent{
ExtraArgs: map[string]string{"address": "1.2.3.4"},
},
},
},
component: kubeadmconstants.KubeControllerManager,
port: 1,
path: "foo",
scheme: v1.URISchemeHTTP,
expected: "1.2.3.4",
},
{
name: "valid IPv6 controller-manager probe",
cfg: &kubeadmapi.InitConfiguration{
ClusterConfiguration: kubeadmapi.ClusterConfiguration{
ControllerManager: kubeadmapi.ControlPlaneComponent{
ExtraArgs: map[string]string{"address": "2001:db8::1"},
},
},
},
component: kubeadmconstants.KubeControllerManager,
port: 1,
path: "foo",
scheme: v1.URISchemeHTTP,
expected: "2001:db8::1",
},
{
name: "valid IPv4 scheduler probe",
cfg: &kubeadmapi.InitConfiguration{
ClusterConfiguration: kubeadmapi.ClusterConfiguration{
Scheduler: kubeadmapi.ControlPlaneComponent{
ExtraArgs: map[string]string{"address": "1.2.3.4"},
},
},
},
component: kubeadmconstants.KubeScheduler,
port: 1,
path: "foo",
scheme: v1.URISchemeHTTP,
expected: "1.2.3.4",
},
{
name: "valid IPv6 scheduler probe",
cfg: &kubeadmapi.InitConfiguration{
ClusterConfiguration: kubeadmapi.ClusterConfiguration{
Scheduler: kubeadmapi.ControlPlaneComponent{
ExtraArgs: map[string]string{"address": "2001:db8::1"},
},
},
},
component: kubeadmconstants.KubeScheduler,
port: 1,
path: "foo",
scheme: v1.URISchemeHTTP,
expected: "2001:db8::1",
expected: "10.10.10.10",
},
}
for _, rt := range tests {
t.Run(rt.name, func(t *testing.T) {
actual := ComponentProbe(rt.cfg, rt.component, rt.port, rt.path, rt.scheme)
if actual.Handler.HTTPGet.Host != rt.expected {
t.Errorf("%s test case failed:\n\texpected: %s\n\t actual: %s",
rt.name, rt.expected,
actual.Handler.HTTPGet.Host)
for _, test := range tests {
t.Run(test.desc, func(t *testing.T) {
actual := GetAPIServerProbeAddress(test.endpoint)
if actual != test.expected {
t.Errorf("Unexpected result from GetAPIServerProbeAddress:\n\texpected: %s\n\tactual: %s", test.expected, actual)
}
if actual.Handler.HTTPGet.Port != intstr.FromInt(rt.port) {
t.Errorf("%s test case failed:\n\texpected: %v\n\t actual: %v",
rt.name, rt.port,
actual.Handler.HTTPGet.Port)
}
if actual.Handler.HTTPGet.Path != rt.path {
t.Errorf("%s test case failed:\n\texpected: %s\n\t actual: %s",
rt.name, rt.path,
actual.Handler.HTTPGet.Path)
}
if actual.Handler.HTTPGet.Scheme != rt.scheme {
t.Errorf("%s test case failed:\n\texpected: %v\n\t actual: %v",
rt.name, rt.scheme,
actual.Handler.HTTPGet.Scheme)
})
}
}
func TestGetControllerManagerProbeAddress(t *testing.T) {
tests := []struct {
desc string
cfg *kubeadmapi.ClusterConfiguration
expected string
}{
{
desc: "no controller manager extra args leads to 127.0.0.1 being used",
cfg: &kubeadmapi.ClusterConfiguration{
ControllerManager: kubeadmapi.ControlPlaneComponent{
ExtraArgs: map[string]string{},
},
},
expected: "127.0.0.1",
},
{
desc: "setting controller manager extra address arg to something acknowledges it",
cfg: &kubeadmapi.ClusterConfiguration{
ControllerManager: kubeadmapi.ControlPlaneComponent{
ExtraArgs: map[string]string{
kubeControllerManagerAddressArg: "10.10.10.10",
},
},
},
expected: "10.10.10.10",
},
}
for _, test := range tests {
t.Run(test.desc, func(t *testing.T) {
actual := GetControllerManagerProbeAddress(test.cfg)
if actual != test.expected {
t.Errorf("Unexpected result from GetControllerManagerProbeAddress:\n\texpected: %s\n\tactual: %s", test.expected, actual)
}
})
}
@@ -194,150 +114,124 @@ func TestComponentProbe(t *testing.T) {
func TestEtcdProbe(t *testing.T) {
var tests = []struct {
name string
cfg *kubeadmapi.ClusterConfiguration
component string
port int
certsDir string
cacert string
cert string
key string
expected string
name string
cfg *kubeadmapi.Etcd
port int
certsDir string
cacert string
cert string
key string
expected string
}{
{
name: "valid etcd probe using listen-client-urls IPv4 addresses",
cfg: &kubeadmapi.ClusterConfiguration{
Etcd: kubeadmapi.Etcd{
Local: &kubeadmapi.LocalEtcd{
ExtraArgs: map[string]string{
"listen-client-urls": "http://1.2.3.4:2379,http://4.3.2.1:2379"},
},
cfg: &kubeadmapi.Etcd{
Local: &kubeadmapi.LocalEtcd{
ExtraArgs: map[string]string{
"listen-client-urls": "http://1.2.3.4:2379,http://4.3.2.1:2379"},
},
},
component: kubeadmconstants.Etcd,
port: 1,
certsDir: "secretsA",
cacert: "ca1",
cert: "cert1",
key: "key1",
expected: "ETCDCTL_API=3 etcdctl --endpoints=https://[1.2.3.4]:1 --cacert=secretsA/ca1 --cert=secretsA/cert1 --key=secretsA/key1 get foo",
port: 1,
certsDir: "secretsA",
cacert: "ca1",
cert: "cert1",
key: "key1",
expected: "ETCDCTL_API=3 etcdctl --endpoints=https://[1.2.3.4]:1 --cacert=secretsA/ca1 --cert=secretsA/cert1 --key=secretsA/key1 get foo",
},
{
name: "valid etcd probe using listen-client-urls unspecified IPv6 address",
cfg: &kubeadmapi.ClusterConfiguration{
Etcd: kubeadmapi.Etcd{
Local: &kubeadmapi.LocalEtcd{
ExtraArgs: map[string]string{
"listen-client-urls": "http://[0:0:0:0:0:0:0:0]:2379"},
},
cfg: &kubeadmapi.Etcd{
Local: &kubeadmapi.LocalEtcd{
ExtraArgs: map[string]string{
"listen-client-urls": "http://[0:0:0:0:0:0:0:0]:2379"},
},
},
component: kubeadmconstants.Etcd,
port: 1,
certsDir: "secretsB",
cacert: "ca2",
cert: "cert2",
key: "key2",
expected: "ETCDCTL_API=3 etcdctl --endpoints=https://[::1]:1 --cacert=secretsB/ca2 --cert=secretsB/cert2 --key=secretsB/key2 get foo",
port: 1,
certsDir: "secretsB",
cacert: "ca2",
cert: "cert2",
key: "key2",
expected: "ETCDCTL_API=3 etcdctl --endpoints=https://[::1]:1 --cacert=secretsB/ca2 --cert=secretsB/cert2 --key=secretsB/key2 get foo",
},
{
name: "valid etcd probe using listen-client-urls unspecified IPv6 address 2",
cfg: &kubeadmapi.ClusterConfiguration{
Etcd: kubeadmapi.Etcd{
Local: &kubeadmapi.LocalEtcd{
ExtraArgs: map[string]string{
"listen-client-urls": "http://[::0:0]:2379"},
},
cfg: &kubeadmapi.Etcd{
Local: &kubeadmapi.LocalEtcd{
ExtraArgs: map[string]string{
"listen-client-urls": "http://[::0:0]:2379"},
},
},
component: kubeadmconstants.Etcd,
port: 1,
certsDir: "secretsB",
cacert: "ca2",
cert: "cert2",
key: "key2",
expected: "ETCDCTL_API=3 etcdctl --endpoints=https://[::1]:1 --cacert=secretsB/ca2 --cert=secretsB/cert2 --key=secretsB/key2 get foo",
port: 1,
certsDir: "secretsB",
cacert: "ca2",
cert: "cert2",
key: "key2",
expected: "ETCDCTL_API=3 etcdctl --endpoints=https://[::1]:1 --cacert=secretsB/ca2 --cert=secretsB/cert2 --key=secretsB/key2 get foo",
},
{
name: "valid etcd probe using listen-client-urls unspecified IPv6 address 3",
cfg: &kubeadmapi.ClusterConfiguration{
Etcd: kubeadmapi.Etcd{
Local: &kubeadmapi.LocalEtcd{
ExtraArgs: map[string]string{
"listen-client-urls": "http://[::]:2379"},
},
cfg: &kubeadmapi.Etcd{
Local: &kubeadmapi.LocalEtcd{
ExtraArgs: map[string]string{
"listen-client-urls": "http://[::]:2379"},
},
},
component: kubeadmconstants.Etcd,
port: 1,
certsDir: "secretsB",
cacert: "ca2",
cert: "cert2",
key: "key2",
expected: "ETCDCTL_API=3 etcdctl --endpoints=https://[::1]:1 --cacert=secretsB/ca2 --cert=secretsB/cert2 --key=secretsB/key2 get foo",
port: 1,
certsDir: "secretsB",
cacert: "ca2",
cert: "cert2",
key: "key2",
expected: "ETCDCTL_API=3 etcdctl --endpoints=https://[::1]:1 --cacert=secretsB/ca2 --cert=secretsB/cert2 --key=secretsB/key2 get foo",
},
{
name: "valid etcd probe using listen-client-urls unspecified IPv4 address",
cfg: &kubeadmapi.ClusterConfiguration{
Etcd: kubeadmapi.Etcd{
Local: &kubeadmapi.LocalEtcd{
ExtraArgs: map[string]string{
"listen-client-urls": "http://1.2.3.4:2379,http://4.3.2.1:2379"},
},
cfg: &kubeadmapi.Etcd{
Local: &kubeadmapi.LocalEtcd{
ExtraArgs: map[string]string{
"listen-client-urls": "http://1.2.3.4:2379,http://4.3.2.1:2379"},
},
},
component: kubeadmconstants.Etcd,
port: 1,
certsDir: "secretsA",
cacert: "ca1",
cert: "cert1",
key: "key1",
expected: "ETCDCTL_API=3 etcdctl --endpoints=https://[1.2.3.4]:1 --cacert=secretsA/ca1 --cert=secretsA/cert1 --key=secretsA/key1 get foo",
port: 1,
certsDir: "secretsA",
cacert: "ca1",
cert: "cert1",
key: "key1",
expected: "ETCDCTL_API=3 etcdctl --endpoints=https://[1.2.3.4]:1 --cacert=secretsA/ca1 --cert=secretsA/cert1 --key=secretsA/key1 get foo",
},
{
name: "valid etcd probe using listen-client-urls IPv6 addresses",
cfg: &kubeadmapi.ClusterConfiguration{
Etcd: kubeadmapi.Etcd{
Local: &kubeadmapi.LocalEtcd{
ExtraArgs: map[string]string{
"listen-client-urls": "http://[2001:db8::1]:2379,http://[2001:db8::2]:2379"},
},
cfg: &kubeadmapi.Etcd{
Local: &kubeadmapi.LocalEtcd{
ExtraArgs: map[string]string{
"listen-client-urls": "http://[2001:db8::1]:2379,http://[2001:db8::2]:2379"},
},
},
component: kubeadmconstants.Etcd,
port: 1,
certsDir: "secretsB",
cacert: "ca2",
cert: "cert2",
key: "key2",
expected: "ETCDCTL_API=3 etcdctl --endpoints=https://[2001:db8::1]:1 --cacert=secretsB/ca2 --cert=secretsB/cert2 --key=secretsB/key2 get foo",
port: 1,
certsDir: "secretsB",
cacert: "ca2",
cert: "cert2",
key: "key2",
expected: "ETCDCTL_API=3 etcdctl --endpoints=https://[2001:db8::1]:1 --cacert=secretsB/ca2 --cert=secretsB/cert2 --key=secretsB/key2 get foo",
},
{
name: "valid IPv4 etcd probe using hostname for listen-client-urls",
cfg: &kubeadmapi.ClusterConfiguration{
Etcd: kubeadmapi.Etcd{
Local: &kubeadmapi.LocalEtcd{
ExtraArgs: map[string]string{
"listen-client-urls": "http://localhost:2379"},
},
cfg: &kubeadmapi.Etcd{
Local: &kubeadmapi.LocalEtcd{
ExtraArgs: map[string]string{
"listen-client-urls": "http://localhost:2379"},
},
},
component: kubeadmconstants.Etcd,
port: 1,
certsDir: "secretsC",
cacert: "ca3",
cert: "cert3",
key: "key3",
expected: "ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:1 --cacert=secretsC/ca3 --cert=secretsC/cert3 --key=secretsC/key3 get foo",
port: 1,
certsDir: "secretsC",
cacert: "ca3",
cert: "cert3",
key: "key3",
expected: "ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:1 --cacert=secretsC/ca3 --cert=secretsC/cert3 --key=secretsC/key3 get foo",
},
}
for _, rt := range tests {
t.Run(rt.name, func(t *testing.T) {
// TODO: Make EtcdProbe accept a ClusterConfiguration object instead of InitConfiguration
initcfg := &kubeadmapi.InitConfiguration{
ClusterConfiguration: *rt.cfg,
}
actual := EtcdProbe(initcfg, rt.component, rt.port, rt.certsDir, rt.cacert, rt.cert, rt.key)
actual := EtcdProbe(rt.cfg, rt.port, rt.certsDir, rt.cacert, rt.cert, rt.key)
if actual.Handler.Exec.Command[2] != rt.expected {
t.Errorf("%s test case failed:\n\texpected: %s\n\t actual: %s",
rt.name, rt.expected,

2
cmd/kubeadm/app/util/system/docker_validator_test.go
View File

@@ -28,7 +28,7 @@ func TestValidateDockerInfo(t *testing.T) {
Reporter: DefaultReporter,
}
spec := &DockerSpec{
Version: []string{`1\.1[1-3]\..*`, `17\.0[3,6,9]\..*`, `18\.06\..*`},
Version: []string{`1\.1[1-3]\..*`, `17\.0[3,6,9]\..*`, `18\.0[6,9]\..*`},
GraphDriver: []string{"driver_1", "driver_2"},
}
for _, test := range []struct {

2
cmd/kubeadm/app/util/system/types_unix.go
View File

@@ -62,7 +62,7 @@ var DefaultSysSpec = SysSpec{
Cgroups: []string{"cpu", "cpuacct", "cpuset", "devices", "freezer", "memory"},
RuntimeSpec: RuntimeSpec{
DockerSpec: &DockerSpec{
Version: []string{`1\.1[1-3]\..*`, `17\.0[3,6,9]\..*`, `18\.06\..*`},
Version: []string{`1\.1[1-3]\..*`, `17\.0[3,6,9]\..*`, `18\.0[6,9]\..*`},
GraphDriver: []string{"aufs", "overlay", "overlay2", "devicemapper", "zfs"},
},
},

2
cmd/kubeadm/app/util/system/types_windows.go
View File

@@ -38,7 +38,7 @@ var DefaultSysSpec = SysSpec{
Cgroups: []string{},
RuntimeSpec: RuntimeSpec{
DockerSpec: &DockerSpec{
Version: []string{`18\.06\..*`}, //Requires [18.06] or later
Version: []string{`18\.0[6,9]\..*`},
GraphDriver: []string{"windowsfilter"},
},
},

2
cmd/kubelet/app/options/options.go
View File

@@ -473,8 +473,6 @@ func AddKubeletConfigFlags(mainfs *pflag.FlagSet, c *kubeletconfig.KubeletConfig
}()
fs.BoolVar(&c.FailSwapOn, "fail-swap-on", c.FailSwapOn, "Makes the Kubelet fail to start if swap is enabled on the node. ")
fs.BoolVar(&c.FailSwapOn, "experimental-fail-swap-on", c.FailSwapOn, "DEPRECATED: please use --fail-swap-on instead.")
fs.StringVar(&c.StaticPodPath, "pod-manifest-path", c.StaticPodPath, "Path to the directory containing static pod files to run, or the path to a single static pod file. Files starting with dots will be ignored.")
fs.DurationVar(&c.SyncFrequency.Duration, "sync-frequency", c.SyncFrequency.Duration, "Max period between synchronizing running containers and config")
fs.DurationVar(&c.FileCheckFrequency.Duration, "file-check-frequency", c.FileCheckFrequency.Duration, "Duration between checking config files for new data")

5
cmd/kubelet/app/server.go
View File

@@ -369,7 +369,10 @@ func UnsecuredDependencies(s *options.KubeletServer) (*kubelet.Dependencies, err
}
mounter = mount.NewNsenterMounter(s.RootDirectory, ne)
// an exec interface which can use nsenter for flex plugin calls
pluginRunner = nsenter.NewNsenterExecutor(nsenter.DefaultHostRootFsPath, exec.New())
pluginRunner, err = nsenter.NewNsenter(nsenter.DefaultHostRootFsPath, exec.New())
if err != nil {
return nil, err
}
}
var dockerClientConfig *dockershim.ClientConfig

5
hack/.golint_failures
View File

@@ -1,5 +1,4 @@
cmd/cloud-controller-manager/app/apis/config/v1alpha1
cmd/hyperkube
cmd/kube-apiserver/app
cmd/kube-controller-manager/app
cmd/kube-proxy/app
@@ -100,7 +99,6 @@ pkg/controller/clusterroleaggregation
pkg/controller/cronjob
pkg/controller/daemon
pkg/controller/deployment
pkg/controller/deployment/util
pkg/controller/disruption
pkg/controller/endpoint
pkg/controller/garbagecollector
@@ -193,8 +191,6 @@ pkg/kubelet/checkpointmanager/checksum
pkg/kubelet/checkpointmanager/testing/example_checkpoint_formats/v1
pkg/kubelet/client
pkg/kubelet/cm
pkg/kubelet/cm/devicemanager/checkpoint
pkg/kubelet/cm/util
pkg/kubelet/config
pkg/kubelet/configmap
pkg/kubelet/container
@@ -359,7 +355,6 @@ pkg/util/bandwidth
pkg/util/config
pkg/util/ebtables
pkg/util/env
pkg/util/file
pkg/util/goroutinemap/exponentialbackoff
pkg/util/initsystem
pkg/util/iptables

19
hack/.shellcheck_failures
View File

@@ -1,4 +1,3 @@
./build/common.sh
./build/lib/release.sh
./cluster/addons/addon-manager/kube-addons.sh
./cluster/addons/fluentd-elasticsearch/es-image/run.sh
@@ -158,19 +157,6 @@
./pkg/kubectl/cmd/edit/testdata/record_testcase.sh
./pkg/util/verify-util-pkg.sh
./plugin/pkg/admission/imagepolicy/gencerts.sh
./staging/src/k8s.io/apiextensions-apiserver/examples/client-go/hack/update-codegen.sh
./staging/src/k8s.io/apiextensions-apiserver/examples/client-go/hack/verify-codegen.sh
./staging/src/k8s.io/apiextensions-apiserver/hack/build-image.sh
./staging/src/k8s.io/apiextensions-apiserver/hack/update-codegen.sh
./staging/src/k8s.io/apiextensions-apiserver/hack/verify-codegen.sh
./staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/testcerts/gencerts.sh
./staging/src/k8s.io/apiserver/pkg/util/webhook/gencerts.sh
./staging/src/k8s.io/apiserver/plugin/pkg/authenticator/token/oidc/testdata/gen.sh
./staging/src/k8s.io/apiserver/plugin/pkg/authorizer/webhook/gencerts.sh
./staging/src/k8s.io/code-generator/generate-groups.sh
./staging/src/k8s.io/code-generator/generate-internal-groups.sh
./staging/src/k8s.io/code-generator/hack/update-codegen.sh
./staging/src/k8s.io/code-generator/hack/verify-codegen.sh
./staging/src/k8s.io/csi-api/hack/update-codegen.sh
./staging/src/k8s.io/csi-api/hack/verify-codegen.sh
./staging/src/k8s.io/kube-aggregator/hack/build-image.sh
@@ -182,11 +168,6 @@
./staging/src/k8s.io/metrics/hack/verify-codegen.sh
./staging/src/k8s.io/node-api/hack/update-codegen.sh
./staging/src/k8s.io/node-api/hack/verify-codegen.sh
./staging/src/k8s.io/sample-apiserver/hack/build-image.sh
./staging/src/k8s.io/sample-apiserver/hack/update-codegen.sh
./staging/src/k8s.io/sample-apiserver/hack/verify-codegen.sh
./staging/src/k8s.io/sample-controller/hack/update-codegen.sh
./staging/src/k8s.io/sample-controller/hack/verify-codegen.sh
./test/cmd/apply.sh
./test/cmd/apps.sh
./test/cmd/authorization.sh

9
hack/lib/util.sh
View File

@@ -697,6 +697,15 @@ function kube::util::ensure-cfssl {
return 0
fi
host_arch=$(kube::util::host_arch)
if [[ "${host_arch}" != "amd64" ]]; then
echo "Cannot download cfssl on non-amd64 hosts and cfssl does not appear to be installed."
echo "Please install cfssl and cfssljson and verify they are in \$PATH."
echo "Hint: export PATH=\$PATH:\$GOPATH/bin; go get -u github.com/cloudflare/cfssl/cmd/..."
exit 1
fi
# Create a temp dir for cfssl if no directory was given
local cfssldir=${1:-}
if [[ -z "${cfssldir}" ]]; then

2
hack/local-up-cluster.sh
View File

@@ -78,6 +78,7 @@ CLOUD_PROVIDER=${CLOUD_PROVIDER:-""}
CLOUD_CONFIG=${CLOUD_CONFIG:-""}
FEATURE_GATES=${FEATURE_GATES:-"AllAlpha=false"}
STORAGE_BACKEND=${STORAGE_BACKEND:-"etcd3"}
STORAGE_MEDIA_TYPE=${STORAGE_MEDIA_TYPE:-""}
# preserve etcd data. you also need to set ETCD_DIR.
PRESERVE_ETCD="${PRESERVE_ETCD:-false}"
# enable Pod priority and preemption
@@ -574,6 +575,7 @@ function start_apiserver {
--insecure-bind-address="${API_HOST_IP}" \
--insecure-port="${API_PORT}" \
--storage-backend=${STORAGE_BACKEND} \
--storage-media-type=${STORAGE_MEDIA_TYPE} \
--etcd-servers="http://${ETCD_HOST}:${ETCD_PORT}" \
--service-cluster-ip-range="${SERVICE_CLUSTER_IP_RANGE}" \
--feature-gates="${FEATURE_GATES}" \

46
hack/verify-generated-files-remake.sh
View File

@@ -37,7 +37,7 @@ function find_genfiles() {
# $1 = filename pattern as in "zz_generated.$1.go"
# $2 timestamp file
function newer() {
find_genfiles "$1" | while read F; do
find_genfiles "$1" | while read -r F; do
if [[ "${F}" -nt "$2" ]]; then
echo "${F}"
fi
@@ -47,7 +47,7 @@ function newer() {
# $1 = filename pattern as in "zz_generated.$1.go"
# $2 timestamp file
function older() {
find_genfiles "$1" | while read F; do
find_genfiles "$1" | while read -r F; do
if [[ "$2" -nt "${F}" ]]; then
echo "${F}"
fi
@@ -61,7 +61,7 @@ function assert_clean() {
X=($(newer deepcopy "${STAMP}"))
if [[ "${#X[*]}" != 0 ]]; then
echo "Generated files changed on back-to-back 'make' runs:"
echo " ${X[@]:-(none)}"
echo " ${X[*]:-(none)}"
return 1
fi
true
@@ -82,19 +82,19 @@ make generated_files >/dev/null
X=($(newer deepcopy "${STAMP}"))
if [[ "${#X[*]}" != 1 || ! ( "${X[0]}" =~ "${DIR}/zz_generated.deepcopy.go" ) ]]; then
echo "Wrong generated deepcopy files changed after touching src file:"
echo " ${X[@]:-(none)}"
echo " ${X[*]:-(none)}"
exit 1
fi
X=($(newer defaults "${STAMP}"))
if [[ "${#X[*]}" != 1 || ! ( "${X[0]}" =~ "${DIR}/zz_generated.defaults.go" ) ]]; then
echo "Wrong generated defaults files changed after touching src file:"
echo " ${X[@]:-(none)}"
echo " ${X[*]:-(none)}"
exit 1
fi
X=($(newer conversion "${STAMP}"))
if [[ "${#X[*]}" != 1 || ! ( "${X[0]}" =~ "${DIR}/zz_generated.conversion.go" ) ]]; then
echo "Wrong generated conversion files changed after touching src file:"
echo " ${X[@]:-(none)}"
echo " ${X[*]:-(none)}"
exit 1
fi
@@ -110,7 +110,7 @@ make generated_files >/dev/null
X=($(older deepcopy "${STAMP}"))
if [[ "${#X[*]}" != 0 ]]; then
echo "Generated deepcopy files did not change after touching code-generator file:"
echo " ${X[@]:-(none)}"
echo " ${X[*]:-(none)}"
exit 1
fi
@@ -122,7 +122,7 @@ make generated_files >/dev/null
X=($(older deepcopy "${STAMP}"))
if [[ "${#X[*]}" != 0 ]]; then
echo "Generated deepcopy files did not change after touching code-generator dir:"
echo " ${X[@]:-(none)}"
echo " ${X[*]}:-(none)"
exit 1
fi
@@ -134,7 +134,7 @@ make generated_files >/dev/null
X=($(older deepcopy "${STAMP}"))
if [[ "${#X[*]}" != 0 ]]; then
echo "Generated deepcopy files did not change after touching code-generator dep file:"
echo " ${X[@]:-(none)}"
echo " ${X[*]:-(none)}"
exit 1
fi
@@ -146,7 +146,7 @@ make generated_files >/dev/null
X=($(older deepcopy "${STAMP}"))
if [[ "${#X[*]}" != 0 ]]; then
echo "Generated deepcopy files did not change after touching code-generator dep dir:"
echo " ${X[@]:-(none)}"
echo " ${X[*]:-(none)}"
exit 1
fi
@@ -162,7 +162,7 @@ make generated_files >/dev/null
X=($(older defaults "${STAMP}"))
if [[ "${#X[*]}" != 0 ]]; then
echo "Generated defaults files did not change after touching code-generator file:"
echo " ${X[@]:-(none)}"
echo " ${X[*]:-(none)}"
exit 1
fi
@@ -174,7 +174,7 @@ make generated_files >/dev/null
X=($(older defaults "${STAMP}"))
if [[ "${#X[*]}" != 0 ]]; then
echo "Generated defaults files did not change after touching code-generator dir:"
echo " ${X[@]:-(none)}"
echo " ${X[*]:-(none)}"
exit 1
fi
@@ -186,7 +186,7 @@ make generated_files >/dev/null
X=($(older defaults "${STAMP}"))
if [[ "${#X[*]}" != 0 ]]; then
echo "Generated defaults files did not change after touching code-generator dep file:"
echo " ${X[@]:-(none)}"
echo " ${X[*]:-(none)}"
exit 1
fi
@@ -198,7 +198,7 @@ make generated_files >/dev/null
X=($(older defaults "${STAMP}"))
if [[ "${#X[*]}" != 0 ]]; then
echo "Generated defaults files did not change after touching code-generator dep dir:"
echo " ${X[@]:-(none)}"
echo " ${X[*]:-(none)}"
exit 1
fi
@@ -214,7 +214,7 @@ make generated_files >/dev/null
X=($(older conversion "${STAMP}"))
if [[ "${#X[*]}" != 0 ]]; then
echo "Generated conversion files did not change after touching code-generator file:"
echo " ${X[@]:-(none)}"
echo " ${X[*]:-(none)}"
exit 1
fi
@@ -226,7 +226,7 @@ make generated_files >/dev/null
X=($(older conversion "${STAMP}"))
if [[ "${#X[*]}" != 0 ]]; then
echo "Generated conversion files did not change after touching code-generator dir:"
echo " ${X[@]:-(none)}"
echo " ${X[*]:-(none)}"
exit 1
fi
@@ -238,7 +238,7 @@ make generated_files >/dev/null
X=($(older conversion "${STAMP}"))
if [[ "${#X[*]}" != 0 ]]; then
echo "Generated conversion files did not change after touching code-generator dep file:"
echo " ${X[@]:-(none)}"
echo " ${X[*]:-(none)}"
exit 1
fi
@@ -250,7 +250,7 @@ make generated_files >/dev/null
X=($(older conversion "${STAMP}"))
if [[ "${#X[*]}" != 0 ]]; then
echo "Generated conversion files did not change after touching code-generator dep dir:"
echo " ${X[@]:-(none)}"
echo " ${X[*]:-(none)}"
exit 1
fi
@@ -266,7 +266,7 @@ make generated_files >/dev/null
X=($(newer openapi "${STAMP}"))
if [[ "${#X[*]}" != 1 || ! ( "${X[0]}" =~ "pkg/generated/openapi/zz_generated.openapi.go" ) ]]; then
echo "Wrong generated openapi files changed after touching src file:"
echo "${X[@]:-(none)}"
echo "${X[*]:-(none)}"
exit 1
fi
@@ -282,7 +282,7 @@ make generated_files >/dev/null
X=($(older openapi "${STAMP}"))
if [[ "${#X[*]}" != 0 ]]; then
echo "Generated openapi files did not change after touching code-generator file:"
echo " ${X[@]:-(none)}"
echo " ${X[*]:-(none)}"
exit 1
fi
@@ -294,7 +294,7 @@ make generated_files >/dev/null
X=($(older openapi "${STAMP}"))
if [[ "${#X[*]}" != 0 ]]; then
echo "Generated openapi files did not change after touching code-generator dir:"
echo " ${X[@]:-(none)}"
echo " ${X[*]:-(none)}"
exit 1
fi
@@ -306,7 +306,7 @@ make generated_files >/dev/null
X=($(older openapi "${STAMP}"))
if [[ "${#X[*]}" != 0 ]]; then
echo "Generated openapi files did not change after touching code-generator dep file:"
echo " ${X[@]:-(none)}"
echo " ${X[*]:-(none)}"
exit 1
fi
@@ -318,6 +318,6 @@ make generated_files >/dev/null
X=($(older openapi "${STAMP}"))
if [[ "${#X[*]}" != 0 ]]; then
echo "Generated openapi files did not change after touching code-generator dep dir:"
echo " ${X[@]:-(none)}"
echo " ${X[*]:-(none)}"
exit 1
fi

27
pkg/cloudprovider/providers/.import-restrictions Normal file
View File

@@ -0,0 +1,27 @@
{
"Rules": [
{
"SelectorRegexp": "k8s[.]io/kubernetes",
"AllowedPrefixes": [
"k8s.io/kubernetes/pkg/api/legacyscheme",
"k8s.io/kubernetes/pkg/api/service",
"k8s.io/kubernetes/pkg/api/v1/service",
"k8s.io/kubernetes/pkg/apis/core",
"k8s.io/kubernetes/pkg/cloudprovider",
"k8s.io/kubernetes/pkg/credentialprovider",
"k8s.io/kubernetes/pkg/features",
"k8s.io/kubernetes/pkg/kubelet/apis",
"k8s.io/kubernetes/pkg/master/ports",
"k8s.io/kubernetes/pkg/util/mount",
"k8s.io/kubernetes/pkg/util/file",
"k8s.io/kubernetes/pkg/util/net/sets",
"k8s.io/kubernetes/pkg/util/resizefs",
"k8s.io/kubernetes/pkg/util/strings",
"k8s.io/kubernetes/pkg/version",
"k8s.io/kubernetes/pkg/volume"
],
"ForbiddenPrefixes": [
]
}
]
}

Some files were not shown because too many files have changed in this diff Show More