While the umask was previously set for builds (and in fact `docker run`
sets it to 0022 by default), the release pipeline `cp`'s files (with
`-a`) which respect the user's umask, and is outside of the build path.
This sets the umask in a common function called by all build tools,
which Stephen agreed was the simplest and most complete fix. The
`verify-prereqs` function is already poorly named for what it does, and
this only makes it epsilon worse.
Some binaries now run as non-root (kube-scheduler). When umask is 0027,
for example, the container image we build has the binary 0750, which is
not executable by the non-root UID.
We have been having issues with making builds reproducible, especially
with the `.note.go.buildid` ELF section. One tip from a golang issue was
to set `-ldflags=-buildid=` which seems to work well. You can confirm
that the buildid is set to empty by inspecting the binaries with the go
command example `go tool buildid _output/local/go/bin/kubectl`
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
In 24d105995d, a fix was made in bazel
based builds to ensure that we add `selinux` tag when we build all
binaries especially the `kubelet`. We need to do the same for in our
hack scripts so things like `make release` will work properly as well.
Some scripts use `GOFLAGS=-tags=providerless` for example, So we should
support the tags to be specified in GOFLAGS as well. We parse out the
tags from there and ensure selinux is added to the list of tags we used
for building the binaries. Note that we add our own `-tags` with the
full set of tags and since we specify our parameter at the end, ours
full list takes precendence
Adds a go app which runs the e2e tests with ginkgo.
- Supports all the existing env vars of the bash script
- Improved flow control to avoid and better report issues
regarding the process PID
- Adds flags for modifying where to find the test binary and
ginkgo binary so that you can run it locally
- Adds 3 flags for specifying extra args before the double-dash,
extra args after the double-dash, and the seperator to use between
values in those env vars. This allows setting arbitrary, complex
values for use on the command such as flags which include spaces
or other characters.
when compile kubectl on platform other than
linux/amd64, we need to check the KUBE_SERVER_PLATFORMS
array emptiness before assign it.
the example command is:
make WHAT=cmd/kubectl KUBE_BUILD_PLATFORMS="darwin/amd64 windows/amd64"
Prior to this change, including windows/amd64 in KUBE_BUILD_PLATFORMS
would, for example, attempt to build the server binaries/tars/images for
Windows, which is not supported. This can break downstream build steps.
Recent change to hack/lib/golang.sh broke the build on MacOS this way:
$ make clean && make generated_files
+++ [0325 13:38:22] Verifying Prerequisites....
+++ [0325 13:38:23] Removing _output directory
k8s.io/kubernetes/vendor/github.com/spf13/pflag
k8s.io/kubernetes/hack/make-rules/helpers/go2make
+++ [0325 13:38:40] Building go targets for darwin/amd64:
./vendor/k8s.io/code-generator/cmd/deepcopy-gen
can't load package: package k8s.io/kubernetes: no Go files in k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes
!!! [0325 13:38:40] Call tree:
!!! [0325 13:38:40] 1: k8s.io/kubernetes/hack/lib/golang.sh:629 kube::golang::build_some_binaries(...)
!!! [0325 13:38:40] 2: k8s.io/kubernetes/hack/lib/golang.sh:764 kube::golang::build_binaries_for_platform(...)
!!! [0325 13:38:40] 3: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
make[1]: *** [_output/bin/deepcopy-gen] Error 1
make: *** [generated_files] Error 2
It was caused by 'binaries' array not being declared with 'local -a'.
It looks like MacOS' old bash version makes an array to contain first
empty element if declared this way.
The fix has been tested on MacOS High Sierra and Linux openSUSE 42.3 (x86_64)
Signed-off-by: Ed Bartosh <eduard.bartosh@intel.com>
Otherwise, calling make followed by bazel might fail, requiring one to
run make clean first.
Additionally, add comments explaining why we must do this.
In go 1.11, go commands will use `GOFLAGS` as default flags, see
https://golang.org/doc/go1.11#go_command.
There is no need to pass GOFLAGS to $goflags, and if we do, go commands
will fail with "duplicate flags" error, e.g.
```
$ make test-integration WHAT=./test/integration/scheduler GOFLAGS="-v"
...
go test: v flag may be set only once
run "go help test" or "go help testflag" for more information
...
```
Pick up some code from https://github.com/heptio/kube-conformance
Fix up build scripts for the new conformance image
Fix Header template and Copyright to make verify job go green
update README and add execute permissions for script
Change-Id: Ib6509acd816cc2fb3a516bfb8e0ff9e32bff8f79
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
Add ability to build with runtime coverage instrumentation
**What this PR does / why we need it**:
This PR adds the ability to instrument a subset of kubernetes binaries to report code coverage information. The specific use-case is to help determine coverage of our end-to-end Conformance tests, as well as provide data that can be used to help determine where to focus. This PR focuses on making it possible to build with instrumentation; collecting and using the generated coverage data will be done in later PRs. For more details as to the intent, see the [design doc](https://docs.google.com/document/d/1FKMBFxz7vtA-6ZgUkA47F8m6yR00fwqLcXMVJqsHt0g/edit?usp=sharing) (google doc; requires kubernetes-dev membership).
Specifically, this PR adds a new `KUBE_BUILD_WITH_COVERAGE` make variable, which when set will cause `kube-apiserver`, `kube-controller-manager`, `kube-scheduler`, `kube-proxy` and `kubelet` to be built with coverage instrumentation. These coverage-instrumented binaries will flush coverage information to disk every five seconds, defaulting to a temporary directory unless the `KUBE_COVERAGE_FILE` environment variable is set at launch, in which case it will write to that file instead.
The mechanism used to achieve coverage instrumentation is to build the targeted binaries as "unit tests" with coverage enabled, and then rigging the unit tests to just execute the binary's usual entry point. This is implemented only for the bash build system.
/sig testing
```release-note
NONE
```
First version of this was only standalone, now the code is being
vendored by anyone who wants to use it. So the standalone binary and
container are no longer useful.
Change-Id: Ib9369de66b4ecb3451f73ba2a252526d6615b96f
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Update all script shebangs to use /usr/bin/env interpreter instead of /bin/interpreter
This is required to support systems where bash doesn't reside in /bin (such as NixOS, or the *BSD family) and allow users to specify a different interpreter version through $PATH manipulation.
https://www.cyberciti.biz/tips/finding-bash-perl-python-portably-using-env.html
```release-note
Use /usr/bin/env in all script shebangs to increase portability.
```
Automatic merge from submit-queue (batch tested with PRs 63046, 62925, 63014). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Bump minimum required go version to 1.10.1
**What this PR does / why we need it**:
We have supported go1.10.x for a few weeks now (#60597). CI has all been updated. This enforces it as the new minimum go version required go k8s >=1.11
**Release note**:
No release note, as #60597 already had one.
```release-note
NONE
```
As of go 1.10, we do not need to explicitly build the stdlib for static
any more. That happens automatically. Timing tests show no discernible
difference between a `go install` with and without it.
Also use 'static' suffix instead of 'cgo'.
It's admittedly extremely unlikely that the host platform name would
contain special characters, but still use double quotes to pattern
matching.
Consider this script:
#!/bin/bash
bar="foobar"
foo="foo*"
[[ $bar == $foo ]] && echo "first true"
[[ "$bar" == "$foo" ]] && echo "second true"
We get the output:
first true
The plan is to move from first case to the second case to prevent
pattern match where there shouldn't be any.
This moves plugin/pkg/scheduler to pkg/scheduler and
plugin/cmd/kube-scheduler to cmd/kube-scheduler.
Bulk of the work was done with gomvpkg, except for kube-scheduler main
package.
Automatic merge from submit-queue (batch tested with PRs 54773, 52523, 47497, 55356, 49429). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
don't check in mounter binary
```release-note
GCI mounter is moved from the manifests tarball to the server tarball.
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
fix typos: remove duplicated word in comments
**What this PR does / why we need it**: Remove the duplicated word `the` in comments
**Which issue this PR fixes** : fixes #
**Special notes for your reviewer**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 49083, 45540, 46862)
Respect KUBE_BUILD_PLATFORMS set by user
**What this PR does / why we need it**:
Currently the only(?) toggle available for custom arch builds is
KUBE_FASTBUILD. We should allow the user to specify a list
of arch(es) in the environment variable KUBE_BUILD_PLATFORMS.
Example:
KUBE_BUILD_PLATFORMS="linux/amd64 linux/arm64" hack/build-cross.sh
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
Fixes#20365
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Currently the only(?) toggle available for custom arch builds is
KUBE_FASTBUILD. We should allow the user to specify a list
of arch(es) in the environment variable KUBE_BUILD_PLATFORMS.
Example:
KUBE_BUILD_PLATFORMS="linux/amd64 linux/arm64" hack/build-cross.sh
Fixes#20365
Drop KUBE_GOFLAGS, KUBE_GOGCFLAGS, KUBE_GOLDFLAGS references
from the build infrastructure. There are some usages still
for KUBE_GOFLAGS, so we should print a deprecation message
until all those are fixed. If both KUBE_GOFLAGS and GOFLAGS
are just then we just bail out.
Automatic merge from submit-queue (batch tested with PRs 44591, 44549)
Update repo-infra bazel dependency and use new gcs_upload rule
This PR provides similar functionality to push-build.sh entirely within Bazel rules (though it relies on gsutil).
It's an alternative to #44306.
Depends on https://github.com/kubernetes/repo-infra/pull/13.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue
Remove the kube-discovery binary from the tree
**What this PR does / why we need it**:
kube-discovery was a temporary solution to implementing proposal: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/bootstrap-discovery.md
However, this functionality is now gonna be implemented in the core for v1.6 and will fully replace kube-discovery:
- https://github.com/kubernetes/kubernetes/pull/36101
- https://github.com/kubernetes/kubernetes/pull/41281
- https://github.com/kubernetes/kubernetes/pull/41417
So due to that `kube-discovery` isn't used in any v1.6 code, it should be removed.
The image `gcr.io/google_containers/kube-discovery-${ARCH}:1.0` should and will continue to exist so kubeadm <= v1.5 continues to work.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
Remove cmd/kube-discovery from the tree since it's not necessary anymore
```
@jbeda @dgoodwin @mikedanese @dmmcquay @lukemarsden @errordeveloper @pires
This adds a new stand-alone certificates controller for use on GKE. It
allows calling GKE to sign certificates instead of requiring the CA
private key locally.
It does not aim for 100% feature parity with kube-controller-manager
yet, so for instance, leader election support is omitted.
Automatic merge from submit-queue
goflags must be after subcommand
**What this PR does / why we need it**:
if GOFLAGS is set when calling make, kubernetes will fail to build
as an example, I often have `GOFLAGS=-v` so I can have some idea of progress during compilation
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*:
no known issue
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
- Remaining spaghetti untangled
- Missed bazel update and a few hardcoded refs
- New instance of framework.ReadOrDie reference removed post rebase
- Resolve new clientset rebase
- Fixed e2e/generated BUILD dep
- A space
- Missed gobindata ref in golang.sh
Automatic merge from submit-queue
Build vendored copy of go-bindata and use that in go generate step
**What this PR does / why we need it**: as the title says, uses the vendored version of `go-bindata` rather than expecting developers to `go get` it (when building outside docker).
**Which issue this PR fixes**: fixes#34067, partially addresses #36655
**Special notes for your reviewer**: we still call `go generate` far too many times:
```console
~/.../src/k8s.io/kubernetes $ which go-bindata
~/.../src/k8s.io/kubernetes $ make
+++ [1116 17:35:28] Building the toolchain targets:
k8s.io/kubernetes/hack/cmd/teststale
k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [1116 17:35:29] Generating bindata:
test/e2e/framework/gobindata_util.go
+++ [1116 17:35:30] Building go targets for linux/amd64:
cmd/libs/go2idl/deepcopy-gen
+++ [1116 17:35:35] Building the toolchain targets:
k8s.io/kubernetes/hack/cmd/teststale
k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [1116 17:35:35] Generating bindata:
test/e2e/framework/gobindata_util.go
+++ [1116 17:35:36] Building go targets for linux/amd64:
cmd/libs/go2idl/defaulter-gen
+++ [1116 17:35:41] Building the toolchain targets:
k8s.io/kubernetes/hack/cmd/teststale
k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [1116 17:35:41] Generating bindata:
test/e2e/framework/gobindata_util.go
+++ [1116 17:35:42] Building go targets for linux/amd64:
cmd/libs/go2idl/conversion-gen
+++ [1116 17:35:47] Building the toolchain targets:
k8s.io/kubernetes/hack/cmd/teststale
k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [1116 17:35:47] Generating bindata:
test/e2e/framework/gobindata_util.go
+++ [1116 17:35:48] Building go targets for linux/amd64:
cmd/libs/go2idl/openapi-gen
+++ [1116 17:35:56] Building the toolchain targets:
k8s.io/kubernetes/hack/cmd/teststale
k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [1116 17:35:56] Generating bindata:
test/e2e/framework/gobindata_util.go
```
Fixing that is a separate effort, though.
cc @sebgoa @ZhangBanger
If dependencies are touched, the Makefile will detect that tools like
deepcopy-gen need to be rebuilt. It will rebuild them, but Go decides the
dependencies are not actually updated (maybe it was a test file - known
limitation of the current Makefile) and does not touch the resulting binary.
Then the Makefile touches the result explicitly to mark thatthe dep change has
been handled. Then rsync blows away that change with the older file, and
helpfully preserves the timestamp.
Now rsync ignores it if the checksum is the same. Result = faster build.
We weren't getting incremental builds because of new test only subpackages. Our
voodoo combo of 'go install' and 'go test -c' didn't cache things like
'test/e2e_node/services'. Add the '-i' flag to 'go test' to install test only
dependencies too.
Automatic merge from submit-queue
build kube-discovery and kubeadm with release
This will start publishing binaries from our ci builds.
@errordeveloper @dgoodwin
Automatic merge from submit-queue
Add separate build process for node test.
This PR is part of https://github.com/kubernetes/kubernetes/pull/31093. However, because currently node e2e is built on `KUBE_TEST_PLATFORMS`, which includes linux/amd64, darwin/amd64, windows/amd64 and linux/arm, it caused #32251 to fail.
In fact, node e2e is running on the same node with kubelet, and it also has built-in apiserver, etcd and namespace controller. All of them are only built on `KUBE_SERVER_PLATFORMS`, so node e2e should also only be built on those platforms.
```
KUBE_SERVER_PLATFORMS=(
linux/amd64
linux/arm
linux/arm64
)
```
This PR added a separate build process for node e2e to address this.
@vishh Do you need this for v1.4? because this blocks your #32251. /cc @dchen1107
Automatic merge from submit-queue
Don't error out if ${go_pkg_dir} already exists
I have 'make' aliased to 'make -j4' for various reasons, and that's always worked fine for Kubernetes in the past. But it recently broke because EEXIST errors from this symlink creation are fatal. So make them not fatal, because if the path already exists, why try to create it again? It exists because two invocations of setup_env() are run in parallel due to the -j4, and they race to create the symlink and one obviously fails.
ln: failed to create symbolic link ‘/home/dcbw/Development/containers/kubernetes/_output/local/go/src/k8s.io/kubernetes’: File exists
Makefile.generated_files:285: recipe for target '_output/bin/deepcopy-gen' failed
make[1]: *** [_output/bin/deepcopy-gen] Error 1
make[1]: *** Waiting for unfinished jobs....
+++ [0826 10:18:02] Generating bindata:
/home/dcbw/Development/containers/kubernetes/test/e2e/framework/gobindata_util.go
+++ [0826 10:18:02] Building the toolchain targets:
k8s.io/kubernetes/hack/cmd/teststale
+++ [0826 10:18:03] Building go targets for linux/amd64:
cmd/libs/go2idl/conversion-gen
Makefile:281: recipe for target 'generated_files' failed
make: *** [generated_files] Error 2
Also build the hyperkube docker image on-the-fly.
This is only a temporary fix until the proposal in issue
https://github.com/kubernetes/kubernetes/issues/28630 is implemented.
Also, the new build/deployment method completely obviates this step.
We use debian image instead of busybox and do not build hyperkube as a
static binary yet. Wait until PR
https://github.com/kubernetes/kubernetes/pull/26028 is merged to build
static hyperkube binaries.
Repro case:
$ make clean generated_files
$ hack/update-generated-protobuf.sh
This would complain about not finding `fmt`, and it was indicating the wrong
GOROOT. The problem was that the first step built binaries for generating
code, which *embeds* the value of GOROOT into the binary. The whole tree was
bind-mounted into the build container and then JUST the dockerized dir was
mounted over it. The in-container build tried to use the existing binaries,
but GOROOT is wrong.
This change whites-out the whole _output dir.
I first made just an anonymous volume for _output, but docker makes that as
root, which means I can't write to it from our non-root build. So I just put
it in the data container. This seems to work. The biggest change this makes
is that the $GOPATH/bin/ and $GOPATH/pkg/ dirs will persist across dockerized
builds.
The KUBE_VERBOSE environment variable sets the verbosity level to
use. Log messages can specify a verbosity by setting the V
variable. e.g.
V=2 kube::log::info foo bar
Would only print "foo bar" if $KUBE_VERBOSE >= 2.
bindata and yaml, Gobindata automation
bindata utils for generating, go generate
match server version
gitignore for dirty, ca, rbase, KUBE_ROOT, buildfix
(rebased jul-25,29)
