As thockin found out here https://github.com/kubernetes/kubernetes/issues/24518,
vast majority of the do-nothing build time is spent in rebuilding the test
binaries. There is no staleness check support for test binaries.
This commit implements the staleness checks for test binaries and uses them
while building packages.
Tests are TBD. I am still trying to figure out how to test this.
Our `realpath` and `readlink -f` functions (required only because of MacOS,
thanks Steve) were poor substitutes at best. Mostly they were downright
broken. This thoroughly overhauls them and adds a test (in comments, since we
don't seem to have shell tests). For all the interesting cases I could think
of, the fakes act just like the real thing.
Then use those and canonicalize KUBE_ROOT. In order to make recursive calls of
our shell tool not additively grow `pwd` we have to essentially make the
sourcing of init.sh idempotent.
Automatic merge from submit-queue
Introduce events flag for describers
Printing events for a given object is not always needed. Thus, introducing --show-events=false to ``kubectl describe`` to skip events printing.
Fixes: #24239
Introduce DescriberSettings for Describer display options
Introduce --show-events flag and DescriberSettings in Describer methods
Introduce unit-tests
Regenerated kubectl describe docs
Add events flag tests to test-cmd.sh
Signed-off-by: dhodovsk@redhat.com
Signed-off-by: jchaloup@redhat.com
Automatic merge from submit-queue
Fix spacing in usage_from_stdin and info_from_stdin (issue #24186).
If "a" is a bash array, then the syntax to append the contents of $line as a
new element to the array is a+=("$line"), not messages+=$line
Using the former syntax just seems to append to the first element, creating a
long string and thus losing newline information.
Fixing this allows us to drop some empty lines from invocations of
usage_from_stdin.
If "a" is a bash array, then the syntax to append the contents of $line as a
new element to the array is a+=("$line"), not messages+=$line
Using the former syntax just seems to append to the first element, creating a
long string and thus losing newline information.
Fixing this allows us to drop some empty lines from invocations of
usage_from_stdin.
Out tests for features that didn't exist in old golang versions (like
gofmt didn't exist in 1.3 or the linker didn't want an = in 1.4) checked
for the new version. But this means every time there is a new version we
have to update the test. This PR inverts the test. We do it the old way
for the old version and if a new version comes along we just assume it
is going to work the new way.
- Add Makefile targets
- Start services in the test harness and connect locally
- Build test into binary and copy to remote host to start services
- Use tar to copy binaries to remote hosts to simplify design
Some functionality in hack/lib is currently depended on by
cluster/common.sh so kube-up from the full release tar (which
does not include hack/) is currently broken. With this PR we
create cluster/lib/ and move the necessary bits from hack/
over to get kube-up working again.
Fixes: 96d1b8d1b2
Signed-off-by: Mike Danese <mikedanese@google.com>
Just do all doc generation in the hack::util::gen-docs instead of spread
around. We also only track the generated docs in a single file for the
whole tree.
kube::util:ensure-temp-dir claims that it will automatically clean it
up. But it obviously doesn't. Since we cannot add multiple trap in bash
add a function that lets us trap and clean up KUBE_TEMP even if someone
already set a trap.
Instead of calling rsync over and over and over and over and over and
over and over and over and over (and probably over) use one `cp`
Before:
real 0m5.247s
user 0m2.294s
sys 0m1.300s
After:
real 0m2.260s
user 0m2.230s
sys 0m0.936s
This PR changes how we version going forward in the following ways:
* mark-new-version.sh is changed to a new policy of just splitting
branches, rather than the old backmerge policy, as discussed in
vX.Y.0, and a tag for vX.(Y+1).0-alpha.0 back to master.
* I eliminated PRs back to master by making the version/base.go
gitVersion and gitCommit just be `export-subst`. I testing that this
works with GitHub's source export tarballs. There's no reason to
bother with forcing the version into `base.go` (especially twice). The
tarball versions outside a git tree aren't perfect (master looks like
"v0.0.0+hash", and the release branches look more accurate), but our
build contract has never allowed that version is perfect in this
situation, so I think we can relax this.
* That master tag gets picked up by "git describe" on master, so e.g.
master would have immediately become v1.1.0-alpha.0
* In order to be more semVer compatible, the gitVersion field for the
master branch now looks something like 1.1.0-alpha.0.6+84c76d1142ea4d.
This is a tiny translation of the "git describe". I did this because
there are a ton of consumers out there of the "gitVersion" field
expecting it to be the full version, but it would be nice if this
field were actually semver compliant. (1.1.0-alpha.0-6-84c76d1142ea4d
is, but it's not *usefully* so.)
Fixes#11495
* Add analytics munger w/ munge heading
* More link autofixes
* Allow running a subset of munges
* Fix repo root detection
* Only process non-preformatted blocks
* Gendocs no longer adds the analytics link; mungedocs does that in a
second pass.
Adds cmd/mungedocs which is framework for processing
all files under docs/ and either verifying that no changes needed or
making in-place changes.
Did not reuse kube::util::gen-docs because that seemed to be
centered around handling added files, and this pass does not
add files.
Planned uses:
- table of contents automatic updating
- linkification
- internal link checker
- link-path-relativizer or absolutizer
- example file syncer
- header inserter.
Just table-of-contents updating in this PR.
Added Table of Contents to docs/networking.md.
Demonstrates use of new TOC generator presubmit.
Other docs will be added in future PRs.
Additional development will be needed to handle some
of the more complex cases.
hack/build-go.sh --use_go_build would fail because e2e.test was not being
excluded from the set of things that we were "go build"ing
Instead of walking all binaries, only walk static and nonstatic
```
$ hack/build-go.sh
+++ [0524 18:09:51] Building go targets for linux/amd64:
cmd/kube-proxy
cmd/kube-apiserver
cmd/kube-controller-manager
cmd/kubelet
cmd/hyperkube
cmd/kubernetes
plugin/cmd/kube-scheduler
cmd/kubectl
cmd/integration
cmd/gendocs
cmd/genman
cmd/genbashcomp
examples/k8petstore/web-server
github.com/onsi/ginkgo/ginkgo
test/e2e/e2e.test
+++ [0524 18:09:51] +++ Warning: stdlib pkg with cgo flag not found.
+++ [0524 18:09:51] +++ Warning: stdlib pkg cannot be rebuilt since /usr/lib/golang/pkg is not writable by eparis
+++ [0524 18:09:51] +++ Warning: Make /usr/lib/golang/pkg writable for eparis for a one-time stdlib install, Or
+++ [0524 18:09:51] +++ Warning: Rebuild stdlib using the command 'CGO_ENABLED=0 go install -a -installsuffix cgo std'
+++ [0524 18:09:51] +++ Falling back to go build, which is slower
**************can't load package: package github.com/GoogleCloudPlatform/kubernetes/test/e2e/e2e.test: cannot find package "github.com/GoogleCloudPlatform/kubernetes/test/e2e/e2e.test" in any of:
/usr/lib/golang/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/e2e.test (from $GOROOT)
/storage/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/e2e.test (from $GOPATH)
/storage/kubernetes/Godeps/_workspace/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/e2e.test
!!! Error in /storage/kubernetes/hack/lib/golang.sh:339
'go build -o "${outfile}" "${goflags[@]:+${goflags[@]}}" -ldflags "${version_ldflags}" "${binary}"' exited with status 1
Call stack:
1: /storage/kubernetes/hack/lib/golang.sh:339 kube::golang::build_binaries_for_platform(...)
2: /storage/kubernetes/hack/lib/golang.sh:488 kube::golang::build_binaries(...)
3: hack/build-go.sh:26 main(...)
Exiting with status 1
!!! Error in /storage/kubernetes/hack/lib/golang.sh:406
'( kube::golang::setup_env; local version_ldflags; version_ldflags=$(kube::version::ldflags); local host_platform; host_platform=$(kube::golang::host_platform); local goflags; eval "goflags=(${KUBE_GOFLAGS:-})"; local use_go_build; local -a targets=(); local arg; for arg in "$@";
do
if [[ "${arg}" == "--use_go_build" ]]; then
use_go_build=true;
else
if [[ "${arg}" == -* ]]; then
goflags+=("${arg}");
else
targets+=("${arg}");
fi;
fi;
done; if [[ ${#targets[@]} -eq 0 ]]; then
targets=("${KUBE_ALL_TARGETS[@]}");
fi; local -a platforms=("${KUBE_BUILD_PLATFORMS[@]:+${KUBE_BUILD_PLATFORMS[@]}}"); if [[ ${#platforms[@]} -eq 0 ]]; then
platforms=("${host_platform}");
fi; local binaries; binaries=($(kube::golang::binaries_from_targets "${targets[@]}")); local parallel=false; if [[ ${#platforms[@]} -gt 1 ]]; then
local gigs; gigs=$(kube::golang::get_physmem); if [[ ${gigs} -gt ${KUBE_PARALLEL_BUILD_MEMORY} ]]; then
kube::log::status "Multiple platforms requested and available ${gigs}G > threshold ${KUBE_PARALLEL_BUILD_MEMORY}G, building platforms in parallel"; parallel=true;
else
kube::log::status "Multiple platforms requested, but available ${gigs}G < threshold ${KUBE_PARALLEL_BUILD_MEMORY}G, building platforms in serial"; parallel=false;
fi;
fi; if [[ "${parallel}" == "true" ]]; then
kube::log::status "Building go targets for ${platforms[@]} in parallel (output will appear in a burst when complete):" "${targets[@]}"; local platform; for platform in "${platforms[@]}";
do
( kube::golang::set_platform_envs "${platform}"; kube::log::status "${platform}: go build started"; kube::golang::build_binaries_for_platform ${platform} ${use_go_build:-}; kube::log::status "${platform}: go build finished" ) &> "/tmp//${platform//\//_}.build" &
done; local fails=0; for job in $(jobs -p);
do
wait ${job} || let "fails+=1";
done; for platform in "${platforms[@]}";
do
cat "/tmp//${platform//\//_}.build";
done; exit ${fails};
else
for platform in "${platforms[@]}";
do
kube::log::status "Building go targets for ${platform}:" "${targets[@]}"; kube::golang::set_platform_envs "${platform}"; kube::golang::build_binaries_for_platform ${platform} ${use_go_build:-};
done;
fi )' exited with status 1
Call stack:
1: /storage/kubernetes/hack/lib/golang.sh:406 kube::golang::build_binaries(...)
2: hack/build-go.sh:26 main(...)
Exiting with status 1
```
Go 1.4 added the -o flag to the "go test" command as well as support for
the TestMain() function, so we must work around these not existing in
Go 1.3.
The version of readlink on OS X does not have the -f flag - so we'll
just skip canonicalizing the path.
This commit deletes cmd/e2e and updates hack/ginkgo-e2e.sh to use the
'ginkgo' command instead. All logic from cmd/e2e/e2e.go and
test/e2e/driver.go have been combined into the new file
test/e2e/e2e_test.go.
The test tarball now includes a built version of the test/e2e test
binary, which includes all tests under test/e2e. This was accomplished
by updating the build scripts to use 'go test -c' when a target name
ended with '.test', and adding a dependency on test/e2e/e2e.test.
This prebuilt test binary is passed to the Ginkgo runner in
hack/ginkgo-e2e.sh. In a future change, we can add support to run
Ginkgo against the source tree if it is available.
This change is generally intended to have no externally visible changes,
aside from the following caveats:
- The -t/--tests flag has been removed
- Calling cmd/e2e/e2e directly obviously won't work, but that was never
intended to be supported anyway
- If the GINKGO_PARALLEL environment variable is set to y, then ginkgo
will run test specs in parallel. (Currently defaults to n, since some
tests are broken in this mode.)
Additionally, several tests which made poor assumptions about cwd or
used testContext before it had been set have been fixed.
This commit deletes cmd/e2e and updates hack/ginkgo-e2e.sh to use the
'ginkgo' command instead. All logic from cmd/e2e/e2e.go and
test/e2e/driver.go have been combined into the new file
test/e2e/e2e_test.go.
Additionally, several tests which made poor assumptions about cwd or
used testContext before it was set have been fixed.
This change is generally intended to have no externally visible changes,
aside from the following caveats:
- The -t/--tests flag has been removed
- Calling cmd/e2e/e2e directly obviously won't work, but that was never
supported anyway
- If the GINKGO_PARALLEL environment variable is set to y, then ginkgo
will run test specs in parallel. (Currently defaults to n, since some
tests are broken in this mode.)
