Merge pull request #24850 from madhusudancs/speedup-do-nothing

Automatic merge from submit-queue

Improve the speed of do-nothing build.

As @thockin found out here https://github.com/kubernetes/kubernetes/issues/24518, vast majority of the do-nothing build time is spent in rebuilding the test binaries. There is no staleness check support for test binaries.

This commit implements the staleness checks for test binaries and uses them while building packages.

On my workstation, do-nothing hack/build-go.sh time goes from ~20 secs to ~4 secs, of which only ~1 sec is from doing test binary staleness check now (as opposed to ~17 secs it took to build the test binaries before). I did some experiments to bring this time down to <1 sec. I measured using go test -bench, but it was not very useful in this case. I believe, a vast majority of that ~1 second is being spent in fork/exec and piping the results back to the staleness check program along with the ser-deser involved, but it needs to be validated. Not a proof, but to provide some supporting evidence to this claim, running `go list -f format packages` in the shell takes about 600ms irrespective of what's in the format.

Tests are TBD. I am still trying to figure out how to test this, but I would like to get early feedback

cc @mikedanese @mml

hack/lib/golang.sh

@@ -377,6 +377,25 @@ kube::golang::fallback_if_stdlib_not_installable() {
   use_go_build=true
 }
 
+# Builds the toolchain necessary for building kube. This needs to be
+# built only on the host platform.
+# TODO: This builds only the `teststale` binary right now. As we expand
+# this function's capabilities we need to find this a right home.
+# Ideally, not a shell script because testing shell scripts is painful.
+kube::golang::build_kube_toolchain() {
+  local targets=(
+    hack/cmd/teststale
+  )
+
+  local binaries
+  binaries=($(kube::golang::binaries_from_targets "${targets[@]}"))
+
+  kube::log::status "Building the toolchain targets:" "${binaries[@]}"
+  go install "${goflags[@]:+${goflags[@]}}" \
+        -ldflags "${goldflags}" \
+        "${binaries[@]:+${binaries[@]}}"
+}
+
 # Try and replicate the native binary placement of go install without
 # calling go install.
 kube::golang::output_filename_for_binary() {
@@ -449,12 +468,39 @@ kube::golang::build_binaries_for_platform() {
   for test in "${tests[@]:+${tests[@]}}"; do
     local outfile=$(kube::golang::output_filename_for_binary "${test}" \
       "${platform}")
+
+    local testpkg="$(dirname ${test})"
+
+    # Staleness check always happens on the host machine, so we don't
+    # have to locate the `teststale` binaries for the other platforms.
+    # Since we place the host binaries in `$KUBE_GOPATH/bin`, we can
+    # assume that the binary exists there, if it exists at all.
+    # Otherwise, something has gone wrong with building the `teststale`
+    # binary and we should safely proceed building the test binaries
+    # assuming that they are stale. There is no good reason to error
+    # out.
+    if test -x "${KUBE_GOPATH}/bin/teststale" && ! "${KUBE_GOPATH}/bin/teststale" -binary "${outfile}" -package "${testpkg}"
+    then
+      continue
+    fi
+
+    # `go test -c` below directly builds the binary. It builds the packages,
+    # but it never installs them. `go test -i` only installs the dependencies
+    # of the test, but not the test package itself. So neither `go test -c`
+    # nor `go test -i` installs, for example, test/e2e.a. And without that,
+    # doing a staleness check on k8s.io/kubernetes/test/e2e package always
+    # returns true (always stale). And that's why we need to install the
+    # test package.
+    go install "${goflags[@]:+${goflags[@]}}" \
+        -ldflags "${goldflags}" \
+        "${testpkg}"
+
     mkdir -p "$(dirname ${outfile})"
     go test -c \
       "${goflags[@]:+${goflags[@]}}" \
       -ldflags "${goldflags}" \
       -o "${outfile}" \
-      "$(dirname ${test})"
+      "${testpkg}"
   done
 }
 
@@ -549,6 +595,9 @@ kube::golang::build_binaries() {
       fi
     fi
 
+    # First build the toolchain before building any other targets
+    kube::golang::build_kube_toolchain
+
     if [[ "${parallel}" == "true" ]]; then
       kube::log::status "Building go targets for ${platforms[@]} in parallel (output will appear in a burst when complete):" "${targets[@]}"
       local platform