Now that they all call setup_env, we don't need find-binary (I think).
That was originally meant to hide the diff between docker and local
builds but all these tools do local builds anyway.
* Moved code and tests out of gengo -> code_generator
* Reworked it to use packages.Load
* Reworked tests (still not comprehensive but pretty good?)
* Dropped test support from gengo (support for tests in
x/tools/go/packages is pretty hostile to gengo, and nobody used it)
Instead of walking paths ourselves, just let Go's packages library do
it. This is a slight CLI change - it wants "./foo" rather than "foo".
This also flagged a few things which seem to be legit failures.
Prior to this it would download Go, setup the environment, then (maybe)
download Go again.
I renamed verify_go_version to be internal and fixed all callers.
This helps 2 things:
1) output is cleaner
2) whether a build is static or non-static depends on string matching,
which is easier after this
If you specify a `...` pattern, this will preserve it, which breaks the
static/non-static matching but really, can we just agree that `make` is
for binaries? It works for other things but so does `go build`.
Example:
```
$ make WHAT="cmd/kubectl/ ./staging/src/k8s.io/api/... ./test/e2e/e2e.test ./test/e2e/e2e.test ./cmd/kubectl/ "
go version go1.21.4 linux/amd64
+++ [1208 22:24:43] Building go targets for linux/amd64
k8s.io/api/... (non-static)
k8s.io/kubernetes/cmd/kubectl (static)
k8s.io/kubernetes/test/e2e/e2e.test (test)
```
Result: `make test` works:
```
$ #### BUILD ####
$ make WHAT="./cmd/kubectl/ ./staging/src/k8s.io/api"
+++ [1211 11:20:46] Building go targets for linux/amd64
./cmd/kubectl/ (non-static)
./staging/src/k8s.io/api (non-static)
$ #### TEST ####
$ make test WHAT=./pkg/proxy/iptables/
+++ [1211 11:22:31] Set GOMAXPROCS automatically to 6
+++ [1211 11:22:31] Running tests without code coverage and with -race
ok k8s.io/kubernetes/pkg/proxy/iptables 9.517s
$ make test WHAT=pkg/proxy/iptables/
+++ [1211 11:22:53] Set GOMAXPROCS automatically to 6
+++ [1211 11:22:53] Running tests without code coverage and with -race
ok k8s.io/kubernetes/pkg/proxy/iptables (cached)
$ make test WHAT=k8s.io/kubernetes/pkg/proxy/iptables/
+++ [1211 11:23:09] Set GOMAXPROCS automatically to 6
+++ [1211 11:23:09] Running tests without code coverage and with -race
ok k8s.io/kubernetes/pkg/proxy/iptables (cached)
$ make test WHAT=./staging/src/k8s.io/api
+++ [1211 11:23:24] Set GOMAXPROCS automatically to 6
+++ [1211 11:23:24] Running tests without code coverage and with -race
ok k8s.io/api 21.981s
$ make test WHAT=staging/src/k8s.io/api
+++ [1211 11:23:54] Set GOMAXPROCS automatically to 6
+++ [1211 11:23:54] Running tests without code coverage and with -race
ok k8s.io/api (cached)
$ make test WHAT=k8s.io/api
+++ [1211 11:24:06] Set GOMAXPROCS automatically to 6
+++ [1211 11:24:06] Running tests without code coverage and with -race
ok k8s.io/api (cached)
```
and use it from test
Result: `make` works, but `make test` still broken
```
$ make kubectl
+++ [1211 11:15:46] Building go targets for linux/amd64
./cmd/kubectl (static)
$ make WHAT=./cmd/kubectl/
+++ [1211 11:15:53] Building go targets for linux/amd64
./cmd/kubectl/ (non-static)
$ make WHAT=cmd/kubectl/
+++ [1211 11:16:02] Building go targets for linux/amd64
./cmd/kubectl/ (non-static)
$ make WHAT=k8s.io/kubernetes/cmd/kubectl
+++ [1211 11:16:16] Building go targets for linux/amd64
k8s.io/kubernetes/cmd/kubectl (static)
$ make WHAT=./staging/src/k8s.io/api
+++ [1211 11:16:31] Building go targets for linux/amd64
./staging/src/k8s.io/api (non-static)
$ make WHAT=staging/src/k8s.io/api
+++ [1211 11:16:43] Building go targets for linux/amd64
./staging/src/k8s.io/api (non-static)
$ make WHAT=k8s.io/api
+++ [1211 11:16:50] Building go targets for linux/amd64
k8s.io/api (non-static)
```
```
$ make test WHAT=./cmd/kubectl
+++ [1211 11:17:23] Set GOMAXPROCS automatically to 6
+++ [1211 11:17:23] Running tests without code coverage and with -race
cmd/kubectl/kubectl.go:25:2: cannot find package "k8s.io/client-go/plugin/pkg/client/auth" in any of:
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/plugin/pkg/client/auth (vendor tree)
/home/thockin/sdk/gotip/src/k8s.io/client-go/plugin/pkg/client/auth (from $GOROOT)
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/client-go/plugin/pkg/client/auth (from $GOPATH)
cmd/kubectl/kubectl.go:20:2: cannot find package "k8s.io/component-base/cli" in any of:
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/component-base/cli (vendor tree)
/home/thockin/sdk/gotip/src/k8s.io/component-base/cli (from $GOROOT)
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/component-base/cli (from $GOPATH)
cmd/kubectl/kubectl.go:21:2: cannot find package "k8s.io/kubectl/pkg/cmd" in any of:
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd (vendor tree)
/home/thockin/sdk/gotip/src/k8s.io/kubectl/pkg/cmd (from $GOROOT)
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubectl/pkg/cmd (from $GOPATH)
cmd/kubectl/kubectl.go:22:2: cannot find package "k8s.io/kubectl/pkg/cmd/util" in any of:
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util (vendor tree)
/home/thockin/sdk/gotip/src/k8s.io/kubectl/pkg/cmd/util (from $GOROOT)
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubectl/pkg/cmd/util (from $GOPATH)
make: *** [Makefile:191: test] Error 1
```
This makes "new" and "old" setup_env functions. In subsequent commits,
all callers of the "old" form will be fixed, and the "new" will be
renamed back.
The old and new functions diff:
```diff
--- /tmp/a 2023-12-14 09:02:57.804092696 -0800
+++ /tmp/b 2023-12-14 09:03:09.679999585 -0800
@@ -1,4 +1,4 @@
-kube::golang::old::setup_env() {
+kube::golang:🆕:setup_env() {
kube::golang::verify_go_version
# Set up GOPATH. We have tools which depend on being in a GOPATH (see
@@ -7,9 +7,9 @@
# Even in module mode, we need to set GOPATH for `go build` and `go install`
# to work. We build various tools (usually via `go install`) from a lot of
# scripts.
- # * We can't set GOBIN because that does not work on cross-compiles.
- # * We could use `go build -o <something>`, but it's subtle when it comes
- # to cross-compiles and whether the <something> is a file or a directory,
+ # * We can't just set GOBIN because that does not work on cross-compiles.
+ # * We could always use `go build -o <something>`, but it's subtle wrt
+ # cross-compiles and whether the <something> is a file or a directory,
# and EVERY caller has to get it *just* right.
# * We could leave GOPATH alone and let `go install` write binaries
# wherever the user's GOPATH says (or doesn't say).
@@ -20,16 +20,6 @@
#
# Eventually, when we no longer rely on run-in-gopath.sh we may be able to
# simplify this some.
- local go_pkg_dir="${KUBE_GOPATH}/src/${KUBE_GO_PACKAGE}"
- local go_pkg_basedir
- go_pkg_basedir=$(dirname "${go_pkg_dir}")
-
- mkdir -p "${go_pkg_basedir}"
-
- # TODO: This symlink should be relative.
- if [[ ! -e "${go_pkg_dir}" || "$(readlink "${go_pkg_dir}")" != "${KUBE_ROOT}" ]]; then
- ln -snf "${KUBE_ROOT}" "${go_pkg_dir}"
- fi
export GOPATH="${KUBE_GOPATH}"
# If these are not set, set them now. This ensures that any subsequent
@@ -40,24 +30,10 @@
# Make sure our own Go binaries are in PATH.
export PATH="${KUBE_GOPATH}/bin:${PATH}"
- # Change directories so that we are within the GOPATH. Some tools get really
- # upset if this is not true. We use a whole fake GOPATH here to collect the
- # resultant binaries.
- local subdir
- subdir=$(kube::realpath . | sed "s|${KUBE_ROOT}||")
- cd "${KUBE_GOPATH}/src/${KUBE_GO_PACKAGE}/${subdir}" || return 1
-
- # Set GOROOT so binaries that parse code can work properly.
- GOROOT=$(go env GOROOT)
- export GOROOT
-
# Unset GOBIN in case it already exists in the current session.
# Cross-compiles will not work with it set.
unset GOBIN
- # This seems to matter to some tools
- export GO15VENDOREXPERIMENT=1
-
- # Disable workspaces
- export GOWORK=off
+ # Explicitly turn on modules.
+ export GO111MODULE=on
}
```
Result: `make` works for k/k:
```
$ make kubectl
+++ [1211 11:07:31] Building go targets for linux/amd64
k8s.io/kubernetes/cmd/kubectl (static)
$ make WHAT=./cmd/kubectl/
+++ [1211 11:08:19] Building go targets for linux/amd64
k8s.io/kubernetes/./cmd/kubectl/ (non-static)
$ make WHAT=k8s.io/kubernetes/cmd/kubectl
+++ [1211 11:08:52] Building go targets for linux/amd64
k8s.io/kubernetes/cmd/kubectl (static)
```
Result: `make` works for staging by package:
```
$ make WHAT=k8s.io/api
+++ [1211 11:11:37] Building go targets for linux/amd64
k8s.io/api (non-static)
```
Result: `make` fails for staging by path:
```
$ make WHAT=./staging/src/k8s.io/api
+++ [1211 11:12:44] Building go targets for linux/amd64
k8s.io/kubernetes/./staging/src/k8s.io/api (non-static)
cannot find module providing package k8s.io/kubernetes/staging/src/k8s.io/api: import lookup disabled by -mod=vendor
(Go version in go.work is at least 1.14 and vendor directory exists.)
!!! [1211 11:12:44] Call tree:
!!! [1211 11:12:44] 1: /home/thockin/src/kubernetes/hack/lib/golang.sh:850 kube::golang::build_some_binaries(...)
!!! [1211 11:12:44] 2: /home/thockin/src/kubernetes/hack/lib/golang.sh:1012 kube::golang::build_binaries_for_platform(...)
!!! [1211 11:12:44] 3: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [1211 11:12:44] Call tree:
!!! [1211 11:12:44] 1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [1211 11:12:44] Call tree:
!!! [1211 11:12:44] 1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
make: *** [Makefile:96: all] Error 1
```
Result: `make test` fails:
```
$ make test WHAT=./cmd/kubectl
+++ [1211 11:13:38] Set GOMAXPROCS automatically to 6
+++ [1211 11:13:38] Running tests without code coverage and with -race
cmd/kubectl/kubectl.go:25:2: cannot find package "k8s.io/client-go/plugin/pkg/client/auth" in any of:
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/plugin/pkg/client/auth (vendor tree)
/home/thockin/sdk/gotip/src/k8s.io/client-go/plugin/pkg/client/auth (from $GOROOT)
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/client-go/plugin/pkg/client/auth (from $GOPATH)
cmd/kubectl/kubectl.go:20:2: cannot find package "k8s.io/component-base/cli" in any of:
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/component-base/cli (vendor tree)
/home/thockin/sdk/gotip/src/k8s.io/component-base/cli (from $GOROOT)
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/component-base/cli (from $GOPATH)
cmd/kubectl/kubectl.go:21:2: cannot find package "k8s.io/kubectl/pkg/cmd" in any of:
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd (vendor tree)
/home/thockin/sdk/gotip/src/k8s.io/kubectl/pkg/cmd (from $GOROOT)
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubectl/pkg/cmd (from $GOPATH)
cmd/kubectl/kubectl.go:22:2: cannot find package "k8s.io/kubectl/pkg/cmd/util" in any of:
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util (vendor tree)
/home/thockin/sdk/gotip/src/k8s.io/kubectl/pkg/cmd/util (from $GOROOT)
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubectl/pkg/cmd/util (from $GOPATH)
make: *** [Makefile:191: test] Error 1
```
This creates go.work and enables Go Workspaces. This is a file that
includes info on k/k and all the staging modules.
This depends on go 1.22 and setting FORCE_HOST_GO=true (for kube
scripts, which try to be hermetic).
Make this part of the normal update/verify sequence.
The top-level go.work file contains no replace statements. Instead, the
replace statements in the individual go.mod files are used. For this to
work, replace statements in the individual go.mod files have to be
consistent.
hack/tools has different dependencies and can't be in the main
workspace, so this adds a go.work just for that. Without this, go tries
to consider all deps in all modules and pick one that works for all.
This is problematic because there are so many of them that it is
difficult to manage.
Likewise for k8s.io/code-generator/examples and
k8s.io/kms/internal/plugins/_mock - add trivial go.work files.
For example k/k depends on an older version of a lib that gloangci-lint
needs (transitively) and it breaks.
This also updates vendor (needed to make go happy), and removes
vendor'ed symlinks. This breaks a LOT of our build tools, which will be
fixed subsequently.
Result: `go` commands work across modules:
Before:
```
$ go list ./pkg/proxy/iptables/ ./staging/src/k8s.io/api/core/v1/
main module (k8s.io/kubernetes) does not contain package k8s.io/kubernetes/staging/src/k8s.io/api/core/v1
$ go build ./pkg/proxy/iptables/ ./staging/src/k8s.io/api
main module (k8s.io/kubernetes) does not contain package k8s.io/kubernetes/staging/src/k8s.io/api
$ go test ./pkg/proxy/iptables/ ./staging/src/k8s.io/api
main module (k8s.io/kubernetes) does not contain package k8s.io/kubernetes/staging/src/k8s.io/api
```
After:
```
$ go list ./pkg/proxy/iptables/ ./staging/src/k8s.io/api/core/v1/
k8s.io/kubernetes/pkg/proxy/iptables
k8s.io/api/core/v1
$ go build ./pkg/proxy/iptables/ ./staging/src/k8s.io/api
$ go test ./pkg/proxy/iptables/ ./staging/src/k8s.io/api
ok k8s.io/kubernetes/pkg/proxy/iptables 0.360s
ok k8s.io/api 2.302s
```
Result: `make` fails:
```
$ make
go version go1.22rc1 linux/amd64
+++ [0106 12:11:03] Building go targets for linux/amd64
k8s.io/kubernetes/cmd/kube-proxy (static)
k8s.io/kubernetes/cmd/kube-apiserver (static)
k8s.io/kubernetes/cmd/kube-controller-manager (static)
k8s.io/kubernetes/cmd/kubelet (non-static)
k8s.io/kubernetes/cmd/kubeadm (static)
k8s.io/kubernetes/cmd/kube-scheduler (static)
k8s.io/component-base/logs/kube-log-runner (static)
k8s.io/kube-aggregator (static)
k8s.io/apiextensions-apiserver (static)
k8s.io/kubernetes/cluster/gce/gci/mounter (static)
k8s.io/kubernetes/cmd/kubectl (static)
k8s.io/kubernetes/cmd/kubectl-convert (static)
github.com/onsi/ginkgo/v2/ginkgo (non-static)
k8s.io/kubernetes/test/e2e/e2e.test (test)
k8s.io/kubernetes/test/conformance/image/go-runner (non-static)
k8s.io/kubernetes/cmd/kubemark (static)
github.com/onsi/ginkgo/v2/ginkgo (non-static)
k8s.io/kubernetes/test/e2e_node/e2e_node.test (test)
test/e2e/e2e.go:35:2: cannot find package "k8s.io/api/apps/v1" in any of:
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/api/apps/v1 (vendor tree)
/home/thockin/src/kubernetes/_output/local/.gimme/versions/go1.22rc1.linux.amd64/src/k8s.io/api/apps/v1 (from $GOROOT)
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/api/apps/v1 (from $GOPATH)
... more ...
... more ...
... more ...
!!! [0106 12:13:41] Call tree:
!!! [0106 12:13:41] 1: /home/thockin/src/kubernetes/hack/lib/golang.sh:948 kube::golang::build_binaries_for_platform(...)
!!! [0106 12:13:41] 2: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0106 12:13:41] Call tree:
!!! [0106 12:13:41] 1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0106 12:13:41] Call tree:
!!! [0106 12:13:41] 1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
make: *** [Makefile:96: all] Error 1
```
Again, this requires go 1.22 (e.g. gotip), as go 1.21.x does not have
`go work vendor` support.
TO REPEAT:
( \
./hack/update-go-workspace.sh; \
./hack/update-vendor.sh; \
./hack/update-go-workspace.sh; \
)
The pull-kubernetes-verify job is using this to run the base
verify-golangci.sh only in the pull-kubernetes-verify-lint job.
Because the file name was not quite right, it still ran.
Wrapping errors may or may not be the right thing to do (see
https://go.dev/blog/go1.13-errors#whether-to-wrap and the discussion in
https://github.com/kubernetes/kubernetes/issues/123234). But developers should
at least think about it, so let's emit linter hints for it: the golangci-lint
config by default enables it for go-errorlint, just not the linter itself, so
we just need to add it for the "hints" config.
Direct error comparisons and assertions also get checked. Those are typically
something that should be replaced by errors.Is and errors.As, but as the
existing code often doesn't do that, let's also treat those as just hints.
Because /tmp/kubelet.log shows trailing log.
Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Signed-off-by: Masashi Honma <masashi.honma@gmail.com>
