Commit Graph

20337 Commits

Author SHA1 Message Date
Tim Hockin
86d2ecbf9a Move gengo/examples/defaulter -> code_generator
This includes tests, which all pass.  Also fixes a bunch of lint.
2024-02-29 22:07:25 -08:00
Tim Hockin
cb7f0593ae Move gengo/examples/deepcopy -> code_generator
Merge comments in main.go and fix a bunch of pre-existing lint errors.
2024-02-29 22:07:24 -08:00
Tim Hockin
01a1865934 Fix lint in code-generator/cmd/...
Later commits touch code around here, which triggers verify.  This
commit JUST fixes lint.
2024-02-29 22:07:24 -08:00
Tim Hockin
b68340e5b6 Get rid of gengo InputDirs flag 2024-02-29 22:07:23 -08:00
Tim Hockin
299b045307 codegen: Remove most uses of InputDirs
This clears the path to getting rid of it completely.
2024-02-29 22:07:22 -08:00
Tim Hockin
e78dc86288 Move import-boss to k/k, retool to not use gengo
* Moved code and tests out of gengo -> code_generator
* Reworked it to use packages.Load
* Reworked tests (still not comprehensive but pretty good?)
* Dropped test support from gengo (support for tests in
  x/tools/go/packages is pretty hostile to gengo, and nobody used it)
2024-02-29 22:07:21 -08:00
Tim Hockin
08ce6a0f14 Get rid of gengo WithoutDefaultFlagParsing 2024-02-29 22:07:20 -08:00
Tim Hockin
5b2d4384c7 codegen: Make import-boss do it's own flags 2024-02-29 22:07:19 -08:00
Tim Hockin
1fed36b984 codegen: Minor: fix wrong comments 2024-02-29 22:07:18 -08:00
Tim Hockin
6420e64662 Move go-header-file flag to each tool
Also remove the build-tag flag (unused, untested).

This changes some output comments (less redundant).
2024-02-29 22:07:17 -08:00
Tim Hockin
3a77592b2c Remove gengo verify-only - unused and untested
All of our own tools emit into a tmp tree and then diff that.
2024-02-29 22:07:17 -08:00
Tim Hockin
d0dd72b01e Move OutputFileBase flag to each tool 2024-02-29 22:07:16 -08:00
Tim Hockin
26a1a13b80 Rename output-package -> output-pkg
To parallel "output-dir".
2024-02-29 22:07:15 -08:00
Tim Hockin
6a375b8f4c Move the OutputBase flag to each tool
...and rename to --output-dir
2024-02-29 22:07:14 -08:00
Tim Hockin
875566f997 Type-assert gengo CustomArgs blindly
If it does fail, it is catastophic.  Don't bother trying to handle it.
2024-02-29 22:07:13 -08:00
Tim Hockin
0ff7dd1943 Rename gengo DefaultGen -> GolangGenerator 2024-02-29 22:07:12 -08:00
Tim Hockin
2ac119bea9 Rename gengo Target.SourcePath() -> Dir() 2024-02-29 22:07:11 -08:00
Tim Hockin
fcd6c19c24 Rename gengo generator.Package -> Target 2024-02-29 22:07:10 -08:00
Tim Hockin
d2c7401910 Drop generator.Packages type 2024-02-29 22:07:10 -08:00
Tim Hockin
b2832697d5 Rename DefaultPackage to SimplePackage 2024-02-29 22:07:09 -08:00
Tim Hockin
4627a15ac0 Get rid of vendor baggage in go-to-proto 2024-02-29 22:07:07 -08:00
Tim Hockin
dd4e456875 codegen:proto: Use revised API to gengo parser 2024-02-29 22:07:03 -08:00
Tim Hockin
4b55936ab0 Remove defunct trim-path-prefix 2024-02-29 22:07:02 -08:00
Tim Hockin
2348e94908 Conversion: Better logging 2024-02-29 22:07:01 -08:00
Tim Hockin
e84adf611f Remove references to "GO111MODULE" 2024-02-29 22:06:55 -08:00
Tim Hockin
d772f7719d Remove defunct references to "vendor" 2024-02-29 22:06:52 -08:00
Tim Hockin
7f5982e708 code-generator: Remove now unused helper function 2024-02-29 22:06:51 -08:00
Tim Hockin
10c32b3e2f Get rid of most references to GOPATH 2024-02-29 22:06:51 -08:00
Tim Hockin
91efe7e571 Fix import-boss wrt gengo/v2 2024-02-29 22:06:40 -08:00
Tim Hockin
c8b61bb98b Fix register-gen wrt gengo/v2 2024-02-29 22:06:39 -08:00
Tim Hockin
f772410082 Make code-gen subprojects work on gengo/v2 2024-02-29 22:06:38 -08:00
Tim Hockin
b0a70dec4a Fix go-to-protobuf wrt gengo/v2
There's some very fishy-smelling logic in here, but this commit is
trying to be as focused as possible.

The *.pb.go diffs are the "name" encoded in the descriptor.  The
descriptor blobs can be decoded by this program (thanks StackOverflow!):

```
package main

import (
	"bytes"
	"compress/gzip"
	"encoding/json"
	"fmt"
	"os"

	"io/ioutil"

	proto "github.com/golang/protobuf/proto"
	dpb "github.com/golang/protobuf/protoc-gen-go/descriptor"
)

func main() {
	m := map[string][]byte{
		"before": blobv1,
		"after":  blobv2,
	}
	arg := os.Args[1]
	dump(m[arg])
}

func dump(bytes []byte) {
	fd, err := decodeFileDesc(bytes)
	if err != nil {
		panic(err)
	}
	b, err := json.MarshalIndent(fd, "", "  ")
	if err != nil {
		panic(err)
	}
	fmt.Println(string(b))
}

// decompress does gzip decompression.
func decompress(b []byte) ([]byte, error) {
	r, err := gzip.NewReader(bytes.NewReader(b))
	if err != nil {
		return nil, fmt.Errorf("bad gzipped descriptor: %v", err)
	}
	out, err := ioutil.ReadAll(r)
	if err != nil {
		return nil, fmt.Errorf("bad gzipped descriptor: %v", err)
	}
	return out, nil
}

func decodeFileDesc(enc []byte) (*dpb.FileDescriptorProto, error) {
	raw, err := decompress(enc)
	if err != nil {
		return nil, fmt.Errorf("failed to decompress enc: %v", err)
	}

	fd := new(dpb.FileDescriptorProto)
	if err := proto.Unmarshal(raw, fd); err != nil {
		return nil, fmt.Errorf("bad descriptor: %v", err)
	}
	return fd, nil
}

var blobv1 = []byte{
	// insert proto "before" blob here
}

var blobv2 = []byte{
	// insert proto "after" blob here
}
```

Running this with "before" and "after" args, and diffing the output
yields something like:

```diff
--- /tmp/a	2023-12-23 23:57:04.748090836 -0800
+++ /tmp/b	2023-12-23 23:57:11.000040973 -0800
@@ -1,5 +1,5 @@
 {
-  "name": "k8s.io/kubernetes/vendor/k8s.io/api/admission/v1/generated.proto",
+  "name": "k8s.io/api/admission/v1/generated.proto",
   "package": "k8s.io.api.admission.v1",
   "dependency": [
     "github.com/gogo/protobuf/gogoproto/gogo.proto",
```
2024-02-29 22:06:37 -08:00
Tim Hockin
b961b39efa Fix client-gen wrt gengo/v2 2024-02-29 22:06:35 -08:00
Tim Hockin
5ddd6772a4 Fix applyconfiguration-gen wrt gengo/v2 2024-02-29 22:06:33 -08:00
Tim Hockin
0aa98ee8e3 Fix informer-gen wrt gengo/v2 2024-02-29 22:06:31 -08:00
Tim Hockin
b82527b6fb Fix lister-gen wrt gengo/v2 2024-02-29 22:06:30 -08:00
Tim Hockin
5475797f43 Make update-codegen conversion work on gengo/v2 2024-02-29 22:06:28 -08:00
Tim Hockin
ed35118664 Fix conversion-gen wrt gengo/v2 2024-02-29 22:06:27 -08:00
Tim Hockin
fd2ca0a344 Fix prerelease-lifecycle-gen wrt gengo/v2 2024-02-29 22:06:25 -08:00
Tim Hockin
514dd0e1f6 Convert all gengo references to gengo/v2
TO REPEAT:

    git ls-files ':!:vendor/k8s.io/gengo' ':!:gengo-v2' ':(glob)**/*.go' \
        | xargs grep -l 'k8s.io/gengo/' \
        | while read F; do \
            sed -i 's|k8s.io/gengo/\([^v]\)|k8s.io/gengo/v2/\1|' $F
        done
2024-02-29 22:06:20 -08:00
Tim Hockin
6f2f3735e0 Re-vendor latest kube-openapi and gengo/v2
./hack/pin-dependency.sh k8s.io/kube-openapi latest
./hack/pin-dependency.sh k8s.io/gengo/v2 latest
./hack/update-vendor.sh
2024-02-29 22:06:19 -08:00
Tim Hockin
b3dd724174 Remove dep on set-gen: gone in gengo/v2 2024-02-29 22:06:18 -08:00
Tim Hockin
7a7dff1eeb Change verify-imports to use new setup_env
Also clean up the tool a bit and drop vendor as an explicit thing to
be handled.
2024-02-29 22:06:16 -08:00
Patrick Ohly
65b841c077 Generate go.work files
This creates go.work and enables Go Workspaces.  This is a file that
includes info on k/k and all the staging modules.

This depends on go 1.22 and setting FORCE_HOST_GO=true (for kube
scripts, which try to be hermetic).

Make this part of the normal update/verify sequence.

The top-level go.work file contains no replace statements. Instead, the
replace statements in the individual go.mod files are used. For this to
work, replace statements in the individual go.mod files have to be
consistent.

hack/tools has different dependencies and can't be in the main
workspace, so this adds a go.work just for that.  Without this, go tries
to consider all deps in all modules and pick one that works for all.
This is problematic because there are so many of them that it is
difficult to manage.

Likewise for k8s.io/code-generator/examples and
k8s.io/kms/internal/plugins/_mock - add trivial go.work files.

For example k/k depends on an older version of a lib that gloangci-lint
needs (transitively) and it breaks.

This also updates vendor (needed to make go happy), and removes
vendor'ed symlinks.  This breaks a LOT of our build tools, which will be
fixed subsequently.

Result: `go` commands work across modules:

Before:
```
$ go list ./pkg/proxy/iptables/ ./staging/src/k8s.io/api/core/v1/
main module (k8s.io/kubernetes) does not contain package k8s.io/kubernetes/staging/src/k8s.io/api/core/v1

$ go build ./pkg/proxy/iptables/ ./staging/src/k8s.io/api
main module (k8s.io/kubernetes) does not contain package k8s.io/kubernetes/staging/src/k8s.io/api

$ go test ./pkg/proxy/iptables/ ./staging/src/k8s.io/api
main module (k8s.io/kubernetes) does not contain package k8s.io/kubernetes/staging/src/k8s.io/api
```

After:
```
$ go list ./pkg/proxy/iptables/ ./staging/src/k8s.io/api/core/v1/
k8s.io/kubernetes/pkg/proxy/iptables
k8s.io/api/core/v1

$ go build ./pkg/proxy/iptables/ ./staging/src/k8s.io/api

$ go test ./pkg/proxy/iptables/ ./staging/src/k8s.io/api
ok  	k8s.io/kubernetes/pkg/proxy/iptables	0.360s
ok  	k8s.io/api	2.302s
```

Result: `make` fails:

```
$ make
go version go1.22rc1 linux/amd64
+++ [0106 12:11:03] Building go targets for linux/amd64
    k8s.io/kubernetes/cmd/kube-proxy (static)
    k8s.io/kubernetes/cmd/kube-apiserver (static)
    k8s.io/kubernetes/cmd/kube-controller-manager (static)
    k8s.io/kubernetes/cmd/kubelet (non-static)
    k8s.io/kubernetes/cmd/kubeadm (static)
    k8s.io/kubernetes/cmd/kube-scheduler (static)
    k8s.io/component-base/logs/kube-log-runner (static)
    k8s.io/kube-aggregator (static)
    k8s.io/apiextensions-apiserver (static)
    k8s.io/kubernetes/cluster/gce/gci/mounter (static)
    k8s.io/kubernetes/cmd/kubectl (static)
    k8s.io/kubernetes/cmd/kubectl-convert (static)
    github.com/onsi/ginkgo/v2/ginkgo (non-static)
    k8s.io/kubernetes/test/e2e/e2e.test (test)
    k8s.io/kubernetes/test/conformance/image/go-runner (non-static)
    k8s.io/kubernetes/cmd/kubemark (static)
    github.com/onsi/ginkgo/v2/ginkgo (non-static)
    k8s.io/kubernetes/test/e2e_node/e2e_node.test (test)
test/e2e/e2e.go:35:2: cannot find package "k8s.io/api/apps/v1" in any of:
	/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/api/apps/v1 (vendor tree)
	/home/thockin/src/kubernetes/_output/local/.gimme/versions/go1.22rc1.linux.amd64/src/k8s.io/api/apps/v1 (from $GOROOT)
	/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/api/apps/v1 (from $GOPATH)
	... more ...
	... more ...
	... more ...
!!! [0106 12:13:41] Call tree:
!!! [0106 12:13:41]  1: /home/thockin/src/kubernetes/hack/lib/golang.sh:948 kube::golang::build_binaries_for_platform(...)
!!! [0106 12:13:41]  2: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0106 12:13:41] Call tree:
!!! [0106 12:13:41]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0106 12:13:41] Call tree:
!!! [0106 12:13:41]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
make: *** [Makefile:96: all] Error 1
```

Again, this requires go 1.22 (e.g. gotip), as go 1.21.x does not have
`go work vendor` support.

TO REPEAT:
    ( \
      ./hack/update-go-workspace.sh; \
      ./hack/update-vendor.sh; \
      ./hack/update-go-workspace.sh; \
    )
2024-02-29 00:22:06 -08:00
Kubernetes Prow Robot
a67973a45c Merge pull request #122717 from jpbetz/crd-object-filters
KEP-4358: Custom Resource Field Selectors
2024-02-28 20:07:12 -08:00
Joe Betz
a0dc949ce3 Improve error handling for json path results 2024-02-28 16:22:34 -05:00
Joe Betz
61d6026d23 Fix field paths 2024-02-28 15:49:44 -05:00
Kubernetes Prow Robot
fe8a12d264 Merge pull request #123562 from jpbetz/bump-cel-go-0_17_8
Bump cel-go to v0.17.8 to pick up CEL estimated cost fix
2024-02-28 12:45:41 -08:00
Joe Betz
45742d1d9a Add integration tests 2024-02-28 14:46:47 -05:00
Joe Betz
31f7efab20 Add cost stability tests for chained and nested CEL comprehensions 2024-02-28 14:20:16 -05:00