Commit Graph

20307 Commits

Author SHA1 Message Date
Tim Hockin
f772410082 Make code-gen subprojects work on gengo/v2 2024-02-29 22:06:38 -08:00
Tim Hockin
b0a70dec4a Fix go-to-protobuf wrt gengo/v2
There's some very fishy-smelling logic in here, but this commit is
trying to be as focused as possible.

The *.pb.go diffs are the "name" encoded in the descriptor.  The
descriptor blobs can be decoded by this program (thanks StackOverflow!):

```
package main

import (
	"bytes"
	"compress/gzip"
	"encoding/json"
	"fmt"
	"os"

	"io/ioutil"

	proto "github.com/golang/protobuf/proto"
	dpb "github.com/golang/protobuf/protoc-gen-go/descriptor"
)

func main() {
	m := map[string][]byte{
		"before": blobv1,
		"after":  blobv2,
	}
	arg := os.Args[1]
	dump(m[arg])
}

func dump(bytes []byte) {
	fd, err := decodeFileDesc(bytes)
	if err != nil {
		panic(err)
	}
	b, err := json.MarshalIndent(fd, "", "  ")
	if err != nil {
		panic(err)
	}
	fmt.Println(string(b))
}

// decompress does gzip decompression.
func decompress(b []byte) ([]byte, error) {
	r, err := gzip.NewReader(bytes.NewReader(b))
	if err != nil {
		return nil, fmt.Errorf("bad gzipped descriptor: %v", err)
	}
	out, err := ioutil.ReadAll(r)
	if err != nil {
		return nil, fmt.Errorf("bad gzipped descriptor: %v", err)
	}
	return out, nil
}

func decodeFileDesc(enc []byte) (*dpb.FileDescriptorProto, error) {
	raw, err := decompress(enc)
	if err != nil {
		return nil, fmt.Errorf("failed to decompress enc: %v", err)
	}

	fd := new(dpb.FileDescriptorProto)
	if err := proto.Unmarshal(raw, fd); err != nil {
		return nil, fmt.Errorf("bad descriptor: %v", err)
	}
	return fd, nil
}

var blobv1 = []byte{
	// insert proto "before" blob here
}

var blobv2 = []byte{
	// insert proto "after" blob here
}
```

Running this with "before" and "after" args, and diffing the output
yields something like:

```diff
--- /tmp/a	2023-12-23 23:57:04.748090836 -0800
+++ /tmp/b	2023-12-23 23:57:11.000040973 -0800
@@ -1,5 +1,5 @@
 {
-  "name": "k8s.io/kubernetes/vendor/k8s.io/api/admission/v1/generated.proto",
+  "name": "k8s.io/api/admission/v1/generated.proto",
   "package": "k8s.io.api.admission.v1",
   "dependency": [
     "github.com/gogo/protobuf/gogoproto/gogo.proto",
```
2024-02-29 22:06:37 -08:00
Tim Hockin
b961b39efa Fix client-gen wrt gengo/v2 2024-02-29 22:06:35 -08:00
Tim Hockin
5ddd6772a4 Fix applyconfiguration-gen wrt gengo/v2 2024-02-29 22:06:33 -08:00
Tim Hockin
0aa98ee8e3 Fix informer-gen wrt gengo/v2 2024-02-29 22:06:31 -08:00
Tim Hockin
b82527b6fb Fix lister-gen wrt gengo/v2 2024-02-29 22:06:30 -08:00
Tim Hockin
5475797f43 Make update-codegen conversion work on gengo/v2 2024-02-29 22:06:28 -08:00
Tim Hockin
ed35118664 Fix conversion-gen wrt gengo/v2 2024-02-29 22:06:27 -08:00
Tim Hockin
fd2ca0a344 Fix prerelease-lifecycle-gen wrt gengo/v2 2024-02-29 22:06:25 -08:00
Tim Hockin
514dd0e1f6 Convert all gengo references to gengo/v2
TO REPEAT:

    git ls-files ':!:vendor/k8s.io/gengo' ':!:gengo-v2' ':(glob)**/*.go' \
        | xargs grep -l 'k8s.io/gengo/' \
        | while read F; do \
            sed -i 's|k8s.io/gengo/\([^v]\)|k8s.io/gengo/v2/\1|' $F
        done
2024-02-29 22:06:20 -08:00
Tim Hockin
6f2f3735e0 Re-vendor latest kube-openapi and gengo/v2
./hack/pin-dependency.sh k8s.io/kube-openapi latest
./hack/pin-dependency.sh k8s.io/gengo/v2 latest
./hack/update-vendor.sh
2024-02-29 22:06:19 -08:00
Tim Hockin
b3dd724174 Remove dep on set-gen: gone in gengo/v2 2024-02-29 22:06:18 -08:00
Tim Hockin
7a7dff1eeb Change verify-imports to use new setup_env
Also clean up the tool a bit and drop vendor as an explicit thing to
be handled.
2024-02-29 22:06:16 -08:00
Patrick Ohly
65b841c077 Generate go.work files
This creates go.work and enables Go Workspaces.  This is a file that
includes info on k/k and all the staging modules.

This depends on go 1.22 and setting FORCE_HOST_GO=true (for kube
scripts, which try to be hermetic).

Make this part of the normal update/verify sequence.

The top-level go.work file contains no replace statements. Instead, the
replace statements in the individual go.mod files are used. For this to
work, replace statements in the individual go.mod files have to be
consistent.

hack/tools has different dependencies and can't be in the main
workspace, so this adds a go.work just for that.  Without this, go tries
to consider all deps in all modules and pick one that works for all.
This is problematic because there are so many of them that it is
difficult to manage.

Likewise for k8s.io/code-generator/examples and
k8s.io/kms/internal/plugins/_mock - add trivial go.work files.

For example k/k depends on an older version of a lib that gloangci-lint
needs (transitively) and it breaks.

This also updates vendor (needed to make go happy), and removes
vendor'ed symlinks.  This breaks a LOT of our build tools, which will be
fixed subsequently.

Result: `go` commands work across modules:

Before:
```
$ go list ./pkg/proxy/iptables/ ./staging/src/k8s.io/api/core/v1/
main module (k8s.io/kubernetes) does not contain package k8s.io/kubernetes/staging/src/k8s.io/api/core/v1

$ go build ./pkg/proxy/iptables/ ./staging/src/k8s.io/api
main module (k8s.io/kubernetes) does not contain package k8s.io/kubernetes/staging/src/k8s.io/api

$ go test ./pkg/proxy/iptables/ ./staging/src/k8s.io/api
main module (k8s.io/kubernetes) does not contain package k8s.io/kubernetes/staging/src/k8s.io/api
```

After:
```
$ go list ./pkg/proxy/iptables/ ./staging/src/k8s.io/api/core/v1/
k8s.io/kubernetes/pkg/proxy/iptables
k8s.io/api/core/v1

$ go build ./pkg/proxy/iptables/ ./staging/src/k8s.io/api

$ go test ./pkg/proxy/iptables/ ./staging/src/k8s.io/api
ok  	k8s.io/kubernetes/pkg/proxy/iptables	0.360s
ok  	k8s.io/api	2.302s
```

Result: `make` fails:

```
$ make
go version go1.22rc1 linux/amd64
+++ [0106 12:11:03] Building go targets for linux/amd64
    k8s.io/kubernetes/cmd/kube-proxy (static)
    k8s.io/kubernetes/cmd/kube-apiserver (static)
    k8s.io/kubernetes/cmd/kube-controller-manager (static)
    k8s.io/kubernetes/cmd/kubelet (non-static)
    k8s.io/kubernetes/cmd/kubeadm (static)
    k8s.io/kubernetes/cmd/kube-scheduler (static)
    k8s.io/component-base/logs/kube-log-runner (static)
    k8s.io/kube-aggregator (static)
    k8s.io/apiextensions-apiserver (static)
    k8s.io/kubernetes/cluster/gce/gci/mounter (static)
    k8s.io/kubernetes/cmd/kubectl (static)
    k8s.io/kubernetes/cmd/kubectl-convert (static)
    github.com/onsi/ginkgo/v2/ginkgo (non-static)
    k8s.io/kubernetes/test/e2e/e2e.test (test)
    k8s.io/kubernetes/test/conformance/image/go-runner (non-static)
    k8s.io/kubernetes/cmd/kubemark (static)
    github.com/onsi/ginkgo/v2/ginkgo (non-static)
    k8s.io/kubernetes/test/e2e_node/e2e_node.test (test)
test/e2e/e2e.go:35:2: cannot find package "k8s.io/api/apps/v1" in any of:
	/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/api/apps/v1 (vendor tree)
	/home/thockin/src/kubernetes/_output/local/.gimme/versions/go1.22rc1.linux.amd64/src/k8s.io/api/apps/v1 (from $GOROOT)
	/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/api/apps/v1 (from $GOPATH)
	... more ...
	... more ...
	... more ...
!!! [0106 12:13:41] Call tree:
!!! [0106 12:13:41]  1: /home/thockin/src/kubernetes/hack/lib/golang.sh:948 kube::golang::build_binaries_for_platform(...)
!!! [0106 12:13:41]  2: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0106 12:13:41] Call tree:
!!! [0106 12:13:41]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0106 12:13:41] Call tree:
!!! [0106 12:13:41]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
make: *** [Makefile:96: all] Error 1
```

Again, this requires go 1.22 (e.g. gotip), as go 1.21.x does not have
`go work vendor` support.

TO REPEAT:
    ( \
      ./hack/update-go-workspace.sh; \
      ./hack/update-vendor.sh; \
      ./hack/update-go-workspace.sh; \
    )
2024-02-29 00:22:06 -08:00
Kubernetes Prow Robot
a67973a45c Merge pull request #122717 from jpbetz/crd-object-filters
KEP-4358: Custom Resource Field Selectors
2024-02-28 20:07:12 -08:00
Joe Betz
a0dc949ce3 Improve error handling for json path results 2024-02-28 16:22:34 -05:00
Joe Betz
61d6026d23 Fix field paths 2024-02-28 15:49:44 -05:00
Kubernetes Prow Robot
fe8a12d264 Merge pull request #123562 from jpbetz/bump-cel-go-0_17_8
Bump cel-go to v0.17.8 to pick up CEL estimated cost fix
2024-02-28 12:45:41 -08:00
Joe Betz
45742d1d9a Add integration tests 2024-02-28 14:46:47 -05:00
Joe Betz
31f7efab20 Add cost stability tests for chained and nested CEL comprehensions 2024-02-28 14:20:16 -05:00
Joe Betz
7ef3eea8a7 Generate code 2024-02-28 14:06:46 -05:00
Joe Betz
291703482d Add selectableFields to CRDs 2024-02-28 14:06:46 -05:00
Kubernetes Prow Robot
e21a2f5d4f Merge pull request #123542 from liggitt/websocket-round-tripper-protocol
Use the websocket protocol header, verify selected protocol
2024-02-28 11:01:44 -08:00
Kubernetes Prow Robot
68a47053d1 Merge pull request #123508 from saschagrunert/image-id-container
Add `image_id` to CRI `Container` message
2024-02-28 11:01:35 -08:00
Joe Betz
d49949b642 Bump cel-go to v0.17.8 to pick up CEL estimated cost fix 2024-02-28 10:52:36 -05:00
Wojciech Tyczyński
92bdc7b387 Ensure that initial events are sorted for WatchList 2024-02-28 12:50:53 +01:00
Kubernetes Prow Robot
d2b4928669 Merge pull request #122830 from p0lyn0mial/upstream-watch-cache-wati-for-bk-after-rv
storage/cacher: ensure the cache is at the Most Recent ResourceVersion when streaming was requested
2024-02-28 01:43:37 -08:00
Sascha Grunert
e663285ccf Add image_id to CRI Container message
This new field allows fixing the kubelet image garbage collection in
container runtimes. The `image_ref` has been historically used by
container runtimes to reference images by digest.

Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
2024-02-28 10:05:07 +01:00
Kubernetes Prow Robot
f7ca532472 Merge pull request #123281 from seans3/remote-command-websocket-beta
RemoteCommand over WebSockets to Beta
2024-02-27 21:01:55 -08:00
Jordan Liggitt
b394aac4ce Use the websocket protocol header, verify selected protocol 2024-02-27 22:47:19 -05:00
Kubernetes Prow Robot
286cdad32d Merge pull request #123538 from jiahuif-forks/fix/cel/mutation-library-map-support
CEL mutation library: add support for map
2024-02-27 18:46:10 -08:00
Kubernetes Prow Robot
872c621bb7 Merge pull request #123475 from alexzielenski/apiserver/apiextensions/cel-error-fieldpath
bugfix: incorrect fieldpath when using multiple crd validation rules
2024-02-27 18:45:57 -08:00
Kubernetes Prow Robot
72c92e975d Merge pull request #123411 from hunshcn/fix/volume-link
fix persistent volume docs link
2024-02-27 16:47:27 -08:00
Sean Sullivan
03812ddb16 streamtranslator counter metric by status code 2024-02-27 23:21:55 +00:00
Monis Khan
e89dddd4af jwt: strictly support compact serialization only
Signed-off-by: Monis Khan <mok@microsoft.com>
2024-02-27 17:16:08 -05:00
Jiahui Feng
dc4c92f5a5 add support for map
to CEL mutation library.
2024-02-27 13:55:08 -08:00
Lukasz Szaszkiewicz
ecaf2093f5 storage/watch_cache: rework getAllEventsSinceLocked 2024-02-27 14:26:46 +01:00
Lukasz Szaszkiewicz
d629d3fa35 storage/cacher: add TestGetWatchCacheResourceVersion, TestGetBookmarkAfterResourceVersionLockedFunc 2024-02-27 14:26:41 +01:00
Lukasz Szaszkiewicz
f90bcf649e storage/cacher: ensure the cache is at the Most Recent ResourceVersion when streaming was requested 2024-02-27 14:26:36 +01:00
Sean Sullivan
a147693deb remote command turn on feature gates 2024-02-27 02:05:24 +00:00
Kubernetes Prow Robot
0a1cb2a4a4 Merge pull request #122589 from pohly/log-contextual-beta
promote contextual logging to beta, enabled by default
2024-02-26 13:56:13 -08:00
Alexander Zielenski
aefaf66d2b remove unused service keys from aggregated discovery 2024-02-26 11:56:07 -05:00
Kubernetes Prow Robot
d171e6733a Merge pull request #123503 from aojea/ccm_ci
Ccm ci
2024-02-26 08:02:58 -08:00
Kubernetes Prow Robot
98bd90fbe2 Merge pull request #114672 from pohly/log-text-split-streams
log: split streams also for text output
2024-02-26 01:44:58 -08:00
Antonio Ojea
3636971c39 add aojea as reviewer
Change-Id: Ie69e01542f9c8420aa0d0275378acd838e77eca9
2024-02-26 08:56:23 +00:00
Kubernetes Prow Robot
689dca08e1 Merge pull request #123479 from Jefftree/sample-apiserver-129
Fix sample-apiserver rbac and bump to 1.29
2024-02-25 16:41:38 -08:00
Kubernetes Prow Robot
4b3da38651 Merge pull request #123417 from soltysh/kube_codegen-applyexternal
kube_codegen: expose external-applyconfigurations flag for client generation
2024-02-25 16:41:31 -08:00
Kubernetes Prow Robot
2a44cb0ab3 Merge pull request #122569 from IvoGoman/wrap-nomatcherror
feat(cli-runtime): wrap meta.NoKindMatchErrors
2024-02-24 10:01:33 -08:00
Jefftree
b3216214f9 Fix sample-apiserver rbac for v1.27 and bump e2e 2024-02-23 23:53:54 -05:00
Alexander Zielenski
247233f7b4 bugfix: modifying argument causes wrong fieldpath 2024-02-23 15:49:31 -08:00