We should expose the SkipUnsafe option, for legacy compatability, so
that conversion-go can be used in other projects, and for platforms
where unsafe is not available.
Make unsafe code generation the default though, and have the help text
hint that the resulting code is sub-optimal.
Automatic merge from submit-queue (batch tested with PRs 35300, 36709, 37643, 37813, 37697)
Add generated informers
Add informer-gen and the informers it generates. We'll do follow-up PRs to convert everything currently using the hand-written informers to the generated ones.
TODO:
- [x] switch to `GroupVersionResource`
- [x] finish godoc
@deads2k @caesarxuchao @sttts @liggitt
Automatic merge from submit-queue
Add tooling to generate listers
Add lister-gen tool to auto-generate listers. So far this PR only demonstrates replacing the manually-written `StoreToLimitRangeLister` with the generated `LimitRangeLister`, as it's a small and easy swap.
cc @deads2k @liggitt @sttts @nikhiljindal @lavalamp @smarterclayton @derekwaynecarr @kubernetes/sig-api-machinery @kubernetes/rh-cluster-infra
Given an object that wishes to have a top level defaulter, traverse the
object looking for nested fields that have defaulters and generate a
single function for that type that invokes all defaulters. The function
will have the name `SetObjectDefaults_NAME`.
Types use `// +k8s:defaulter-gen=true` to indicate they wish a defaulter
generated. If a function already exists with the desired name
`SetObjectDefaults_NAME` then no generation will occur. At a package
level, authors can bulk select the types to generate by setting the
value of the comment to the name of a field - all objects with that
field name without `// +k8s:defaulter-gen=false` defined on the type
will get a defaulter.
Because the defaulting behavior from conversions happens recursively,
all defaulters are expected to be invoked. We call these defaulters
"non-covering" (other defaulters may be invoked beneath them). The
defaulters we generate, by comparison, are "covering" - no nested
defaulters should be invoked. To distinguish between these two types, we
introduce the `// +k8s:defaulter-gen=covers` comment on a defaulter
function which will instruct the generator that the function should
terminate recursion.
This sets the stage for future defaulter generation from comments by
subsuming our existing generators
Automatic merge from submit-queue
Speed up dockerized builds
This PR speeds up dockerized builds. First, we make sure that we are as incremental as possible. The bigger change is that now we use rsync to move sources into the container and get data back out.
To do yet:
* [x] Add a random password to rsync. This is 128bit MD4, but it is better than nothing.
* [x] Lock down rsync to only come from the host.
* [x] Deal with remote docker engines -- this should be necessary for docker-machine on the mac.
* [x] Allow users to specify the port for the rsync daemon. Perhaps randomize this or let docker pick an ephemeral port and detect the port?
* [x] Copy back generated files so that users can check them in. This is done for `zz_generated.*` files generated by `make generated_files`
* [x] This should include generated proto files so that we can remove the hack-o-rama that is `hack/hack/update-*-dockerized.sh`
* [x] Start "versioning" the build container and the data container so that the CI system doesn't have to be manually kicked.
* [x] Get some benchmarks to qualify how much faster.
This replaces #28518 and is related to #30600.
cc @thockin @spxtr @david-mcmahon @MHBauer
Benchmarks by running `make clean ; sync ; time bash -xc 'time build/make-build-image.sh ; time sync ; time build/run.sh make ; time sync; time build/run.sh make'` on a GCE n1-standard-8 with PD-SSD.
| setup | build image | sync | first build | sync | second build | total |
|-------|-------------|----- |----------|------|--------------|------|
| baseline | 0m11.420s | 0m0.812s | 7m2.353s | 0m42.380s | 7m8.381s | 15m5.348s |
| this pr | 0m10.977s | 0m15.168s | 7m31.096s | 1m55.692s | 0m16.514s | 10m9.449s |
make-generated-{protobuf,runtime}.sh was doing some really nasty stuff with how
the build container was managed in order to copy results out. Since we have
more flexibility to grab results out of the build container, we can now avoid
all of this. Ideally we wouldn't have `hack` calling `build` at all, but we
aren't there yet.
Automatic merge from submit-queue
Autoconvert cleanups
A few accumulated cleanups to conversion, and then a big change. Please review commit-by-commit. Verified by nuking all generated files and comparing the new results.
cc @lavalamp
Previously we refused to emit 'autoConvert_*' functions if any field was not
convertible. The way around this was to write manual Conversion functions, but
to do so safely you must handle every fields. Huge opportunity for errors.
This PR cleans up the filtering such that it only operates on types that should
be converted (remove a lot of code) and tracks when fields are skipped. In
that case, it emits an 'autoConvert' function but not a public 'Convert'
function. If there is no manual function, the compile will fail.
This also means that manual conversion functions can call autoConvert functions
and then "patch up" what they need.
rename plugin/pkg/client/auth/plugins.go package name to auth
add the plugin import line in client-gen
update import_known_versions for release_1_5 clientset
change copy.sh
Automatic merge from submit-queue
Switch ScheduledJob controller to use clientset
**What this PR does / why we need it**:
This is part of #25442. I've applied here the same fix I've applied in the manual client in #29187, see the 1st commit for that (@caesarxuchao we've talked about it in #29856).
@deads2k as promised
@janetkuo ptal
Automatic merge from submit-queue
Improvements on OpenAPI spec generation
- Generating models using go2idl library (no reflection anymore)
- Remove dependencies on go-restful/swagger
- Generate one swagger.json file for each web-service
- Bugfix: fixed a bug in trie implementation
Reference: #13414
**Release note**:
```release-note
Generate separate OpenAPI spec for each API GroupVersion on /<Group>/<Version>/swagger.json
```
- Generating models using go2idl library (no reflection anymore)
- Remove dependencies on go-restful/swagger
- Generate one swagger.json file for each web-service
- Bugfix: fixed a bug in trie implementation
Automatic merge from submit-queue
Avoid sorting lists when unnecessary
I've seen ThreadSafeMap::List consuming ~30% of whole CPU usage, spending the whole time in sorting (while it is in fact completely unneded).
Automatic merge from submit-queue
Cut the client repo, staging it in the main repo
Tracking issue: #28559
ref: https://github.com/kubernetes/kubernetes/pull/25978#issuecomment-232710174
This PR implements the plan a few of us came up with last week for cutting client into its own repo:
1. creating "_staging" (name is tentative) directory in the main repo, using a script to copy the client and its dependencies to this directory
2. periodically publishing the contents of this staging client to k8s.io/client-go repo
3. converting k8s components in the main repo to use the staged client. They should import the staged client as if the client were vendored. (i.e., the import line should be `import "k8s.io/client-go/<pacakge name>`). This requirement is to ease step 4.
4. In the future, removing the staging area, and vendoring the real client-go repo.
The advantage of having the staging area is that we can continuously run integration/e2e tests with the latest client repo and the latest main repo, without waiting for the client repo to be vendored back into the main repo. This staging area will exist until our test matrix is vendoring both the client and the server.
In the above plan, the tricky part is step 3. This PR achieves it by creating a symlink under ./vendor, pointing to the staging area, so packages in the main repo can refer to the client repo as if it's vendored. To prevent the godep tool from messing up the staging area, we export the staged client to GOPATH in hack/godep-save.sh so godep will think the client packages are local and won't attempt to manage ./vendor/k8s.io/client-go.
This is a POC. We'll rearrange the directory layout of the client before merge.
@thockin @lavalamp @bgrant0607 @kubernetes/sig-api-machinery
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.kubernetes.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.kubernetes.io/reviews/kubernetes/kubernetes/29147)
<!-- Reviewable:end -->
Automatic merge from submit-queue
Unable to have optional message slice
When making a message slice optional, we need to handle one more
conditional case in the rewriter.
@wojtek-t
Automatic merge from submit-queue
Generate a better Stringer method for proto types
This replaces the bad string output generated by golang/proto with gogo/protobuf stringer generation. Makes the output similar to %#v and more debuggable. We have to have a String() method to implement proto.Message, so this is strictly better.
@wojtek-t, @thockin for after your PR merges
Fixes#28756
Automatic merge from submit-queue
Allow proto tag to define field name
When we introduce a new field for backwards compatibility, we may want
to specify a different protobuf field name (one that matches JSON) than
the automatic transformation applied to the struct field. This allows an
API field to define the name of its protobuf tag.
@wojtek-t hit this while adding protobuf to OpenShift - we have fields we
keep for backward compatibility that need to be assigned a field by name.
Has no impact on existing (already tagged) fields.
This mostly takes the previously checked in files and removes them, and moves
the generation to be on-demand instead of manual. Manually verified no change
in generated output.
Automatic merge from submit-queue
Deepcopy: avoid struct copies and reflection Call
- make signature of generated deepcopy methods symmetric with `in *type, out *type`, avoiding copies of big structs on the stack
- switch to `in interface{}, out interface{}` which allows us to call them with without `reflect.Call`
The first change reduces runtime of BenchmarkPodCopy-4 from `> 3500ns` to around `2300ns`.
The second change reduces runtime to around `1900ns`.
When we introduce a new field for backwards compatibility, we may want
to specify a different protobuf field name (one that matches JSON) than
the automatic transformation applied to the struct field. This allows an
API field to define the name of its protobuf tag.
This fixes PodSpec to generate cleanly. No other types only half-generate (so
now we Fatalf), though several fail to generate at all (only Errorf for now).
There are ample opportunities to optimize and streamline here. For example,
there's no reason to have a function to convert IntStr to IntStr. Removing the
function does generate the right assignment, but it is unclear whether the
registered function is needed or not. I opted to leave it alone for now.
Another example is Convert_Slice_byte_To_Slice_byte, which just seems silly.
This drives conversion generation from file tags like:
// +conversion-gen=k8s.io/my/internal/version
.. rather than hardcoded lists of packages.
The only net change in generated code can be explained as correct. Previously
it didn't know that conversion was available.
This is used subsequently to simplify the conversion generation, so each
package can declare what peer-packages it uses, and have those imported
dynamically, rather than having one mega list of packages to import and not
really being clear why, for any given list item.
This is the last piece of Clayton's #26179 to be implemented with file tags.
All diffs are accounted for. Followup will use this to streamline some
packages.
Also add some V(5) debugging - it was helpful in diagnosing various issues, it
may be helpful again.
This drives most of the logic of deep-copy generation from tags like:
// +deepcopy-gen=package
..rather than hardcoded lists of packages. This will make it possible to
subsequently generate code ONLY for packages that need it *right now*, rather
than all of them always.
Also remove pkgs that really do not need deep-copies (no symbols used
anywhere).
Previously we just tracked comments on the 'package' declaration. Treat all
file comments as one comment-block, for simplicity. Can be revisited if
needed.
This means that tags like:
// +foo=bar
// +foo=bat
..will produce []string{"bar", "bat"}. This is needed for later commits which
will want to use this to make code generation more self contained.
In bringing back Clayton's PR piece-by-piece this was almost as easy to
implement as his version, and is much more like what I think we should be
doing.
Specifically, any time which defines a .DeepCopy() method will have that method
called preferentially. Otherwise we generate our own functions for
deep-copying. This affected exactly one type - resource.Quantity. In applying
this heuristic, several places in the generated code were simplified.
To achieve this I had to convert types.Type.Methods from a slice to a map,
which seems correct anyway (to do by-name lookups).
This re-institutes some of the rolled-back logic from previous commits. It
bounds the scope of what the deepcopy generator is willing to do with regards
to generating and calling generated functions.
