Automatic merge from submit-queue
Switch ScheduledJob controller to use clientset
**What this PR does / why we need it**:
This is part of #25442. I've applied here the same fix I've applied in the manual client in #29187, see the 1st commit for that (@caesarxuchao we've talked about it in #29856).
@deads2k as promised
@janetkuo ptal
Automatic merge from submit-queue
Improvements on OpenAPI spec generation
- Generating models using go2idl library (no reflection anymore)
- Remove dependencies on go-restful/swagger
- Generate one swagger.json file for each web-service
- Bugfix: fixed a bug in trie implementation
Reference: #13414
**Release note**:
```release-note
Generate separate OpenAPI spec for each API GroupVersion on /<Group>/<Version>/swagger.json
```
- Generating models using go2idl library (no reflection anymore)
- Remove dependencies on go-restful/swagger
- Generate one swagger.json file for each web-service
- Bugfix: fixed a bug in trie implementation
Automatic merge from submit-queue
Avoid sorting lists when unnecessary
I've seen ThreadSafeMap::List consuming ~30% of whole CPU usage, spending the whole time in sorting (while it is in fact completely unneded).
Automatic merge from submit-queue
Cut the client repo, staging it in the main repo
Tracking issue: #28559
ref: https://github.com/kubernetes/kubernetes/pull/25978#issuecomment-232710174
This PR implements the plan a few of us came up with last week for cutting client into its own repo:
1. creating "_staging" (name is tentative) directory in the main repo, using a script to copy the client and its dependencies to this directory
2. periodically publishing the contents of this staging client to k8s.io/client-go repo
3. converting k8s components in the main repo to use the staged client. They should import the staged client as if the client were vendored. (i.e., the import line should be `import "k8s.io/client-go/<pacakge name>`). This requirement is to ease step 4.
4. In the future, removing the staging area, and vendoring the real client-go repo.
The advantage of having the staging area is that we can continuously run integration/e2e tests with the latest client repo and the latest main repo, without waiting for the client repo to be vendored back into the main repo. This staging area will exist until our test matrix is vendoring both the client and the server.
In the above plan, the tricky part is step 3. This PR achieves it by creating a symlink under ./vendor, pointing to the staging area, so packages in the main repo can refer to the client repo as if it's vendored. To prevent the godep tool from messing up the staging area, we export the staged client to GOPATH in hack/godep-save.sh so godep will think the client packages are local and won't attempt to manage ./vendor/k8s.io/client-go.
This is a POC. We'll rearrange the directory layout of the client before merge.
@thockin @lavalamp @bgrant0607 @kubernetes/sig-api-machinery
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.kubernetes.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.kubernetes.io/reviews/kubernetes/kubernetes/29147)
<!-- Reviewable:end -->
Automatic merge from submit-queue
Unable to have optional message slice
When making a message slice optional, we need to handle one more
conditional case in the rewriter.
@wojtek-t
Automatic merge from submit-queue
Generate a better Stringer method for proto types
This replaces the bad string output generated by golang/proto with gogo/protobuf stringer generation. Makes the output similar to %#v and more debuggable. We have to have a String() method to implement proto.Message, so this is strictly better.
@wojtek-t, @thockin for after your PR merges
Fixes#28756
Automatic merge from submit-queue
Allow proto tag to define field name
When we introduce a new field for backwards compatibility, we may want
to specify a different protobuf field name (one that matches JSON) than
the automatic transformation applied to the struct field. This allows an
API field to define the name of its protobuf tag.
@wojtek-t hit this while adding protobuf to OpenShift - we have fields we
keep for backward compatibility that need to be assigned a field by name.
Has no impact on existing (already tagged) fields.
This mostly takes the previously checked in files and removes them, and moves
the generation to be on-demand instead of manual. Manually verified no change
in generated output.
Automatic merge from submit-queue
Deepcopy: avoid struct copies and reflection Call
- make signature of generated deepcopy methods symmetric with `in *type, out *type`, avoiding copies of big structs on the stack
- switch to `in interface{}, out interface{}` which allows us to call them with without `reflect.Call`
The first change reduces runtime of BenchmarkPodCopy-4 from `> 3500ns` to around `2300ns`.
The second change reduces runtime to around `1900ns`.
When we introduce a new field for backwards compatibility, we may want
to specify a different protobuf field name (one that matches JSON) than
the automatic transformation applied to the struct field. This allows an
API field to define the name of its protobuf tag.
This fixes PodSpec to generate cleanly. No other types only half-generate (so
now we Fatalf), though several fail to generate at all (only Errorf for now).
There are ample opportunities to optimize and streamline here. For example,
there's no reason to have a function to convert IntStr to IntStr. Removing the
function does generate the right assignment, but it is unclear whether the
registered function is needed or not. I opted to leave it alone for now.
Another example is Convert_Slice_byte_To_Slice_byte, which just seems silly.
This drives conversion generation from file tags like:
// +conversion-gen=k8s.io/my/internal/version
.. rather than hardcoded lists of packages.
The only net change in generated code can be explained as correct. Previously
it didn't know that conversion was available.
This is used subsequently to simplify the conversion generation, so each
package can declare what peer-packages it uses, and have those imported
dynamically, rather than having one mega list of packages to import and not
really being clear why, for any given list item.
This is the last piece of Clayton's #26179 to be implemented with file tags.
All diffs are accounted for. Followup will use this to streamline some
packages.
Also add some V(5) debugging - it was helpful in diagnosing various issues, it
may be helpful again.
This drives most of the logic of deep-copy generation from tags like:
// +deepcopy-gen=package
..rather than hardcoded lists of packages. This will make it possible to
subsequently generate code ONLY for packages that need it *right now*, rather
than all of them always.
Also remove pkgs that really do not need deep-copies (no symbols used
anywhere).
Previously we just tracked comments on the 'package' declaration. Treat all
file comments as one comment-block, for simplicity. Can be revisited if
needed.
This means that tags like:
// +foo=bar
// +foo=bat
..will produce []string{"bar", "bat"}. This is needed for later commits which
will want to use this to make code generation more self contained.
In bringing back Clayton's PR piece-by-piece this was almost as easy to
implement as his version, and is much more like what I think we should be
doing.
Specifically, any time which defines a .DeepCopy() method will have that method
called preferentially. Otherwise we generate our own functions for
deep-copying. This affected exactly one type - resource.Quantity. In applying
this heuristic, several places in the generated code were simplified.
To achieve this I had to convert types.Type.Methods from a slice to a map,
which seems correct anyway (to do by-name lookups).
This re-institutes some of the rolled-back logic from previous commits. It
bounds the scope of what the deepcopy generator is willing to do with regards
to generating and calling generated functions.
His PR cam during the middle of this development cycle, and it was easier to
burn it down and recreate it than try to patch it into an existing series and
re-test every assumption. This behavior will be re-introduced in subsequent
commits.
Automatic merge from submit-queue
Cleanup third party (pt 2)
Move forked-and-hacked golang code to the forked/ directory. Remove ast/build/parse code that is now in stdlib. Remove unused shell2junit
Automatic merge from submit-queue
WIP - Handle map[]struct{} in DeepCopy
Deep copy was not properly handling the empty struct case we use for Sets.
@lavalamp I need your expertise when you have some time - the go2idl parser is turning sets.String into the following tree:
type: sets.String kind: Alias
underlying: map[string]sets.Empty kind: Map
key: string kind: Builtin
elem: set.Empty kind: Struct
^
should be Alias
Looking at tc.Named, I'm not sure what the expected outcome would be and why you flatten there.
While testing this fix in OpenShift it was discovered that the
PackageConstraint was overly aggressive - types that declare a public
copy function should always return true. PackageConstraint is intended
to limit packages where we might generate deep copy function, rather
than to prevent external packages from being consumed.
Downstream generators that want to reuse the upstream generated types
need to be able to define a different ignore tag (so that they can see
the already generated types).
Automatic merge from submit-queue
Track object modifications in fake clientset
Fake clientset is used by unit tests extensively but it has some
shortcomings:
- no filtering on namespace and name: tests that want to test objects in
multiple namespaces end up getting all objects from this clientset,
as it doesn't perform any filtering based on name and namespace;
- updates and deletes don't modify the clientset state, so some tests
can get unexpected results if they modify/delete objects using the
clientset;
- it's possible to insert multiple objects with the same
kind/name/namespace, this leads to confusing behavior, as retrieval is
based on the insertion order, but anchors on the last added object as
long as no more objects are added.
This change changes core.ObjectRetriever implementation to track object
adds, updates and deletes.
Some unit tests were depending on the previous (and somewhat incorrect)
behavior. These are fixed in the following few commits.
Automatic merge from submit-queue
TLS bootstrap API group (alpha)
This PR only covers the new types and related client/storage code- the vast majority of the line count is codegen. The implementation differs slightly from the current proposal document based on discussions in design thread (#20439). The controller logic and kubelet support mentioned in the proposal are forthcoming in separate requests.
I submit that #18762 ("Creating a new API group is really hard") is, if anything, understating it. I've tried to structure the commits to illustrate the process.
@mikedanese @erictune @smarterclayton @deads2k
```release-note-experimental
An alpha implementation of the the TLS bootstrap API described in docs/proposals/kubelet-tls-bootstrap.md.
```
[]()
Fake clientset is used by unit tests extensively but it has some
shortcomings:
- no filtering on namespace and name: tests that want to test objects in
multiple namespaces end up getting all objects from this clientset,
as it doesn't perform any filtering based on name and namespace;
- updates and deletes don't modify the clientset state, so some tests
can get unexpected results if they modify/delete objects using the
clientset;
- it's possible to insert multiple objects with the same
kind/name/namespace, this leads to confusing behavior, as retrieval is
based on the insertion order, but anchors on the last added object as
long as no more objects are added.
This change changes core.ObjectRetriever implementation to track object
adds, updates and deletes.
Some unit tests were depending on the previous (and somewhat incorrect)
behavior. These are fixed in the following few commits.
Specifying // +protobuf.nullable=true on a Go type that is an alias of a
map or slice will generate a synthetic protobuf message with the type
name that will serialize to the wire in a way that allows the difference
between empty and nil to be recorded.
For instance:
// +protobuf.nullable=true
types OptionalMap map[string]string
will create the following message:
message OptionalMap {
map<string, string> Items = 1
}
and generate marshallers that use the presence of OptionalMap to
determine whether the map is nil (rather than Items, which protobuf
provides no way to delineate between empty and nil).
