4e20339916
Automatic merge from submit-queue Upgrade addon-manager with kubectl apply The first step of #33698. Use `kubectl apply` to replace addon-manager's previous logic. The most important issue this PR is targeting is the upgrade from 1.4 to 1.5. Procedure as below: 1. Precondition: After the master is upgraded, new addon-manager starts and all the old resources on nodes are running normally. 2. Annotate the old ReplicationController resources with kubectl.kubernetes.io/last-applied-configuration="" 3. Call `kubectl apply --prune=false` on addons folder to create new addons, including the new Deployments. 4. Wait for one minute for new addons to be spinned up. 5. Enter the periodical loop of `kubectl apply --prune=true`. The old RCs will be pruned at the first call. Procedure of a normal startup: 1. Addon-manager starts and no addon resources are running. 2. Annotate nothing. 3. Call `kubectl apply --prune=false` to create all new addons. 4. No need to explain the remain. Remained Issues: - Need to add `--type` flag to `kubectl apply --prune`, mentioned [here](https://github.com/kubernetes/kubernetes/pull/33075#discussion_r80814070). - This addon manager is not working properly with the current Deployment heapster, which runs [addon-resizer](https://github.com/kubernetes/contrib/tree/master/addon-resizer) in the same pod and changes resource limit configuration through the apiserver. `kubectl apply` fights with the addon-resizers. May be we should remove the initial resource limit field in the configuration file for this specific Deployment as we removed the replica count. @mikedanese @thockin @bprashanth --- Below are some logical things that may need to be clarified, feel free to **OMIT** them as they are too verbose: - For upgrade, the old RCs will not fight with the new Deployments during the overlap period even if they use the same label in template: - Deployment will not recognize the old pods because it need to match an additional "pod-template-hash" label. - ReplicationController will not manage the new pods (created by deployment) because the [`controllerRef`](https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/controller-ref.md) feature. - As we are moving all addons to Deployment, all old RCs would be removed. Attach empty annotation to RCs is only for letting `kubectl apply --prune` to recognize them, the content does not matter. - We might need to also annotate other resource types if we plan to upgrade them in 1.5 release: - They don't need to be attached this fake annotation if they remain in the same name. `kubectl apply` can recognize them by name/type/namespace. - In the other case, attaching empty annotations to them will still work. As the plan is to use label selector for annotate, some innocence old resources may also be attached empty annotations, they work as below two cases: - Resources that need to be bumped up to a newer version (mainly due to some significant update --- change disallowed fields --- that could not be managed by the update feature of `kubectl apply`) are good to go with this fake annotation, as old resources will be deleted and new sources will be created. The content in annotation does not matter. - Resources that need to stay inside the management of `kubectl apply` is also good to go. As `kubectl apply` will [generate a 3-way merge patch](https://github.com/kubernetes/kubernetes/blob/master/pkg/util/strategicpatch/patch.go#L1202-L1226). This empty annotation is harmless enough. |
||
|---|---|---|
| .. | ||
| addon-manager | ||
| calico-policy-controller | ||
| cluster-loadbalancing | ||
| cluster-monitoring | ||
| dashboard | ||
| dns | ||
| etcd-empty-dir-cleanup | ||
| fluentd-elasticsearch | ||
| fluentd-gcp/fluentd-gcp-image | ||
| gci | ||
| node-problem-detector | ||
| podsecuritypolicies | ||
| python-image | ||
| registry | ||
| README.md | ||
Cluster add-ons
Cluster add-ons are Services and Replication Controllers (with pods) that are
shipped with the Kubernetes binaries and are considered an inherent part of the
Kubernetes clusters. The add-ons are visible through the API (they can be listed
using kubectl), but manipulation of these objects is discouraged because
the system will bring them back to the original state, in particular:
- if an add-on is stopped, it will be restarted automatically
- if an add-on is rolling-updated (for Replication Controllers), the system will stop the new version and start the old one again (or perform rolling update to the old version, in the future).
On the cluster, the add-ons are kept in /etc/kubernetes/addons on the master node, in yaml files
(json is not supported at the moment). A system daemon periodically checks if
the contents of this directory is consistent with the add-on objects on the API
server. If any difference is spotted, the system updates the API objects
accordingly. (Limitation: for now, the system compares only the names of objects
in the directory and on the API server. So changes in parameters may not be
noticed). So the only persistent way to make changes in add-ons is to update the
manifests on the master server. But still, users are discouraged to do it
on their own - they should rather wait for a new release of
Kubernetes that will also contain new versions of add-ons.
Each add-on must specify the following label: kubernetes.io/cluster-service: true.
Yaml files that do not define this label will be ignored.
The naming convention for Replication Controllers is
<basename>-<version>, where <basename> is the same in consecutive
versions and <version> changes when the component is updated
(<version> must not contain -). For instance,
heapster-controller-v1 and heapster-controller-12 are the
same controllers with two different versions, while heapster-controller-v1
and heapster-newcontroller-12 are treated as two different applications.
When a new version of a Replication Controller add-on is found, the system will
stop the old (current) replication controller and start the new one
(in the future, rolling update will be performed).
For services, the naming scheme is just <basename> (with empty version number)
because we do not expect the service name to change in consecutive versions (and
rolling-update of services does not exist).
Add-on update procedure
To update add-ons, just update the contents of /etc/kubernetes/addons
directory with the desired definition of add-ons. Then the system will take care
of:
- Removing the objects from the API server whose manifest was removed.
- This is done for add-ons in the system that do not have a manifest file with the same basename
- Creating objects from new manifests
- This is done for manifests that do not correspond to existing API objects with the same basename
- Updating objects whose basename is the same, but whose versions changed.
- The update is currently performed by removing the old object and creating the new one. In the future, rolling update of replication controllers will be implemented to keep the add-on services up and running during update of add-on pods.
- Note that this cannot happen for Services as their version is always empty.
Note that in order to run the updator script, python is required on the machine. For OS distros that don't have python installed, a python container will be used.