add auto update for apf bootstrap configuration

Take the following approach:
On a fresh install, all bootstrap configuration objects will
have auto update enabled via the following annotation :
`apf.kubernetes.io/autoupdate: 'true'`

The kube-apiserver periodically checks the bootstrap configuration
objects on the cluster and applies update if necessary.

We enforce an 'always auto-update' policy for the mandatory
configuration object(s).

We update the suggested configuration objects when:
- auto update is enabled (`apf.kubernetes.io/autoupdate: 'true'`) or
- auto update annotation key is missing but `generation` is `1`

If the configuration object is missing the annotation key, we add
it appropriately:
it is set to `true` if `generation` is `1`, `false` otherwise.

The above approach ensures that we don't squash changes made by an
operator. Please note, we can't protect the changes made by the
operator in the following scenario:
- the user changes the spec and then deletes and recreates
  the same object. (generation resets to 1)

remove using a marker

staging/src/k8s.io/api/flowcontrol/v1beta1/types.go

@@ -57,6 +57,50 @@ const (
 	ResponseHeaderMatchedFlowSchemaUID                 = "X-Kubernetes-PF-FlowSchema-UID"
 )
 
+const (
+	// AutoUpdateAnnotationKey is the name of an annotation that enables
+	// automatic update of the spec of the bootstrap configuration
+	// object(s), if set to 'true'.
+	//
+	// On a fresh install, all bootstrap configuration objects will have auto
+	// update enabled with the following annotation key:
+	//    apf.kubernetes.io/autoupdate-spec: 'true'
+	//
+	// The kube-apiserver periodically checks the bootstrap configuration
+	// objects on the cluster and applies updates if necessary.
+	//
+	// kube-apiserver enforces an 'always auto-update' policy for the
+	// mandatory configuration object(s). This implies:
+	// - the auto-update annotation key is added with a value of 'true'
+	//   if it is missing.
+	// - the auto-update annotation key is set to 'true' if its current value
+	//   is a boolean false or has an invalid boolean representation
+	//   (if the cluster operator sets it to 'false' it will be stomped)
+	// - any changes to the spec made by the cluster operator will be
+	//   stomped.
+	//
+	// The kube-apiserver will apply updates on the suggested configuration if:
+	// - the cluster operator has enabled auto-update by setting the annotation
+	//   (apf.kubernetes.io/autoupdate-spec: 'true') or
+	// - the annotation key is missing but the generation is 1
+	//
+	// If the suggested configuration object is missing the annotation key,
+	// kube-apiserver will update the annotation appropriately:
+	// - it is set to 'true' if generation of the object is '1' which usually
+	//   indicates that the spec of the object has not been changed.
+	// - it is set to 'false' if generation of the object is greater than 1.
+	//
+	// The goal is to enable the kube-apiserver to apply update on suggested
+	// configuration objects installed by previous releases but not overwrite
+	// changes made by the cluster operators.
+	// Note that this distinction is imperfectly detected: in the case where an
+	// operator deletes a suggested configuration object and later creates it
+	// but with a variant spec and then does no updates of the object
+	// (generation is 1), the technique outlined above will incorrectly
+	// determine that the object should be auto-updated.
+	AutoUpdateAnnotationKey = "apf.kubernetes.io/autoupdate-spec"
+)
+
 // +genclient
 // +genclient:nonNamespaced
 // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

staging/src/k8s.io/apiserver/pkg/apis/flowcontrol/bootstrap/default.go

@@ -455,8 +455,14 @@ var (
 
 func newPriorityLevelConfiguration(name string, spec flowcontrol.PriorityLevelConfigurationSpec) *flowcontrol.PriorityLevelConfiguration {
 	return &flowcontrol.PriorityLevelConfiguration{
-		ObjectMeta: metav1.ObjectMeta{Name: name},
-		Spec:       spec}
+		ObjectMeta: metav1.ObjectMeta{
+			Name: name,
+			Annotations: map[string]string{
+				flowcontrol.AutoUpdateAnnotationKey: "true",
+			},
+		},
+		Spec: spec,
+	}
 }
 
 func newFlowSchema(name, plName string, matchingPrecedence int32, dmType flowcontrol.FlowDistinguisherMethodType, rules ...flowcontrol.PolicyRulesWithSubjects) *flowcontrol.FlowSchema {
@@ -465,7 +471,12 @@ func newFlowSchema(name, plName string, matchingPrecedence int32, dmType flowcon
 		dm = &flowcontrol.FlowDistinguisherMethod{Type: dmType}
 	}
 	return &flowcontrol.FlowSchema{
-		ObjectMeta: metav1.ObjectMeta{Name: name},
+		ObjectMeta: metav1.ObjectMeta{
+			Name: name,
+			Annotations: map[string]string{
+				flowcontrol.AutoUpdateAnnotationKey: "true",
+			},
+		},
 		Spec: flowcontrol.FlowSchemaSpec{
 			PriorityLevelConfiguration: flowcontrol.PriorityLevelConfigurationReference{
 				Name: plName,