When scaling down a ReplicaSet, delete doubled up replicas first, where a
"doubled up replica" is defined as one that is on the same node as an
active replica belonging to a related ReplicaSet. ReplicaSets are
considered "related" if they have a common controller (typically a
Deployment).
The intention of this change is to make a rolling update of a Deployment
scale down the old ReplicaSet as it scales up the new ReplicaSet by
deleting pods from the old ReplicaSet that are colocated with ready pods of
the new ReplicaSet. This change in the behavior of rolling updates can be
combined with pod affinity rules to preserve the locality of a Deployment's
pods over rollout.
A specific scenario that benefits from this change is when a Deployment's
pods are exposed by a Service that has type "LoadBalancer" and external
traffic policy "Local". In this scenario, the load balancer uses health
checks to determine whether it should forward traffic for the Service to a
particular node. If the node has no local endpoints for the Service, the
health check will fail for that node. Eventually, the load balancer will
stop forwarding traffic to that node. In the meantime, the service proxy
drops traffic for that Service. Thus, in order to reduce risk of dropping
traffic during a rolling update, it is desirable preserve node locality of
endpoints.
* pkg/controller/controller_utils.go (ActivePodsWithRanks): New type to
sort pods using a given ranking.
* pkg/controller/controller_utils_test.go (TestSortingActivePodsWithRanks):
New test for ActivePodsWithRanks.
* pkg/controller/replicaset/replica_set.go
(getReplicaSetsWithSameController): New method. Given a ReplicaSet, return
all ReplicaSets that have the same owner.
(manageReplicas): Call getIndirectlyRelatedPods, and pass its result to
getPodsToDelete.
(getIndirectlyRelatedPods): New method. Given a ReplicaSet, return all
pods that are owned by any ReplicaSet with the same owner.
(getPodsToDelete): Add an argument for related pods. Use related pods and
the new getPodsRankedByRelatedPodsOnSameNode function to take into account
whether a pod is doubled up when sorting pods for deletion.
(getPodsRankedByRelatedPodsOnSameNode): New function. Return an
ActivePodsWithRanks value that wraps the given slice of pods and computes
ranks where each pod's rank is equal to the number of active related pods
that are colocated on the same node.
* pkg/controller/replicaset/replica_set_test.go (newReplicaSet): Set
OwnerReferences on the ReplicaSet.
(newPod): Set a unique UID on the pod.
(byName): New type to sort pods by name.
(TestGetReplicaSetsWithSameController): New test for
getReplicaSetsWithSameController.
(TestRelatedPodsLookup): New test for getIndirectlyRelatedPods.
(TestGetPodsToDelete): Augment the "various pod phases and conditions, diff
= len(pods)" test case to ensure that scale-down still selects doubled-up
pods if there are not enough other pods to scale down. Add a "various pod
phases and conditions, diff = len(pods), relatedPods empty" test case to
verify that getPodsToDelete works even if related pods could not be
determined. Add a "ready and colocated with another ready pod vs not
colocated, diff < len(pods)" test case to verify that a doubled-up pod gets
preferred for deletion. Augment the "various pod phases and conditions,
diff < len(pods)" test case to ensure that not-ready pods are preferred
over ready but doubled-up pods.
* pkg/controller/replicaset/BUILD: Regenerate.
* test/e2e/apps/deployment.go
(testRollingUpdateDeploymentWithLocalTrafficLoadBalancer): New end-to-end
test. Create a deployment with a rolling update strategy and affinity
rules and a load balancer with "Local" external traffic policy, and verify
that set of nodes with local endponts for the service remains unchanged
during rollouts.
(setAffinity): New helper, used by
testRollingUpdateDeploymentWithLocalTrafficLoadBalancer.
* test/e2e/framework/service/jig.go (GetEndpointNodes): Factor building the
set of node names out...
(GetEndpointNodeNames): ...into this new method.
CreateNginxPod() is called from flexvolume_online_resize only and
that seems storage specific function because that requires a PVC.
So this moves the function to the place which calls it for the code
cleanup.
The following functions are used locally in e2e framework subpackages.
- RunSSHCommandViaBastion
- MakeNginxPod
- LogPodTerminationMessages
- CheckPodsCondition
- SetNodeAffinityRequirement
This renames them to clarify them as local ones.
test_verify.go contained the function TestPodSuccessOrFail() only,
and the function is used in the package only.
This moves the function to create.go and remove test_verify.go.
The function is used at e2e framework util module only.
So this moves the function to the module for trying to remove
dependencies to subpackages from core e2e framework.
These functions are only used in fixtures.go module.
So it is not necessary to define them for exposing.
This renames these functions for making them local functions clearly.
test/e2e/storage/testsuites creates volumes dynamically. Initially, the size of those volumes was
hard-coded in the test, which prevented using the tests with storage backends that couldn't support
that hard-coded size
This cleans up the framework.util.go file and moves various Expect*
functions into their own file for better organization.
Signed-off-by: alejandrox1 <alarcj137@gmail.com>
This continues the work in
https://github.com/kubernetes/kubernetes/pull/81426 by also moving the
logger_test.go, moving the log helper code from util.go to log.go (a
more logical place, as it is only used there) and updating comments.
After merging
259bb3bef5 (diff-eb7b79470992813ea1905e96c298b47b)
ExpectEqual and some of the other wrappers logged the failure twice,
once inside the wrapper itself and once in the failure handler.
Logging the stack backtrace is useful because many assertions still
don't contain an explanation and therefore knowing where they occur is
crucial. Now all failures are logged with a "Full Stack Trace", not
just those with a wrapper. The stack is pruned to skip over wrapper
functions and removes Ginkgo internal functions to keep the stack
trace smaller.
Failures occuring in the wrappers were recorded as occuring in those
wrappers. Now the wrappers are skipped and the caller is recorded
instead.
The full stack trace recorded by Ginkgo 1.10.0 is currently off by one
entry. This needs to be fixed in Ginkgo, then the test can be updated.
Ginkgo 1.10.0 includes the relevant fix for dumping the full stack
(https://github.com/onsi/ginkgo/pull/590), so when using that release
we can simplify the logging unit test.
By changing the skipping, we can avoid the rather volatile util.go
entries. However, that gomega is part of the stack trace still needs
to be fixed in Gingko.
Locked down IAM configurations will prevent mounting of arbitrary
volumes, so when we create volumes that we intend the cluster to
mount, we must set the appropriate EC2 tags.
A number of tests were using hardcoded image paths instead of
going through the imageutils package. The reason for centralizing
the logic there is to keep an eye on what images we use and where
they come from.
Now that namespace deletion is reliable, use the suite tear down
to catch non terminated namespaces and stop waiting within each test
for deletion.
Serial tests that need a clean set of namespaces must use the
appropriate flag to control whether they wait for clean namespaces
on startup.
This is an effort to clean up the framework util.go file so that it
makes more sense and it is more managable.
Signed-off-by: alejandrox1 <alarcj137@gmail.com>
