Merge pull request #55242 from sttts/sttts-stop-cache-deadlock

Automatic merge from submit-queue (batch tested with PRs 55331, 55272, 55228, 49763, 55242). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

client-go/tools/cache: fix possible deadlock when stopping a reflector

While getting Connection-Refused error, a reflector was retrying without stopping when the stopCh is closed.

The flaking TestCRD #54095 sometimes shows a deadlock in reflectors that should be shutdown, called from the storage cacher. So possibly this is related.

staging/src/k8s.io/client-go/tools/cache/reflector.go

@@ -295,6 +295,13 @@ func (r *Reflector) ListAndWatch(stopCh <-chan struct{}) error {
 	}()
 
 	for {
+		// give the stopCh a chance to stop the loop, even in case of continue statements further down on errors
+		select {
+		case <-stopCh:
+			return nil
+		default:
+		}
+
 		timemoutseconds := int64(minWatchTimeout.Seconds() * (rand.Float64() + 1.0))
 		options = metav1.ListOptions{
 			ResourceVersion: resourceVersion,