github/kubernetes
4
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
2c438a8f21
kubernetes/pkg/kubelet/cm/deviceplugin
History
Kubernetes Submit Queue a4eb2f96d0
Merge pull request #57610 from vikaschoudhary16/remove-redundant-sleep 
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Remove redundant sleep from ReRegistration unit test case

/kind cleanup
/sig node

**What this PR does / why we need it**:
Once upon a time, there was a race in the device plugin registration logic.  At that time, [list()](5cac9fc984/pkg/kubelet/deviceplugin/manager.go (L206)) and [listAndWatch()](5cac9fc984/pkg/kubelet/deviceplugin/manager.go (L224)) used to be separate functions. Race was there for taking manager.mutex lock from two places. [One, from within the m.addEndpoint()](5cac9fc984/pkg/kubelet/deviceplugin/manager.go (L214)) and the [second, from within m.Devices()](5cac9fc984/pkg/kubelet/deviceplugin/manager.go (L137)).  This race was making `TestDevicePluginReRegistration` flaky as explained below.
 	
```
1.     p1.Register(socketName, testResourceName)
2.  	// Wait for the first callback to be issued.
3.  	<-callbackChan
4.        devices := m.Devices()  
```
* L#1 leads to eventually **asynchronous** invocation of m.addEndpoint(), let say **thread1**.
* L#3 holds the test case execution till the [callback gets invoked](5cac9fc984/pkg/kubelet/deviceplugin/endpoint.go (L108)). This means test case execution waits on channel till the **thread1**  reaches the point where [e.list() call completes in the addEndpoint.](5cac9fc984/pkg/kubelet/deviceplugin/manager.go (L206)) 
* L#4 triggers a new thread. thread1 and this new thread are both racing for m.mutex.Lock(). Former, in the addEndpoint() and later one in the m.Devices(). If m.Devices wins the race, result is the test case failure because endpoint gets added in the manager only after taking mutex.Lock() in the addEndpoint().

To deal with this flake, we added `Sleep` between L#3 and L#4.  `Sleep` was getting some extra time to addEndpoint() and thus making thread1 win the race each time.

Above explained race scenario got fixed and merged sometime back in this PR:
[Deviceplugin refactoring: merge func list and listwatch in endpoint into one](https://github.com/kubernetes/kubernetes/pull/52149)
With the above PR, callback function is invoked from e.run() which makes sure that test case waits on channel till the endpoint is added and devices are updated
Above explained race scenario does not exist now, therefore removing redundant sleeps from the test case.

Tested:
go test -race -count 500 k8s.io/kubernetes/pkg/kubelet/cm/deviceplugin -run TestDevicePluginReRegistration  -timeout 5h

Related #52616 #56026 

**Special notes for your reviewer**:

**Release note**:

```release-note
None
```
/cc @vishh @derekwaynecarr @jiayingz @RenaudWasTaken @lichuqiang @ScorpioCPH @tengqm @mindprince @ConnorDoyle @jeremyeder
2017-12-27 14:53:21 -08:00
..
BUILD Autogenerate BUILD files 2017-12-23 13:12:11 -08:00
device_plugin_stub.go Move device plugin related files under pkg/kubelet/cm/deviceplugin/. 2017-10-24 14:17:20 -07:00
endpoint_test.go Merging handler into manager API 2017-11-20 21:37:46 +00:00
endpoint.go Fix a race in the endpoint.go 2017-12-23 03:02:33 -05:00
manager_stub.go Extends deviceplugin to gracefully handle full device plugin lifecycle. 2017-11-20 23:40:14 -08:00
manager_test.go Merge pull request #57610 from vikaschoudhary16/remove-redundant-sleep 2017-12-27 14:53:21 -08:00
manager.go Revert back #57278 2017-12-22 18:55:53 -05:00
OWNERS Add vikaschoudhary16 as reviewer in pkg/kubelet/cm/deviceplugin 2017-12-16 08:19:36 -05:00
pod_devices.go Re-uses device plugin resources allocated to init containers. 2017-12-04 22:01:28 -08:00
types.go Extends deviceplugin to gracefully handle full device plugin lifecycle. 2017-11-20 23:40:14 -08:00