Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Split scheduler latency metric to fine-grained steps
This splits the summary metric we recently added into finer steps. It should be very useful for performance experiments.
/cc @wojtek-t
fyi - @bsalamat @misterikkit
Strictly speaking this is a breaking change, but since this metric was added only ~week ago I think it should fine (we should port this change to 1.11).
```release-note
Split 'scheduling_latency_seconds' metric into finer steps (predicate, priority, premption)
```
Automatic merge from submit-queue (batch tested with PRs 64285, 63660, 63661, 63662, 64883). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
use subtest for table units (pkg/scheduler/algorithmprovider)
**What this PR does / why we need it**: Update scheduler's unit table tests to use subtest
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
**Special notes for your reviewer**:
breaks up PR: https://github.com/kubernetes/kubernetes/pull/63281
/ref #63267
**Release note**:
```release-note
This PR will leverage subtests on the existing table tests for the scheduler units.
Some refactoring of error/status messages and functions to align with new approach.
```
Automatic merge from submit-queue (batch tested with PRs 64285, 63660, 63661, 63662, 64883). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
use subtest for table units (pkg/scheduler)
**What this PR does / why we need it**: Update scheduler's unit table tests to use subtest
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
**Special notes for your reviewer**:
breaks up PR: https://github.com/kubernetes/kubernetes/pull/63281
/ref #63267
**Release note**:
```release-note
This PR will leverage subtests on the existing table tests for the scheduler units.
Some refactoring of error/status messages and functions to align with new approach.
```
Automatic merge from submit-queue (batch tested with PRs 64285, 63660, 63661, 63662, 64883). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
use subtest for table units (pkg/scheduler/algorithm/predicates)
**What this PR does / why we need it**: Update scheduler's unit table tests to use subtest
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
**Special notes for your reviewer**:
breaks up PR: https://github.com/kubernetes/kubernetes/pull/63281
/ref #63267
**Release note**:
```release-note
This PR will leverage subtests on the existing table tests for the scheduler units.
Some refactoring of error/status messages and functions to align with new approach.
```
This moves the equivalence cache implementation out of the 'core'
package and into k8s.io/kubernetes/pkg/scheduler/core/equivalence.
Separating the equiv. cache from the genericScheduler implementation
make their interaction points easier to follow, and prevents us from
accidentally accessing unexported fields.
Automatic merge from submit-queue (batch tested with PRs 64882, 64692, 64389, 60626, 64840). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
update pod state of scheduler cache when UpdatePod
update pod state map in scheduler cache when call UpdatePod. @k82cn @bsalamat
```release-note
keep pod state consistent when scheduler cache UpdatePod
```
Automatic merge from submit-queue (batch tested with PRs 64142, 64426, 62910, 63942, 64548). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
scheduler: further cleanup of equivalence cache
**What this PR does / why we need it**:
This improves comments and simplifies some names/logic in equivalence_cache.go, as well as changing the order of some items in the file.
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
/kind cleanup
Automatic merge from submit-queue (batch tested with PRs 64252, 64307, 64163, 64378, 64179). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Remove unused parameter (pod) in `pkg/scheduler/core/generic_scheduler`
**What this PR does / why we need it**:
Remove unused parameter (pod) in `pkg/scheduler/core/generic_scheduler`
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 65230, 57355, 59174, 63698, 63659). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
use subtest for table units (pkg-scheduler-algorithm-priorities-util)
**What this PR does / why we need it**: Update scheduler's unit table tests to use subtest
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
**Special notes for your reviewer**:
breaks up PR: https://github.com/kubernetes/kubernetes/pull/63281
/ref #63267
**Release note**:
```release-note
This PR will leverage subtests on the existing table tests for the scheduler units.
Some refactoring of error/status messages and functions to align with new approach.
```
Automatic merge from submit-queue (batch tested with PRs 61803, 64305, 64170, 64361, 64339). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Remove some unnecessarily gendered pronouns in comments
**What this PR does / why we need it**:
A bunch of comments are unnecessarily gendered. I've changed them to gender-neutral they/theirs.
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
This makes the lookup behave like a normal map lookup, so it is easier
for readers to follow the logic. It also inverts the "invalid" bool to
an "ok" bool because `!invalid` is a double negative.
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Adding tests for ImageLocalityPriority
**What this PR does / why we need it**:
This PR adds tests for ImageLocalityPriority scheduling policy, as follow-ups of [#63842](https://github.com/kubernetes/kubernetes/issues/63842) and [#63345](https://github.com/kubernetes/kubernetes/issues/63345). It includes the unit test for ImageSizes function of NodeInfo in the scheduler cache.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
@resouer
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 61963, 64279, 64130, 64125, 64049). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix TestSchedulerWithVolumeBinding to avoid setting predicate ordering.
It is causing data race condition as predicate ordering is changing global
variable `predicatesOrdering`. Infact this test does not require any special
predicate order and should work on default predicate ordering as far as
VolumeScheduling feature is enabled.
See these logs:
```
==================
==================
WARNING: DATA RACE
Read at 0x00c420894180 by goroutine 156:
k8s.io/kubernetes/pkg/scheduler/core.podFitsOnNode()
/home/avagarwa/upstream-code/gocode/src/k8s.io/kubernetes/pkg/scheduler/core/generic_scheduler.go:503 +0xbb
k8s.io/kubernetes/pkg/scheduler/core.(*genericScheduler).findNodesThatFit.func1()
/home/avagarwa/upstream-code/gocode/src/k8s.io/kubernetes/pkg/scheduler/core/generic_scheduler.go:353 +0x2f0
k8s.io/kubernetes/vendor/k8s.io/client-go/util/workqueue.Parallelize.func1()
/home/avagarwa/upstream-code/gocode/src/k8s.io/kubernetes/vendor/k8s.io/client-go/util/workqueue/parallelizer.go:47 +0xa3
Previous write at 0x00c420894180 by goroutine 186:
k8s.io/kubernetes/pkg/scheduler.TestSchedulerWithVolumeBinding()
/home/avagarwa/upstream-code/gocode/src/k8s.io/kubernetes/pkg/scheduler/scheduler_test.go:663 +0x71
testing.tRunner()
/usr/lib/golang/src/testing/testing.go:777 +0x16d
Goroutine 156 (running) created at:
k8s.io/kubernetes/vendor/k8s.io/client-go/util/workqueue.Parallelize()
/home/avagarwa/upstream-code/gocode/src/k8s.io/kubernetes/vendor/k8s.io/client-go/util/workqueue/parallelizer.go:43 +0x139
k8s.io/kubernetes/pkg/scheduler/core.(*genericScheduler).findNodesThatFit()
/home/avagarwa/upstream-code/gocode/src/k8s.io/kubernetes/pkg/scheduler/core/generic_scheduler.go:378 +0xe8a
k8s.io/kubernetes/pkg/scheduler/core.(*genericScheduler).Schedule()
/home/avagarwa/upstream-code/gocode/src/k8s.io/kubernetes/pkg/scheduler/core/generic_scheduler.go:131 +0x385
k8s.io/kubernetes/pkg/scheduler.(*Scheduler).schedule()
/home/avagarwa/upstream-code/gocode/src/k8s.io/kubernetes/pkg/scheduler/scheduler.go:192 +0xcd
k8s.io/kubernetes/pkg/scheduler.(*Scheduler).scheduleOne()
/home/avagarwa/upstream-code/gocode/src/k8s.io/kubernetes/pkg/scheduler/scheduler.go:447 +0x598
k8s.io/kubernetes/pkg/scheduler.(*Scheduler).(k8s.io/kubernetes/pkg/scheduler.scheduleOne)-fm()
/home/avagarwa/upstream-code/gocode/src/k8s.io/kubernetes/pkg/scheduler/scheduler.go:182 +0x41
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1()
/home/avagarwa/upstream-code/gocode/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x61
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil()
/home/avagarwa/upstream-code/gocode/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xcd
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Until()
/home/avagarwa/upstream-code/gocode/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x5a
Goroutine 186 (running) created at:
testing.(*T).Run()
/usr/lib/golang/src/testing/testing.go:824 +0x564
testing.runTests.func1()
/usr/lib/golang/src/testing/testing.go:1063 +0xa4
testing.tRunner()
/usr/lib/golang/src/testing/testing.go:777 +0x16d
testing.runTests()
/usr/lib/golang/src/testing/testing.go:1061 +0x4e1
testing.(*M).Run()
/usr/lib/golang/src/testing/testing.go:978 +0x2cd
main.main()
_testmain.go:52 +0x22a
==================
--- FAIL: TestSchedulerWithVolumeBinding (18.04s)
testing.go:730: race detected during execution of test
FAIL
```
It is pretty easy to reproduce this race by following these steps:
```
cd pkg/scheduler
go test -c -race
stress -p 100 ./scheduler.test
```
Predicate ordering to this unit test was added here: https://github.com/kubernetes/kubernetes/pull/57168
Since the whole scheduler instance uses just one ordering at time, not sure what is the advantage.
@kubernetes/sig-scheduling-bugs @bsalamat @k82cn @frobware @smarterclayton @sjenning
```release-note
None
```
Automatic merge from submit-queue (batch tested with PRs 63434, 64172, 63975, 64180, 63755). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Optimize the lock which in the RunPredicate
**What this PR does / why we need it**:
Enhance the performance of scheduler
- Change the lock in the RunPredicate from lock to rlock
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Could solve part of #63784
**Special notes for your reviewer**:
_Run benchmark test by scheduler_perf_:
`Before` BenchmarkScheduling/1000Nodes/0Pods-32 1000 11689758 ns/op
`After` BenchmarkScheduling/1000Nodes/0Pods-32 1000 5951510 ns/op
_Run integration (density) test by scheduler_perf_:
Schedule 3000 Pods On 3000 Nodes
`Before` rate 19 per second on average
`After` rate 58 per second on average
_Cpu profile test result_:
`Before` [click](https://cdn.rawgit.com/godliness/files/master/63784_before.svg)
`After` [click](https://cdn.rawgit.com/godliness/files/master/63784_after.svg)
**Release note**:
```release-note
`None`
```
/sig scheduling
/cc @misterikkit
/cc @bsalamat
/cc @ravisantoshgudimetla
/cc @resouer
Automatic merge from submit-queue (batch tested with PRs 64127, 63895, 64066, 64215, 64202). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add warnings about cache invalidation.
Part of https://github.com/kubernetes/kubernetes/pull/63040 is the
assumption that scheduler cache updates must happen before equivalence
cache updates for any given informer event.
The reason for this is that the equivalence cache implementation checks
the main cache for staleness while holding the equiv. cache write lock.
case 1: If an informer invalidates an equiv. cache entry before the
staleness check, then we know that the main cache update completed.
case 2: If an informer blocks trying to grab the equiv. cache lock, then
invalidation will occur right after the potentially stale update is
written.
This patch adds a note to places where we invalidate the equivalence
cache so that hopefully nobody violates this invariant.
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
/kind cleanup
/sig scheduling
Automatic merge from submit-queue (batch tested with PRs 64174, 64187, 64216, 63265, 64223). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Do not use DeepEqual to compare slices in test.
This wraps DeepEqual with a helper that considers nil slices and empty
slices to be equal.
Scheduler code might use a nil slice or empty slice to represent an
empty list, so tests should not be sensitive to the difference. Tests
could fail because DeepEqual considers nil to be different from an empty
slice.
**What this PR does / why we need it**:
Avoid breaking tests in cases where application behavior is not changed.
**Special notes for your reviewer**:
This brittle test keeps breaking in a number of my PRs. Hoping to get this fix merged independently.
**Release note**:
```release-note
NONE
```
/sig scheduling
/kind cleanup
This wraps DeepEqual with a helper that considers nil slices and empty
slices to be equal.
Scheduler code might use a nil slice or empty slice to represent an
empty list, so tests should not be sensitive to the difference. Tests
could fail because DeepEqual considers nil to be different from an empty
slice.
Automatic merge from submit-queue (batch tested with PRs 63283, 64032, 64159, 64126, 64098). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
remove unused code of (pkg/scheduler)
**What this PR does / why we need it**:
/kind cleanup
remove unused code
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Part of https://github.com/kubernetes/kubernetes/pull/63040 is the
assumption that scheduler cache updates must happen before equivalence
cache updates for any given informer event.
The reason for this is that the equivalence cache implementation checks
the main cache for staleness while holding the equiv. cache write lock.
case 1: If an informer invalidates an equiv. cache entry before the
staleness check, then we know that the main cache update completed.
case 2: If an informer blocks trying to grab the equiv. cache lock, then
invalidation will occur right after the potentially stale update is
written.
This patch adds a note to places where we invalidate the equivalence
cache so that hopefully nobody violates this invariant.
Automatic merge from submit-queue (batch tested with PRs 63598, 63913, 63459, 63963, 60464). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Check nodeInfo before ecache predicate
**What this PR does / why we need it**:
There's chances during test when nodeInfo is nil which may cause ecache predicate fail with nil pointer.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#63427
**Special notes for your reviewer**:
Not sure how to reproduce the original issue yet. i.e. why and when `nodeInfo` will become nil in tests is not clear to me, that's why I label it as WIP.
cc @bsalamat who may have more inputs.
**Release note**:
```release-note
NONE
```
It is causing data race condition as predicate ordering is changing global
variable predicatesOrdering. Infact this test does not require any special
predicate order and should work on default predicate ordering as far as
VolumeScheduling feature is enabled.
