github/kubernetes
4
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge pull request #39608 from peay/cronjob-too-many-times-to-list

Browse Source 
Automatic merge from submit-queue

Do not list CronJob unmet starting times beyond deadline

**What this PR does / why we need it**:

See #36311. `getRecentUnmetScheduleTimes` gives up after 100 unmet times to avoid wasting too much CPU or memory generating all the times, as it generates them sequentially.

When concurrency is forbidden, this is conceptually un-necessary: we only need the last unmet start time. This suggests that when concurrency is forbidden, we could generate times by going backward in time from now. This is not very practical as CronJob currently relies on a package that only provides `Next` and no `Prev`. Hand-cooking a `Prev` does not seem like a good idea. I could submit a PR to the cron library to add a `Prev` method, and use that when concurrency is forbidden through something like `getLastUnmetScheduleTime`. This would be `O(1)` and there would be no limit involved.

(edit: actually, even for the other concurrency settings, we only start the last unmet start times -- there is a `TODO` in the controller to actually start all of them, but that is not implemented at the moment. This means the solution would apply, at least temporarily, to all concurrency settings).

cc @soltysh what do you think?

In the meantime, I would suggest to do something simple. Currently, the user has no way to configure anything to ensure that his CronJob will not get stuck if one job takes more that 100 unmet times.

 `getRecentUnmetScheduleTimes` starts with an initial time corresponding to the last start (or to the creation of the CronJob, if nothing has started yet). However, when `StartingDeadlineSeconds` is set, the controller will not start anything that is older than the deadline, so if the last start is way beyond the deadline, we are generating potentially lots of unmet start times that will not be considered by the scheduler for scheduling anyway.

Consider a job running every minute, where the last instance has taken 120 minutes. This means there are more than 100 unmet times when we start counting from the last start time.

**The PR makes `getRecentUnmetScheduleTimes` only consider times that do not fall beyond the deadline.** Here, the CronJob can be configured with a `StartingDeadlineSeconds` of, say, 10 minutes. After the 120min job has run, `getRecentUnmetScheduleTimes` will only consider the times in the last 10 minutes from now, and will not get stuck.

As a side note on the max. number of unmet times to use as limits in terms of CPU used by the controller: I have run a quick benchmark on my i7 mac. Schedules corresponding to "once a week" tend to be more expensive to generate unmet times for. Just FYI.

```
+--------------+---------------+--------------+
|   SCHEDULE   | MISSED STARTS |    TIMING    |
+--------------+---------------+--------------+
| */1 * * * ?  |           100 | 383.645µs    |
| */30 * * * ? |           100 | 354.765µs    |
| 30 1 * * ?   |           100 | 1.065124ms   |
| 30 1 * * 0   |           100 | 1.80034ms    |
| */1 * * * ?  |           500 | 1.341365ms   |
| */30 * * * ? |           500 | 1.814441ms   |
| 30 1 * * ?   |           500 | 8.475012ms   |
| 30 1 * * 0   |           500 | 10.020613ms  |
| */1 * * * ?  |          1000 | 2.551697ms   |
| */30 * * * ? |          1000 | 4.075813ms   |
| 30 1 * * ?   |          1000 | 17.674945ms  |
| 30 1 * * 0   |          1000 | 19.149324ms  |
| */1 * * * ?  |         10000 | 25.725531ms  |
| */30 * * * ? |         10000 | 87.520022ms  |
| 30 1 * * ?   |         10000 | 174.29216ms  |
| 30 1 * * 0   |         10000 | 196.565748ms |
+--------------+---------------+--------------+
```

using

```.go
package main

import (
    "fmt"
    "time"
    "os"
    "strconv"

    "github.com/robfig/cron"
    "github.com/olekukonko/tablewriter"
)

func timeSchedule(schedule string, iterations int) (time.Duration) {
    sched, err := cron.ParseStandard(schedule)

    if err != nil {
        panic(fmt.Sprintf("Unparseable schedule: %s", err))
    }

    start := time.Now()
    t := time.Now()

    for i := 1; i <= iterations; i++ {
        t = sched.Next(t)
    }

    return time.Since(start)
}

func main() {
    table := tablewriter.NewWriter(os.Stdout)
    table.SetHeader([]string{"Schedule", "Missed starts", "Timing"})

    schedules := []string{"*/1 * * * ?", "*/30 * * * ?", "30 1 * * ?", "30 1 * * 0"}
    iteration_nums := []int{100, 500, 1000, 10000}

    for _, iterations := range iteration_nums {
        for _, schedule := range schedules {
            table.Append([]string{schedule,
                                  strconv.Itoa(iterations),
                                  timeSchedule(schedule, iterations).String()})
        }
    }
    table.Render()
}
```

**Which issue this PR fixes**: fixes #36311

**Special notes for your reviewer**:

**Release note**:

```release-note
```
This commit is contained in:
Kubernetes Submit Queue 2017-01-17 00:41:45 -08:00 committed by GitHub
commit 9d2fce7c22
3 changed files with 71 additions and 10 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

56
pkg/controller/cronjob/cronjob_controller_test.go
View File

@ -56,6 +56,14 @@ func justAfterTheHour() time.Time {
return T1
}
func weekAfterTheHour() time.Time {
T1, err := time.Parse(time.RFC3339, "2016-05-26T10:00:00Z")
if err != nil {
panic("test setup error")
}
return T1
}
func justBeforeThePriorHour() time.Time {
T1, err := time.Parse(time.RFC3339, "2016-05-19T08:59:00Z")
if err != nil {
@ -129,17 +137,31 @@ func newJob(UID string) batch.Job {
}
var (
shortDead int64 = 10
longDead int64 = 1000000
noDead int64 = -12345
A batch.ConcurrencyPolicy = batch.AllowConcurrent
f batch.ConcurrencyPolicy = batch.ForbidConcurrent
R batch.ConcurrencyPolicy = batch.ReplaceConcurrent
T bool = true
F bool = false
shortDead int64 = 10
mediumDead int64 = 2 * 60 * 60
longDead int64 = 1000000
noDead int64 = -12345
A batch.ConcurrencyPolicy = batch.AllowConcurrent
f batch.ConcurrencyPolicy = batch.ForbidConcurrent
R batch.ConcurrencyPolicy = batch.ReplaceConcurrent
T bool = true
F bool = false
)
func TestSyncOne_RunOrNot(t *testing.T) {
// Check expectations on deadline parameters
if shortDead/60/60 >= 1 {
t.Errorf("shortDead should be less than one hour")
}
if mediumDead/60/60 < 1 || mediumDead/60/60 >= 24 {
t.Errorf("mediumDead should be between one hour and one day")
}
if longDead/60/60/24 < 10 {
t.Errorf("longDead should be at least ten days")
}
testCases := map[string]struct {
// sj spec
concurrencyPolicy batch.ConcurrencyPolicy
@ -188,6 +210,24 @@ func TestSyncOne_RunOrNot(t *testing.T) {
"still active, is time, suspended": {A, T, onTheHour, noDead, T, T, justAfterTheHour(), F, F, 1},
"still active, is time, past deadline": {A, F, onTheHour, shortDead, T, T, justAfterTheHour(), F, F, 1},
"still active, is time, not past deadline": {A, F, onTheHour, longDead, T, T, justAfterTheHour(), T, F, 2},
// Controller should fail to schedule these, as there are too many missed starting times
// and either no deadline or a too long deadline.
"prev ran but done, long overdue, not past deadline, A": {A, F, onTheHour, longDead, T, F, weekAfterTheHour(), F, F, 0},
"prev ran but done, long overdue, not past deadline, R": {R, F, onTheHour, longDead, T, F, weekAfterTheHour(), F, F, 0},
"prev ran but done, long overdue, not past deadline, F": {f, F, onTheHour, longDead, T, F, weekAfterTheHour(), F, F, 0},
"prev ran but done, long overdue, no deadline, A": {A, F, onTheHour, noDead, T, F, weekAfterTheHour(), F, F, 0},
"prev ran but done, long overdue, no deadline, R": {R, F, onTheHour, noDead, T, F, weekAfterTheHour(), F, F, 0},
"prev ran but done, long overdue, no deadline, F": {f, F, onTheHour, noDead, T, F, weekAfterTheHour(), F, F, 0},
"prev ran but done, long overdue, past medium deadline, A": {A, F, onTheHour, mediumDead, T, F, weekAfterTheHour(), T, F, 1},
"prev ran but done, long overdue, past short deadline, A": {A, F, onTheHour, shortDead, T, F, weekAfterTheHour(), T, F, 1},
"prev ran but done, long overdue, past medium deadline, R": {R, F, onTheHour, mediumDead, T, F, weekAfterTheHour(), T, F, 1},
"prev ran but done, long overdue, past short deadline, R": {R, F, onTheHour, shortDead, T, F, weekAfterTheHour(), T, F, 1},
"prev ran but done, long overdue, past medium deadline, F": {f, F, onTheHour, mediumDead, T, F, weekAfterTheHour(), T, F, 1},
"prev ran but done, long overdue, past short deadline, F": {f, F, onTheHour, shortDead, T, F, weekAfterTheHour(), T, F, 1},
}
for name, tc := range testCases {
sj := cronJob()

10
pkg/controller/cronjob/utils.go
View File

@ -125,6 +125,7 @@ func getRecentUnmetScheduleTimes(sj batch.CronJob, now time.Time) ([]time.Time,
if err != nil {
return starts, fmt.Errorf("Unparseable schedule: %s : %s", sj.Spec.Schedule, err)
}
var earliestTime time.Time
if sj.Status.LastScheduleTime != nil {
earliestTime = sj.Status.LastScheduleTime.Time
@ -137,7 +138,14 @@ func getRecentUnmetScheduleTimes(sj batch.CronJob, now time.Time) ([]time.Time,
// CronJob as last known start time.
earliestTime = sj.ObjectMeta.CreationTimestamp.Time
}
if sj.Spec.StartingDeadlineSeconds != nil {
// Controller is not going to schedule anything below this point
schedulingDeadline := now.Add(-time.Second * time.Duration(*sj.Spec.StartingDeadlineSeconds))
if schedulingDeadline.After(earliestTime) {
earliestTime = schedulingDeadline
}
}
if earliestTime.After(now) {
return []time.Time{}, nil
}
@ -163,7 +171,7 @@ func getRecentUnmetScheduleTimes(sj batch.CronJob, now time.Time) ([]time.Time,
// but less than "lots".
if len(starts) > 100 {
// We can't get the most recent times so just return an empty slice
return []time.Time{}, fmt.Errorf("Too many missed start times to list")
return []time.Time{}, fmt.Errorf("Too many missed start time (> 100). Set or decrease .spec.startingDeadlineSeconds or check clock skew.")
}
}
return starts, nil

15
pkg/controller/cronjob/utils_test.go
View File

@ -364,7 +364,7 @@ func TestGetRecentUnmetScheduleTimes(t *testing.T) {
}
}
{
// Case 6: now is way way ahead of last start time.
// Case 6: now is way way ahead of last start time, and there is no deadline.
sj.ObjectMeta.CreationTimestamp = metav1.Time{Time: T1.Add(-2 * time.Hour)}
sj.Status.LastScheduleTime = &metav1.Time{Time: T1.Add(-1 * time.Hour)}
now := T2.Add(10 * 24 * time.Hour)
@ -373,5 +373,18 @@ func TestGetRecentUnmetScheduleTimes(t *testing.T) {
t.Errorf("unexpected lack of error")
}
}
{
// Case 7: now is way way ahead of last start time, but there is a short deadline.
sj.ObjectMeta.CreationTimestamp = metav1.Time{Time: T1.Add(-2 * time.Hour)}
sj.Status.LastScheduleTime = &metav1.Time{Time: T1.Add(-1 * time.Hour)}
now := T2.Add(10 * 24 * time.Hour)
// Deadline is short
deadline := int64(2 * 60 * 60)
sj.Spec.StartingDeadlineSeconds = &deadline
_, err := getRecentUnmetScheduleTimes(sj, now)
if err != nil {
t.Errorf("unexpected error")
}
}
}