Azure's cloud provider VMSS VMs API accesses are mediated through
a cache holding and refreshing all VMSS together.
Due to that we hit VMSSVM.List API more often than we could: an
instance's cache miss or expiration should only require a single
VMSS re-list, while it's currently O(n) relative to the number of
attached Scale Sets.
Under hard pressure (clusters with many attached VMSS that can't all
be listed in one sequence of successive API calls) the controller
manager might be stuck trying to re-list everything from scratch,
then aborting the whole operation; then re-trying and re-triggering
API rate-limits, affecting the whole Subscription.
This patch replaces the global VMSS VMs cache by per-VMSS VMs caches.
Refreshes (VMSS VMs lists) are scoped to the single relevant VMSS; under
severe throttling the various caches can be incrementally refreshed.
Signed-off-by: Benjamin Pineau <benjamin.pineau@datadoghq.com>
The `default-go-version` field specifies the go version used for the
master branch, and if the go version is not explicitly specified for a
release branch.
This commit also uses go 1.14.6 for the `release-1.19` branch.
Updated etcd to v3.4.10 to include this fix:
- change protobuf field type from int to int64
This should fix increased flakyness in a lot of node e2e tests.
Introduce min, average, and standard deviation for the number of
executing mutating and readOnly requests.
Introduce min, max, average, and standard deviation for the number
waiting and number waiting per priority level.
Later:
Revised to use a series of windows
Use three individuals instead of array of powers
Later:
Add coarse queue count metrics, removed windowed avg and stddev
Add metrics for number of queued mutating and readOnly requests,
to complement metrics for number executing.
Later:
Removed windowed average and standard deviation because consumers can
derive such from integrals of consumer's chosen window.
Also replaced "requestKind" Prometheus label with "request_kind".
Later:
Revised to focus on sampling
Make the clock intrinsic to a TimedObserver
... so that the clock can be read while holding the observer's lock;
otherwise, forward progress is not guaranteed (and violations were
observed in testing).
Bug fixes and histogram buckets revision
SetX1 to 1 when queue length limit is zero, beause dividing by zero is nasty.
Remove obsolete argument in gen_test.go.
Add a bucket boundary at 0 for sample-and-water-mark histograms, to
distinguish zeroes from non-zeros.
This includes adding Integrator test.
Simplified test code.
More pervasively used "ctlr" instead of "ctl" as abbreviation for
"controller".
The current design for Fair Queueing for Server Requests has a
problem: if the min-max fair result stays different from an even
division for a long time and no queue involved in the imbalance goes
empty then the imbalance keeps accruing in queue virtual state times.
This commit adds a test that demonstrates the problem.
It also has some other tweaks to make other tests less flaky.
Factor the big scenario-testing func into pieces, with supporting
structs.
Such nodes should not be added to backend pools. This has particular impact
for nodes which are not master nodes (such as nodes running an etcd cluster)
and are incorrectly added to the pool prior to this change.
