When trying to bring up a cluster via kubeadm with these feature gates enabled,
kube-proxy fails because it didn't know about them:
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
featureGates: {"DynamicResourceAllocation":true,"ContextualLogging":true}
runtimeConfig: {"resource.k8s.io/v1alpha1":"true"}
=>
2023-01-20T07:07:54.474966617Z stderr F E0120 07:07:54.474846 1 run.go:74] "command failed" err="failed complete: unrecognized feature gate: ContextualLogging"
The effect of the logging feature gates is minor for kube-proxy, supporting
them is mostly useful for the sake of consistency and to support kubeadm.
ContextForChannel uses a goroutine to transform a channel close to
a context cancel. However, this exposes a synchronization issue if
we want to unify the underlying implementation between contextless
and with context - a ConditionFunc that closes the channel today
expects the behavior that no subsequent conditions will be invoked
(we have a test in wait_test.go TestUntilReturnsImmediately that
verifies this expectation). We can't unify the implementation
without ensuring this property holds.
To do that this commit changes from the goroutine propagation to
implementing context.Context and using stopCh as the Done(). We
then implement Err() by returning context.Canceled and stub the
other methods. Since our context cannot be explicitly cancelled
by users, we cease to return the cancelFn and callers that need
that behavior must wrap the context as normal.
This should be invisible to clients - they would already observe
the same behavior from the context, and the existing error
behavior of Poll* is preserved (which ignores ctx.Err()).
As a side effect, one less goroutine is created making it more
efficient.
run the testcase with `-v` flag will reveal the warning,
e.g.
`W1103 ... Usage of CRI endpoints without URL scheme is deprecated...`
Signed-off-by: Dave Chen <dave.chen@arm.com>
Set the `cri-socket` both in flags and config file will hit errors,
this should not be a valid case to validate in current testcases.
Signed-off-by: Dave Chen <dave.chen@arm.com>
enforceRequirements will run preflight checks, including whether the user
is privileged is not. Because of this, the test will make different assertions
based on the user's UID. However, we don't have UIDs on Windows, so we're asserting
the wrong thing.
This fix addresses the issue.
`genCSRConfig.kubeadmConfig` is possible to be nil if there any error
from the config loading, so access the field should only be done if
there is no error in the previous step.
Signed-off-by: Dave Chen <dave.chen@arm.com>
In the dual-stack case, iptables.NewDualStackProxier and
ipvs.NewDualStackProxier filtered the nodeport addresses values by IP
family before creating the single-stack proxiers. But in the
single-stack case, the kube-proxy startup code just passed the value
to the single-stack proxiers without validation, so they had to
re-check it themselves. Fix that.
Kube-proxy was checking that iptables supports both IPv4 and IPv6 and
falling back to single-stack if not. But it always fell back to the
primary IP family, regardless of which family iptables supported...
Fix it so that if the primary IP family isn't supported then it bails
out entirely.
It was just saying the copy of file failed with `exit status 1`,
no much details for what's going wrong.
Combine the stderr and stdout and show those info will be easier
for us to fix the problem.
Signed-off-by: Dave Chen <dave.chen@arm.com>
The path module has a few different functions:
Clean, Split, Join, Ext, Dir, Base, IsAbs. These functions do not
take into account the OS-specific path separator, meaning that they
won't behave as intended on Windows.
For example, Dir is supposed to return all but the last element of the
path. For the path "C:\some\dir\somewhere", it is supposed to return
"C:\some\dir\", however, it returns ".".
Instead of these functions, the ones in filepath should be used instead.
If we are dry-running, do not attempt to fetch the /version
resource and just return the stored FakeServerVersion,
which is done when constructing the dry-run client in
upgrade/common.go#getClient().
The problem here is that during upgrade
dry-run client reactors are backed by a dynamic client
via NewClientBackedDryRunGetterFromKubeconfig() and
for GetActions there seems to be no analog to
Discovery().Serverversion() resource for a dynamic client(?).
The kubeadm dry run client reactor code is flawed as it assumes
all invoked "get" verb actions can be casted to GetAction.
Apparently that is not the case when Discovery().ServerVersion()
and other discovery calls are made. In such cases the action
type is the bare ActionImpl.
Catch if an action can be casted to ActionImpl and construct a
GetAction from it. GetActionImpl only suppersets ActionImpl with
a Name field (empty string in this case).
Add unit test for Discovery().ServerVersion().
There seems to be a bug where it's not possible to write to
/etc/kubernetes/tmp... at the time of backing up the old kubelet
config.yaml
Also this kubelet config backup only targets "upgrade node"
and it should also target "upgrade apply".
Revert the related changes until a fully working feature
is implemented.
This addresses the TODO item so that the old kubelet config file could
be recovered if something goes wrong.
Signed-off-by: Dave Chen <dave.chen@arm.com>
The --contention-profiling flag enables block profiling by calling
SetBlockProfileRate(). It, however does not call SetMutexProfileFraction
which enables mutex profiling.
Mutex profiling gives us information about lock contention whereas
block profiling on the other hand gives us information on gorotuines
being blocked on sync primitives.
Updating the docs to "block profiling" in order to make it more accurate.
Eventhough a block profile may reflect contention points on locks to some
extent, it also talks about other sources where goroutines may be blocked.
Signed-off-by: Madhav Jivrajani <madhav.jiv@gmail.com>
Create a new conversion Factory interface for CRDs, and split out
NewDelegatingConverter as a standalone package-level function, instead
of being part of CRConverterFactory.
Signed-off-by: Andy Goldstein <andy.goldstein@redhat.com>
Priority is normally set by the admission controller when missing.
For static pods there seems to be some corner cases, as, if priority
is not explicitly set, Graceful Node Shutdown will shutdown
static pods with the other pods that have priority 0 or unset.
Signed-off-by: Etienne Champetier <e.champetier@ateme.com>
These testcases are too vague, they are not test against the scenario
they want, but instead all of them are failed due to client cannot
be created.
`kubeconfig` file is created and mocked the function of `loadConfig` in
order to make those testcases valid.
Signed-off-by: Dave Chen <dave.chen@arm.com>
Dependencies need to be updated to use
github.com/container-orchestrated-devices/container-device-interface.
It's not decided yet whether we will implement Topology support
for DRA or not. Not having any toppology-related code
will help to avoid wrong impression that DRA is used as a hint
provider for the Topology Manager.
The controller uses the exact same logic as the generic ephemeral inline volume
controller, just for inline ResourceClaimTemplate -> ResourceClaim.
In addition, it supports removal of pods from the ReservedFor field when those
pods are known to not need the claim anymore. At the moment, only this special
case is supported. Removal of arbitrary objects would imply granting full read
access to all types to determine whether a) an object is gone and b) if the
current incarnation is the one which is listed in ReservedFor. This may get
added later.
This adds a new resource.k8s.io API group with v1alpha1 as version. It contains
four new types: resource.ResourceClaim, resource.ResourceClass, resource.ResourceClaimTemplate, and
resource.PodScheduling.
This change enables hot reload of encryption config file when api server
flag --encryption-provider-config-automatic-reload is set to true. This
allows the user to change the encryption config file without restarting
kube-apiserver. The change is detected by polling the file and is done
by using fsnotify watcher. When file is updated it's process to generate
new set of transformers and close the old ones.
Signed-off-by: Nilekh Chaudhari <1626598+nilekhc@users.noreply.github.com>
This change adds a flag --encryption-provider-config-automatic-reload
which will be used to drive automatic reloading of the encryption
config at runtime. While this flag is set to true, or when KMS v2
plugins are used without KMS v1 plugins, the /healthz endpoints
associated with said plugins are collapsed into a single endpoint at
/healthz/kms-providers - in this state, it is not possible to
configure exclusions for specific KMS providers while including the
remaining ones - ex: using /readyz?exclude=kms-provider-1 to exclude
a particular KMS is not possible. This single healthz check handles
checking all configured KMS providers. When reloading is enabled
but no KMS providers are configured, it is a no-op.
k8s.io/apiserver does not support dynamic addition and removal of
healthz checks at runtime. Reloading will instead have a single
static healthz check and swap the underlying implementation at
runtime when a config change occurs.
Signed-off-by: Monis Khan <mok@microsoft.com>
This patch adds new Kubelet option topologyManagerPolicyOptions.
To introduce new TopologyManager options, first we need to introduce new
flag called `topology-manager-policy-options` to allow users to modify
behaviour of best-effort and restricted policies.
Signed-off-by: PiotrProkop <pprokop@nvidia.com>
CPUManager is going GA, thus it makes little sense
to keep the names of the internal configuration
variables `Experimental*`.
Trivial rename only.
Signed-off-by: Francesco Romani <fromani@redhat.com>
With graduation of device plugins to GA in 1.26, the feature gate is
enabled by default so `devicePluginEnabled` field no longer needs to
be passed at the time of Container Manager creation.
In addition to that, we remove the `ManagerStub` as it is no longer
needed.
Signed-off-by: Swati Sehgal <swsehgal@redhat.com>
For parity with `init` and `join`, the test file only
covers some basic scenarios and is expected to be expanded
when the config file for `kubeadm reset` is ready.
Signed-off-by: Dave Chen <dave.chen@arm.com>
The init config file fetched from cluster won't have `ignorePreflightErrors`
set, and shouldn't try to reuse it even it's set with any value.
Instead, it should be defined in reset config file and fetch from it when the
reset config is ready.
Signed-off-by: Dave Chen <dave.chen@arm.com>
It was saying `discovery.bootstrapToken` is invalid with empty value, while
it is actually `discovery.bootstrapToken.caCertHashes` invalid.
Signed-off-by: Dave Chen <dave.chen@arm.com>
Currently, there are some unit tests that are failing on Windows due to
various reasons:
- config options not supported on Windows.
- files not closed, which means that they cannot be removed / renamed.
- paths not properly joined (filepath.Join should be used).
- time.Now() is not as precise on Windows, which means that 2
consecutive calls may return the same timestamp.
- different error messages on Windows.
- files have \r\n line endings on Windows.
- /tmp directory being used, which might not exist on Windows. Instead,
the OS-specific Temp directory should be used.
- the default value for Kubelet's EvictionHard field was containing
OS-specific fields. This is now moved, the field is now set during
Kubelet's initialization, after the config file is read.
Kube/proxy, in NodeCIDR local detector mode, uses the node.Spec.PodCIDRs
field to build the Services iptables rules.
The Node object depends on the kubelet, but if kube-proxy runs as a
static pods or as a standalone binary, it is not possible to guarantee
that the values obtained at bootsrap are valid, causing traffic outages.
Kube-proxy has to react on node changes to avoid this problems, it
simply restarts if detect that the node PodCIDRs have changed.
In case that the Node has been deleted, kube-proxy will only log an
error and keep working, since it may break graceful shutdowns of the
node.
Some of the unit tests cannot pass on Windows due to various reasons:
- fsnotify does not have a Windows implementation.
- Proxy Mode IPVS not supported on Windows.
- Seccomp not supported on Windows.
- VolumeMode=Block is not supported on Windows.
- iSCSI volumes are mounted differently on Windows, and iscsiadm is a
Linux utility.
- All certs will be created under the folder of `/etc/kubernetes/tmp/kubeadm-join-dryrunxxx`
if the `dry-run` mode is enabled.
- Try to make each phase idempotent by resetting the cert dir with `dry-run` mode
Signed-off-by: Dave Chen <dave.chen@arm.com>
Previously, the err msg will repeat the token ID of the form "[a-z0-9]{6}"
if the token doesn't match with the desired format.
Signed-off-by: Dave Chen <dave.chen@arm.com>
Fix conversion errors
Changed the order
update
update
fix manaul coversions
keep the global parameter for backward compatibility
Address Wei's comments
Fix an error
Fix issues
Add unit tests for validation
Fix a comment
Address comments
Update comments
fix verifiation errors
Add tests for scheme_test.go
Convert percentageOfNodesToScore to pointer
Fix errors
Resolve conflicts
Fix testing errors
Address Wei's comments
Revert IntPtr to Int changes
Address comments
Not overrite percentageOfNodesToScore
Fix a bug
Fix a bug
change errs to err
Fix a nit
Remove duplication
Address comments
Fix lint warning
Fix an issue
Update comments
Clean up
Address comments
Revert changes to defaults
fix unit test error
Update
Fix tests
Use default PluginConfigs
The phase `kubeadm init phase upload-certs` intends to be used if
secret "kubeadm-certs" is expired and deleted.
But the help msg was saying this command cannot run on its own.
It's fine to run this command on its own to upload local certs to secret.
Signed-off-by: Dave Chen <dave.chen@arm.com>
This change updates the API server code to load the encryption
config once at start up instead of multiple times. Previously the
code would set up the storage transformers and the etcd healthz
checks in separate parse steps. This is problematic for KMS v2 key
ID based staleness checks which need to be able to assert that the
API server has a single view into the KMS plugin's current key ID.
Signed-off-by: Monis Khan <mok@microsoft.com>
- The sub-phases like `kubeadm reset phase cleanup-node` which
could be run independently would be able to support the `dry-run`
mode as well.
- Consistent with the sub-phases which support the `dry-run` mode
already, such as `kubeadm init phase control-plane apiserver`.
- Prepare for the day when each of those sub-phases could be run
independently.
Signed-off-by: Dave Chen <dave.chen@arm.com>
phase `kubeadm init phase kubeconfig/*` must run after the phase
`kubeadm init phase certs/ca` to get the client cert and key singed
by the CA.
print out the way to generate the CA file is more useful than just say
the file cannot be loaded.
Signed-off-by: Dave Chen <dave.chen@arm.com>
wire up feature_gate.go with metrics via AddMetrics method
Change-Id: I9b4f6b04c0f4eb9bcb198b16284393d21c774ad8
wire in metrics to kubernetes components
Change-Id: I6d4ef8b26f149f62b03f32d1658f04f3056fe4dc
rename metric since we're using the value to determine if enabled is true or false
Change-Id: I13a6b6df90a5ffb4b9c5b34fa187562413bea029
Update staging/src/k8s.io/component-base/featuregate/feature_gate.go
Co-authored-by: Jordan Liggitt <jordan@liggitt.net>
The generation of `RunData` for init and join is relative time consuming,
the data should be shared / reused whenever possible.
Signed-off-by: Dave Chen <dave.chen@arm.com>
Hi team, hope u all doing well.
I have checked TODO that to remove "retry" if feasible but it's important i think that it shouldn't be removed because it was used in every file on your repo.
Update idempotency.go
Update idempotency.go
Update idempotency.go
Having to list all packages isn't very manageable and requires constant
maintenance that is easy to miss. For example, test/e2e/framework/daemonset was
created without adding an entry for it. Promptly one file doesn't use the
intended e2edaemonset alias.
A simpler solution is to support matching the import path against a regular
expression and deriving the intended alias by expanding placeholders (${name},
$1, etc.) with the corresponding submatches from the import path. Example:
k8s.io/kubernetes/test/e2e/framework/([^/]*) -> e2e$1
This support is backwards compatible because normal import paths don't contain
characters that are special in a regular expression and normal aliases don't
contain placeholders.
A regular expression must match the entire import path, otherwise it is
skipped.
The `tmp` is created by `kubeadm` but is never removed, the
size is expected to be expanded as time goes by.
Add one bool option to cleanup the `tmp` dir, the flag is
off by default.
Signed-off-by: Dave Chen <dave.chen@arm.com>
This change updates TestAggregatedAPIServer and the related test
server wiring to exercise the full network path between the Kube API
server and the aggregated API server. We now assert that the wardle
API service and Kube API server discovery endpoints are fully healthy.
CRUD operations are performed through the Kube API server to the
wardle API server.
Signed-off-by: Monis Khan <mok@microsoft.com>
Currently, there are some unit tests that are failing on Windows due to
various reasons:
- filepath.IsAbs does not consider "/" or "\" as absolute paths, even
though files can be addressed as such.
- paths not properly joined (filepath.Join should be used).
- files not closed, which means that they cannot be removed / renamed.
- some assertions fail due to slashes / backslashes not matching.
- backslashes need to be escaped in yaml files, or put between ''
instead of "".
Currently, there are some unit tests that are failing on Windows due to
various reasons:
- Windows file permissions do not work the same way as the Linux ones.
- cp does not exist on Windows, and xcopy should be used instead.
- Get-Item does not work for hidden files / folders like AppData, but
works if given the -Force flag.
If the user passes "--proxy-mode ipvs", and it is not possible to use
IPVS, then error out rather than falling back to iptables.
There was never any good reason to be doing fallback; this was
presumably erroneously added to parallel the iptables-to-userspace
fallback (which only existed because we had wanted iptables to be the
default but not all systems could support it).
In particular, if the user passed configuration options for ipvs, then
they presumably *didn't* pass configuration options for iptables, and
so even if the iptables proxy is able to run, it is likely to be
misconfigured.
Back when iptables was first made the default, there were
theoretically some users who wouldn't have been able to support it due
to having an old /sbin/iptables. But kube-proxy no longer does the
things that didn't work with old iptables, and we removed that check a
long time ago. There is also a check for a new-enough kernel version,
but it's checking for a feature which was added in kernel 3.6, and no
one could possibly be running Kubernetes with a kernel that old. So
the fallback code now never actually falls back, so it should just be
removed.
This was implemented partly in server.go and partly in
server_others.go even though even the parts in server.go were totally
linux-specific. Simplify things by putting it all in server_others.go
and get rid of some unnecessary abstraction.
Introduce networking/v1alpha1 api group.
Add `ClusterCIDR` type to networking/v1alpha1 api group, this type
will enable the NodeIPAM controller to support multiple ClusterCIDRs.
This change is to promote local storage capacity isolation feature to GA
At the same time, to allow rootless system disable this feature due to
unable to get root fs, this change introduced a new kubelet config
"localStorageCapacityIsolation". By default it is set to true. For
rootless systems, they can set this configuration to false to disable
the feature. Once it is set, user cannot set ephemeral-storage
request/limit because capacity and allocatable will not be set.
Change-Id: I48a52e737c6a09e9131454db6ad31247b56c000a
Flocker storage plugin removed from k8s codebase.
Flocker, an early external storage plugin in k8s,
has not been in maintenance and their business is
down. As far as I know, the plugin is not being
used anymore.
This PR removes the whole flocker dependency and
codebase from core k8s to reduce potential security
risks and reduce maintenance work from the sig-storage community.
the fix introduced in #110634 also introduces a bug preventing `kubeadm
upgrade plan` from running on nodes having a different `os.Hostname()`
than node name. concretely, for a node `titi.company.ch`,
`os.Hostname()` will return `titi`, while the full node name is actually
`titi.company.ch`. this simple fix uses the `cfg.NodeRegistration.Name`
instead, which fixes the issue on my nodes with a FQDN node name
Keep previous hostname retrieval as fallback for dupURL CRI fix
`getClientSet` is used by both cmd `token` and `reset`, move this
method to cmd utils to decouple it from one specific cmd.
Signed-off-by: Dave Chen <dave.chen@arm.com>
- Run hack/update-codegen.sh
- Run hack/update-generated-device-plugin.sh
- Run hack/update-generated-protobuf.sh
- Run hack/update-generated-runtime.sh
- Run hack/update-generated-swagger-docs.sh
- Run hack/update-openapi-spec.sh
- Run hack/update-gofmt.sh
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
The experimental-watch-progress-notify-interval flag specifies an interval
at which etcd sends data to the kube-api server.
It is used by the WatchBookmark feature which is GA since 1.17.
It will be used by a new WatchList feature which is Alpha since 1.25
In addition to that the feature was graduated to GA (non-experiment) in etcd 3.5 without any code changes
Fix a TODO to plumb an update filter from above in the resource quota
monitor code that was handling update events for quota-able objects,
instead of hard-coding the logic in the resource quota monitor.
Signed-off-by: Andy Goldstein <andy.goldstein@redhat.com>
* kubelet: silence flag output on errors
Currently, the `--help` text is output on kubelet errors. Currently on
my machine this is 280 lines. Typically kubelet is run by systemd or
similar, starting it a loop. This means when an issue is encountered, we
are spammed by 100s of logs per second, masking the real error.
With this PR, the list of all flags is silenced. Users can still access
them by `kubelet --help` as normal. This same `SilenceUsage` is already
set in the api-server command.
* Update cmd/kubelet/app/server.go
Co-authored-by: Paco Xu <paco.xu@daocloud.io>
Co-authored-by: Paco Xu <paco.xu@daocloud.io>
- `cert-dir` could be specified to a value other than the default value
- we have tests that should be executed successfully on the working cluster
Signed-off-by: Dave Chen <dave.chen@arm.com>
It is useful to have the ability to control whether alpha or beta features are
enabled. We can group features under LoggingAlphaOptions and LoggingBetaOptions
because the configuration is designed so that each feature individually must be
enabled via its own option.
Currently, the JSON format itself is beta (graduated in 1.23) but additional
options for it were only added in 1.23 and thus are still alpha:
$ go run ./staging/src/k8s.io/component-base/logs/example/cmd/logger.go --logging-format=json --log-json-split-stream --log-json-info-buffer-size 1M --feature-gates LoggingBetaOptions=false
[format: Forbidden: Log format json is BETA and disabled, see LoggingBetaOptions feature, options.json.splitStream: Forbidden: Feature LoggingAlphaOptions is disabled, options.json.infoBufferSize: Forbidden: Feature LoggingAlphaOptions is disabled]
$ go run ./staging/src/k8s.io/component-base/logs/example/cmd/logger.go --logging-format=json --log-json-split-stream --log-json-info-buffer-size 1M
[options.json.splitStream: Forbidden: Feature LoggingAlphaOptions is disabled, options.json.infoBufferSize: Forbidden: Feature LoggingAlphaOptions is disabled]
This is the same approach that was taken for CPUManagerPolicyAlphaOptions and
CPUManagerPolicyBetaOptions.
In order to test this without modifying the global feature gate in a test file,
ValidateKubeletConfiguration must take a feature gate as argument.
Making the LoggingConfiguration part of the versioned component-base/config API
had the theoretic advantage that components could have offered different
configuration APIs with experimental features limited to alpha versions (for
example, sanitization offered only in a v1alpha1.KubeletConfiguration). Some
components could have decided to only use stable logging options.
In practice, this wasn't done. Furthermore, we don't want different components
to make different choices regarding which logging features they offer to
users. It should always be the same everywhere, for the sake of consistency.
This can be achieved with a saner Go API by dropping the distinction between
internal and external LoggingConfiguration types. Different stability levels of
indidividual fields have to be covered by documentation (done) and potentially
feature gates (not currently done).
Advantages:
- everything related to logging is under component-base/logs;
previously this was scattered across different packages and
different files under "logs" (why some code was in logs/config.go
vs. logs/options.go vs. logs/logs.go always confused me again
and again when coming back to the code):
- long-term config and command line API are clearly separated
into the "api" package underneath that
- logs/logs.go itself only deals with legacy global flags and
logging configuration
- removal of separate Go APIs like logs.BindLoggingFlags and
logs.Options
- LogRegistry becomes an implementation detail, with less code
and less exported functionality (only registration needs to
be exported, querying is internal)
After the removal of the dynamic kubelet configuration feature it became
possible to initialize logging directly after configuration parsing. The
advantage is that logs emitted by
kubeletconfigvalidation.ValidateKubeletConfiguration and
`klog.InfoS("unsupported configuration ...` already use the intended log
output.
After the code was originally added, Run was replaced by RunE. Taking advantage
of that and returning an error is cleaner.
