github/kubernetes
4
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Remove gcp in-tree cloud provider and credential provider

Browse Source 
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
This commit is contained in:
Davanum Srinivas
2024-01-25 08:47:20 -05:00
parent b041e54e70
commit bf268f02a3
381 changed files with 12 additions and 1071357 deletions
Download Patch File Download Diff File Expand all files Collapse all files

206
LICENSES/vendor/cloud.google.com/go/compute/LICENSE generated vendored
View File

@@ -1,206 +0,0 @@
= vendor/cloud.google.com/go/compute licensed under: =
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
= vendor/cloud.google.com/go/compute/LICENSE 3b83ef96387f14655fc854ddc3c6bd57

206
LICENSES/vendor/cloud.google.com/go/compute/metadata/LICENSE generated vendored
View File

@@ -1,206 +0,0 @@
= vendor/cloud.google.com/go/compute/metadata licensed under: =
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
= vendor/cloud.google.com/go/compute/metadata/LICENSE 3b83ef96387f14655fc854ddc3c6bd57

206
LICENSES/vendor/github.com/GoogleCloudPlatform/k8s-cloud-provider/LICENSE generated vendored
View File

@@ -1,206 +0,0 @@
= vendor/github.com/GoogleCloudPlatform/k8s-cloud-provider licensed under: =
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
= vendor/github.com/GoogleCloudPlatform/k8s-cloud-provider/LICENSE 3b83ef96387f14655fc854ddc3c6bd57

206
LICENSES/vendor/github.com/google/s2a-go/LICENSE generated vendored
View File

@@ -1,206 +0,0 @@
= vendor/github.com/google/s2a-go licensed under: =
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
= vendor/github.com/google/s2a-go/LICENSE.md 3b83ef96387f14655fc854ddc3c6bd57

206
LICENSES/vendor/github.com/googleapis/enterprise-certificate-proxy/LICENSE generated vendored
View File

@@ -1,206 +0,0 @@
= vendor/github.com/googleapis/enterprise-certificate-proxy licensed under: =
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
= vendor/github.com/googleapis/enterprise-certificate-proxy/LICENSE 3b83ef96387f14655fc854ddc3c6bd57

31
LICENSES/vendor/github.com/googleapis/gax-go/v2/LICENSE generated vendored
View File

@@ -1,31 +0,0 @@
= vendor/github.com/googleapis/gax-go/v2 licensed under: =
Copyright 2016, Google Inc.
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of Google Inc. nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
= vendor/github.com/googleapis/gax-go/v2/LICENSE 0dd48ae8103725bd7b401261520cdfbb

31
LICENSES/vendor/google.golang.org/api/LICENSE generated vendored
View File

@@ -1,31 +0,0 @@
= vendor/google.golang.org/api licensed under: =
Copyright (c) 2011 Google Inc. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of Google Inc. nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
= vendor/google.golang.org/api/LICENSE a651bb3d8b1c412632e28823bb432b40

27
LICENSES/vendor/google.golang.org/api/internal/third_party/uritemplates/LICENSE generated vendored
View File

@@ -1,27 +0,0 @@
Copyright (c) 2013 Joshua Tacoma. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of Google Inc. nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

32
LICENSES/vendor/gopkg.in/gcfg.v1/LICENSE generated vendored
View File

@@ -1,32 +0,0 @@
= vendor/gopkg.in/gcfg.v1 licensed under: =
Copyright (c) 2012 Péter Surányi. Portions Copyright (c) 2009 The Go
Authors. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of Google Inc. nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
= vendor/gopkg.in/gcfg.v1/LICENSE 13cea479df204c85485b5db6eb1bc9d5

28
LICENSES/vendor/gopkg.in/warnings.v0/LICENSE generated vendored
View File

@@ -1,28 +0,0 @@
= vendor/gopkg.in/warnings.v0 licensed under: =
Copyright (c) 2016 Péter Surányi.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
= vendor/gopkg.in/warnings.v0/LICENSE c6775875c9d604beb22447dfae3d7049

29
cmd/cloud-controller-manager/providers.go
View File

@@ -1,29 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2020 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// The external controller manager is responsible for running controller loops that
// are cloud provider dependent. It uses the API to listen to new events on resources.
package main
import (
// NOTE: Importing all in-tree cloud-providers is not required when
// implementing an out-of-tree cloud-provider.
_ "k8s.io/legacy-cloud-providers/gce"
)

3
cmd/kube-apiserver/app/options/globalflags.go
View File

@@ -32,9 +32,6 @@ import (
func AddCustomGlobalFlags(fs *pflag.FlagSet) {
// Lookup flags in global flag set and re-register the values with our flagset.
// Adds flags from k8s.io/kubernetes/pkg/cloudprovider/providers.
registerLegacyGlobalFlags(fs)
// Adds flags from k8s.io/apiserver/pkg/admission.
globalflag.Register(fs, "default-not-ready-toleration-seconds")
globalflag.Register(fs, "default-unreachable-toleration-seconds")

28
cmd/kube-apiserver/app/options/globalflags_providerless.go
View File

@@ -1,28 +0,0 @@
//go:build providerless
// +build providerless
/*
Copyright 2019 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package options
import (
"github.com/spf13/pflag"
)
func registerLegacyGlobalFlags(fs *pflag.FlagSet) {
// no-op when no legacy providers are compiled in
}

32
cmd/kube-apiserver/app/options/globalflags_providers.go
View File

@@ -1,32 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2019 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package options
import (
"github.com/spf13/pflag"
"k8s.io/component-base/cli/globalflag"
)
func registerLegacyGlobalFlags(fs *pflag.FlagSet) {
globalflag.Register(fs, "cloud-provider-gce-lb-src-cidrs")
globalflag.Register(fs, "cloud-provider-gce-l7lb-src-cidrs")
fs.MarkDeprecated("cloud-provider-gce-lb-src-cidrs", "This flag will be removed once the GCE Cloud Provider is removed from kube-apiserver")
}

1
cmd/kube-controller-manager/app/controllermanager.go
View File

@@ -159,7 +159,6 @@ controller, and serviceaccounts controller.`,
namedFlagSets := s.Flags(KnownControllers(), ControllersDisabledByDefault(), ControllerAliases())
verflag.AddFlags(namedFlagSets.FlagSet("global"))
globalflag.AddGlobalFlags(namedFlagSets.FlagSet("global"), cmd.Name(), logs.SkipLoggingConfigurationFlags())
registerLegacyGlobalFlags(namedFlagSets)
for _, f := range namedFlagSets.FlagSets {
fs.AddFlagSet(f)
}

28
cmd/kube-controller-manager/app/flags_providerless.go
View File

@@ -1,28 +0,0 @@
//go:build providerless
// +build providerless
/*
Copyright 2019 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package app
import (
cliflag "k8s.io/component-base/cli/flag"
)
func registerLegacyGlobalFlags(namedFlagSets cliflag.NamedFlagSets) {
// no-op when legacy cloud providers are not compiled
}

32
cmd/kube-controller-manager/app/flags_providers.go
View File

@@ -1,32 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2019 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package app
import (
cliflag "k8s.io/component-base/cli/flag"
"k8s.io/component-base/cli/globalflag"
)
func registerLegacyGlobalFlags(namedFlagSets cliflag.NamedFlagSets) {
// hoist this flag from the global flagset to preserve the commandline until
// the gce cloudprovider is removed.
globalflag.Register(namedFlagSets.FlagSet("generic"), "cloud-provider-gce-lb-src-cidrs")
namedFlagSets.FlagSet("generic").MarkDeprecated("cloud-provider-gce-lb-src-cidrs", "This flag will be removed once the GCE Cloud Provider is removed from kube-controller-manager")
}

18
cmd/kube-controller-manager/app/plugins.go
View File

@@ -38,7 +38,6 @@ import (
"k8s.io/kubernetes/pkg/volume/nfs"
volumeutil "k8s.io/kubernetes/pkg/volume/util"
utilfeature "k8s.io/apiserver/pkg/util/feature"
persistentvolumeconfig "k8s.io/kubernetes/pkg/controller/volume/persistentvolume/config"
"k8s.io/utils/exec"
)
@@ -48,12 +47,7 @@ import (
// The list of plugins is manually compiled. This code and the plugin
// initialization code for kubelet really, really need a through refactor.
func ProbeAttachableVolumePlugins(logger klog.Logger) ([]volume.VolumePlugin, error) {
var err error
allPlugins := []volume.VolumePlugin{}
allPlugins, err = appendAttachableLegacyProviderVolumes(logger, allPlugins, utilfeature.DefaultFeatureGate)
if err != nil {
return allPlugins, err
}
allPlugins = append(allPlugins, fc.ProbeVolumePlugins()...)
allPlugins = append(allPlugins, iscsi.ProbeVolumePlugins()...)
allPlugins = append(allPlugins, csi.ProbeVolumePlugins()...)
@@ -69,12 +63,7 @@ func GetDynamicPluginProber(config persistentvolumeconfig.VolumeConfiguration) v
// ProbeExpandableVolumePlugins returns volume plugins which are expandable
func ProbeExpandableVolumePlugins(logger klog.Logger, config persistentvolumeconfig.VolumeConfiguration) ([]volume.VolumePlugin, error) {
var err error
allPlugins := []volume.VolumePlugin{}
allPlugins, err = appendExpandableLegacyProviderVolumes(logger, allPlugins, utilfeature.DefaultFeatureGate)
if err != nil {
return allPlugins, err
}
allPlugins = append(allPlugins, fc.ProbeVolumePlugins()...)
return allPlugins, nil
}
@@ -116,13 +105,6 @@ func ProbeControllerVolumePlugins(logger klog.Logger, config persistentvolumecon
klog.FlushAndExit(klog.ExitFlushTimeout, 1)
}
allPlugins = append(allPlugins, nfs.ProbeVolumePlugins(nfsConfig)...)
var err error
allPlugins, err = appendExpandableLegacyProviderVolumes(logger, allPlugins, utilfeature.DefaultFeatureGate)
if err != nil {
return allPlugins, err
}
allPlugins = append(allPlugins, local.ProbeVolumePlugins()...)
allPlugins = append(allPlugins, csi.ProbeVolumePlugins()...)

42
cmd/kube-controller-manager/app/plugins_providerless.go
View File

@@ -1,42 +0,0 @@
//go:build providerless
// +build providerless
/*
Copyright 2019 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package app
import (
"k8s.io/component-base/featuregate"
"k8s.io/klog/v2"
"k8s.io/kubernetes/pkg/volume"
)
func appendAttachableLegacyProviderVolumes(logger klog.Logger, allPlugins []volume.VolumePlugin, featureGate featuregate.FeatureGate) ([]volume.VolumePlugin, error) {
// no-op when compiled without legacy cloud providers
return allPlugins, nil
}
func appendExpandableLegacyProviderVolumes(logger klog.Logger, allPlugins []volume.VolumePlugin, featureGate featuregate.FeatureGate) ([]volume.VolumePlugin, error) {
// no-op when compiled without legacy cloud providers
return allPlugins, nil
}
func appendLegacyProviderVolumes(logger klog.Logger, allPlugins []volume.VolumePlugin, featureGate featuregate.FeatureGate) ([]volume.VolumePlugin, error) {
// no-op when compiled without legacy cloud providers
return allPlugins, nil
}

80
cmd/kube-controller-manager/app/plugins_providers.go
View File

@@ -1,80 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2019 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package app
import (
"k8s.io/component-base/featuregate"
"k8s.io/csi-translation-lib/plugins"
"k8s.io/klog/v2"
"k8s.io/kubernetes/pkg/features"
"k8s.io/kubernetes/pkg/volume"
"k8s.io/kubernetes/pkg/volume/csimigration"
"k8s.io/kubernetes/pkg/volume/portworx"
"k8s.io/kubernetes/pkg/volume/rbd"
)
type probeFn func() []volume.VolumePlugin
func appendPluginBasedOnFeatureFlags(logger klog.Logger, plugins []volume.VolumePlugin, inTreePluginName string, featureGate featuregate.FeatureGate, pluginInfo pluginInfo) ([]volume.VolumePlugin, error) {
_, err := csimigration.CheckMigrationFeatureFlags(featureGate, pluginInfo.pluginMigrationFeature, pluginInfo.pluginUnregisterFeature)
if err != nil {
logger.Error(err, "Unexpected CSI Migration Feature Flags combination detected. CSI Migration may not take effect")
klog.FlushAndExit(klog.ExitFlushTimeout, 1)
// TODO: fail and return here once alpha only tests can set the feature flags for a plugin correctly
}
// Skip appending the in-tree plugin to the list of plugins to be probed/initialized
// if the plugin unregister feature flag is set
if featureGate.Enabled(pluginInfo.pluginUnregisterFeature) {
logger.Info("Skip registration of plugin since feature flag is enabled", "plugin", inTreePluginName, "feature", pluginInfo.pluginUnregisterFeature)
return plugins, nil
}
plugins = append(plugins, pluginInfo.pluginProbeFunction()...)
return plugins, nil
}
type pluginInfo struct {
pluginMigrationFeature featuregate.Feature
pluginUnregisterFeature featuregate.Feature
pluginProbeFunction probeFn
}
func appendAttachableLegacyProviderVolumes(logger klog.Logger, allPlugins []volume.VolumePlugin, featureGate featuregate.FeatureGate) ([]volume.VolumePlugin, error) {
pluginMigrationStatus := make(map[string]pluginInfo)
pluginMigrationStatus[plugins.PortworxVolumePluginName] = pluginInfo{pluginMigrationFeature: features.CSIMigrationPortworx, pluginUnregisterFeature: features.InTreePluginPortworxUnregister, pluginProbeFunction: portworx.ProbeVolumePlugins}
pluginMigrationStatus[plugins.RBDVolumePluginName] = pluginInfo{pluginMigrationFeature: features.CSIMigrationRBD, pluginUnregisterFeature: features.InTreePluginRBDUnregister, pluginProbeFunction: rbd.ProbeVolumePlugins}
var err error
for pluginName, pluginInfo := range pluginMigrationStatus {
allPlugins, err = appendPluginBasedOnFeatureFlags(logger, allPlugins, pluginName, featureGate, pluginInfo)
if err != nil {
return allPlugins, err
}
}
return allPlugins, nil
}
func appendExpandableLegacyProviderVolumes(logger klog.Logger, allPlugins []volume.VolumePlugin, featureGate featuregate.FeatureGate) ([]volume.VolumePlugin, error) {
return appendLegacyProviderVolumes(logger, allPlugins, featureGate)
}
func appendLegacyProviderVolumes(logger klog.Logger, allPlugins []volume.VolumePlugin, featureGate featuregate.FeatureGate) ([]volume.VolumePlugin, error) {
return appendAttachableLegacyProviderVolumes(logger, allPlugins, featureGate)
}

3
cmd/kubelet/app/options/globalflags.go
View File

@@ -27,9 +27,6 @@ import (
// libs that provide registration functions
"k8s.io/component-base/logs"
"k8s.io/component-base/version/verflag"
// ensure libs have a chance to globally register their flags
_ "k8s.io/kubernetes/pkg/credentialprovider/gcp"
)
// AddGlobalFlags explicitly registers flags that libraries (glog, verflag, etc.) register

28
cmd/kubelet/app/options/globalflags_providerless.go
View File

@@ -1,28 +0,0 @@
//go:build providerless
// +build providerless
/*
Copyright 2020 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package options
import (
"github.com/spf13/pflag"
)
func addLegacyCloudProviderCredentialProviderFlags(global, local *pflag.FlagSet) {
// no-op when no legacy providers are compiled in
}

5
cmd/kubelet/app/plugins.go
View File

@@ -52,11 +52,6 @@ func ProbeVolumePlugins(featureGate featuregate.FeatureGate) ([]volume.VolumePlu
//
// Kubelet does not currently need to configure volume plugins.
// If/when it does, see kube-controller-manager/app/plugins.go for example of using volume.VolumeConfig
var err error
allPlugins, err = appendLegacyProviderVolumes(allPlugins, featureGate)
if err != nil {
return allPlugins, err
}
allPlugins = append(allPlugins, emptydir.ProbeVolumePlugins()...)
allPlugins = append(allPlugins, git_repo.ProbeVolumePlugins()...)
allPlugins = append(allPlugins, hostpath.ProbeVolumePlugins(volume.VolumeConfig{})...)

31
cmd/kubelet/app/plugins_providerless.go
View File

@@ -1,31 +0,0 @@
//go:build providerless
// +build providerless
/*
Copyright 2019 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package app
import (
"k8s.io/component-base/featuregate"
"k8s.io/kubernetes/pkg/volume"
)
func appendLegacyProviderVolumes(allPlugins []volume.VolumePlugin, featureGate featuregate.FeatureGate) ([]volume.VolumePlugin, error) {
// no-op when we didn't compile in support for these
return allPlugins, nil
}

75
cmd/kubelet/app/plugins_providers.go
View File

@@ -1,75 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2019 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package app
import (
// Credential providers
_ "k8s.io/kubernetes/pkg/credentialprovider/gcp"
"k8s.io/component-base/featuregate"
"k8s.io/csi-translation-lib/plugins"
"k8s.io/klog/v2"
"k8s.io/kubernetes/pkg/features"
"k8s.io/kubernetes/pkg/volume"
"k8s.io/kubernetes/pkg/volume/csimigration"
"k8s.io/kubernetes/pkg/volume/portworx"
"k8s.io/kubernetes/pkg/volume/rbd"
)
type probeFn func() []volume.VolumePlugin
func appendPluginBasedOnFeatureFlags(plugins []volume.VolumePlugin, inTreePluginName string,
featureGate featuregate.FeatureGate, pluginInfo pluginInfo) ([]volume.VolumePlugin, error) {
_, err := csimigration.CheckMigrationFeatureFlags(featureGate, pluginInfo.pluginMigrationFeature, pluginInfo.pluginUnregisterFeature)
if err != nil {
klog.InfoS("Unexpected CSI Migration Feature Flags combination detected, CSI Migration may not take effect", "err", err)
// TODO: fail and return here once alpha only tests can set the feature flags for a plugin correctly
}
// Skip appending the in-tree plugin to the list of plugins to be probed/initialized
// if the plugin unregister feature flag is set
if featureGate.Enabled(pluginInfo.pluginUnregisterFeature) {
klog.InfoS("Skipped registration of plugin since feature flag is enabled", "pluginName", inTreePluginName, "featureFlag", pluginInfo.pluginUnregisterFeature)
return plugins, nil
}
plugins = append(plugins, pluginInfo.pluginProbeFunction()...)
return plugins, nil
}
type pluginInfo struct {
pluginMigrationFeature featuregate.Feature
pluginUnregisterFeature featuregate.Feature
pluginProbeFunction probeFn
}
func appendLegacyProviderVolumes(allPlugins []volume.VolumePlugin, featureGate featuregate.FeatureGate) ([]volume.VolumePlugin, error) {
pluginMigrationStatus := make(map[string]pluginInfo)
pluginMigrationStatus[plugins.PortworxVolumePluginName] = pluginInfo{pluginMigrationFeature: features.CSIMigrationPortworx, pluginUnregisterFeature: features.InTreePluginPortworxUnregister, pluginProbeFunction: portworx.ProbeVolumePlugins}
pluginMigrationStatus[plugins.RBDVolumePluginName] = pluginInfo{pluginMigrationFeature: features.CSIMigrationRBD, pluginUnregisterFeature: features.InTreePluginRBDUnregister, pluginProbeFunction: rbd.ProbeVolumePlugins}
var err error
for pluginName, pluginInfo := range pluginMigrationStatus {
allPlugins, err = appendPluginBasedOnFeatureFlags(allPlugins, pluginName, featureGate, pluginInfo)
if err != nil {
return allPlugins, err
}
}
return allPlugins, nil
}

10
go.mod
View File

@@ -10,7 +10,6 @@ go 1.22.0
require (
bitbucket.org/bertimus9/systemstat v0.5.0
github.com/GoogleCloudPlatform/k8s-cloud-provider v1.18.1-0.20220218231025-f11817397a1b
github.com/JeffAshton/win_pdh v0.0.0-20161109143554-76bb4ee9f0ab
github.com/Microsoft/go-winio v0.6.0
github.com/Microsoft/hcsshim v0.8.25
@@ -82,7 +81,6 @@ require (
golang.org/x/term v0.18.0
golang.org/x/time v0.3.0
golang.org/x/tools v0.18.0
google.golang.org/api v0.126.0
google.golang.org/genproto/googleapis/rpc v0.0.0-20230822172742-b8732ec3820d
google.golang.org/grpc v1.59.0
google.golang.org/protobuf v1.33.0
@@ -114,7 +112,6 @@ require (
k8s.io/kube-scheduler v0.0.0
k8s.io/kubectl v0.0.0
k8s.io/kubelet v0.0.0
k8s.io/legacy-cloud-providers v0.0.0
k8s.io/metrics v0.0.0
k8s.io/mount-utils v0.0.0
k8s.io/pod-security-admission v0.0.0
@@ -127,8 +124,6 @@ require (
)
require (
cloud.google.com/go/compute v1.23.0 // indirect
cloud.google.com/go/compute/metadata v0.2.3 // indirect
github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 // indirect
github.com/MakeNowJust/heredoc v1.0.0 // indirect
github.com/NYTimes/gziphandler v1.1.1 // indirect
@@ -163,10 +158,7 @@ require (
github.com/golang-jwt/jwt/v4 v4.5.0 // indirect
github.com/google/btree v1.0.1 // indirect
github.com/google/pprof v0.0.0-20210720184732-4bb14d4b1be1 // indirect
github.com/google/s2a-go v0.1.7 // indirect
github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.2.3 // indirect
github.com/googleapis/gax-go/v2 v2.11.0 // indirect
github.com/gorilla/websocket v1.5.0 // indirect
github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7 // indirect
github.com/grpc-ecosystem/go-grpc-middleware v1.3.0 // indirect
@@ -223,10 +215,8 @@ require (
google.golang.org/appengine v1.6.7 // indirect
google.golang.org/genproto v0.0.0-20230822172742-b8732ec3820d // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20230822172742-b8732ec3820d // indirect
gopkg.in/gcfg.v1 v1.2.3 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect
gopkg.in/warnings.v0 v0.1.2 // indirect
k8s.io/gengo/v2 v2.0.0-20240228010128-51d4e06bde70 // indirect
sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.29.0 // indirect
sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect

316
go.sum
View File

@@ -8,26 +8,7 @@ cloud.google.com/go v0.44.1/go.mod h1:iSa0KzasP4Uvy3f1mN/7PiObzGgflwredwwASm/v6A
cloud.google.com/go v0.44.2/go.mod h1:60680Gw3Yr4ikxnPRS/oxxkBccT6SA1yMk63TGekxKY=
cloud.google.com/go v0.45.1/go.mod h1:RpBamKRgapWJb87xiFSdk4g1CME7QZg3uwTez+TSTjc=
cloud.google.com/go v0.46.3/go.mod h1:a6bKKbmY7er1mI7TEI4lsAkts/mkhTSZK8w33B4RAg0=
cloud.google.com/go v0.50.0/go.mod h1:r9sluTvynVuxRIOHXQEHMFffphuXHOMZMycpNR5e6To=
cloud.google.com/go v0.52.0/go.mod h1:pXajvRH/6o3+F9jDHZWQ5PbGhn+o8w9qiu/CffaVdO4=
cloud.google.com/go v0.53.0/go.mod h1:fp/UouUEsRkN6ryDKNW/Upv/JBKnv6WDthjR6+vze6M=
cloud.google.com/go v0.54.0/go.mod h1:1rq2OEkV3YMf6n/9ZvGWI3GWw0VoqH/1x2nd8Is/bPc=
cloud.google.com/go v0.56.0/go.mod h1:jr7tqZxxKOVYizybht9+26Z/gUq7tiRzu+ACVAMbKVk=
cloud.google.com/go v0.57.0/go.mod h1:oXiQ6Rzq3RAkkY7N6t3TcE6jE+CIBBbA36lwQ1JyzZs=
cloud.google.com/go v0.62.0/go.mod h1:jmCYTdRCQuc1PHIIJ/maLInMho30T/Y0M4hTdTShOYc=
cloud.google.com/go v0.65.0/go.mod h1:O5N8zS7uWy9vkA9vayVHs65eM1ubvY4h553ofrNHObY=
cloud.google.com/go v0.72.0/go.mod h1:M+5Vjvlc2wnp6tjzE102Dw08nGShTscUx2nZMufOKPI=
cloud.google.com/go v0.74.0/go.mod h1:VV1xSbzvo+9QJOxLDaJfTjx5e+MePCpCWwvftOeQmWk=
cloud.google.com/go v0.78.0/go.mod h1:QjdrLG0uq+YwhjoVOLsS1t7TW8fs36kLs4XO5R5ECHg=
cloud.google.com/go v0.79.0/go.mod h1:3bzgcEeQlzbuEAYu4mrWhKqWjmpprinYgKJLgKHnbb8=
cloud.google.com/go v0.81.0/go.mod h1:mk/AM35KwGk/Nm2YSeZbxXdrNK3KZOYHmLkOqC2V6E0=
cloud.google.com/go v0.83.0/go.mod h1:Z7MJUsANfY0pYPdw0lbnivPx4/vhy/e2FEkSkF7vAVY=
cloud.google.com/go v0.84.0/go.mod h1:RazrYuxIK6Kb7YrzzhPoLmCVzl7Sup4NrbKPg8KHSUM=
cloud.google.com/go v0.87.0/go.mod h1:TpDYlFy7vuLzZMMZ+B6iRiELaY7z/gJPaqbMx6mlWcY=
cloud.google.com/go v0.90.0/go.mod h1:kRX0mNRHe0e2rC6oNakvwQqzyDmg57xJ+SZU1eT2aDQ=
cloud.google.com/go v0.93.3/go.mod h1:8utlLll2EF5XMAV15woO4lSbWQlk8rer9aLOfLh7+YI=
cloud.google.com/go v0.94.1/go.mod h1:qAlAugsXlC+JWO+Bke5vCtc9ONxjQT3drlTTnAplMW4=
cloud.google.com/go v0.97.0/go.mod h1:GF7l59pYBVlXQIBLx3a761cZ41F9bBH3JUlihCt2Udc=
cloud.google.com/go v0.110.7 h1:rJyC7nWRg2jWGZ4wSJ5nY65GTdYJkg0cd/uXb+ACI6o=
cloud.google.com/go v0.110.7/go.mod h1:+EYjdK8e5RME/VY/qLCAtuyALQ9q67dvuum8i+H5xsI=
cloud.google.com/go/accessapproval v1.7.1/go.mod h1:JYczztsHRMK7NTXb6Xw+dwbs/WnOJxbo/2mTI+Kgg68=
cloud.google.com/go/accesscontextmanager v1.8.1/go.mod h1:JFJHfvuaTC+++1iL1coPiG1eu5D24db2wXCDWDjIrxo=
@@ -46,11 +27,6 @@ cloud.google.com/go/baremetalsolution v1.1.1/go.mod h1:D1AV6xwOksJMV4OSlWHtWuFNZ
cloud.google.com/go/batch v1.3.1/go.mod h1:VguXeQKXIYaeeIYbuozUmBR13AfL4SJP7IltNPS+A4A=
cloud.google.com/go/beyondcorp v1.0.0/go.mod h1:YhxDWw946SCbmcWo3fAhw3V4XZMSpQ/VYfcKGAEU8/4=
cloud.google.com/go/bigquery v1.0.1/go.mod h1:i/xbL2UlR5RvWAURpBYZTtm/cXjCha9lbfbpx4poX+o=
cloud.google.com/go/bigquery v1.3.0/go.mod h1:PjpwJnslEMmckchkHFfq+HTD2DmtT67aNFKH1/VBDHE=
cloud.google.com/go/bigquery v1.4.0/go.mod h1:S8dzgnTigyfTmLBfrtrhyYhwRxG72rYxvftPBK2Dvzc=
cloud.google.com/go/bigquery v1.5.0/go.mod h1:snEHRnqQbz117VIFhE8bmtwIDY80NLUZUMb4Nv6dBIg=
cloud.google.com/go/bigquery v1.7.0/go.mod h1://okPTzCYNXSlb24MZs83e2Do+h+VXtc4gLoIoXIAPc=
cloud.google.com/go/bigquery v1.8.0/go.mod h1:J5hqkt3O0uAFnINi6JXValWIb1v0goeZM77hZzJN/fQ=
cloud.google.com/go/bigquery v1.53.0/go.mod h1:3b/iXjRQGU4nKa87cXeg6/gogLjO8C6PmuM8i5Bi/u4=
cloud.google.com/go/billing v1.16.0/go.mod h1:y8vx09JSSJG02k5QxbycNRrN7FGZB6F3CAcgum7jvGA=
cloud.google.com/go/binaryauthorization v1.6.1/go.mod h1:TKt4pa8xhowwffiBmbrbcxijJRZED4zrqnwZ1lKH51U=
@@ -75,7 +51,6 @@ cloud.google.com/go/dataplex v1.9.0/go.mod h1:7TyrDT6BCdI8/38Uvp0/ZxBslOslP2X2MP
cloud.google.com/go/dataproc/v2 v2.0.1/go.mod h1:7Ez3KRHdFGcfY7GcevBbvozX+zyWGcwLJvvAMwCaoZ4=
cloud.google.com/go/dataqna v0.8.1/go.mod h1:zxZM0Bl6liMePWsHA8RMGAfmTG34vJMapbHAxQ5+WA8=
cloud.google.com/go/datastore v1.0.0/go.mod h1:LXYbyblFSglQ5pkeyhO+Qmw7ukd3C+pD7TKLgZqpHYE=
cloud.google.com/go/datastore v1.1.0/go.mod h1:umbIZjpQpHh4hmRpGhH4tLFup+FVzqBi1b3c64qFpCk=
cloud.google.com/go/datastore v1.13.0/go.mod h1:KjdB88W897MRITkvWWJrg2OUtrR5XVj1EoLgSp6/N70=
cloud.google.com/go/datastream v1.10.0/go.mod h1:hqnmr8kdUBmrnk65k5wNRoHSCYksvpdZIcZIEl8h43Q=
cloud.google.com/go/deploy v1.13.0/go.mod h1:tKuSUV5pXbn67KiubiUNUejqLs4f5cxxiCNCeyl0F2g=
@@ -124,9 +99,6 @@ cloud.google.com/go/phishingprotection v0.8.1/go.mod h1:AxonW7GovcA8qdEk13NfHq9h
cloud.google.com/go/policytroubleshooter v1.8.0/go.mod h1:tmn5Ir5EToWe384EuboTcVQT7nTag2+DuH3uHmKd1HU=
cloud.google.com/go/privatecatalog v0.9.1/go.mod h1:0XlDXW2unJXdf9zFz968Hp35gl/bhF4twwpXZAW50JA=
cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I=
cloud.google.com/go/pubsub v1.1.0/go.mod h1:EwwdRX2sKPjnvnqCa270oGRyludottCI76h+R3AArQw=
cloud.google.com/go/pubsub v1.2.0/go.mod h1:jhfEVHT8odbXTkndysNHCcx0awwzvfOlguIAii9o8iA=
cloud.google.com/go/pubsub v1.3.1/go.mod h1:i+ucay31+CNRpDW4Lu78I4xXG+O1r/MAHgjpRVR+TSU=
cloud.google.com/go/pubsub v1.33.0/go.mod h1:f+w71I33OMyxf9VpMVcZbnG5KSUkCOUHYpFd5U1GdRc=
cloud.google.com/go/pubsublite v1.8.1/go.mod h1:fOLdU4f5xldK4RGJrBMm+J7zMWNj/k4PxwEZXy39QS0=
cloud.google.com/go/recaptchaenterprise/v2 v2.7.2/go.mod h1:kR0KjsJS7Jt1YSyWFkseQ756D45kaYNTlDPPaRAvDBU=
@@ -146,10 +118,6 @@ cloud.google.com/go/shell v1.7.1/go.mod h1:u1RaM+huXFaTojTbW4g9P5emOrrmLE69KrxqQ
cloud.google.com/go/spanner v1.47.0/go.mod h1:IXsJwVW2j4UKs0eYDqodab6HgGuA1bViSqW4uH9lfUI=
cloud.google.com/go/speech v1.19.0/go.mod h1:8rVNzU43tQvxDaGvqOhpDqgkJTFowBpDvCJ14kGlJYo=
cloud.google.com/go/storage v1.0.0/go.mod h1:IhtSnM/ZTZV8YYJWCY8RULGVqBDmpoyjwiyrjsg+URw=
cloud.google.com/go/storage v1.5.0/go.mod h1:tpKbwo567HUNpVclU5sGELwQWBDZ8gh0ZeosJ0Rtdos=
cloud.google.com/go/storage v1.6.0/go.mod h1:N7U0C8pVQ/+NIKOBQyamJIeKQKkZ+mxpohlUTyfDhBk=
cloud.google.com/go/storage v1.8.0/go.mod h1:Wv1Oy7z6Yz3DshWRJFhqM/UCfaWIRTdp0RXyy7KQOVs=
cloud.google.com/go/storage v1.10.0/go.mod h1:FLPqc6j+Ki4BU591ie1oL6qBQGu2Bl/tZ9ullr3+Kg0=
cloud.google.com/go/storagetransfer v1.10.0/go.mod h1:DM4sTlSmGiNczmV6iZyceIh2dbs+7z2Ayg6YAiQlYfA=
cloud.google.com/go/talent v1.6.2/go.mod h1:CbGvmKCG61mkdjcqTcLOkb2ZN1SrQI8MDyma2l7VD24=
cloud.google.com/go/texttospeech v1.7.1/go.mod h1:m7QfG5IXxeneGqTapXNxv2ItxP/FS0hCZBwXYqucgSk=
@@ -170,8 +138,6 @@ github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 h1:UQHMgLO+TxOEl
github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
github.com/GoogleCloudPlatform/k8s-cloud-provider v1.18.1-0.20220218231025-f11817397a1b h1:Heo1J/ttaQFgGJSVnCZquy3e5eH5j1nqxBuomztB3P0=
github.com/GoogleCloudPlatform/k8s-cloud-provider v1.18.1-0.20220218231025-f11817397a1b/go.mod h1:FNj4KYEAAHfYu68kRYolGoxkaJn+6mdEsaM12VTwuI0=
github.com/JeffAshton/win_pdh v0.0.0-20161109143554-76bb4ee9f0ab h1:UKkYhof1njT1/xq4SEg5z+VpTgjmNeHwPGRQl7takDI=
github.com/JeffAshton/win_pdh v0.0.0-20161109143554-76bb4ee9f0ab/go.mod h1:3VYc5hodBMJ5+l/7J4xAyMeuM2PNuepvHlGs8yilUCA=
github.com/MakeNowJust/heredoc v1.0.0 h1:cXCdzVdstXyiTqTvfqk9SDHpKNjxuom+DOlyEeQ4pzQ=
@@ -229,10 +195,7 @@ github.com/cilium/ebpf v0.9.1 h1:64sn2K3UKw8NbP/blsixRpF3nXuyhz/VjRlRzvlBRu4=
github.com/cilium/ebpf v0.9.1/go.mod h1:+OhNOIXx/Fnu1IE8bJz2dzOA+VSfyTfdNUVdlQnxUFY=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc=
github.com/cncf/udpa/go v0.0.0-20200629203442-efcf912fb354/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk=
github.com/cncf/udpa/go v0.0.0-20201120205902-5459f2c99403/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk=
github.com/cncf/udpa/go v0.0.0-20220112060539-c52dc94e7fbe/go.mod h1:6pvJx4me5XPnfI9Z40ddWsdw2W/uZgQLFXToKeRcDiI=
github.com/cncf/xds/go v0.0.0-20210312221358-fbca930ec8ed/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
github.com/cncf/xds/go v0.0.0-20230607035331-e9ce68804cb4 h1:/inchEIKaYC1Akx+H+gqO04wryn5h75LSazbRlnya1k=
github.com/cncf/xds/go v0.0.0-20230607035331-e9ce68804cb4/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
github.com/cockroachdb/datadriven v1.0.2 h1:H9MtNqVoVhvd9nCBwOyDjUEdZCREqbIdCJD93PBm/jA=
@@ -309,10 +272,6 @@ github.com/emicklei/go-restful/v3 v3.11.0/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRr
github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98=
github.com/envoyproxy/go-control-plane v0.9.7/go.mod h1:cwu0lG7PUMfa9snN8LXBig5ynNVH9qI8YYLbd1fK2po=
github.com/envoyproxy/go-control-plane v0.9.9-0.20201210154907-fd9021fe5dad/go.mod h1:cXg6YxExXjJnVBQHBLXeUAgxn2UodCpnH306RInaBQk=
github.com/envoyproxy/go-control-plane v0.9.9-0.20210217033140-668b12f5399d/go.mod h1:cXg6YxExXjJnVBQHBLXeUAgxn2UodCpnH306RInaBQk=
github.com/envoyproxy/go-control-plane v0.9.9-0.20210512163311-63b5d3c536b0/go.mod h1:hliV/p42l8fGbc6Y9bQ70uLwIvmJyVE5k4iMKlh8wCQ=
github.com/envoyproxy/go-control-plane v0.11.1/go.mod h1:uhMcXKCQMEJHiAb0w+YGefQLaTEw+YhGluxZkrTmD0g=
github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
github.com/envoyproxy/protoc-gen-validate v1.0.2 h1:QkIBuU5k+x7/QXPvPPnWXWlCdaBFApVqftFV6k087DA=
@@ -343,15 +302,12 @@ github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeME
github.com/go-errors/errors v1.4.2 h1:J6MZopCL4uSllY1OfXM374weqZFFItUbrImctkmUxIA=
github.com/go-errors/errors v1.4.2/go.mod h1:sIVyrIiJhuEF+Pj9Ebtd6P/rEYROXFi3BopGUQ5a5Og=
github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
github.com/go-kit/kit v0.9.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
github.com/go-kit/log v0.2.1/go.mod h1:NwTd00d/i8cPZ3xOwwiv2PO5MOcx78fFErGNcVmBjv0=
github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE=
github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk=
github.com/go-logfmt/logfmt v0.5.1/go.mod h1:WYhtIu8zTZfxdn5+rREduYbwxfcBr/Vr6KEVveWlfTs=
github.com/go-logr/logr v0.1.0/go.mod h1:ixOQHD9gLJUVQQ2ZOR7zLEifBX6tGkNJF4QyIY7sIas=
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-logr/logr v1.4.1 h1:pKouT5E8xu9zeFC39JXRDukb6JFQPXM5p5I91188VAQ=
github.com/go-logr/logr v1.4.1/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
@@ -384,24 +340,16 @@ github.com/golang/glog v1.1.2 h1:DVjP2PbBOzHyzA+dn3WhHIq4NdVu3Q+pvivFICf/7fo=
github.com/golang/glog v1.1.2/go.mod h1:zR+okUeTbrL6EL3xHUDxZuEtGv04p5shwip1+mL/rLQ=
github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da h1:oI5xCqsCo564l8iNU+DwB5epxmsaqB+rhGL0m5jtYqE=
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y=
github.com/golang/mock v1.4.0/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw=
github.com/golang/mock v1.4.1/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw=
github.com/golang/mock v1.4.3/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw=
github.com/golang/mock v1.4.4/go.mod h1:l3mdAwkq5BuhzHwde/uurv3sEJeZMXNpwsxVWU71h+4=
github.com/golang/mock v1.5.0/go.mod h1:CWnOUgYIOo4TcNZ0wHX3YZCqsaM1I1Jvs6v3mP3KVu8=
github.com/golang/mock v1.6.0/go.mod h1:p6yTPP+5HYm5mzsMV8JkE6ZKdX+/wYM6Hr+LicevLPs=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.3/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw=
github.com/golang/protobuf v1.3.4/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw=
github.com/golang/protobuf v1.3.5/go.mod h1:6O5/vntMXwX2lRkT1hjjk0nAC1IDOTvTlVgjlRvqsdk=
github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8=
github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA=
@@ -409,14 +357,10 @@ github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrU
github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w=
github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0=
github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8=
github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
github.com/golang/protobuf v1.4.3/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=
github.com/golang/protobuf v1.5.1/go.mod h1:DopwsBzvsk0Fs44TXzsVbJyPhcCPeIwnvohx4u74HPM=
github.com/golang/protobuf v1.5.2/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY=
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
github.com/golang/snappy v0.0.3/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/golangplus/bytes v0.0.0-20160111154220-45c989fe5450/go.mod h1:Bk6SMAONeMXrxql8uvOKuAZSu8aM5RUGv+1C6IJaEho=
github.com/golangplus/bytes v1.0.0/go.mod h1:AdRaCFwmc/00ZzELMWb01soso6W1R/++O1XL80yAn+A=
github.com/golangplus/fmt v1.0.0/go.mod h1:zpM0OfbMCjPtd2qkTD/jX2MgiFCqklhSUFyDW44gVQE=
@@ -436,14 +380,11 @@ github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5a
github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.4.1/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.1/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.3/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.4/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.6/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
@@ -451,41 +392,19 @@ github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/
github.com/google/gofuzz v1.2.0 h1:xRy4A+RhZaiKjJ1bPfwQ8sedCA+YS2YcCHW6ec7JMi0=
github.com/google/gofuzz v1.2.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
github.com/google/martian/v3 v3.0.0/go.mod h1:y5Zk1BBys9G+gd6Jrk0W3cC1+ELVxBWuIGO+w/tUAp0=
github.com/google/martian/v3 v3.1.0/go.mod h1:y5Zk1BBys9G+gd6Jrk0W3cC1+ELVxBWuIGO+w/tUAp0=
github.com/google/martian/v3 v3.2.1/go.mod h1:oBOf6HBosgwRXnUGWUB05QECsc6uvmMiJ3+6W4l/CUk=
github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
github.com/google/pprof v0.0.0-20190515194954-54271f7e092f/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
github.com/google/pprof v0.0.0-20191218002539-d4f498aebedc/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/pprof v0.0.0-20200212024743-f11f1df84d12/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/pprof v0.0.0-20200229191704-1ebb73c60ed3/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/pprof v0.0.0-20200430221834-fc25d7d30c6d/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/pprof v0.0.0-20200708004538-1a94d8640e99/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/pprof v0.0.0-20201023163331-3e6fc7fc9c4c/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE=
github.com/google/pprof v0.0.0-20201203190320-1bf35d6f28c2/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE=
github.com/google/pprof v0.0.0-20210122040257-d980be63207e/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE=
github.com/google/pprof v0.0.0-20210226084205-cbba55b83ad5/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE=
github.com/google/pprof v0.0.0-20210601050228-01bbb1931b22/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE=
github.com/google/pprof v0.0.0-20210609004039-a478d1d731e9/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE=
github.com/google/pprof v0.0.0-20210720184732-4bb14d4b1be1 h1:K6RDEckDVWvDI9JAJYCmNdQXq6neHJOYx3V6jnqNEec=
github.com/google/pprof v0.0.0-20210720184732-4bb14d4b1be1/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE=
github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
github.com/google/s2a-go v0.1.7 h1:60BLSyTrOV4/haCDW4zb1guZItoSq8foHCXrAnjBo/o=
github.com/google/s2a-go v0.1.7/go.mod h1:50CgR4k1jNlWBu4UfS4AcfhVe1r6pdZPygJ3R8F0Qdw=
github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 h1:El6M4kTTCOh6aBiKaUGG7oYTSPP8MxqL4YI3kZKwcP4=
github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510/go.mod h1:pupxD2MaaD3pAXIBCelhxNneeOaAeabZDe5s4K6zSpQ=
github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/google/uuid v1.3.1 h1:KjJaJ9iWZ3jOFZIf1Lqf4laDRCasjl0BCmnEGxkdLb4=
github.com/google/uuid v1.3.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/googleapis/enterprise-certificate-proxy v0.2.3 h1:yk9/cqRKtT9wXZSsRH9aurXEpJX+U6FLtpYTdC3R06k=
github.com/googleapis/enterprise-certificate-proxy v0.2.3/go.mod h1:AwSRAtLfXpU5Nm3pW+v7rGDHp09LsPtGY9MduiEsR9k=
github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg=
github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
github.com/googleapis/gax-go/v2 v2.1.0/go.mod h1:Q3nei7sK6ybPYH7twZdmQpAd1MKb7pfu6SK+H1/DsU0=
github.com/googleapis/gax-go/v2 v2.1.1/go.mod h1:hddJymUZASv3XPyGkUpKj8pPO47Rmb0eJc8R6ouapiM=
github.com/googleapis/gax-go/v2 v2.11.0 h1:9V9PWXEsWnPpQhu/PeQIkS4eGzMlTLGgt80cUUI8Ki4=
github.com/googleapis/gax-go/v2 v2.11.0/go.mod h1:DxmR61SGKkGLa2xigwuZIQpkCI2S5iydzRfb3peWZJI=
github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ=
github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
@@ -523,7 +442,6 @@ github.com/hashicorp/logutils v1.0.0/go.mod h1:QIAnNjmIWmVIIkWDTG1z5v++HQmx9WQRO
github.com/hashicorp/mdns v1.0.0/go.mod h1:tL+uN++7HEJ6SQLQ2/p+z2pH24WQKWjBPkE0mNTz8vQ=
github.com/hashicorp/memberlist v0.1.3/go.mod h1:ajVTdAv/9Im8oMAAj5G31PhhMCZJV2pPBoIllUwCN7I=
github.com/hashicorp/serf v0.8.2/go.mod h1:6hOLApaqBFA1NXqRQAsxw9QxuDEvNxSQRwA/JwenrHc=
github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
github.com/ianlancetaylor/demangle v0.0.0-20200824232613-28f6c0f3b639/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
github.com/imdario/mergo v0.3.6 h1:xTNEAn+kxVO7dTZGu0CegyqKZmoWFI0rF8UxjlB2d28=
github.com/imdario/mergo v0.3.6/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA=
@@ -543,7 +461,6 @@ github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCV
github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU=
github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk=
github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w=
github.com/julienschmidt/httprouter v1.3.0/go.mod h1:JR6WtHb+2LUe8TCKY3cZOxFyyO8IZAc4RVcycCCAKdM=
@@ -746,11 +663,8 @@ github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q
github.com/xlab/treeprint v1.2.0 h1:HzHnuAF1plUN2zGlAFHbSQP2qJ0ZAD3XF5XD7OesXRQ=
github.com/xlab/treeprint v1.2.0/go.mod h1:gj5Gd3gPdKtR1ikdDK6fnFLdmIS0X30kTTuNd/WEJu0=
github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q=
github.com/yuin/goldmark v1.1.25/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.1.32/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.3.5/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k=
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
go.etcd.io/bbolt v1.3.9 h1:8x7aARPEXiXbHmtUwAIv7eV2fQFHrLLavdiJ3uzJXoI=
@@ -772,11 +686,7 @@ go.etcd.io/etcd/server/v3 v3.5.13/go.mod h1:K/8nbsGupHqmr5MkgaZpLlH1QdX1pcNQLAkO
go.etcd.io/gofail v0.1.0/go.mod h1:VZBCXYGZhHAinaBiiqYvuDynvahNsAyLFwB3kEHKz1M=
go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=
go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
go.opencensus.io v0.22.3/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
go.opencensus.io v0.22.4/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
go.opencensus.io v0.22.5/go.mod h1:5pWMHQbX5EPX2/62yrJeAkowc+lfs/XD7Uxpq3pI6kk=
go.opencensus.io v0.23.0/go.mod h1:XItmlyltB5F7CS4xOC1DcqMoFqwtC6OG2xF7mCv7P7E=
go.opencensus.io v0.24.0 h1:y73uSU6J157QMP2kn2r30vwW1A2W2WFwSCGnAVxeaD0=
go.opencensus.io v0.24.0/go.mod h1:vNK8G9p7aAivkbmorf4v+7Hgx+Zs0yY+0fOtgBfjQKo=
go.opentelemetry.io/contrib/instrumentation/github.com/emicklei/go-restful/otelrestful v0.42.0 h1:Z6SbqeRZAl2OczfkFOqLx1BeYBDYehNjEnqluD7581Y=
@@ -799,7 +709,6 @@ go.opentelemetry.io/otel/sdk v1.20.0 h1:5Jf6imeFZlZtKv9Qbo6qt2ZkmWtdWx/wzcCbNUlA
go.opentelemetry.io/otel/sdk v1.20.0/go.mod h1:rmkSx1cZCm/tn16iWDn1GQbLtsW/LvsdEEFzCSRM6V0=
go.opentelemetry.io/otel/trace v1.20.0 h1:+yxVAPZPbQhbC3OfAkeIVTky6iTFpcr4SiY9om7mXSQ=
go.opentelemetry.io/otel/trace v1.20.0/go.mod h1:HJSK7F/hA5RlzpZ0zKDCHCDHm556LCDtKaAo6JmBFUU=
go.opentelemetry.io/proto/otlp v0.7.0/go.mod h1:PqfVotwruBrMGOCsRd/89rSnXhoiJIqeYNgFYFoEGnI=
go.opentelemetry.io/proto/otlp v1.0.0 h1:T0TX0tmXU8a3CbNXzEKGeU5mIVOdf0oykP+u2lIVU/I=
go.opentelemetry.io/proto/otlp v1.0.0/go.mod h1:Sy6pihPLfYHkr3NkUbEhGHFhINUSI/v80hjKIs5JXpM=
go.starlark.net v0.0.0-20230525235612-a134d8f9ddca h1:VdD38733bfYv5tUZwEIskMM93VanwNIi5bIKnDrJdEY=
@@ -830,11 +739,6 @@ golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL
golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
golang.org/x/exp v0.0.0-20190829153037-c13cbed26979/go.mod h1:86+5VVa7VpoJ4kLfm080zCjGlMRFzhUhsZKEZO7MGek=
golang.org/x/exp v0.0.0-20191030013958-a1ab85dbe136/go.mod h1:JXzH8nQsPlswgeRAPE3MuO9GYsAcnJvJ4vnMwN/5qkY=
golang.org/x/exp v0.0.0-20191129062945-2f5052295587/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/exp v0.0.0-20200119233911-0405dc783f0a/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/exp v0.0.0-20200207192155-f17229e696bd/go.mod h1:J/WKrq2StrnmMY6+EHIKF9dgMWnmCNThgcyBT1FY9mM=
golang.org/x/exp v0.0.0-20200224162631-6cc2880d07d6/go.mod h1:3jZMyOhIsHpP37uCMkUooju7aAi5cS1Q23tOzKc+0MU=
golang.org/x/exp v0.0.0-20230515195305-f3d0a9c9a5cc h1:mCRnTeVUjcrhlRmO0VK8a6k6Rrf6TF9htwo2pJVSjIU=
golang.org/x/exp v0.0.0-20230515195305-f3d0a9c9a5cc/go.mod h1:V1LtkGg67GoY2N1AnLN78QLrzxkLyJw7RJb1gzOOz9w=
golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
@@ -846,22 +750,12 @@ golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHl
golang.org/x/lint v0.0.0-20190409202823-959b441ac422/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/lint v0.0.0-20190909230951-414d861bb4ac/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f/go.mod h1:5qLYkcX4OjUUV8bRuDixDT3tpyyb+LUpUlRWLxfhWrs=
golang.org/x/lint v0.0.0-20200130185559-910be7a94367/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY=
golang.org/x/lint v0.0.0-20200302205851-738671d3881b/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY=
golang.org/x/lint v0.0.0-20201208152925-83fdc39ff7b5/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY=
golang.org/x/lint v0.0.0-20210508222113-6edffad5e616/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY=
golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6/go.mod h1:z+o9i4GpDbdi3rU15maQ/Ox0txvL9dWGYEHz965HBQE=
golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028/go.mod h1:E/iHnbuqvinMTCcRqshq8CkpyQDoeVncDDYHnLhea+o=
golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc=
golang.org/x/mod v0.1.0/go.mod h1:0QHyrYULN0/3qlju5TqG8bIK38QM8yzMo5ekMj3DlcY=
golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/mod v0.1.1-0.20191107180719-034126e5016b/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.4.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.4.1/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.15.0 h1:SernR4v+D55NyBH2QiEQrlBAnj1ECL6AGrA5+dPaMY8=
golang.org/x/mod v0.15.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@@ -879,51 +773,18 @@ golang.org/x/net v0.0.0-20190503192946-f4e77d36d62c/go.mod h1:t9HGtf8HONx5eT2rtn
golang.org/x/net v0.0.0-20190522155817-f3200d17e092/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190628185345-da137c7871d7/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190724013045-ca1201d0de80/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200222125558-5a598a2470a0/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200301022130-244492dfa37a/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200501053045-e0ff5e5a1de5/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200506145744-7e3656a0809f/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200513185701-a91f0712d120/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200520182314-0ba52f642ac2/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20201031054903-ff519b6c9102/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20201202161906-c7110b5ffcbb/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20201209123823-ac852fbbde11/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20210119194325-5f4716e94777/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20210316092652-d523dce5a7f4/go.mod h1:RBQZq4jEuRlivfhVLdyRGr576XBO4/greRjx4P4O3yc=
golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM=
golang.org/x/net v0.0.0-20210503060351-7fd8e65b6420/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20211123203042-d83791d6bcd9/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.23.0 h1:7EYJ93RZ9vYSZAIb2x3lnuvqO5zneoD6IvWjuhfxjTs=
golang.org/x/net v0.23.0/go.mod h1:JKghWKKOSdJwpW2GEx0Ja7fmaKnMsbu+MWVZTokSYmg=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20200902213428-5d25da1a8d43/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
golang.org/x/oauth2 v0.0.0-20201109201403-9fd604954f58/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
golang.org/x/oauth2 v0.0.0-20201208152858-08078c50e5b5/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
golang.org/x/oauth2 v0.0.0-20210218202405-ba52d332ba99/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
golang.org/x/oauth2 v0.0.0-20210220000619-9bb904979d93/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
golang.org/x/oauth2 v0.0.0-20210313182246-cd4f82c27b84/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
golang.org/x/oauth2 v0.0.0-20210514164344-f6687ab2804c/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
golang.org/x/oauth2 v0.0.0-20210628180205-a41e5a781914/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
golang.org/x/oauth2 v0.0.0-20210805134026-6f1e6394065a/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
golang.org/x/oauth2 v0.0.0-20210819190943-2bc19b11175f/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
golang.org/x/oauth2 v0.0.0-20211005180243-6b3c2da341f1/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
golang.org/x/oauth2 v0.18.0 h1:09qnuIAgzdx1XplqJvW6CQqMCtGZykZWcXzPMPUusvI=
golang.org/x/oauth2 v0.18.0/go.mod h1:Wf7knwG0MPoWIMMBgFlEaSUDaKskp0dCfrlJRJXbBi8=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -932,11 +793,8 @@ golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJ
golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20200317015054-43a5402ce75a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.6.0 h1:5BMeUDZ7vkXGfEr1x9B4bRcTH4lpkTkpdh0T/J+qjbQ=
golang.org/x/sync v0.6.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sys v0.0.0-20180823144017-11551d06cbcc/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@@ -954,50 +812,19 @@ golang.org/x/sys v0.0.0-20190507160741-ecd444e8653b/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20190606165138-5da285871e9c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190606203320-7fc4e5ec1444/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190624142023-c5567b49c5d0/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190726091711-fc99dfbffb4e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191001151750-bb3f8db39f24/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191022100944-742c48ecaeb7/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191228213918-04cbcbbfeed8/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200113162924-86b910548bc1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200122134326-e047566fdf82/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200212091648-12a6c2dcc1e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200302150141-5c8b2ff67527/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200331124033-c3d80250170d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200501052902-10377860bb8e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200511232937-7e40ca221e25/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200515095857-1151b9dac4a9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200523222454-059865788121/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200803210538-64077c9b5642/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200905004654-be1d3432aa8f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200916030750-2334cc1a136f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201201145000-ef89a241ccb3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210104204734-6f8348627aad/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210119212857-b64e53b001e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210220050731-9a76102bfb43/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210305230114-8fe3ee5dd75b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210315160823-c6e025ad8005/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210320140829-1e4c9ba3b0c4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210324051608-47abb6519492/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210514084401-e8d321eab015/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210603125802-9665404d3644/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210616094352-59db8d763f22/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210806184541-e5e7981a1069/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210823070655-63515b42dcdf/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210908233432-aa78b53d3365/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211025201205-69cdffdb9359/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
@@ -1008,19 +835,15 @@ golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9sn
golang.org/x/term v0.0.0-20220526004731-065cf7ba2467/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.18.0 h1:FcHjZXDMxI8mM3nwhX9HlKop4C0YQvCVCdwYl2wOtE8=
golang.org/x/term v0.18.0/go.mod h1:ILwASektA3OnRv7amZ1xhE/KTR+u50pbXfZ03+6Nx58=
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.4/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.5/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ=
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.3.0 h1:rg5rLMjNzMS1RkNLzCG38eapWhnYLFYXDXj2gOlr8j4=
golang.org/x/time v0.3.0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
@@ -1042,44 +865,9 @@ golang.org/x/tools v0.0.0-20190816200558-6889da9d5479/go.mod h1:b+2E5dAYhXwXZwtn
golang.org/x/tools v0.0.0-20190911174233-4f2ddba30aff/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191012152004-8de300cfc20a/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191112195655-aa38f8e97acc/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191113191852-77e3bb0ad9e7/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191115202509-3a792d9c32b2/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191125144606-a911d9008d1f/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191130070609-6e064ea0cf2d/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191216173652-a0e659d51361/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20191227053925-7b8e75db28f4/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200117161641-43d50277825c/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200122220014-bf1340f18c4a/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200130002326-2f3ba24bd6e7/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200204074204-1cc6d1ef6c74/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200207183749-b753a1ba74fa/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200212150539-ea181f53ac56/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200224181240-023911ca70b2/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200227222343-706bc42d1f0d/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200304193943-95d2e580d8eb/go.mod h1:o4KQGtdN14AW+yjsvvwRTJJuXz8XRtIHtEnmAXLyFUw=
golang.org/x/tools v0.0.0-20200312045724-11d5b4c81c7d/go.mod h1:o4KQGtdN14AW+yjsvvwRTJJuXz8XRtIHtEnmAXLyFUw=
golang.org/x/tools v0.0.0-20200331025713-a30bf2db82d4/go.mod h1:Sl4aGygMT6LrqrWclx+PTx3U+LnKx/seiNR+3G19Ar8=
golang.org/x/tools v0.0.0-20200501065659-ab2804fb9c9d/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20200512131952-2bc93b1c0c88/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20200515010526-7d3b6ebf133d/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20200618134242-20370b0cb4b2/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20200729194436-6467de6f59a7/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA=
golang.org/x/tools v0.0.0-20200804011535-6c149bb5ef0d/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA=
golang.org/x/tools v0.0.0-20200825202427-b303f430e36d/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA=
golang.org/x/tools v0.0.0-20200904185747-39188db58858/go.mod h1:Cj7w3i3Rnn0Xh82ur9kSqwfTHTeVxaDqrfMjpcNT6bE=
golang.org/x/tools v0.0.0-20201110124207-079ba7bd75cd/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.0.0-20201201161351-ac6f37ff4c2a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.0.0-20201208233053-a543418bbed2/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.0.0-20210105154028-b0ab187a4818/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.1.0/go.mod h1:xkSsbof2nBLbhDlRMhhhyNLN/zl3eTqcnHD5viDpcZ0=
golang.org/x/tools v0.1.1/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
golang.org/x/tools v0.1.2/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
golang.org/x/tools v0.1.3/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
golang.org/x/tools v0.1.4/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
golang.org/x/tools v0.1.5/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
golang.org/x/tools v0.18.0 h1:k8NLag8AGHnn+PHbl7g43CtqZAwG60vZkLqgyZgIHgQ=
golang.org/x/tools v0.18.0/go.mod h1:GL7B4CwcLLeo59yx/9UWWuNOW1n3VZ4f5axWfML7Lcg=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
@@ -1092,39 +880,10 @@ google.golang.org/api v0.7.0/go.mod h1:WtwebWUNSVBH/HAw79HIFXZNqEvBhG+Ra+ax0hx3E
google.golang.org/api v0.8.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg=
google.golang.org/api v0.9.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg=
google.golang.org/api v0.13.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI=
google.golang.org/api v0.14.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI=
google.golang.org/api v0.15.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI=
google.golang.org/api v0.17.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE=
google.golang.org/api v0.18.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE=
google.golang.org/api v0.19.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE=
google.golang.org/api v0.20.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE=
google.golang.org/api v0.22.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE=
google.golang.org/api v0.24.0/go.mod h1:lIXQywCXRcnZPGlsd8NbLnOjtAoL6em04bJ9+z0MncE=
google.golang.org/api v0.28.0/go.mod h1:lIXQywCXRcnZPGlsd8NbLnOjtAoL6em04bJ9+z0MncE=
google.golang.org/api v0.29.0/go.mod h1:Lcubydp8VUV7KeIHD9z2Bys/sm/vGKnG1UHuDBSrHWM=
google.golang.org/api v0.30.0/go.mod h1:QGmEvQ87FHZNiUVJkT14jQNYJ4ZJjdRF23ZXz5138Fc=
google.golang.org/api v0.35.0/go.mod h1:/XrVsuzM0rZmrsbjJutiuftIzeuTQcEeaYcSk/mQ1dg=
google.golang.org/api v0.36.0/go.mod h1:+z5ficQTmoYpPn8LCUNVpK5I7hwkpjbcgqA7I34qYtE=
google.golang.org/api v0.40.0/go.mod h1:fYKFpnQN0DsDSKRVRcQSDQNtqWPfM9i+zNPxepjRCQ8=
google.golang.org/api v0.41.0/go.mod h1:RkxM5lITDfTzmyKFPt+wGrCJbVfniCr2ool8kTBzRTU=
google.golang.org/api v0.43.0/go.mod h1:nQsDGjRXMo4lvh5hP0TKqF244gqhGcr/YSIykhUk/94=
google.golang.org/api v0.47.0/go.mod h1:Wbvgpq1HddcWVtzsVLyfLp8lDg6AA241LmgIL59tHXo=
google.golang.org/api v0.48.0/go.mod h1:71Pr1vy+TAZRPkPs/xlCf5SsU8WjuAWv1Pfjbtukyy4=
google.golang.org/api v0.50.0/go.mod h1:4bNT5pAuq5ji4SRZm+5QIkjny9JAyVD/3gaSihNefaw=
google.golang.org/api v0.51.0/go.mod h1:t4HdrdoNgyN5cbEfm7Lum0lcLDLiise1F8qDKX00sOU=
google.golang.org/api v0.54.0/go.mod h1:7C4bFFOvVDGXjfDTAsgGwDgAxRDeQ4X8NvUedIt6z3k=
google.golang.org/api v0.55.0/go.mod h1:38yMfeP1kfjsl8isn0tliTjIb1rJXcQi4UXlbqivdVE=
google.golang.org/api v0.56.0/go.mod h1:38yMfeP1kfjsl8isn0tliTjIb1rJXcQi4UXlbqivdVE=
google.golang.org/api v0.57.0/go.mod h1:dVPlbZyBo2/OjBpmvNdpn2GRm6rPy75jyU7bmhdrMgI=
google.golang.org/api v0.60.0/go.mod h1:d7rl65NZAkEQ90JFzqBjcRq1TVeG5ZoGV3sSpEnnVb4=
google.golang.org/api v0.126.0 h1:q4GJq+cAdMAC7XP7njvQ4tvohGLiSlytuL4BQxbIZ+o=
google.golang.org/api v0.126.0/go.mod h1:mBwVAtz+87bEN6CbA1GtZPDOqY2R5ONPqJeIlvyo4Aw=
google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.6.1/go.mod h1:i06prIuMbXzDqacNJfV5OdTW448YApPu5ww/cMBSeb0=
google.golang.org/appengine v1.6.5/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
google.golang.org/appengine v1.6.6/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
google.golang.org/appengine v1.6.7 h1:FZR1q0exgwxzPzp/aF+VccGrSfxfPpkBqjIIEq3ru6c=
google.golang.org/appengine v1.6.7/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
@@ -1136,60 +895,14 @@ google.golang.org/genproto v0.0.0-20190801165951-fa694d86fc64/go.mod h1:DMBHOl98
google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
google.golang.org/genproto v0.0.0-20190911173649-1774047e7e51/go.mod h1:IbNlFCBrqXvoKpeg0TB2l7cyZUmoaFKYIwrEpbDKLA8=
google.golang.org/genproto v0.0.0-20191108220845-16a3f7862a1a/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20191115194625-c23dd37a84c9/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20191216164720-4f79533eabd1/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20191230161307-f3c370f40bfb/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20200115191322-ca5a22157cba/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20200122232147-0452cf42e150/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20200204135345-fa8e72b47b90/go.mod h1:GmwEX6Z4W5gMy59cAlVYjN9JhxgbQH6Gn+gFDQe2lzA=
google.golang.org/genproto v0.0.0-20200212174721-66ed5ce911ce/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200224152610-e50cd9704f63/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200228133532-8c2c7df3a383/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200305110556-506484158171/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200312145019-da6875a35672/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200331122359-1ee6d9798940/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200423170343-7949de9c1215/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200430143042-b979b6f78d84/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200511104702-f5ebc3bea380/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200513103714-09dca8ec2884/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200515170657-fc4c6c6a6587/go.mod h1:YsZOwe1myG/8QRHRsmBRE1LrgQY60beZKjly0O1fX9U=
google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo=
google.golang.org/genproto v0.0.0-20200618031413-b414f8b61790/go.mod h1:jDfRM7FcilCzHH/e9qn6dsT145K34l5v+OpcnNgKAAA=
google.golang.org/genproto v0.0.0-20200729003335-053ba62fc06f/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20200804131852-c06518451d9c/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20200825200019-8632dd797987/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20200904004341-0bd0a958aa1d/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20201109203340-2640f1f9cdfb/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20201201144952-b05cb90ed32e/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20201210142538-e3217bee35cc/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20201214200347-8c77b98c765d/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20210222152913-aa3ee6e6a81c/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20210303154014-9728d6b83eeb/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20210310155132-4ce2db91004e/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20210319143718-93e7006c17a6/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20210402141018-6c239bbf2bb1/go.mod h1:9lPAdzaEmUacj36I+k7YKbEc5CXzPIeORRgDAUOu28A=
google.golang.org/genproto v0.0.0-20210513213006-bf773b8c8384/go.mod h1:P3QM42oQyzQSnHPnZ/vqoCdDmzH28fzWByN9asMeM8A=
google.golang.org/genproto v0.0.0-20210602131652-f16073e35f0c/go.mod h1:UODoCrxHCcBojKKwX1terBiRUaqAsFqJiF615XL43r0=
google.golang.org/genproto v0.0.0-20210604141403-392c879c8b08/go.mod h1:UODoCrxHCcBojKKwX1terBiRUaqAsFqJiF615XL43r0=
google.golang.org/genproto v0.0.0-20210608205507-b6d2f5bf0d7d/go.mod h1:UODoCrxHCcBojKKwX1terBiRUaqAsFqJiF615XL43r0=
google.golang.org/genproto v0.0.0-20210624195500-8bfb893ecb84/go.mod h1:SzzZ/N+nwJDaO1kznhnlzqS8ocJICar6hYhVyhi++24=
google.golang.org/genproto v0.0.0-20210713002101-d411969a0d9a/go.mod h1:AxrInvYm1dci+enl5hChSFPOmmUF1+uAa/UsgNRWd7k=
google.golang.org/genproto v0.0.0-20210716133855-ce7ef5c701ea/go.mod h1:AxrInvYm1dci+enl5hChSFPOmmUF1+uAa/UsgNRWd7k=
google.golang.org/genproto v0.0.0-20210728212813-7823e685a01f/go.mod h1:ob2IJxKrgPT52GcgX759i1sleT07tiKowYBGbczaW48=
google.golang.org/genproto v0.0.0-20210805201207-89edb61ffb67/go.mod h1:ob2IJxKrgPT52GcgX759i1sleT07tiKowYBGbczaW48=
google.golang.org/genproto v0.0.0-20210813162853-db860fec028c/go.mod h1:cFeNkxwySK631ADgubI+/XFU/xp8FD5KIVV4rj8UC5w=
google.golang.org/genproto v0.0.0-20210821163610-241b8fcbd6c8/go.mod h1:eFjDcFEctNawg4eG61bRv87N7iHBWyVhJu7u1kqDUXY=
google.golang.org/genproto v0.0.0-20210828152312-66f60bf46e71/go.mod h1:eFjDcFEctNawg4eG61bRv87N7iHBWyVhJu7u1kqDUXY=
google.golang.org/genproto v0.0.0-20210831024726-fe130286e0e2/go.mod h1:eFjDcFEctNawg4eG61bRv87N7iHBWyVhJu7u1kqDUXY=
google.golang.org/genproto v0.0.0-20210903162649-d08c68adba83/go.mod h1:eFjDcFEctNawg4eG61bRv87N7iHBWyVhJu7u1kqDUXY=
google.golang.org/genproto v0.0.0-20210909211513-a8c4777a87af/go.mod h1:eFjDcFEctNawg4eG61bRv87N7iHBWyVhJu7u1kqDUXY=
google.golang.org/genproto v0.0.0-20210924002016-3dee208752a0/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc=
google.golang.org/genproto v0.0.0-20211021150943-2b146023228c/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc=
google.golang.org/genproto v0.0.0-20230822172742-b8732ec3820d h1:VBu5YqKPv6XiJ199exd8Br+Aetz+o08F+PLMnwJQHAY=
google.golang.org/genproto v0.0.0-20230822172742-b8732ec3820d/go.mod h1:yZTlhN0tQnXo3h00fuXNCxJdLdIdnVFVBaRJ5LWBbw4=
google.golang.org/genproto/googleapis/api v0.0.0-20230822172742-b8732ec3820d h1:DoPTO70H+bcDXcd39vOqb2viZxgqeBeSGtZ55yZU4/Q=
google.golang.org/genproto/googleapis/api v0.0.0-20230822172742-b8732ec3820d/go.mod h1:KjSP20unUpOx5kyQUFa7k4OJg0qeJ7DEZflGDu2p6Bk=
google.golang.org/genproto/googleapis/bytestream v0.0.0-20230530153820-e85fd2cbaebc/go.mod h1:ylj+BE99M198VPbBh6A8d9n3w8fChvyLK3wwBOjXBFA=
google.golang.org/genproto/googleapis/rpc v0.0.0-20230822172742-b8732ec3820d h1:uvYuEyMHKNt+lT4K3bN6fGswmK8qSvcreM3BwjDh+y4=
google.golang.org/genproto/googleapis/rpc v0.0.0-20230822172742-b8732ec3820d/go.mod h1:+Bk1OCOj40wS2hwAMA+aCW9ypzm63QTBBHp6lQ3p+9M=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
@@ -1198,29 +911,13 @@ google.golang.org/grpc v1.21.0/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ij
google.golang.org/grpc v1.21.1/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM=
google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY=
google.golang.org/grpc v1.26.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.27.1/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.28.0/go.mod h1:rpkK4SK4GF4Ach/+MFLZUBavHOvF2JJB5uozKKal+60=
google.golang.org/grpc v1.29.1/go.mod h1:itym6AZVZYACWQqET3MqgPpjcuV5QH3BxFS3IjizoKk=
google.golang.org/grpc v1.30.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
google.golang.org/grpc v1.31.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
google.golang.org/grpc v1.31.1/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
google.golang.org/grpc v1.33.1/go.mod h1:fr5YgcSWrqhRRxogOsw7RzIpsmvOZ6IcH4kBYTpR3n0=
google.golang.org/grpc v1.33.2/go.mod h1:JMHMWHQWaTccqQQlmk3MJZS+GWXOdAesneDmEnv2fbc=
google.golang.org/grpc v1.34.0/go.mod h1:WotjhfgOW/POjDeRt8vscBtXq+2VjORFy659qA51WJ8=
google.golang.org/grpc v1.35.0/go.mod h1:qjiiYl8FncCW8feJPdyg3v6XW24KsRHe+dy9BAGRRjU=
google.golang.org/grpc v1.36.0/go.mod h1:qjiiYl8FncCW8feJPdyg3v6XW24KsRHe+dy9BAGRRjU=
google.golang.org/grpc v1.36.1/go.mod h1:qjiiYl8FncCW8feJPdyg3v6XW24KsRHe+dy9BAGRRjU=
google.golang.org/grpc v1.37.0/go.mod h1:NREThFqKR1f3iQ6oBuvc5LadQuXVGo9rkm5ZGrQdJfM=
google.golang.org/grpc v1.37.1/go.mod h1:NREThFqKR1f3iQ6oBuvc5LadQuXVGo9rkm5ZGrQdJfM=
google.golang.org/grpc v1.38.0/go.mod h1:NREThFqKR1f3iQ6oBuvc5LadQuXVGo9rkm5ZGrQdJfM=
google.golang.org/grpc v1.39.0/go.mod h1:PImNr+rS9TWYb2O4/emRugxiyHZ5JyHW5F+RPnDzfrE=
google.golang.org/grpc v1.39.1/go.mod h1:PImNr+rS9TWYb2O4/emRugxiyHZ5JyHW5F+RPnDzfrE=
google.golang.org/grpc v1.40.0/go.mod h1:ogyxbiOoUXAkP+4+xa6PZSE9DZgIHtSpzjDTB9KAK34=
google.golang.org/grpc v1.59.0 h1:Z5Iec2pjwb+LEOqzpB2MR12/eKFhDPhuqW91O+4bwUk=
google.golang.org/grpc v1.59.0/go.mod h1:aUPDwccQo6OTjy7Hct4AfBPD1GptF4fyUjIkQ9YtF98=
google.golang.org/grpc/cmd/protoc-gen-go-grpc v1.1.0/go.mod h1:6Kw0yEErY5E/yWrBtf03jp27GLLJujG4z/JK95pnjjw=
google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
@@ -1229,10 +926,8 @@ google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzi
google.golang.org/protobuf v1.22.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.24.0/go.mod h1:r/3tXBNzIEhYS9I1OUVjXDlt8tc493IdKGjtUeSXeh4=
google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c=
google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
google.golang.org/protobuf v1.27.1/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
google.golang.org/protobuf v1.33.0 h1:uNO2rsAINq/JlFpSdYEKIZ0uKD/R9cpdv0T+yoGwGmI=
google.golang.org/protobuf v1.33.0/go.mod h1:c6P6GXX6sHbq/GpV6MGZEdwhWPcYBgnhAHhKbcUYpos=
@@ -1242,8 +937,6 @@ gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI=
gopkg.in/gcfg.v1 v1.2.3 h1:m8OOJ4ccYHnx2f4gQwpno8nAX5OGOh7RLaaz0pj3Ogs=
gopkg.in/gcfg.v1 v1.2.3/go.mod h1:yesOnuUOFQAhST5vPY4nbZsb/huCgGGXlipJsBn0b3o=
gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc=
gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw=
gopkg.in/ini.v1 v1.51.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
@@ -1252,8 +945,6 @@ gopkg.in/natefinch/lumberjack.v2 v2.2.1/go.mod h1:YD8tP3GAjkrDg1eZH7EGmyESg/lsYs
gopkg.in/resty.v1 v1.12.0/go.mod h1:mDo4pnntr5jdWRML875a/NmxYqAlA73dVijT2AXvQQo=
gopkg.in/square/go-jose.v2 v2.6.0 h1:NGk74WTnPKBNUhNzQX7PYcTLUjoq7mzKk2OKbvwk2iI=
gopkg.in/square/go-jose.v2 v2.6.0/go.mod h1:M9dMgbHiYLoDGQrXy7OpJDJWiKiU//h+vD76mk0e1AI=
gopkg.in/warnings.v0 v0.1.2 h1:wFXVbFY8DY5/xOe1ECiWdKCzZlxgshcYVNkBHstARME=
gopkg.in/warnings.v0 v0.1.2/go.mod h1:jksf8JmL6Qr/oQM2OXTHunEvvTAsrWBLb6OOjuVWRNI=
gopkg.in/yaml.v2 v2.0.0-20170812160011-eb3733d160e7/go.mod h1:JAlM8MvJe8wmxCU4Bli9HhUf9+ttbYbLASfIpnQbh74=
gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
@@ -1271,11 +962,8 @@ honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWh
honnef.co/go/tools v0.0.0-20190418001031-e561f6794a2a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg=
honnef.co/go/tools v0.0.1-2020.1.3/go.mod h1:X/FiERA/W4tHapMX5mGpAtMSVEeEUOyHaw9vFzvIQ3k=
honnef.co/go/tools v0.0.1-2020.1.4/go.mod h1:X/FiERA/W4tHapMX5mGpAtMSVEeEUOyHaw9vFzvIQ3k=
k8s.io/gengo/v2 v2.0.0-20240228010128-51d4e06bde70 h1:NGrVE502P0s0/1hudf8zjgwki1X/TByhmAoILTarmzo=
k8s.io/gengo/v2 v2.0.0-20240228010128-51d4e06bde70/go.mod h1:VH3AT8AaQOqiGjMF9p0/IM1Dj+82ZwjfxUP1IxaHE+8=
k8s.io/klog/v2 v2.0.0/go.mod h1:PBfzABfn139FHAV07az/IF9Wp1bkk3vpT2XSJ76fSDE=
k8s.io/klog/v2 v2.120.1 h1:QXU6cPEOIslTGvZaXvFWiP9VKyeet3sawzTOvdXb4Vw=
k8s.io/klog/v2 v2.120.1/go.mod h1:3Jpz1GvMt720eyJH1ckRHK1EDfpxISzJ7I9OYgaDtPE=
k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340 h1:BZqlfIlq5YbRMFko6/PM7FjZpUb45WallggurYhKGag=
@@ -1285,8 +973,6 @@ k8s.io/system-validators v1.8.0/go.mod h1:gP1Ky+R9wtrSiFbrpEPwWMeYz9yqyy1S/KOh0V
k8s.io/utils v0.0.0-20230726121419-3b25d923346b h1:sgn3ZU783SCgtaSJjpcVVlRqd6GSnlTLKgpAAttJvpI=
k8s.io/utils v0.0.0-20230726121419-3b25d923346b/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0=
rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8=
rsc.io/quote/v3 v3.1.0/go.mod h1:yEA65RcK8LyAZtP9Kv3t0HmxON59tX3rD+tICJqUlj0=
rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9JXDnKaTXpA=
sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.29.0 h1:/U5vjBbQn3RChhv7P11uhYvCSm5G2GaIi5AIGBS6r4c=
sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.29.0/go.mod h1:z7+wmGM2dfIiLRfrC6jb5kV2Mq/sK1ZP303cxzkV5Y4=
sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd h1:EDPBXCAspyGV4jQlpZSudPeMmr1bNJefnuqLsRAsHZo=

7
go.work.sum
View File

@@ -105,6 +105,7 @@ cloud.google.com/go/servicedirectory v1.11.0 h1:pBWpjCFVGWkzVTkqN3TBBIqNSoSHY86/
cloud.google.com/go/shell v1.7.1 h1:aHbwH9LSqs4r2rbay9f6fKEls61TAjT63jSyglsw7sI=
cloud.google.com/go/spanner v1.47.0 h1:aqiMP8dhsEXgn9K5EZBWxPG7dxIiyM2VaikqeU4iteg=
cloud.google.com/go/speech v1.19.0 h1:MCagaq8ObV2tr1kZJcJYgXYbIn8Ai5rp42tyGYw9rls=
cloud.google.com/go/storage v1.0.0 h1:VV2nUM3wwLLGh9lSABFgZMjInyUbJeaRSE64WuAIQ+4=
cloud.google.com/go/storage v1.10.0 h1:STgFzyU5/8miMl0//zKh2aQeTyeaUH3WN9bSUiJ09bA=
cloud.google.com/go/storage v1.30.1/go.mod h1:NfxhC0UJE1aXSx7CIIbCf7y9HKT7BiccwkR7+P7gN8E=
cloud.google.com/go/storagetransfer v1.10.0 h1:+ZLkeXx0K0Pk5XdDmG0MnUVqIR18lllsihU/yq39I8Q=
@@ -175,6 +176,7 @@ github.com/go-logfmt/logfmt v0.5.1 h1:otpy5pqBCBZ1ng9RQ0dPu4PN7ba75Y/aA+UpowDyNV
github.com/go-stack/stack v1.8.0 h1:5SgMzNM5HxrEjV0ww2lTmX6E2Izsfxas4+YHWRs3Lsk=
github.com/goccy/go-json v0.9.11/go.mod h1:6MelG93GURQebXPDq3khkgXZkazVtN9CRI+MGFi0w8I=
github.com/gogo/googleapis v1.4.1 h1:1Yx4Myt7BxzvUr5ldGSbwYiZG6t9wGBZ+8/fX3Wvtq0=
github.com/golang/mock v1.3.1 h1:qGJ6qTW+x6xX/my+8YUVl4WNpX9B7+/l2tRsHGZ7f2s=
github.com/golang/mock v1.6.0 h1:ErTB+efbowRARo13NNdxyJji2egdxLGQhRaY+DUumQc=
github.com/golang/protobuf v1.5.3/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY=
github.com/golang/snappy v0.0.3 h1:fHPg5GQYlCeLIPB9BZqMVR5nR9A+IM5zcgeTdjMYmLA=
@@ -187,6 +189,7 @@ github.com/google/martian/v3 v3.2.1 h1:d8MncMlErDFTwQGBK1xhv026j9kqhvw1Qv9IbWT1V
github.com/google/martian/v3 v3.3.2/go.mod h1:oBOf6HBosgwRXnUGWUB05QECsc6uvmMiJ3+6W4l/CUk=
github.com/google/renameio v0.1.0 h1:GOZbcHa3HfsPKPlmyPyN2KEohoMXOhdMbHrvbpl2QaA=
github.com/google/s2a-go v0.1.4/go.mod h1:Ej+mSEMGRnqRzjc7VtF+jdBwYG5fuJfiZ8ELkjEwM0A=
github.com/googleapis/gax-go/v2 v2.0.5 h1:sjZBwGj9Jlw33ImPtvFviGYvseOtDM7hkSKB7+Tv3SM=
github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1 h1:EGx4pi6eqNxGaHF6qqu48+N2wcFQ5qg5FXgOdqsJ5d8=
github.com/hashicorp/consul/api v1.1.0 h1:BNQPM9ytxj6jbjjdRPioQ94T6YXriSopn0i8COv6SRA=
github.com/hashicorp/consul/sdk v0.1.1 h1:LnuDWGNsoajlhGyHJvuWW6FVqRl8JOTPqS6CPTsYjhY=
@@ -210,6 +213,7 @@ github.com/iancoleman/strcase v0.2.0/go.mod h1:iwCmte+B7n89clKwxIoIXy/HfoL7AsD47
github.com/ianlancetaylor/demangle v0.0.0-20200824232613-28f6c0f3b639 h1:mV02weKRL81bEnm8A0HT1/CAelMQDBuQIfLw8n+d6xI=
github.com/jmespath/go-jmespath v0.4.0 h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg=
github.com/jpillora/backoff v1.0.0 h1:uvFg412JmmHBHw7iwprIxkPMI+sGQ4kzOWsMeHnm2EA=
github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024 h1:rBMNdlhTLzJjJSDIjNEXX1Pz3Hmwmz91v+zycvx9PJc=
github.com/jstemmer/go-junit-report v0.9.1 h1:6QPYqodiu3GuPL+7mfx+NwDdp2eTkp9IfEUpgAwUN0o=
github.com/jtolds/gls v4.20.0+incompatible h1:xdiiI2gbIgH/gLH7ADydsJ1uDOEzR8yvV7C0MuV77Wo=
github.com/julienschmidt/httprouter v1.3.0 h1:U0609e9tgbseu3rBINet9P48AI/D3oJs4dN7jwJOQ1U=
@@ -272,6 +276,7 @@ golang.org/x/crypto v0.0.0-20220314234659-1baeb1ce4c0b/go.mod h1:IxCIyHEi3zRg3s0
golang.org/x/crypto v0.9.0/go.mod h1:yrmDGqONDYtNj3tH8X9dzUun2m2lzPa9ngI6/RUPGR0=
golang.org/x/crypto v0.12.0/go.mod h1:NF0Gs7EO5K4qLn+Ylc+fih8BSTeIjAP05siRnAh98yw=
golang.org/x/image v0.0.0-20190802002840-cff245a6509b h1:+qEpEAPhDZ1o0x3tHzZTQDArnOixOzGD9HUJfcg0mb4=
golang.org/x/lint v0.0.0-20190930215403-16217165b5de h1:5hukYrvBGR8/eNkX5mdUezrA6JiaEZDtJb9Ei+1LlBs=
golang.org/x/lint v0.0.0-20210508222113-6edffad5e616 h1:VLliZ0d+/avPrXXH+OakdXhpJuEoBZuwh1m2j7U6Iug=
golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028 h1:4+4C/Iv2U4fMZBiMCc98MG1In4gJY5YRhtpDNeDeHWs=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
@@ -304,6 +309,7 @@ golang.org/x/text v0.12.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/tools v0.13.0/go.mod h1:HvlwmtVNQAhOuCjW7xxvovg8wbNq7LwfXh/k7wXUl58=
golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 h1:H2TDz8ibqkAF6YGhCdN3jS9O0/s90v0rJh3X/OLHEUk=
google.golang.org/api v0.13.0 h1:Q3Ui3V3/CVinFWFiW39Iw0kMuVrRzYX0wN6OPFp0lTA=
google.golang.org/genproto v0.0.0-20230530153820-e85fd2cbaebc/go.mod h1:xZnkP7mREFX5MORlOPEzLMr+90PPZQ2QWzrVTWfAq64=
google.golang.org/genproto v0.0.0-20230706204954-ccb25ca9f130/go.mod h1:O9kGHb51iE/nOGvQaDUuadVYqovW56s5emA88lQnj6Y=
google.golang.org/genproto v0.0.0-20230711160842-782d3b101e98/go.mod h1:S7mY02OqCJTD0E1OiQy1F72PWFB4bZJ87cAtLPYgDR0=
@@ -326,6 +332,7 @@ gopkg.in/errgo.v2 v2.1.0 h1:0vLT13EuvQ0hNvakwLuFZ/jYrLp5F3kcWHXdRggjCE8=
gopkg.in/ini.v1 v1.51.0 h1:AQvPpx3LzTDM0AjnIRlVFwFFGC+npRopjZxLJj6gdno=
gopkg.in/resty.v1 v1.12.0 h1:CuXP0Pjfw9rOuY6EP+UvtNvt5DSqHpIxILZKT/quCZI=
gotest.tools/v3 v3.0.3 h1:4AuOwCGf4lLR9u3YOe2awrHygurzhO/HeQ6laiA6Sx0=
honnef.co/go/tools v0.0.1-2019.2.3 h1:3JgtbtFHMiCmsznwGVTUWbgGov+pVqnlf1dEJTNAXeM=
honnef.co/go/tools v0.0.1-2020.1.4 h1:UoveltGrhghAA7ePc+e+QYDHXrBps2PqFZiHkGR/xK8=
rsc.io/binaryregexp v0.2.0 h1:HfqmD5MEmC0zvwBuF187nq9mdnXjXsSivRiXN7SmRkE=
rsc.io/quote/v3 v3.1.0 h1:9JKUTTIUgS6kzR9mK1YuGKv6Nl+DijDNIc0ghT58FaY=

39
hack/unwanted-dependencies.json
View File

@@ -74,14 +74,12 @@
"status": {
"unwantedReferences": {
"cloud.google.com/go": [
"cloud.google.com/go/compute",
"google.golang.org/genproto"
],
"cloud.google.com/go/bigquery": [
"google.golang.org/genproto"
],
"cloud.google.com/go/compute": [
"cloud.google.com/go/compute/metadata",
"github.com/google/cadvisor",
"go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",
"golang.org/x/oauth2",
@@ -91,10 +89,6 @@
"cloud.google.com/go/firestore": [
"google.golang.org/genproto"
],
"github.com/GoogleCloudPlatform/k8s-cloud-provider": [
"k8s.io/kubernetes",
"k8s.io/legacy-cloud-providers"
],
"github.com/go-kit/kit": [
"github.com/grpc-ecosystem/go-grpc-middleware"
],
@@ -125,21 +119,10 @@
"k8s.io/kubernetes",
"k8s.io/metrics"
],
"github.com/google/s2a-go": [
"cloud.google.com/go/compute",
"google.golang.org/api"
],
"github.com/google/shlex": [
"sigs.k8s.io/kustomize/api",
"sigs.k8s.io/kustomize/kustomize/v5"
],
"github.com/googleapis/enterprise-certificate-proxy": [
"cloud.google.com/go/compute",
"google.golang.org/api"
],
"github.com/googleapis/gax-go/v2": [
"cloud.google.com/go/compute"
],
"github.com/gregjones/httpcache": [
"k8s.io/client-go"
],
@@ -183,56 +166,37 @@
"sigs.k8s.io/kustomize/kustomize/v5"
],
"go.opencensus.io": [
"cloud.google.com/go/compute",
"github.com/Microsoft/hcsshim",
"google.golang.org/api"
"github.com/Microsoft/hcsshim"
],
"golang.org/x/exp": [
"github.com/antlr4-go/antlr/v4",
"github.com/google/cel-go"
],
"google.golang.org/api": [
"cloud.google.com/go/compute",
"github.com/GoogleCloudPlatform/k8s-cloud-provider",
"github.com/googleapis/gax-go/v2",
"k8s.io/kubernetes",
"k8s.io/legacy-cloud-providers"
],
"google.golang.org/appengine": [
"cloud.google.com/go/compute",
"github.com/grpc-ecosystem/grpc-gateway/v2",
"github.com/prometheus/client_golang",
"github.com/prometheus/common",
"go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",
"golang.org/x/oauth2",
"google.golang.org/api",
"google.golang.org/grpc"
],
"google.golang.org/genproto": [
"cloud.google.com/go/compute",
"github.com/container-storage-interface/spec",
"github.com/containerd/ttrpc",
"github.com/googleapis/gax-go/v2",
"github.com/grpc-ecosystem/go-grpc-middleware",
"github.com/grpc-ecosystem/grpc-gateway",
"github.com/grpc-ecosystem/grpc-gateway/v2",
"go.etcd.io/etcd/api/v3",
"go.etcd.io/etcd/client/v3",
"go.etcd.io/etcd/server/v3",
"google.golang.org/api",
"google.golang.org/genproto/googleapis/api",
"google.golang.org/genproto/googleapis/rpc",
"google.golang.org/grpc"
]
},
"unwantedVendored": [
"cloud.google.com/go/compute",
"github.com/GoogleCloudPlatform/k8s-cloud-provider",
"github.com/gogo/protobuf",
"github.com/google/s2a-go",
"github.com/google/shlex",
"github.com/googleapis/enterprise-certificate-proxy",
"github.com/googleapis/gax-go/v2",
"github.com/gregjones/httpcache",
"github.com/grpc-ecosystem/go-grpc-prometheus",
"github.com/grpc-ecosystem/grpc-gateway",
@@ -241,7 +205,6 @@
"github.com/pkg/errors",
"go.opencensus.io",
"golang.org/x/exp",
"google.golang.org/api",
"google.golang.org/appengine",
"google.golang.org/genproto"
]

25
pkg/cloudprovider/providers/providers.go
View File

@@ -1,25 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2015 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package cloudprovider
import (
// Cloud providers
_ "k8s.io/legacy-cloud-providers/gce"
)

147
pkg/controller/nodeipam/ipam/adapter.go
View File

@@ -1,147 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package ipam
import (
"context"
"encoding/json"
"fmt"
"net"
"k8s.io/klog/v2"
netutils "k8s.io/utils/net"
v1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/types"
utilruntime "k8s.io/apimachinery/pkg/util/runtime"
clientset "k8s.io/client-go/kubernetes"
v1core "k8s.io/client-go/kubernetes/typed/core/v1"
"k8s.io/client-go/tools/record"
nodeutil "k8s.io/component-helpers/node/util"
"k8s.io/legacy-cloud-providers/gce"
"k8s.io/metrics/pkg/client/clientset/versioned/scheme"
)
type adapter struct {
k8s clientset.Interface
cloud *gce.Cloud
broadcaster record.EventBroadcaster
recorder record.EventRecorder
}
func newAdapter(ctx context.Context, k8s clientset.Interface, cloud *gce.Cloud) *adapter {
broadcaster := record.NewBroadcaster(record.WithContext(ctx))
ret := &adapter{
k8s: k8s,
cloud: cloud,
broadcaster: broadcaster,
recorder: broadcaster.NewRecorder(scheme.Scheme, v1.EventSource{Component: "cloudCIDRAllocator"}),
}
return ret
}
func (a *adapter) Run(ctx context.Context) {
defer utilruntime.HandleCrash()
// Start event processing pipeline.
a.broadcaster.StartStructuredLogging(3)
a.broadcaster.StartRecordingToSink(&v1core.EventSinkImpl{Interface: a.k8s.CoreV1().Events("")})
defer a.broadcaster.Shutdown()
<-ctx.Done()
}
func (a *adapter) Alias(ctx context.Context, node *v1.Node) (*net.IPNet, error) {
if node.Spec.ProviderID == "" {
return nil, fmt.Errorf("node %s doesn't have providerID", node.Name)
}
cidrs, err := a.cloud.AliasRangesByProviderID(node.Spec.ProviderID)
if err != nil {
return nil, err
}
switch len(cidrs) {
case 0:
return nil, nil
case 1:
break
default:
klog.FromContext(ctx).Info("Node has more than one alias assigned, defaulting to the first", "node", klog.KObj(node), "CIDRs", cidrs)
}
_, cidrRange, err := netutils.ParseCIDRSloppy(cidrs[0])
if err != nil {
return nil, err
}
return cidrRange, nil
}
func (a *adapter) AddAlias(ctx context.Context, node *v1.Node, cidrRange *net.IPNet) error {
if node.Spec.ProviderID == "" {
return fmt.Errorf("node %s doesn't have providerID", node.Name)
}
return a.cloud.AddAliasToInstanceByProviderID(node.Spec.ProviderID, cidrRange)
}
func (a *adapter) Node(ctx context.Context, name string) (*v1.Node, error) {
return a.k8s.CoreV1().Nodes().Get(context.TODO(), name, metav1.GetOptions{})
}
func (a *adapter) UpdateNodePodCIDR(ctx context.Context, node *v1.Node, cidrRange *net.IPNet) error {
patch := map[string]interface{}{
"apiVersion": node.APIVersion,
"kind": node.Kind,
"metadata": map[string]interface{}{"name": node.Name},
"spec": map[string]interface{}{"podCIDR": cidrRange.String()},
}
bytes, err := json.Marshal(patch)
if err != nil {
return err
}
_, err = a.k8s.CoreV1().Nodes().Patch(context.TODO(), node.Name, types.StrategicMergePatchType, bytes, metav1.PatchOptions{})
return err
}
func (a *adapter) UpdateNodeNetworkUnavailable(nodeName string, unavailable bool) error {
condition := v1.ConditionFalse
if unavailable {
condition = v1.ConditionTrue
}
return nodeutil.SetNodeCondition(a.k8s, types.NodeName(nodeName), v1.NodeCondition{
Type: v1.NodeNetworkUnavailable,
Status: condition,
Reason: "RouteCreated",
Message: "NodeController created an implicit route",
LastTransitionTime: metav1.Now(),
})
}
func (a *adapter) EmitNodeWarningEvent(nodeName, reason, fmt string, args ...interface{}) {
ref := &v1.ObjectReference{Kind: "Node", Name: nodeName}
a.recorder.Eventf(ref, v1.EventTypeNormal, reason, fmt, args...)
}

2
pkg/controller/nodeipam/ipam/cidr_allocator.go
View File

@@ -116,8 +116,6 @@ func New(ctx context.Context, kubeClient clientset.Interface, cloud cloudprovide
switch allocatorType {
case RangeAllocatorType:
return NewCIDRRangeAllocator(ctx, kubeClient, nodeInformer, allocatorParams, nodeList)
case CloudAllocatorType:
return NewCloudCIDRAllocator(ctx, kubeClient, cloud, nodeInformer)
default:
return nil, fmt.Errorf("invalid CIDR allocator type: %v", allocatorType)
}

378
pkg/controller/nodeipam/ipam/cloud_cidr_allocator.go
View File

@@ -1,378 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2016 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package ipam
import (
"context"
"fmt"
"math/rand"
"net"
"sync"
"time"
"github.com/google/go-cmp/cmp"
"k8s.io/klog/v2"
v1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/types"
utilruntime "k8s.io/apimachinery/pkg/util/runtime"
informers "k8s.io/client-go/informers/core/v1"
corelisters "k8s.io/client-go/listers/core/v1"
"k8s.io/client-go/tools/cache"
"k8s.io/client-go/tools/record"
clientset "k8s.io/client-go/kubernetes"
"k8s.io/client-go/kubernetes/scheme"
v1core "k8s.io/client-go/kubernetes/typed/core/v1"
cloudprovider "k8s.io/cloud-provider"
nodeutil "k8s.io/component-helpers/node/util"
controllerutil "k8s.io/kubernetes/pkg/controller/util/node"
utiltaints "k8s.io/kubernetes/pkg/util/taints"
"k8s.io/legacy-cloud-providers/gce"
netutils "k8s.io/utils/net"
)
// nodeProcessingInfo tracks information related to current nodes in processing
type nodeProcessingInfo struct {
retries int
}
// cloudCIDRAllocator allocates node CIDRs according to IP address aliases
// assigned by the cloud provider. In this case, the allocation and
// deallocation is delegated to the external provider, and the controller
// merely takes the assignment and updates the node spec.
type cloudCIDRAllocator struct {
client clientset.Interface
cloud *gce.Cloud
// nodeLister is able to list/get nodes and is populated by the shared informer passed to
// NewCloudCIDRAllocator.
nodeLister corelisters.NodeLister
// nodesSynced returns true if the node shared informer has been synced at least once.
nodesSynced cache.InformerSynced
// Channel that is used to pass updating Nodes to the background.
// This increases the throughput of CIDR assignment by parallelization
// and not blocking on long operations (which shouldn't be done from
// event handlers anyway).
nodeUpdateChannel chan string
broadcaster record.EventBroadcaster
recorder record.EventRecorder
// Keep a set of nodes that are currectly being processed to avoid races in CIDR allocation
lock sync.Mutex
nodesInProcessing map[string]*nodeProcessingInfo
}
var _ CIDRAllocator = (*cloudCIDRAllocator)(nil)
// NewCloudCIDRAllocator creates a new cloud CIDR allocator.
func NewCloudCIDRAllocator(ctx context.Context, client clientset.Interface, cloud cloudprovider.Interface, nodeInformer informers.NodeInformer) (CIDRAllocator, error) {
logger := klog.FromContext(ctx)
if client == nil {
logger.Error(nil, "kubeClient is nil when starting cloud CIDR allocator")
klog.FlushAndExit(klog.ExitFlushTimeout, 1)
}
eventBroadcaster := record.NewBroadcaster(record.WithContext(ctx))
recorder := eventBroadcaster.NewRecorder(scheme.Scheme, v1.EventSource{Component: "cidrAllocator"})
gceCloud, ok := cloud.(*gce.Cloud)
if !ok {
err := fmt.Errorf("cloudCIDRAllocator does not support %v provider", cloud.ProviderName())
return nil, err
}
ca := &cloudCIDRAllocator{
client: client,
cloud: gceCloud,
nodeLister: nodeInformer.Lister(),
nodesSynced: nodeInformer.Informer().HasSynced,
nodeUpdateChannel: make(chan string, cidrUpdateQueueSize),
broadcaster: eventBroadcaster,
recorder: recorder,
nodesInProcessing: map[string]*nodeProcessingInfo{},
}
nodeInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
AddFunc: controllerutil.CreateAddNodeHandler(
func(node *v1.Node) error {
return ca.AllocateOrOccupyCIDR(ctx, node)
}),
UpdateFunc: controllerutil.CreateUpdateNodeHandler(func(_, newNode *v1.Node) error {
if newNode.Spec.PodCIDR == "" {
return ca.AllocateOrOccupyCIDR(ctx, newNode)
}
// Even if PodCIDR is assigned, but NetworkUnavailable condition is
// set to true, we need to process the node to set the condition.
networkUnavailableTaint := &v1.Taint{Key: v1.TaintNodeNetworkUnavailable, Effect: v1.TaintEffectNoSchedule}
_, cond := controllerutil.GetNodeCondition(&newNode.Status, v1.NodeNetworkUnavailable)
if cond == nil || cond.Status != v1.ConditionFalse || utiltaints.TaintExists(newNode.Spec.Taints, networkUnavailableTaint) {
return ca.AllocateOrOccupyCIDR(ctx, newNode)
}
return nil
}),
DeleteFunc: controllerutil.CreateDeleteNodeHandler(logger, func(node *v1.Node) error {
return ca.ReleaseCIDR(logger, node)
}),
})
logger.Info("Using cloud CIDR allocator", "provider", cloud.ProviderName())
return ca, nil
}
func (ca *cloudCIDRAllocator) Run(ctx context.Context) {
defer utilruntime.HandleCrash()
// Start event processing pipeline.
ca.broadcaster.StartStructuredLogging(3)
logger := klog.FromContext(ctx)
logger.Info("Sending events to api server")
ca.broadcaster.StartRecordingToSink(&v1core.EventSinkImpl{Interface: ca.client.CoreV1().Events("")})
defer ca.broadcaster.Shutdown()
logger.Info("Starting cloud CIDR allocator")
defer logger.Info("Shutting down cloud CIDR allocator")
if !cache.WaitForNamedCacheSync("cidrallocator", ctx.Done(), ca.nodesSynced) {
return
}
for i := 0; i < cidrUpdateWorkers; i++ {
go ca.worker(ctx)
}
<-ctx.Done()
}
func (ca *cloudCIDRAllocator) worker(ctx context.Context) {
logger := klog.FromContext(ctx)
for {
select {
case workItem, ok := <-ca.nodeUpdateChannel:
if !ok {
logger.Info("Channel nodeCIDRUpdateChannel was unexpectedly closed")
return
}
if err := ca.updateCIDRAllocation(ctx, workItem); err == nil {
logger.V(3).Info("Updated CIDR", "workItem", workItem)
} else {
logger.Error(err, "Error updating CIDR", "workItem", workItem)
if canRetry, timeout := ca.retryParams(logger, workItem); canRetry {
logger.V(2).Info("Retrying update on next period", "workItem", workItem, "timeout", timeout)
time.AfterFunc(timeout, func() {
// Requeue the failed node for update again.
ca.nodeUpdateChannel <- workItem
})
continue
}
logger.Error(nil, "Exceeded retry count, dropping from queue", "workItem", workItem)
}
ca.removeNodeFromProcessing(workItem)
case <-ctx.Done():
return
}
}
}
func (ca *cloudCIDRAllocator) insertNodeToProcessing(nodeName string) bool {
ca.lock.Lock()
defer ca.lock.Unlock()
if _, found := ca.nodesInProcessing[nodeName]; found {
return false
}
ca.nodesInProcessing[nodeName] = &nodeProcessingInfo{}
return true
}
func (ca *cloudCIDRAllocator) retryParams(logger klog.Logger, nodeName string) (bool, time.Duration) {
ca.lock.Lock()
defer ca.lock.Unlock()
entry, ok := ca.nodesInProcessing[nodeName]
if !ok {
logger.Error(nil, "Cannot get retryParams for node as entry does not exist", "node", klog.KRef("", nodeName))
return false, 0
}
count := entry.retries + 1
if count > updateMaxRetries {
return false, 0
}
ca.nodesInProcessing[nodeName].retries = count
return true, nodeUpdateRetryTimeout(count)
}
func nodeUpdateRetryTimeout(count int) time.Duration {
timeout := updateRetryTimeout
for i := 0; i < count && timeout < maxUpdateRetryTimeout; i++ {
timeout *= 2
}
if timeout > maxUpdateRetryTimeout {
timeout = maxUpdateRetryTimeout
}
return time.Duration(timeout.Nanoseconds()/2 + rand.Int63n(timeout.Nanoseconds()))
}
func (ca *cloudCIDRAllocator) removeNodeFromProcessing(nodeName string) {
ca.lock.Lock()
defer ca.lock.Unlock()
delete(ca.nodesInProcessing, nodeName)
}
// WARNING: If you're adding any return calls or defer any more work from this
// function you have to make sure to update nodesInProcessing properly with the
// disposition of the node when the work is done.
func (ca *cloudCIDRAllocator) AllocateOrOccupyCIDR(ctx context.Context, node *v1.Node) error {
if node == nil {
return nil
}
logger := klog.FromContext(ctx)
if !ca.insertNodeToProcessing(node.Name) {
logger.V(2).Info("Node is already in a process of CIDR assignment", "node", klog.KObj(node))
return nil
}
logger.V(4).Info("Putting node into the work queue", "node", klog.KObj(node))
ca.nodeUpdateChannel <- node.Name
return nil
}
// updateCIDRAllocation assigns CIDR to Node and sends an update to the API server.
func (ca *cloudCIDRAllocator) updateCIDRAllocation(ctx context.Context, nodeName string) error {
logger := klog.FromContext(ctx)
node, err := ca.nodeLister.Get(nodeName)
if err != nil {
if errors.IsNotFound(err) {
return nil // node no longer available, skip processing
}
logger.Error(err, "Failed while getting the node for updating Node.Spec.PodCIDR", "node", klog.KRef("", nodeName))
return err
}
if node.Spec.ProviderID == "" {
return fmt.Errorf("node %s doesn't have providerID", nodeName)
}
cidrStrings, err := ca.cloud.AliasRangesByProviderID(node.Spec.ProviderID)
if err != nil {
controllerutil.RecordNodeStatusChange(logger, ca.recorder, node, "CIDRNotAvailable")
return fmt.Errorf("failed to get cidr(s) from provider: %v", err)
}
if len(cidrStrings) == 0 {
controllerutil.RecordNodeStatusChange(logger, ca.recorder, node, "CIDRNotAvailable")
return fmt.Errorf("failed to allocate cidr: Node %v has no CIDRs", node.Name)
}
//Can have at most 2 ips (one for v4 and one for v6)
if len(cidrStrings) > 2 {
logger.Info("Got more than 2 ips, truncating to 2", "cidrStrings", cidrStrings)
cidrStrings = cidrStrings[:2]
}
cidrs, err := netutils.ParseCIDRs(cidrStrings)
if err != nil {
return fmt.Errorf("failed to parse strings %v as CIDRs: %v", cidrStrings, err)
}
needUpdate, err := needPodCIDRsUpdate(logger, node, cidrs)
if err != nil {
return fmt.Errorf("err: %v, CIDRS: %v", err, cidrStrings)
}
if needUpdate {
if node.Spec.PodCIDR != "" {
logger.Error(nil, "PodCIDR being reassigned", "node", klog.KObj(node), "podCIDRs", node.Spec.PodCIDRs, "cidrStrings", cidrStrings)
// We fall through and set the CIDR despite this error. This
// implements the same logic as implemented in the
// rangeAllocator.
//
// See https://github.com/kubernetes/kubernetes/pull/42147#discussion_r103357248
}
for i := 0; i < cidrUpdateRetries; i++ {
if err = nodeutil.PatchNodeCIDRs(ctx, ca.client, types.NodeName(node.Name), cidrStrings); err == nil {
logger.Info("Set the node PodCIDRs", "node", klog.KObj(node), "cidrStrings", cidrStrings)
break
}
}
}
if err != nil {
controllerutil.RecordNodeStatusChange(logger, ca.recorder, node, "CIDRAssignmentFailed")
logger.Error(err, "Failed to update the node PodCIDR after multiple attempts", "node", klog.KObj(node), "cidrStrings", cidrStrings)
return err
}
err = nodeutil.SetNodeCondition(ca.client, types.NodeName(node.Name), v1.NodeCondition{
Type: v1.NodeNetworkUnavailable,
Status: v1.ConditionFalse,
Reason: "RouteCreated",
Message: "NodeController create implicit route",
LastTransitionTime: metav1.Now(),
})
if err != nil {
logger.Error(err, "Error setting route status for the node", "node", klog.KObj(node))
}
return err
}
func needPodCIDRsUpdate(logger klog.Logger, node *v1.Node, podCIDRs []*net.IPNet) (bool, error) {
if node.Spec.PodCIDR == "" {
return true, nil
}
_, nodePodCIDR, err := netutils.ParseCIDRSloppy(node.Spec.PodCIDR)
if err != nil {
logger.Error(err, "Found invalid node.Spec.PodCIDR", "podCIDR", node.Spec.PodCIDR)
// We will try to overwrite with new CIDR(s)
return true, nil
}
nodePodCIDRs, err := netutils.ParseCIDRs(node.Spec.PodCIDRs)
if err != nil {
logger.Error(err, "Found invalid node.Spec.PodCIDRs", "podCIDRs", node.Spec.PodCIDRs)
// We will try to overwrite with new CIDR(s)
return true, nil
}
if len(podCIDRs) == 1 {
if cmp.Equal(nodePodCIDR, podCIDRs[0]) {
logger.V(4).Info("Node already has allocated CIDR. It matches the proposed one", "node", klog.KObj(node), "podCIDR", podCIDRs[0])
return false, nil
}
} else if len(nodePodCIDRs) == len(podCIDRs) {
if dualStack, _ := netutils.IsDualStackCIDRs(podCIDRs); !dualStack {
return false, fmt.Errorf("IPs are not dual stack")
}
for idx, cidr := range podCIDRs {
if !cmp.Equal(nodePodCIDRs[idx], cidr) {
return true, nil
}
}
logger.V(4).Info("Node already has allocated CIDRs. It matches the proposed one", "node", klog.KObj(node), "podCIDRs", podCIDRs)
return false, nil
}
return true, nil
}
func (ca *cloudCIDRAllocator) ReleaseCIDR(logger klog.Logger, node *v1.Node) error {
logger.V(2).Info("Node's PodCIDR will be released by external cloud provider (not managed by controller)",
"node", klog.KObj(node), "podCIDR", node.Spec.PodCIDR)
return nil
}

34
pkg/controller/nodeipam/ipam/cloud_cidr_allocator_nolegacyproviders.go
View File

@@ -1,34 +0,0 @@
//go:build providerless
// +build providerless
/*
Copyright 2019 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package ipam
import (
"context"
"errors"
informers "k8s.io/client-go/informers/core/v1"
clientset "k8s.io/client-go/kubernetes"
cloudprovider "k8s.io/cloud-provider"
)
// NewCloudCIDRAllocator creates a new cloud CIDR allocator.
func NewCloudCIDRAllocator(ctx context.Context, client clientset.Interface, cloud cloudprovider.Interface, nodeInformer informers.NodeInformer) (CIDRAllocator, error) {
return nil, errors.New("legacy cloud provider support not built")
}

193
pkg/controller/nodeipam/ipam/cloud_cidr_allocator_test.go
View File

@@ -1,193 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2018 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package ipam
import (
"fmt"
"testing"
"time"
v1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/informers"
"k8s.io/client-go/kubernetes/fake"
"k8s.io/klog/v2/ktesting"
netutils "k8s.io/utils/net"
)
func hasNodeInProcessing(ca *cloudCIDRAllocator, name string) bool {
ca.lock.Lock()
defer ca.lock.Unlock()
_, found := ca.nodesInProcessing[name]
return found
}
func TestBoundedRetries(t *testing.T) {
clientSet := fake.NewSimpleClientset()
updateChan := make(chan string, 1) // need to buffer as we are using only on go routine
sharedInfomer := informers.NewSharedInformerFactory(clientSet, 1*time.Hour)
ca := &cloudCIDRAllocator{
client: clientSet,
nodeUpdateChannel: updateChan,
nodeLister: sharedInfomer.Core().V1().Nodes().Lister(),
nodesSynced: sharedInfomer.Core().V1().Nodes().Informer().HasSynced,
nodesInProcessing: map[string]*nodeProcessingInfo{},
}
_, ctx := ktesting.NewTestContext(t)
go ca.worker(ctx)
nodeName := "testNode"
if err := ca.AllocateOrOccupyCIDR(ctx, &v1.Node{
ObjectMeta: metav1.ObjectMeta{
Name: nodeName,
},
}); err != nil {
t.Errorf("unexpected error in AllocateOrOccupyCIDR: %v", err)
}
for hasNodeInProcessing(ca, nodeName) {
// wait for node to finish processing (should terminate and not time out)
}
}
func withinExpectedRange(got time.Duration, expected time.Duration) bool {
return got >= expected/2 && got <= 3*expected/2
}
func TestNodeUpdateRetryTimeout(t *testing.T) {
for _, tc := range []struct {
count int
want time.Duration
}{
{count: 0, want: 250 * time.Millisecond},
{count: 1, want: 500 * time.Millisecond},
{count: 2, want: 1000 * time.Millisecond},
{count: 3, want: 2000 * time.Millisecond},
{count: 50, want: 5000 * time.Millisecond},
} {
t.Run(fmt.Sprintf("count %d", tc.count), func(t *testing.T) {
if got := nodeUpdateRetryTimeout(tc.count); !withinExpectedRange(got, tc.want) {
t.Errorf("nodeUpdateRetryTimeout(tc.count) = %v; want %v", got, tc.want)
}
})
}
}
func TestNeedPodCIDRsUpdate(t *testing.T) {
for _, tc := range []struct {
desc string
cidrs []string
nodePodCIDR string
nodePodCIDRs []string
want bool
wantErr bool
}{
{
desc: "want error - invalid cidr",
cidrs: []string{"10.10.10.0/24"},
nodePodCIDR: "10.10..0/24",
nodePodCIDRs: []string{"10.10..0/24"},
want: true,
},
{
desc: "want error - cidr len 2 but not dual stack",
cidrs: []string{"10.10.10.0/24", "10.10.11.0/24"},
nodePodCIDR: "10.10.10.0/24",
nodePodCIDRs: []string{"10.10.10.0/24", "2001:db8::/64"},
wantErr: true,
},
{
desc: "want false - matching v4 only cidr",
cidrs: []string{"10.10.10.0/24"},
nodePodCIDR: "10.10.10.0/24",
nodePodCIDRs: []string{"10.10.10.0/24"},
want: false,
},
{
desc: "want false - nil node.Spec.PodCIDR",
cidrs: []string{"10.10.10.0/24"},
want: true,
},
{
desc: "want true - non matching v4 only cidr",
cidrs: []string{"10.10.10.0/24"},
nodePodCIDR: "10.10.11.0/24",
nodePodCIDRs: []string{"10.10.11.0/24"},
want: true,
},
{
desc: "want false - matching v4 and v6 cidrs",
cidrs: []string{"10.10.10.0/24", "2001:db8::/64"},
nodePodCIDR: "10.10.10.0/24",
nodePodCIDRs: []string{"10.10.10.0/24", "2001:db8::/64"},
want: false,
},
{
desc: "want false - matching v4 and v6 cidrs, different strings but same CIDRs",
cidrs: []string{"10.10.10.0/24", "2001:db8::/64"},
nodePodCIDR: "10.10.10.0/24",
nodePodCIDRs: []string{"10.10.10.0/24", "2001:db8:0::/64"},
want: false,
},
{
desc: "want true - matching v4 and non matching v6 cidrs",
cidrs: []string{"10.10.10.0/24", "2001:db8::/64"},
nodePodCIDR: "10.10.10.0/24",
nodePodCIDRs: []string{"10.10.10.0/24", "2001:dba::/64"},
want: true,
},
{
desc: "want true - nil node.Spec.PodCIDRs",
cidrs: []string{"10.10.10.0/24", "2001:db8::/64"},
want: true,
},
{
desc: "want true - matching v6 and non matching v4 cidrs",
cidrs: []string{"10.10.10.0/24", "2001:db8::/64"},
nodePodCIDR: "10.10.1.0/24",
nodePodCIDRs: []string{"10.10.1.0/24", "2001:db8::/64"},
want: true,
},
{
desc: "want true - missing v6",
cidrs: []string{"10.10.10.0/24", "2001:db8::/64"},
nodePodCIDR: "10.10.10.0/24",
nodePodCIDRs: []string{"10.10.10.0/24"},
want: true,
},
} {
var node v1.Node
node.Spec.PodCIDR = tc.nodePodCIDR
node.Spec.PodCIDRs = tc.nodePodCIDRs
netCIDRs, err := netutils.ParseCIDRs(tc.cidrs)
if err != nil {
t.Errorf("failed to parse %v as CIDRs: %v", tc.cidrs, err)
}
logger, _ := ktesting.NewTestContext(t)
t.Run(tc.desc, func(t *testing.T) {
got, err := needPodCIDRsUpdate(logger, &node, netCIDRs)
if tc.wantErr == (err == nil) {
t.Errorf("err: %v, wantErr: %v", err, tc.wantErr)
}
if err == nil && got != tc.want {
t.Errorf("got: %v, want: %v", got, tc.want)
}
})
}
}

236
pkg/controller/nodeipam/ipam/controller_legacyprovider.go
View File

@@ -1,236 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package ipam
import (
"context"
"fmt"
"net"
"sync"
"time"
"k8s.io/klog/v2"
netutils "k8s.io/utils/net"
v1 "k8s.io/api/core/v1"
utilruntime "k8s.io/apimachinery/pkg/util/runtime"
informers "k8s.io/client-go/informers/core/v1"
clientset "k8s.io/client-go/kubernetes"
"k8s.io/client-go/tools/cache"
cloudprovider "k8s.io/cloud-provider"
"k8s.io/kubernetes/pkg/controller/nodeipam/ipam/cidrset"
nodesync "k8s.io/kubernetes/pkg/controller/nodeipam/ipam/sync"
controllerutil "k8s.io/kubernetes/pkg/controller/util/node"
"k8s.io/legacy-cloud-providers/gce"
)
// Config for the IPAM controller.
type Config struct {
// Resync is the default timeout duration when there are no errors.
Resync time.Duration
// MaxBackoff is the maximum timeout when in a error backoff state.
MaxBackoff time.Duration
// InitialRetry is the initial retry interval when an error is reported.
InitialRetry time.Duration
// Mode to use to synchronize.
Mode nodesync.NodeSyncMode
}
// Controller is the controller for synchronizing cluster and cloud node
// pod CIDR range assignments.
type Controller struct {
config *Config
adapter *adapter
lock sync.Mutex
syncers map[string]*nodesync.NodeSync
set *cidrset.CidrSet
}
// NewController returns a new instance of the IPAM controller.
func NewController(
ctx context.Context,
config *Config,
kubeClient clientset.Interface,
cloud cloudprovider.Interface,
clusterCIDR, serviceCIDR *net.IPNet,
nodeCIDRMaskSize int) (*Controller, error) {
if !nodesync.IsValidMode(config.Mode) {
return nil, fmt.Errorf("invalid IPAM controller mode %q", config.Mode)
}
gceCloud, ok := cloud.(*gce.Cloud)
if !ok {
return nil, fmt.Errorf("cloud IPAM controller does not support %q provider", cloud.ProviderName())
}
set, err := cidrset.NewCIDRSet(clusterCIDR, nodeCIDRMaskSize)
if err != nil {
return nil, err
}
c := &Controller{
config: config,
adapter: newAdapter(ctx, kubeClient, gceCloud),
syncers: make(map[string]*nodesync.NodeSync),
set: set,
}
if err := occupyServiceCIDR(c.set, clusterCIDR, serviceCIDR); err != nil {
return nil, err
}
//check whether there is a remaining cidr after occupyServiceCIDR
cidr, err := c.set.AllocateNext()
switch err {
case cidrset.ErrCIDRRangeNoCIDRsRemaining:
return nil, fmt.Errorf("failed after occupy serviceCIDR: %v", err)
case nil:
err := c.set.Release(cidr)
return c, err
default:
return nil, fmt.Errorf("unexpected error when check remaining CIDR range: %v", err)
}
}
// Start initializes the Controller with the existing list of nodes and
// registers the informers for node changes. This will start synchronization
// of the node and cloud CIDR range allocations.
func (c *Controller) Start(logger klog.Logger, nodeInformer informers.NodeInformer) error {
logger.Info("Starting IPAM controller", "config", c.config)
ctx := klog.NewContext(context.TODO(), logger)
nodes, err := listNodes(ctx, c.adapter.k8s)
if err != nil {
return err
}
for _, node := range nodes.Items {
if node.Spec.PodCIDR != "" {
_, cidrRange, err := netutils.ParseCIDRSloppy(node.Spec.PodCIDR)
if err == nil {
c.set.Occupy(cidrRange)
logger.V(3).Info("Occupying CIDR for node", "CIDR", node.Spec.PodCIDR, "node", klog.KObj(&node))
} else {
logger.Error(err, "Node has an invalid CIDR", "node", klog.KObj(&node), "CIDR", node.Spec.PodCIDR)
}
}
func() {
c.lock.Lock()
defer c.lock.Unlock()
// XXX/bowei -- stagger the start of each sync cycle.
syncer := c.newSyncer(node.Name)
c.syncers[node.Name] = syncer
go syncer.Loop(logger, nil)
}()
}
nodeInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
AddFunc: controllerutil.CreateAddNodeHandler(func(node *v1.Node) error {
return c.onAdd(logger, node)
}),
UpdateFunc: controllerutil.CreateUpdateNodeHandler(func(_, newNode *v1.Node) error {
return c.onUpdate(logger, newNode)
}),
DeleteFunc: controllerutil.CreateDeleteNodeHandler(logger, func(node *v1.Node) error {
return c.onDelete(logger, node)
}),
})
return nil
}
func (c *Controller) Run(ctx context.Context) {
defer utilruntime.HandleCrash()
go c.adapter.Run(ctx)
<-ctx.Done()
}
type nodeState struct {
t Timeout
}
func (ns *nodeState) ReportResult(err error) {
ns.t.Update(err == nil)
}
func (ns *nodeState) ResyncTimeout() time.Duration {
return ns.t.Next()
}
func (c *Controller) newSyncer(name string) *nodesync.NodeSync {
ns := &nodeState{
Timeout{
Resync: c.config.Resync,
MaxBackoff: c.config.MaxBackoff,
InitialRetry: c.config.InitialRetry,
},
}
return nodesync.New(ns, c.adapter, c.adapter, c.config.Mode, name, c.set)
}
func (c *Controller) onAdd(logger klog.Logger, node *v1.Node) error {
c.lock.Lock()
defer c.lock.Unlock()
syncer, ok := c.syncers[node.Name]
if !ok {
syncer = c.newSyncer(node.Name)
c.syncers[node.Name] = syncer
go syncer.Loop(logger, nil)
} else {
logger.Info("Add for node that already exists", "node", klog.KObj(node))
}
syncer.Update(node)
return nil
}
func (c *Controller) onUpdate(logger klog.Logger, node *v1.Node) error {
c.lock.Lock()
defer c.lock.Unlock()
if sync, ok := c.syncers[node.Name]; ok {
sync.Update(node)
} else {
logger.Error(nil, "Received update for non-existent node", "node", klog.KObj(node))
return fmt.Errorf("unknown node %q", node.Name)
}
return nil
}
func (c *Controller) onDelete(logger klog.Logger, node *v1.Node) error {
c.lock.Lock()
defer c.lock.Unlock()
if syncer, ok := c.syncers[node.Name]; ok {
syncer.Delete(node)
delete(c.syncers, node.Name)
} else {
logger.Info("Node was already deleted", "node", klog.KObj(node))
}
return nil
}

75
pkg/controller/nodeipam/legacyprovider.go
View File

@@ -1,75 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2019 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package nodeipam
import (
"context"
"fmt"
"net"
coreinformers "k8s.io/client-go/informers/core/v1"
clientset "k8s.io/client-go/kubernetes"
cloudprovider "k8s.io/cloud-provider"
"k8s.io/klog/v2"
"k8s.io/kubernetes/pkg/controller/nodeipam/ipam"
nodesync "k8s.io/kubernetes/pkg/controller/nodeipam/ipam/sync"
)
func createLegacyIPAM(
ctx context.Context,
ic *Controller,
nodeInformer coreinformers.NodeInformer,
cloud cloudprovider.Interface,
kubeClient clientset.Interface,
clusterCIDRs []*net.IPNet,
serviceCIDR *net.IPNet,
nodeCIDRMaskSizes []int,
) (*ipam.Controller, error) {
cfg := &ipam.Config{
Resync: ipamResyncInterval,
MaxBackoff: ipamMaxBackoff,
InitialRetry: ipamInitialBackoff,
}
switch ic.allocatorType {
case ipam.IPAMFromClusterAllocatorType:
cfg.Mode = nodesync.SyncFromCluster
case ipam.IPAMFromCloudAllocatorType:
cfg.Mode = nodesync.SyncFromCloud
}
// we may end up here with no cidr at all in case of FromCloud/FromCluster
var cidr *net.IPNet
if len(clusterCIDRs) > 0 {
cidr = clusterCIDRs[0]
}
logger := klog.FromContext(ctx)
if len(clusterCIDRs) > 1 {
logger.Info("Multiple cidrs were configured with FromCluster or FromCloud. cidrs except first one were discarded")
}
ipamc, err := ipam.NewController(ctx, cfg, kubeClient, cloud, cidr, serviceCIDR, nodeCIDRMaskSizes[0])
if err != nil {
return nil, fmt.Errorf("error creating ipam controller: %w", err)
}
if err := ipamc.Start(logger, nodeInformer); err != nil {
return nil, fmt.Errorf("error trying to Init(): %w", err)
}
return ipamc, nil
}

118
pkg/controller/nodeipam/node_ipam_controller_test.go
View File

@@ -1,118 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2018 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package nodeipam
import (
"context"
"errors"
"net"
"strings"
"testing"
v1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/informers"
"k8s.io/client-go/kubernetes/fake"
"k8s.io/klog/v2/ktesting"
"k8s.io/kubernetes/pkg/controller/nodeipam/ipam"
"k8s.io/kubernetes/pkg/controller/testutil"
"k8s.io/legacy-cloud-providers/gce"
netutils "k8s.io/utils/net"
)
func newTestNodeIpamController(ctx context.Context, clusterCIDR []*net.IPNet, serviceCIDR *net.IPNet, secondaryServiceCIDR *net.IPNet, nodeCIDRMaskSizes []int, allocatorType ipam.CIDRAllocatorType) (*Controller, error) {
clientSet := fake.NewSimpleClientset()
fakeNodeHandler := &testutil.FakeNodeHandler{
Existing: []*v1.Node{
{ObjectMeta: metav1.ObjectMeta{Name: "node0"}},
},
Clientset: fake.NewSimpleClientset(),
}
fakeClient := &fake.Clientset{}
fakeInformerFactory := informers.NewSharedInformerFactory(fakeClient, 0)
fakeNodeInformer := fakeInformerFactory.Core().V1().Nodes()
for _, node := range fakeNodeHandler.Existing {
fakeNodeInformer.Informer().GetStore().Add(node)
}
fakeGCE := gce.NewFakeGCECloud(gce.DefaultTestClusterValues())
return NewNodeIpamController(
ctx,
fakeNodeInformer, fakeGCE, clientSet,
clusterCIDR, serviceCIDR, secondaryServiceCIDR, nodeCIDRMaskSizes, allocatorType,
)
}
// TestNewNodeIpamControllerWithCIDRMasks tests if the controller can be
// created with combinations of network CIDRs and masks.
func TestNewNodeIpamControllerWithCIDRMasks(t *testing.T) {
emptyServiceCIDR := ""
for _, tc := range []struct {
desc string
clusterCIDR string
serviceCIDR string
secondaryServiceCIDR string
maskSize []int
allocatorType ipam.CIDRAllocatorType
expectedError error
}{
{"valid_range_allocator", "10.0.0.0/21", "10.1.0.0/21", emptyServiceCIDR, []int{24}, ipam.RangeAllocatorType, nil},
{"valid_range_allocator_dualstack", "10.0.0.0/21,2000::/48", "10.1.0.0/21", emptyServiceCIDR, []int{24, 64}, ipam.RangeAllocatorType, nil},
{"valid_range_allocator_dualstack_dualstackservice", "10.0.0.0/21,2000::/48", "10.1.0.0/21", "3000::/112", []int{24, 64}, ipam.RangeAllocatorType, nil},
{"valid_cloud_allocator", "10.0.0.0/21", "10.1.0.0/21", emptyServiceCIDR, []int{24}, ipam.CloudAllocatorType, nil},
{"valid_ipam_from_cluster", "10.0.0.0/21", "10.1.0.0/21", emptyServiceCIDR, []int{24}, ipam.IPAMFromClusterAllocatorType, nil},
{"valid_ipam_from_cloud", "10.0.0.0/21", "10.1.0.0/21", emptyServiceCIDR, []int{24}, ipam.IPAMFromCloudAllocatorType, nil},
{"valid_skip_cluster_CIDR_validation_for_cloud_allocator", "invalid", "10.1.0.0/21", emptyServiceCIDR, []int{24}, ipam.CloudAllocatorType, nil},
{"valid_CIDR_larger_than_mask_cloud_allocator", "10.0.0.0/16", "10.1.0.0/21", emptyServiceCIDR, []int{24}, ipam.CloudAllocatorType, nil},
{"invalid_cluster_CIDR", "", "10.1.0.0/21", emptyServiceCIDR, []int{24}, ipam.IPAMFromClusterAllocatorType, errors.New("Controller: Must specify --cluster-cidr if --allocate-node-cidrs is set")},
{"invalid_CIDR_smaller_than_mask_other_allocators", "10.0.0.0/26", "10.1.0.0/21", emptyServiceCIDR, []int{24}, ipam.IPAMFromCloudAllocatorType, errors.New("Controller: Invalid --cluster-cidr, mask size of cluster CIDR must be less than or equal to --node-cidr-mask-size configured for CIDR family")},
{"invalid_serviceCIDR_contains_clusterCIDR", "10.0.0.0/16", "10.0.0.0/8", emptyServiceCIDR, []int{24}, ipam.IPAMFromClusterAllocatorType, errors.New("error creating ipam controller: failed after occupy serviceCIDR: CIDR allocation failed; there are no remaining CIDRs left to allocate in the accepted range")},
{"invalid_CIDR_mask_size", "10.0.0.0/24,2000::/64", "10.1.0.0/21", emptyServiceCIDR, []int{24, 48}, ipam.IPAMFromClusterAllocatorType, errors.New("Controller: Invalid --cluster-cidr, mask size of cluster CIDR must be less than or equal to --node-cidr-mask-size configured for CIDR family")},
} {
test := tc
_, ctx := ktesting.NewTestContext(t)
t.Run(test.desc, func(t *testing.T) {
t.Parallel()
clusterCidrs, err := netutils.ParseCIDRs(strings.Split(test.clusterCIDR, ","))
if err != nil {
clusterCidrs = nil
}
_, serviceCIDRIpNet, err := netutils.ParseCIDRSloppy(test.serviceCIDR)
if err != nil {
serviceCIDRIpNet = nil
}
_, secondaryServiceCIDRIpNet, err := netutils.ParseCIDRSloppy(test.secondaryServiceCIDR)
if err != nil {
secondaryServiceCIDRIpNet = nil
}
_, err = newTestNodeIpamController(ctx, clusterCidrs, serviceCIDRIpNet, secondaryServiceCIDRIpNet, test.maskSize, test.allocatorType)
if test.expectedError == nil {
if err != nil {
t.Errorf("Test %s, unexpected error: %v", test.desc, err)
}
} else {
if err.Error() != test.expectedError.Error() {
t.Errorf("Test %s, got error: %v, expected error: %v", test.desc, err, test.expectedError)
}
}
})
}
}

3
pkg/controller/nodeipam/nolegacyprovider.go
View File

@@ -1,6 +1,3 @@
//go:build providerless
// +build providerless
/*
Copyright 2019 The Kubernetes Authors.

9
pkg/credentialprovider/gcp/OWNERS
View File

@@ -1,9 +0,0 @@
# See the OWNERS docs at https://go.k8s.io/owners
reviewers:
- thockin
- deads2k
- yujuhong
- derekwaynecarr
- mikedanese
- dims

19
pkg/credentialprovider/gcp/doc.go
View File

@@ -1,19 +0,0 @@
/*
Copyright 2014 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Package gcp contains implementations of DockerConfigProvider
// for Google Cloud Platform.
package gcp // import "k8s.io/kubernetes/pkg/credentialprovider/gcp"

273
pkg/credentialprovider/gcp/metadata.go
View File

@@ -1,273 +0,0 @@
/*
Copyright 2014 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gcp
import (
"encoding/json"
"net/http"
"os"
"os/exec"
"runtime"
"strings"
"sync"
"time"
utilnet "k8s.io/apimachinery/pkg/util/net"
"k8s.io/cloud-provider/credentialconfig"
"k8s.io/klog/v2"
"k8s.io/kubernetes/pkg/credentialprovider"
"k8s.io/legacy-cloud-providers/gce/gcpcredential"
)
const (
metadataURL = "http://metadata.google.internal./computeMetadata/v1/"
metadataAttributes = metadataURL + "instance/attributes/"
// DockerConfigKey is the URL of the dockercfg metadata key used by DockerConfigKeyProvider.
DockerConfigKey = metadataAttributes + "google-dockercfg"
// DockerConfigURLKey is the URL of the dockercfg metadata key used by DockerConfigURLKeyProvider.
DockerConfigURLKey = metadataAttributes + "google-dockercfg-url"
serviceAccounts = metadataURL + "instance/service-accounts/"
metadataScopes = metadataURL + "instance/service-accounts/default/scopes"
// StorageScopePrefix is the prefix checked by ContainerRegistryProvider.Enabled.
StorageScopePrefix = "https://www.googleapis.com/auth/devstorage"
cloudPlatformScopePrefix = "https://www.googleapis.com/auth/cloud-platform"
defaultServiceAccount = "default/"
)
// gceProductNameFile is the product file path that contains the cloud service name.
// This is a variable instead of a const to enable testing.
var gceProductNameFile = "/sys/class/dmi/id/product_name"
var metadataHeader = &http.Header{
"Metadata-Flavor": []string{"Google"},
}
var warnOnce sync.Once
// init registers the various means by which credentials may
// be resolved on GCP.
func init() {
tr := utilnet.SetTransportDefaults(&http.Transport{})
metadataHTTPClientTimeout := time.Second * 10
httpClient := &http.Client{
Transport: tr,
Timeout: metadataHTTPClientTimeout,
}
credentialprovider.RegisterCredentialProvider("google-dockercfg",
&credentialprovider.CachingDockerConfigProvider{
Provider: &DockerConfigKeyProvider{
MetadataProvider: MetadataProvider{Client: httpClient},
},
Lifetime: 60 * time.Second,
})
credentialprovider.RegisterCredentialProvider("google-dockercfg-url",
&credentialprovider.CachingDockerConfigProvider{
Provider: &DockerConfigURLKeyProvider{
MetadataProvider: MetadataProvider{Client: httpClient},
},
Lifetime: 60 * time.Second,
})
credentialprovider.RegisterCredentialProvider("google-container-registry",
// Never cache this. The access token is already
// cached by the metadata service.
&ContainerRegistryProvider{
MetadataProvider: MetadataProvider{Client: httpClient},
})
}
// MetadataProvider is a DockerConfigProvider that reads its configuration from Google
// Compute Engine metadata.
type MetadataProvider struct {
Client *http.Client
}
// DockerConfigKeyProvider is a DockerConfigProvider that reads its configuration from a specific
// Google Compute Engine metadata key: 'google-dockercfg'.
type DockerConfigKeyProvider struct {
MetadataProvider
}
// DockerConfigURLKeyProvider is a DockerConfigProvider that reads its configuration from a URL read from
// a specific Google Compute Engine metadata key: 'google-dockercfg-url'.
type DockerConfigURLKeyProvider struct {
MetadataProvider
}
// ContainerRegistryProvider is a DockerConfigProvider that provides a dockercfg with:
//
// Username: "_token"
// Password: "{access token from metadata}"
type ContainerRegistryProvider struct {
MetadataProvider
}
// Returns true if it finds a local GCE VM.
// Looks at a product file that is an undocumented API.
func onGCEVM() bool {
var name string
if runtime.GOOS == "windows" {
data, err := exec.Command("wmic", "computersystem", "get", "model").Output()
if err != nil {
return false
}
fields := strings.Split(strings.TrimSpace(string(data)), "\r\n")
if len(fields) != 2 {
klog.V(2).Infof("Received unexpected value retrieving system model: %q", string(data))
return false
}
name = fields[1]
} else {
data, err := os.ReadFile(gceProductNameFile)
if err != nil {
klog.V(2).Infof("Error while reading product_name: %v", err)
return false
}
name = strings.TrimSpace(string(data))
}
return name == "Google" || name == "Google Compute Engine"
}
// Enabled implements DockerConfigProvider for all of the Google implementations.
func (g *MetadataProvider) Enabled() bool {
onGCE := onGCEVM()
if !onGCE {
return false
}
if credentialprovider.AreLegacyCloudCredentialProvidersDisabled() {
warnOnce.Do(func() {
klog.V(4).Infof("GCP credential provider is now disabled. Please refer to sig-cloud-provider for guidance on external credential provider integration for GCP")
})
return false
}
return true
}
// Provide implements DockerConfigProvider
func (g *DockerConfigKeyProvider) Provide(image string) credentialprovider.DockerConfig {
return registryToDocker(gcpcredential.ProvideConfigKey(g.Client, image))
}
// Provide implements DockerConfigProvider
func (g *DockerConfigURLKeyProvider) Provide(image string) credentialprovider.DockerConfig {
return registryToDocker(gcpcredential.ProvideURLKey(g.Client, image))
}
// runWithBackoff runs input function `f` with an exponential backoff.
// Note that this method can block indefinitely.
func runWithBackoff(f func() ([]byte, error)) []byte {
var backoff = 100 * time.Millisecond
const maxBackoff = time.Minute
for {
value, err := f()
if err == nil {
return value
}
time.Sleep(backoff)
backoff = backoff * 2
if backoff > maxBackoff {
backoff = maxBackoff
}
}
}
// Enabled implements a special metadata-based check, which verifies the
// storage scope is available on the GCE VM.
// If running on a GCE VM, check if 'default' service account exists.
// If it does not exist, assume that registry is not enabled.
// If default service account exists, check if relevant scopes exist in the default service account.
// The metadata service can become temporarily inaccesible. Hence all requests to the metadata
// service will be retried until the metadata server returns a `200`.
// It is expected that "http://metadata.google.internal./computeMetadata/v1/instance/service-accounts/" will return a `200`
// and "http://metadata.google.internal./computeMetadata/v1/instance/service-accounts/default/scopes" will also return `200`.
// More information on metadata service can be found here - https://cloud.google.com/compute/docs/storing-retrieving-metadata
func (g *ContainerRegistryProvider) Enabled() bool {
if !onGCEVM() {
return false
}
if credentialprovider.AreLegacyCloudCredentialProvidersDisabled() {
warnOnce.Do(func() {
klog.V(4).Infof("GCP credential provider is now disabled. Please refer to sig-cloud-provider for guidance on external credential provider integration for GCP")
})
return false
}
// Given that we are on GCE, we should keep retrying until the metadata server responds.
value := runWithBackoff(func() ([]byte, error) {
value, err := gcpcredential.ReadURL(serviceAccounts, g.Client, metadataHeader)
if err != nil {
klog.V(2).Infof("Failed to Get service accounts from gce metadata server: %v", err)
}
return value, err
})
// We expect the service account to return a list of account directories separated by newlines, e.g.,
// sv-account-name1/
// sv-account-name2/
// ref: https://cloud.google.com/compute/docs/storing-retrieving-metadata
defaultServiceAccountExists := false
for _, sa := range strings.Split(string(value), "\n") {
if strings.TrimSpace(sa) == defaultServiceAccount {
defaultServiceAccountExists = true
break
}
}
if !defaultServiceAccountExists {
klog.V(2).Infof("'default' service account does not exist. Found following service accounts: %q", string(value))
return false
}
url := metadataScopes + "?alt=json"
value = runWithBackoff(func() ([]byte, error) {
value, err := gcpcredential.ReadURL(url, g.Client, metadataHeader)
if err != nil {
klog.V(2).Infof("Failed to Get scopes in default service account from gce metadata server: %v", err)
}
return value, err
})
var scopes []string
if err := json.Unmarshal(value, &scopes); err != nil {
klog.Errorf("Failed to unmarshal scopes: %v", err)
return false
}
for _, v := range scopes {
// cloudPlatformScope implies storage scope.
if strings.HasPrefix(v, StorageScopePrefix) || strings.HasPrefix(v, cloudPlatformScopePrefix) {
return true
}
}
klog.Warningf("Google container registry is disabled, no storage scope is available: %s", value)
return false
}
// Provide implements DockerConfigProvider
func (g *ContainerRegistryProvider) Provide(image string) credentialprovider.DockerConfig {
return registryToDocker(gcpcredential.ProvideContainerRegistry(g.Client, image))
}
func registryToDocker(registryConfig credentialconfig.RegistryConfig) credentialprovider.DockerConfig {
dockerConfig := credentialprovider.DockerConfig{}
for k, v := range registryConfig {
dockerConfig[k] = credentialprovider.DockerConfigEntry{
Username: v.Username,
Password: v.Password,
Email: v.Email,
}
}
return dockerConfig
}

454
pkg/credentialprovider/gcp/metadata_test.go
View File

@@ -1,454 +0,0 @@
/*
Copyright 2014 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gcp
import (
"encoding/base64"
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"net/url"
"os"
"reflect"
"runtime"
"strings"
"testing"
utilnet "k8s.io/apimachinery/pkg/util/net"
utilfeature "k8s.io/apiserver/pkg/util/feature"
featuregatetesting "k8s.io/component-base/featuregate/testing"
"k8s.io/kubernetes/pkg/credentialprovider"
kubefeatures "k8s.io/kubernetes/pkg/features"
"k8s.io/legacy-cloud-providers/gce/gcpcredential"
)
func createProductNameFile() (string, error) {
file, err := os.CreateTemp("", "")
if err != nil {
return "", fmt.Errorf("failed to create temporary test file: %v", err)
}
return file.Name(), os.WriteFile(file.Name(), []byte("Google"), 0600)
}
// The tests here are run in this fashion to ensure TestAllProvidersNoMetadata
// is run after the others, since that test currently relies upon the file
// referenced by gceProductNameFile being removed, which is the opposite of
// the other tests
func TestMetadata(t *testing.T) {
// This test requires onGCEVM to return True. On Linux, this can be faked by creating a
// Product Name File. But on Windows, onGCEVM makes the following syscall instead:
// wmic computersystem get model
if runtime.GOOS == "windows" && !onGCEVM() {
t.Skip("Skipping test on Windows, not on GCE.")
}
featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, kubefeatures.DisableKubeletCloudCredentialProviders, false)
var err error
gceProductNameFile, err = createProductNameFile()
if err != nil {
t.Errorf("failed to create gce product name file: %v", err)
}
defer os.Remove(gceProductNameFile)
t.Run("productNameDependent", func(t *testing.T) {
t.Run("DockerKeyringFromGoogleDockerConfigMetadata",
DockerKeyringFromGoogleDockerConfigMetadata)
t.Run("DockerKeyringFromGoogleDockerConfigMetadataUrl",
DockerKeyringFromGoogleDockerConfigMetadataURL)
t.Run("ContainerRegistryNoServiceAccount",
ContainerRegistryNoServiceAccount)
t.Run("ContainerRegistryBasics",
ContainerRegistryBasics)
t.Run("ContainerRegistryNoStorageScope",
ContainerRegistryNoStorageScope)
t.Run("ComputePlatformScopeSubstitutesStorageScope",
ComputePlatformScopeSubstitutesStorageScope)
})
// We defer os.Remove in case of an unexpected exit, but this os.Remove call
// is the normal teardown call so AllProvidersNoMetadata executes properly
os.Remove(gceProductNameFile)
t.Run("AllProvidersNoMetadata",
AllProvidersNoMetadata)
}
func DockerKeyringFromGoogleDockerConfigMetadata(t *testing.T) {
t.Parallel()
registryURL := "hello.kubernetes.io"
email := "foo@bar.baz"
username := "foo"
password := "bar" // Fake value for testing.
auth := base64.StdEncoding.EncodeToString([]byte(fmt.Sprintf("%s:%s", username, password)))
sampleDockerConfig := fmt.Sprintf(`{
"https://%s": {
"email": %q,
"auth": %q
}
}`, registryURL, email, auth)
const probeEndpoint = "/computeMetadata/v1/"
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// Only serve the one metadata key.
if probeEndpoint == r.URL.Path {
w.WriteHeader(http.StatusOK)
} else if strings.HasSuffix(gcpcredential.DockerConfigKey, r.URL.Path) {
w.WriteHeader(http.StatusOK)
w.Header().Set("Content-Type", "application/json")
fmt.Fprintln(w, sampleDockerConfig)
} else {
http.Error(w, "", http.StatusNotFound)
}
}))
defer server.Close()
// Make a transport that reroutes all traffic to the example server
transport := utilnet.SetTransportDefaults(&http.Transport{
Proxy: func(req *http.Request) (*url.URL, error) {
return url.Parse(server.URL + req.URL.Path)
},
})
keyring := &credentialprovider.BasicDockerKeyring{}
provider := &DockerConfigKeyProvider{
MetadataProvider: MetadataProvider{Client: &http.Client{Transport: transport}},
}
if !provider.Enabled() {
t.Errorf("Provider is unexpectedly disabled")
}
keyring.Add(provider.Provide(""))
creds, ok := keyring.Lookup(registryURL)
if !ok {
t.Errorf("Didn't find expected URL: %s", registryURL)
return
}
if len(creds) > 1 {
t.Errorf("Got more hits than expected: %s", creds)
}
val := creds[0]
if username != val.Username {
t.Errorf("Unexpected username value, want: %s, got: %s", username, val.Username)
}
if password != val.Password {
t.Errorf("Unexpected password value, want: %s, got: %s", password, val.Password)
}
if email != val.Email {
t.Errorf("Unexpected email value, want: %s, got: %s", email, val.Email)
}
}
func DockerKeyringFromGoogleDockerConfigMetadataURL(t *testing.T) {
t.Parallel()
registryURL := "hello.kubernetes.io"
email := "foo@bar.baz"
username := "foo"
password := "bar" // Fake value for testing.
auth := base64.StdEncoding.EncodeToString([]byte(fmt.Sprintf("%s:%s", username, password)))
sampleDockerConfig := fmt.Sprintf(`{
"https://%s": {
"email": %q,
"auth": %q
}
}`, registryURL, email, auth)
const probeEndpoint = "/computeMetadata/v1/"
const valueEndpoint = "/my/value"
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// Only serve the URL key and the value endpoint
if probeEndpoint == r.URL.Path {
w.WriteHeader(http.StatusOK)
} else if valueEndpoint == r.URL.Path {
w.WriteHeader(http.StatusOK)
w.Header().Set("Content-Type", "application/json")
fmt.Fprintln(w, sampleDockerConfig)
} else if strings.HasSuffix(gcpcredential.DockerConfigURLKey, r.URL.Path) {
w.WriteHeader(http.StatusOK)
w.Header().Set("Content-Type", "application/text")
fmt.Fprint(w, "http://foo.bar.com"+valueEndpoint)
} else {
http.Error(w, "", http.StatusNotFound)
}
}))
defer server.Close()
// Make a transport that reroutes all traffic to the example server
transport := utilnet.SetTransportDefaults(&http.Transport{
Proxy: func(req *http.Request) (*url.URL, error) {
return url.Parse(server.URL + req.URL.Path)
},
})
keyring := &credentialprovider.BasicDockerKeyring{}
provider := &DockerConfigURLKeyProvider{
MetadataProvider: MetadataProvider{Client: &http.Client{Transport: transport}},
}
if !provider.Enabled() {
t.Errorf("Provider is unexpectedly disabled")
}
keyring.Add(provider.Provide(""))
creds, ok := keyring.Lookup(registryURL)
if !ok {
t.Errorf("Didn't find expected URL: %s", registryURL)
return
}
if len(creds) > 1 {
t.Errorf("Got more hits than expected: %s", creds)
}
val := creds[0]
if username != val.Username {
t.Errorf("Unexpected username value, want: %s, got: %s", username, val.Username)
}
if password != val.Password {
t.Errorf("Unexpected password value, want: %s, got: %s", password, val.Password)
}
if email != val.Email {
t.Errorf("Unexpected email value, want: %s, got: %s", email, val.Email)
}
}
func ContainerRegistryBasics(t *testing.T) {
t.Parallel()
registryURLs := []string{"container.cloud.google.com", "eu.gcr.io", "us-west2-docker.pkg.dev"}
for _, registryURL := range registryURLs {
t.Run(registryURL, func(t *testing.T) {
email := "1234@project.gserviceaccount.com"
token := &gcpcredential.TokenBlob{AccessToken: "ya26.lots-of-indiscernible-garbage"} // Fake value for testing.
const (
serviceAccountsEndpoint = "/computeMetadata/v1/instance/service-accounts/"
defaultEndpoint = "/computeMetadata/v1/instance/service-accounts/default/"
scopeEndpoint = defaultEndpoint + "scopes"
emailEndpoint = defaultEndpoint + "email"
tokenEndpoint = defaultEndpoint + "token"
)
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// Only serve the URL key and the value endpoint
if scopeEndpoint == r.URL.Path {
w.WriteHeader(http.StatusOK)
w.Header().Set("Content-Type", "application/json")
fmt.Fprintf(w, `["%s.read_write"]`, gcpcredential.StorageScopePrefix)
} else if emailEndpoint == r.URL.Path {
w.WriteHeader(http.StatusOK)
fmt.Fprint(w, email)
} else if tokenEndpoint == r.URL.Path {
w.WriteHeader(http.StatusOK)
w.Header().Set("Content-Type", "application/json")
bytes, err := json.Marshal(token)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
fmt.Fprintln(w, string(bytes))
} else if serviceAccountsEndpoint == r.URL.Path {
w.WriteHeader(http.StatusOK)
fmt.Fprintln(w, "default/\ncustom")
} else {
http.Error(w, "", http.StatusNotFound)
}
}))
defer server.Close()
// Make a transport that reroutes all traffic to the example server
transport := utilnet.SetTransportDefaults(&http.Transport{
Proxy: func(req *http.Request) (*url.URL, error) {
return url.Parse(server.URL + req.URL.Path)
},
})
keyring := &credentialprovider.BasicDockerKeyring{}
provider := &ContainerRegistryProvider{
MetadataProvider: MetadataProvider{Client: &http.Client{Transport: transport}},
}
if !provider.Enabled() {
t.Errorf("Provider is unexpectedly disabled")
}
keyring.Add(provider.Provide(""))
creds, ok := keyring.Lookup(registryURL)
if !ok {
t.Errorf("Didn't find expected URL: %s", registryURL)
return
}
if len(creds) > 1 {
t.Errorf("Got more hits than expected: %s", creds)
}
val := creds[0]
if val.Username != "_token" {
t.Errorf("Unexpected username value, want: %s, got: %s", "_token", val.Username)
}
if token.AccessToken != val.Password {
t.Errorf("Unexpected password value, want: %s, got: %s", token.AccessToken, val.Password)
}
if email != val.Email {
t.Errorf("Unexpected email value, want: %s, got: %s", email, val.Email)
}
})
}
}
func ContainerRegistryNoServiceAccount(t *testing.T) {
const (
serviceAccountsEndpoint = "/computeMetadata/v1/instance/service-accounts/"
)
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// Only serve the URL key and the value endpoint
if serviceAccountsEndpoint == r.URL.Path {
w.WriteHeader(http.StatusOK)
w.Header().Set("Content-Type", "application/json")
bytes, err := json.Marshal([]string{})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
fmt.Fprintln(w, string(bytes))
} else {
http.Error(w, "", http.StatusNotFound)
}
}))
defer server.Close()
// Make a transport that reroutes all traffic to the example server
transport := utilnet.SetTransportDefaults(&http.Transport{
Proxy: func(req *http.Request) (*url.URL, error) {
return url.Parse(server.URL + req.URL.Path)
},
})
provider := &ContainerRegistryProvider{
MetadataProvider: MetadataProvider{Client: &http.Client{Transport: transport}},
}
if provider.Enabled() {
t.Errorf("Provider is unexpectedly enabled")
}
}
func ContainerRegistryNoStorageScope(t *testing.T) {
t.Parallel()
const (
serviceAccountsEndpoint = "/computeMetadata/v1/instance/service-accounts/"
defaultEndpoint = "/computeMetadata/v1/instance/service-accounts/default/"
scopeEndpoint = defaultEndpoint + "scopes"
)
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// Only serve the URL key and the value endpoint
if scopeEndpoint == r.URL.Path {
w.WriteHeader(http.StatusOK)
w.Header().Set("Content-Type", "application/json")
fmt.Fprint(w, `["https://www.googleapis.com/auth/compute.read_write"]`)
} else if serviceAccountsEndpoint == r.URL.Path {
w.WriteHeader(http.StatusOK)
fmt.Fprintln(w, "default/\ncustom")
} else {
http.Error(w, "", http.StatusNotFound)
}
}))
defer server.Close()
// Make a transport that reroutes all traffic to the example server
transport := utilnet.SetTransportDefaults(&http.Transport{
Proxy: func(req *http.Request) (*url.URL, error) {
return url.Parse(server.URL + req.URL.Path)
},
})
provider := &ContainerRegistryProvider{
MetadataProvider: MetadataProvider{Client: &http.Client{Transport: transport}},
}
if provider.Enabled() {
t.Errorf("Provider is unexpectedly enabled")
}
}
func ComputePlatformScopeSubstitutesStorageScope(t *testing.T) {
t.Parallel()
const (
serviceAccountsEndpoint = "/computeMetadata/v1/instance/service-accounts/"
defaultEndpoint = "/computeMetadata/v1/instance/service-accounts/default/"
scopeEndpoint = defaultEndpoint + "scopes"
)
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// Only serve the URL key and the value endpoint
if scopeEndpoint == r.URL.Path {
w.WriteHeader(http.StatusOK)
w.Header().Set("Content-Type", "application/json")
fmt.Fprint(w, `["https://www.googleapis.com/auth/compute.read_write","https://www.googleapis.com/auth/cloud-platform.read-only"]`)
} else if serviceAccountsEndpoint == r.URL.Path {
w.WriteHeader(http.StatusOK)
w.Header().Set("Content-Type", "application/json")
fmt.Fprintln(w, "default/\ncustom")
} else {
http.Error(w, "", http.StatusNotFound)
}
}))
defer server.Close()
// Make a transport that reroutes all traffic to the example server
transport := utilnet.SetTransportDefaults(&http.Transport{
Proxy: func(req *http.Request) (*url.URL, error) {
return url.Parse(server.URL + req.URL.Path)
},
})
provider := &ContainerRegistryProvider{
MetadataProvider: MetadataProvider{Client: &http.Client{Transport: transport}},
}
if !provider.Enabled() {
t.Errorf("Provider is unexpectedly disabled")
}
}
func AllProvidersNoMetadata(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
http.Error(w, "", http.StatusNotFound)
}))
defer server.Close()
// Make a transport that reroutes all traffic to the example server
transport := utilnet.SetTransportDefaults(&http.Transport{
Proxy: func(req *http.Request) (*url.URL, error) {
return url.Parse(server.URL + req.URL.Path)
},
})
providers := []credentialprovider.DockerConfigProvider{
&DockerConfigKeyProvider{
MetadataProvider: MetadataProvider{Client: &http.Client{Transport: transport}},
},
&DockerConfigURLKeyProvider{
MetadataProvider: MetadataProvider{Client: &http.Client{Transport: transport}},
},
&ContainerRegistryProvider{
MetadataProvider: MetadataProvider{Client: &http.Client{Transport: transport}},
},
}
for _, provider := range providers {
if provider.Enabled() {
t.Errorf("Provider %s is unexpectedly enabled", reflect.TypeOf(provider).String())
}
}
}

27
pkg/kubelet/cadvisor/cadvisor_cloudproviders.go
View File

@@ -1,27 +0,0 @@
//go:build linux && !providerless
// +build linux,!providerless
/*
Copyright 2021 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package cadvisor
import (
// Register cloud info providers.
// TODO(#68522): Remove this in 1.20+ once the cAdvisor endpoints are removed.
_ "github.com/google/cadvisor/utils/cloudinfo/azure"
_ "github.com/google/cadvisor/utils/cloudinfo/gce"
)

36
staging/src/k8s.io/client-go/plugin/pkg/client/auth/azure/azure_stub.go
View File

@@ -1,36 +0,0 @@
/*
Copyright 2022 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package azure
import (
"errors"
"k8s.io/client-go/rest"
"k8s.io/klog/v2"
)
func init() {
if err := rest.RegisterAuthProviderPlugin("azure", newAzureAuthProvider); err != nil {
klog.Fatalf("Failed to register azure auth plugin: %v", err)
}
}
func newAzureAuthProvider(_ string, _ map[string]string, _ rest.AuthProviderConfigPersister) (rest.AuthProvider, error) {
return nil, errors.New(`The azure auth plugin has been removed.
Please use the https://github.com/Azure/kubelogin kubectl/client-go credential plugin instead.
See https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins for further details`)
}

36
staging/src/k8s.io/client-go/plugin/pkg/client/auth/gcp/gcp_stub.go
View File

@@ -1,36 +0,0 @@
/*
Copyright 2022 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gcp
import (
"errors"
"k8s.io/client-go/rest"
"k8s.io/klog/v2"
)
func init() {
if err := rest.RegisterAuthProviderPlugin("gcp", newGCPAuthProvider); err != nil {
klog.Fatalf("Failed to register gcp auth plugin: %v", err)
}
}
func newGCPAuthProvider(_ string, _ map[string]string, _ rest.AuthProviderConfigPersister) (rest.AuthProvider, error) {
return nil, errors.New(`The gcp auth plugin has been removed.
Please use the "gke-gcloud-auth-plugin" kubectl/client-go credential plugin instead.
See https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke for further details`)
}

26
staging/src/k8s.io/client-go/plugin/pkg/client/auth/plugins_providers.go
View File

@@ -1,26 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2016 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package auth
import (
// Initialize client auth plugins for cloud providers.
_ "k8s.io/client-go/plugin/pkg/client/auth/azure"
_ "k8s.io/client-go/plugin/pkg/client/auth/gcp"
)

10
staging/src/k8s.io/legacy-cloud-providers/gce/OWNERS
View File

@@ -1,10 +0,0 @@
# See the OWNERS docs at https://go.k8s.io/owners
# We are no longer accepting features into k8s.io/legacy-cloud-providers.
# Any kind/feature PRs must be approved by SIG Cloud Provider going forward.
emeritus_approvers:
- saad-ali
- jingxu97
- bowei
- freehan
- mrhohn
- cheftako

19
staging/src/k8s.io/legacy-cloud-providers/gce/doc.go
View File

@@ -1,19 +0,0 @@
/*
Copyright 2014 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Package gce is an implementation of Interface, LoadBalancer
// and Instances for Google Compute Engine.
package gce // import "k8s.io/legacy-cloud-providers/gce"

967
staging/src/k8s.io/legacy-cloud-providers/gce/gce.go
View File

@@ -1,967 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2014 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
"context"
"fmt"
"io"
"net/http"
"runtime"
"strconv"
"strings"
"sync"
"time"
gcfg "gopkg.in/gcfg.v1"
"cloud.google.com/go/compute/metadata"
"golang.org/x/oauth2"
"golang.org/x/oauth2/google"
computealpha "google.golang.org/api/compute/v0.alpha"
computebeta "google.golang.org/api/compute/v0.beta"
compute "google.golang.org/api/compute/v1"
container "google.golang.org/api/container/v1"
"google.golang.org/api/option"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud"
v1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/util/sets"
"k8s.io/apimachinery/pkg/util/wait"
"k8s.io/client-go/informers"
clientset "k8s.io/client-go/kubernetes"
"k8s.io/client-go/kubernetes/scheme"
v1core "k8s.io/client-go/kubernetes/typed/core/v1"
"k8s.io/client-go/pkg/version"
"k8s.io/client-go/tools/cache"
"k8s.io/client-go/tools/record"
"k8s.io/client-go/util/flowcontrol"
cloudprovider "k8s.io/cloud-provider"
"k8s.io/klog/v2"
)
const (
// ProviderName is the official const representation of the Google Cloud Provider
ProviderName = "gce"
k8sNodeRouteTag = "k8s-node-route"
// AffinityTypeNone - no session affinity.
gceAffinityTypeNone = "NONE"
// AffinityTypeClientIP - affinity based on Client IP.
gceAffinityTypeClientIP = "CLIENT_IP"
operationPollInterval = time.Second
maxTargetPoolCreateInstances = 200
maxInstancesPerTargetPoolUpdate = 1000
// HTTP Load Balancer parameters
// Configure 8 second period for external health checks.
gceHcCheckIntervalSeconds = int64(8)
gceHcTimeoutSeconds = int64(1)
// Start sending requests as soon as a pod is found on the node.
gceHcHealthyThreshold = int64(1)
// Defaults to 3 * 8 = 24 seconds before the LB will steer traffic away.
gceHcUnhealthyThreshold = int64(3)
gceComputeAPIEndpoint = "https://www.googleapis.com/compute/v1/"
gceComputeAPIEndpointBeta = "https://www.googleapis.com/compute/beta/"
)
var _ cloudprovider.Interface = (*Cloud)(nil)
var _ cloudprovider.Instances = (*Cloud)(nil)
var _ cloudprovider.LoadBalancer = (*Cloud)(nil)
var _ cloudprovider.Routes = (*Cloud)(nil)
var _ cloudprovider.Zones = (*Cloud)(nil)
var _ cloudprovider.PVLabeler = (*Cloud)(nil)
var _ cloudprovider.Clusters = (*Cloud)(nil)
// Cloud is an implementation of Interface, LoadBalancer and Instances for Google Compute Engine.
type Cloud struct {
// ClusterID contains functionality for getting (and initializing) the ingress-uid. Call Cloud.Initialize()
// for the cloudprovider to start watching the configmap.
ClusterID ClusterID
// initializer is used for lazy initialization of subnetworkURL
// and isLegacyNetwork fields if they are not passed via the config.
// The reason is to avoid GCE API calls to initialize them if they
// will never be used. This is especially important when
// it is run from Kubelets, as there can be thousands of them.
subnetworkURLAndIsLegacyNetworkInitializer sync.Once
service *compute.Service
serviceBeta *computebeta.Service
serviceAlpha *computealpha.Service
containerService *container.Service
tpuService *tpuService
client clientset.Interface
clientBuilder cloudprovider.ControllerClientBuilder
eventBroadcaster record.EventBroadcaster
eventRecorder record.EventRecorder
projectID string
region string
regional bool
localZone string // The zone in which we are running
// managedZones will be set to the 1 zone if running a single zone cluster
// it will be set to ALL zones in region for any multi-zone cluster
// Use GetAllCurrentZones to get only zones that contain nodes
managedZones []string
networkURL string
// unsafeIsLegacyNetwork should be used only via IsLegacyNetwork() accessor,
// to ensure it was properly initialized.
unsafeIsLegacyNetwork bool
// unsafeSubnetworkURL should be used only via SubnetworkURL() accessor,
// to ensure it was properly initialized.
unsafeSubnetworkURL string
// DEPRECATED: Do not rely on this value as it may be incorrect.
secondaryRangeName string
networkProjectID string
onXPN bool
nodeTags []string // List of tags to use on firewall rules for load balancers
lastComputedNodeTags []string // List of node tags calculated in GetHostTags()
lastKnownNodeNames sets.String // List of hostnames used to calculate lastComputedHostTags in GetHostTags(names)
computeNodeTagLock sync.Mutex // Lock for computing and setting node tags
nodeInstancePrefix string // If non-"", an advisory prefix for all nodes in the cluster
useMetadataServer bool
operationPollRateLimiter flowcontrol.RateLimiter
manager diskServiceManager
// Lock for access to nodeZones
nodeZonesLock sync.Mutex
// nodeZones is a mapping from Zone to a sets.String of Node's names in the Zone
// it is updated by the nodeInformer
nodeZones map[string]sets.String
nodeInformerSynced cache.InformerSynced
// sharedResourceLock is used to serialize GCE operations that may mutate shared state to
// prevent inconsistencies. For example, load balancers manipulation methods will take the
// lock to prevent shared resources from being prematurely deleted while the operation is
// in progress.
sharedResourceLock sync.Mutex
// AlphaFeatureGate gates gce alpha features in Cloud instance.
// Related wrapper functions that interacts with gce alpha api should examine whether
// the corresponding api is enabled.
// If not enabled, it should return error.
AlphaFeatureGate *AlphaFeatureGate
// New code generated interface to the GCE compute library.
c cloud.Cloud
// Keep a reference of this around so we can inject a new cloud.RateLimiter implementation.
s *cloud.Service
metricsCollector loadbalancerMetricsCollector
// the compute API endpoint with the `projects/` element.
projectsBasePath string
}
// ConfigGlobal is the in memory representation of the gce.conf config data
// TODO: replace gcfg with json
type ConfigGlobal struct {
TokenURL string `gcfg:"token-url"`
TokenBody string `gcfg:"token-body" datapolicy:"token"`
// ProjectID and NetworkProjectID can either be the numeric or string-based
// unique identifier that starts with [a-z].
ProjectID string `gcfg:"project-id"`
// NetworkProjectID refers to the project which owns the network being used.
NetworkProjectID string `gcfg:"network-project-id"`
NetworkName string `gcfg:"network-name"`
SubnetworkName string `gcfg:"subnetwork-name"`
// DEPRECATED: Do not rely on this value as it may be incorrect.
// SecondaryRangeName is the name of the secondary range to allocate IP
// aliases. The secondary range must be present on the subnetwork the
// cluster is attached to.
SecondaryRangeName string `gcfg:"secondary-range-name"`
NodeTags []string `gcfg:"node-tags"`
NodeInstancePrefix string `gcfg:"node-instance-prefix"`
Regional bool `gcfg:"regional"`
Multizone bool `gcfg:"multizone"`
// APIEndpoint is the GCE compute API endpoint to use. If this is blank,
// then the default endpoint is used.
APIEndpoint string `gcfg:"api-endpoint"`
// ContainerAPIEndpoint is the GCE container API endpoint to use. If this is blank,
// then the default endpoint is used.
ContainerAPIEndpoint string `gcfg:"container-api-endpoint"`
// LocalZone specifies the GCE zone that gce cloud client instance is
// located in (i.e. where the controller will be running). If this is
// blank, then the local zone will be discovered via the metadata server.
LocalZone string `gcfg:"local-zone"`
// Default to none.
// For example: MyFeatureFlag
AlphaFeatures []string `gcfg:"alpha-features"`
}
// ConfigFile is the struct used to parse the /etc/gce.conf configuration file.
// NOTE: Cloud config files should follow the same Kubernetes deprecation policy as
// flags or CLIs. Config fields should not change behavior in incompatible ways and
// should be deprecated for at least 2 release prior to removing.
// See https://kubernetes.io/docs/reference/using-api/deprecation-policy/#deprecating-a-flag-or-cli
// for more details.
type ConfigFile struct {
Global ConfigGlobal `gcfg:"global"`
}
// CloudConfig includes all the necessary configuration for creating Cloud
type CloudConfig struct {
APIEndpoint string
ContainerAPIEndpoint string
ProjectID string
NetworkProjectID string
Region string
Regional bool
Zone string
ManagedZones []string
NetworkName string
NetworkURL string
SubnetworkName string
SubnetworkURL string
// DEPRECATED: Do not rely on this value as it may be incorrect.
SecondaryRangeName string
NodeTags []string
NodeInstancePrefix string
TokenSource oauth2.TokenSource
UseMetadataServer bool
AlphaFeatureGate *AlphaFeatureGate
}
func init() {
cloudprovider.RegisterCloudProvider(
ProviderName,
func(config io.Reader) (cloudprovider.Interface, error) {
return newGCECloud(config)
})
}
// Services is the set of all versions of the compute service.
type Services struct {
// GA, Alpha, Beta versions of the compute API.
GA *compute.Service
Alpha *computealpha.Service
Beta *computebeta.Service
}
// ComputeServices returns access to the internal compute services.
func (g *Cloud) ComputeServices() *Services {
return &Services{g.service, g.serviceAlpha, g.serviceBeta}
}
// Compute returns the generated stubs for the compute API.
func (g *Cloud) Compute() cloud.Cloud {
return g.c
}
// ContainerService returns the container service.
func (g *Cloud) ContainerService() *container.Service {
return g.containerService
}
// newGCECloud creates a new instance of Cloud.
func newGCECloud(config io.Reader) (gceCloud *Cloud, err error) {
var cloudConfig *CloudConfig
var configFile *ConfigFile
if config != nil {
configFile, err = readConfig(config)
if err != nil {
return nil, err
}
klog.Infof("Using GCE provider config %+v", configFile)
}
cloudConfig, err = generateCloudConfig(configFile)
if err != nil {
return nil, err
}
return CreateGCECloud(cloudConfig)
}
func readConfig(reader io.Reader) (*ConfigFile, error) {
cfg := &ConfigFile{}
if err := gcfg.FatalOnly(gcfg.ReadInto(cfg, reader)); err != nil {
klog.Errorf("Couldn't read config: %v", err)
return nil, err
}
return cfg, nil
}
func generateCloudConfig(configFile *ConfigFile) (cloudConfig *CloudConfig, err error) {
cloudConfig = &CloudConfig{}
// By default, fetch token from GCE metadata server
cloudConfig.TokenSource = google.ComputeTokenSource("")
cloudConfig.UseMetadataServer = true
cloudConfig.AlphaFeatureGate = NewAlphaFeatureGate([]string{})
if configFile != nil {
if configFile.Global.APIEndpoint != "" {
cloudConfig.APIEndpoint = configFile.Global.APIEndpoint
}
if configFile.Global.ContainerAPIEndpoint != "" {
cloudConfig.ContainerAPIEndpoint = configFile.Global.ContainerAPIEndpoint
}
if configFile.Global.TokenURL != "" {
// if tokenURL is nil, set tokenSource to nil. This will force the OAuth client to fall
// back to use DefaultTokenSource. This allows running gceCloud remotely.
if configFile.Global.TokenURL == "nil" {
cloudConfig.TokenSource = nil
} else {
cloudConfig.TokenSource = NewAltTokenSource(configFile.Global.TokenURL, configFile.Global.TokenBody)
}
}
cloudConfig.NodeTags = configFile.Global.NodeTags
cloudConfig.NodeInstancePrefix = configFile.Global.NodeInstancePrefix
cloudConfig.AlphaFeatureGate = NewAlphaFeatureGate(configFile.Global.AlphaFeatures)
}
// retrieve projectID and zone
if configFile == nil || configFile.Global.ProjectID == "" || configFile.Global.LocalZone == "" {
cloudConfig.ProjectID, cloudConfig.Zone, err = getProjectAndZone()
if err != nil {
return nil, err
}
}
if configFile != nil {
if configFile.Global.ProjectID != "" {
cloudConfig.ProjectID = configFile.Global.ProjectID
}
if configFile.Global.LocalZone != "" {
cloudConfig.Zone = configFile.Global.LocalZone
}
if configFile.Global.NetworkProjectID != "" {
cloudConfig.NetworkProjectID = configFile.Global.NetworkProjectID
}
}
// retrieve region
cloudConfig.Region, err = GetGCERegion(cloudConfig.Zone)
if err != nil {
return nil, err
}
// Determine if its a regional cluster
if configFile != nil && configFile.Global.Regional {
cloudConfig.Regional = true
}
// generate managedZones
cloudConfig.ManagedZones = []string{cloudConfig.Zone}
if configFile != nil && (configFile.Global.Multizone || configFile.Global.Regional) {
cloudConfig.ManagedZones = nil // Use all zones in region
}
// Determine if network parameter is URL or Name
if configFile != nil && configFile.Global.NetworkName != "" {
if strings.Contains(configFile.Global.NetworkName, "/") {
cloudConfig.NetworkURL = configFile.Global.NetworkName
} else {
cloudConfig.NetworkName = configFile.Global.NetworkName
}
} else {
cloudConfig.NetworkName, err = getNetworkNameViaMetadata()
if err != nil {
return nil, err
}
}
// Determine if subnetwork parameter is URL or Name
// If cluster is on a GCP network of mode=custom, then `SubnetName` must be specified in config file.
if configFile != nil && configFile.Global.SubnetworkName != "" {
if strings.Contains(configFile.Global.SubnetworkName, "/") {
cloudConfig.SubnetworkURL = configFile.Global.SubnetworkName
} else {
cloudConfig.SubnetworkName = configFile.Global.SubnetworkName
}
}
if configFile != nil {
cloudConfig.SecondaryRangeName = configFile.Global.SecondaryRangeName
}
return cloudConfig, err
}
// CreateGCECloud creates a Cloud object using the specified parameters.
// If no networkUrl is specified, loads networkName via rest call.
// If no tokenSource is specified, uses oauth2.DefaultTokenSource.
// If managedZones is nil / empty all zones in the region will be managed.
func CreateGCECloud(config *CloudConfig) (*Cloud, error) {
// Remove any pre-release version and build metadata from the semver,
// leaving only the MAJOR.MINOR.PATCH portion. See http://semver.org/.
version := strings.TrimLeft(strings.Split(strings.Split(version.Get().GitVersion, "-")[0], "+")[0], "v")
// Create a user-agent header append string to supply to the Google API
// clients, to identify Kubernetes as the origin of the GCP API calls.
userAgent := fmt.Sprintf("Kubernetes/%s (%s %s)", version, runtime.GOOS, runtime.GOARCH)
// Use ProjectID for NetworkProjectID, if it wasn't explicitly set.
if config.NetworkProjectID == "" {
config.NetworkProjectID = config.ProjectID
}
service, err := compute.NewService(context.Background(), option.WithTokenSource(config.TokenSource))
if err != nil {
return nil, err
}
service.UserAgent = userAgent
serviceBeta, err := computebeta.NewService(context.Background(), option.WithTokenSource(config.TokenSource))
if err != nil {
return nil, err
}
serviceBeta.UserAgent = userAgent
serviceAlpha, err := computealpha.NewService(context.Background(), option.WithTokenSource(config.TokenSource))
if err != nil {
return nil, err
}
serviceAlpha.UserAgent = userAgent
if config.APIEndpoint != "" {
if strings.HasSuffix(service.BasePath, "/projects/") {
service.BasePath = getProjectsBasePath(config.APIEndpoint)
serviceBeta.BasePath = getProjectsBasePath(strings.Replace(config.APIEndpoint, "v1", "beta", -1))
serviceAlpha.BasePath = getProjectsBasePath(strings.Replace(config.APIEndpoint, "v1", "alpha", -1))
} else {
service.BasePath = config.APIEndpoint
serviceBeta.BasePath = strings.Replace(config.APIEndpoint, "v1", "beta", -1)
serviceAlpha.BasePath = strings.Replace(config.APIEndpoint, "v1", "alpha", -1)
}
}
containerService, err := container.NewService(context.Background(), option.WithTokenSource(config.TokenSource))
if err != nil {
return nil, err
}
containerService.UserAgent = userAgent
if config.ContainerAPIEndpoint != "" {
containerService.BasePath = config.ContainerAPIEndpoint
}
client, err := newOauthClient(config.TokenSource)
if err != nil {
return nil, err
}
tpuService, err := newTPUService(client)
if err != nil {
return nil, err
}
// ProjectID and.NetworkProjectID may be project number or name.
projID, netProjID := tryConvertToProjectNames(config.ProjectID, config.NetworkProjectID, service)
onXPN := projID != netProjID
var networkURL string
var subnetURL string
var isLegacyNetwork bool
if config.NetworkURL != "" {
networkURL = config.NetworkURL
} else if config.NetworkName != "" {
networkURL = gceNetworkURL(config.APIEndpoint, netProjID, config.NetworkName)
} else {
// Other consumers may use the cloudprovider without utilizing the wrapped GCE API functions
// or functions requiring network/subnetwork URLs (e.g. Kubelet).
klog.Warningf("No network name or URL specified.")
}
if config.SubnetworkURL != "" {
subnetURL = config.SubnetworkURL
} else if config.SubnetworkName != "" {
subnetURL = gceSubnetworkURL(config.APIEndpoint, netProjID, config.Region, config.SubnetworkName)
}
// If neither SubnetworkURL nor SubnetworkName are provided, defer to
// lazy initialization. Determining subnetURL and isLegacyNetwork requires
// GCE API call. Given that it's not used in many cases and the fact that
// the provider is initialized also for Kubelets (and there can be thousands
// of them) we defer to lazy initialization here.
if len(config.ManagedZones) == 0 {
config.ManagedZones, err = getZonesForRegion(service, config.ProjectID, config.Region)
if err != nil {
return nil, err
}
}
if len(config.ManagedZones) > 1 {
klog.Infof("managing multiple zones: %v", config.ManagedZones)
}
operationPollRateLimiter := flowcontrol.NewTokenBucketRateLimiter(5, 5) // 5 qps, 5 burst.
gce := &Cloud{
service: service,
serviceAlpha: serviceAlpha,
serviceBeta: serviceBeta,
containerService: containerService,
tpuService: tpuService,
projectID: projID,
networkProjectID: netProjID,
onXPN: onXPN,
region: config.Region,
regional: config.Regional,
localZone: config.Zone,
managedZones: config.ManagedZones,
networkURL: networkURL,
unsafeIsLegacyNetwork: isLegacyNetwork,
unsafeSubnetworkURL: subnetURL,
secondaryRangeName: config.SecondaryRangeName,
nodeTags: config.NodeTags,
nodeInstancePrefix: config.NodeInstancePrefix,
useMetadataServer: config.UseMetadataServer,
operationPollRateLimiter: operationPollRateLimiter,
AlphaFeatureGate: config.AlphaFeatureGate,
nodeZones: map[string]sets.String{},
metricsCollector: newLoadBalancerMetrics(),
projectsBasePath: getProjectsBasePath(service.BasePath),
}
gce.manager = &gceServiceManager{gce}
gce.s = &cloud.Service{
GA: service,
Alpha: serviceAlpha,
Beta: serviceBeta,
ProjectRouter: &gceProjectRouter{gce},
RateLimiter: &gceRateLimiter{gce},
}
gce.c = cloud.NewGCE(gce.s)
return gce, nil
}
// initializeNetworkConfig() is supposed to be called under sync.Once()
// for accessors to subnetworkURL and isLegacyNetwork fields.
func (g *Cloud) initializeSubnetworkURLAndIsLegacyNetwork() {
if g.unsafeSubnetworkURL != "" {
// This has already been initialized via the config.
return
}
var subnetURL string
var isLegacyNetwork bool
// Determine the type of network and attempt to discover the correct subnet for AUTO mode.
// Gracefully fail because kubelet calls CreateGCECloud without any config, and minions
// lack the proper credentials for API calls.
if networkName := lastComponent(g.NetworkURL()); networkName != "" {
if n, err := getNetwork(g.service, g.NetworkProjectID(), networkName); err != nil {
klog.Warningf("Could not retrieve network %q; err: %v", networkName, err)
} else {
switch typeOfNetwork(n) {
case netTypeLegacy:
klog.Infof("Network %q is type legacy - no subnetwork", networkName)
isLegacyNetwork = true
case netTypeCustom:
klog.Warningf("Network %q is type custom - cannot auto select a subnetwork", networkName)
case netTypeAuto:
subnetURL, err = determineSubnetURL(g.service, g.NetworkProjectID(), networkName, g.Region())
if err != nil {
klog.Warningf("Could not determine subnetwork for network %q and region %v; err: %v", networkName, g.Region(), err)
} else {
klog.Infof("Auto selecting subnetwork %q", subnetURL)
}
}
}
}
g.unsafeSubnetworkURL = subnetURL
g.unsafeIsLegacyNetwork = isLegacyNetwork
}
// SetRateLimiter adds a custom cloud.RateLimiter implementation.
// WARNING: Calling this could have unexpected behavior if you have in-flight
// requests. It is best to use this immediately after creating a Cloud.
func (g *Cloud) SetRateLimiter(rl cloud.RateLimiter) {
if rl != nil {
g.s.RateLimiter = rl
}
}
// determineSubnetURL queries for all subnetworks in a region for a given network and returns
// the URL of the subnetwork which exists in the auto-subnet range.
func determineSubnetURL(service *compute.Service, networkProjectID, networkName, region string) (string, error) {
subnets, err := listSubnetworksOfNetwork(service, networkProjectID, networkName, region)
if err != nil {
return "", err
}
autoSubnets, err := subnetsInCIDR(subnets, autoSubnetIPRange)
if err != nil {
return "", err
}
if len(autoSubnets) == 0 {
return "", fmt.Errorf("no subnet exists in auto CIDR")
}
if len(autoSubnets) > 1 {
return "", fmt.Errorf("multiple subnetworks in the same region exist in auto CIDR")
}
return autoSubnets[0].SelfLink, nil
}
func tryConvertToProjectNames(configProject, configNetworkProject string, service *compute.Service) (projID, netProjID string) {
projID = configProject
if isProjectNumber(projID) {
projName, err := getProjectID(service, projID)
if err != nil {
klog.Warningf("Failed to retrieve project %v while trying to retrieve its name. err %v", projID, err)
} else {
projID = projName
}
}
netProjID = projID
if configNetworkProject != configProject {
netProjID = configNetworkProject
}
if isProjectNumber(netProjID) {
netProjName, err := getProjectID(service, netProjID)
if err != nil {
klog.Warningf("Failed to retrieve network project %v while trying to retrieve its name. err %v", netProjID, err)
} else {
netProjID = netProjName
}
}
return projID, netProjID
}
// Initialize takes in a clientBuilder and spawns a goroutine for watching the clusterid configmap.
// This must be called before utilizing the funcs of gce.ClusterID
func (g *Cloud) Initialize(clientBuilder cloudprovider.ControllerClientBuilder, stop <-chan struct{}) {
g.clientBuilder = clientBuilder
g.client = clientBuilder.ClientOrDie("cloud-provider")
g.eventBroadcaster = record.NewBroadcaster()
g.eventBroadcaster.StartRecordingToSink(&v1core.EventSinkImpl{Interface: g.client.CoreV1().Events("")})
g.eventRecorder = g.eventBroadcaster.NewRecorder(scheme.Scheme, v1.EventSource{Component: "g-cloudprovider"})
go func() {
defer g.eventBroadcaster.Shutdown()
<-stop
}()
go g.watchClusterID(stop)
go g.metricsCollector.Run(stop)
}
// LoadBalancer returns an implementation of LoadBalancer for Google Compute Engine.
func (g *Cloud) LoadBalancer() (cloudprovider.LoadBalancer, bool) {
return g, true
}
// Instances returns an implementation of Instances for Google Compute Engine.
func (g *Cloud) Instances() (cloudprovider.Instances, bool) {
return g, true
}
// InstancesV2 returns an implementation of InstancesV2 for Google Compute Engine.
// Implement ONLY for external cloud provider
func (g *Cloud) InstancesV2() (cloudprovider.InstancesV2, bool) {
return g, true
}
// Zones returns an implementation of Zones for Google Compute Engine.
func (g *Cloud) Zones() (cloudprovider.Zones, bool) {
return g, true
}
// Clusters returns an implementation of Clusters for Google Compute Engine.
func (g *Cloud) Clusters() (cloudprovider.Clusters, bool) {
return g, true
}
// Routes returns an implementation of Routes for Google Compute Engine.
func (g *Cloud) Routes() (cloudprovider.Routes, bool) {
return g, true
}
// ProviderName returns the cloud provider ID.
func (g *Cloud) ProviderName() string {
return ProviderName
}
// ProjectID returns the ProjectID corresponding to the project this cloud is in.
func (g *Cloud) ProjectID() string {
return g.projectID
}
// NetworkProjectID returns the ProjectID corresponding to the project this cluster's network is in.
func (g *Cloud) NetworkProjectID() string {
return g.networkProjectID
}
// Region returns the region
func (g *Cloud) Region() string {
return g.region
}
// OnXPN returns true if the cluster is running on a cross project network (XPN)
func (g *Cloud) OnXPN() bool {
return g.onXPN
}
// NetworkURL returns the network url
func (g *Cloud) NetworkURL() string {
return g.networkURL
}
// SubnetworkURL returns the subnetwork url
func (g *Cloud) SubnetworkURL() string {
g.subnetworkURLAndIsLegacyNetworkInitializer.Do(g.initializeSubnetworkURLAndIsLegacyNetwork)
return g.unsafeSubnetworkURL
}
// IsLegacyNetwork returns true if the cluster is still running a legacy network configuration.
func (g *Cloud) IsLegacyNetwork() bool {
g.subnetworkURLAndIsLegacyNetworkInitializer.Do(g.initializeSubnetworkURLAndIsLegacyNetwork)
return g.unsafeIsLegacyNetwork
}
// SetInformers sets up the zone handlers we need watching for node changes.
func (g *Cloud) SetInformers(informerFactory informers.SharedInformerFactory) {
klog.Infof("Setting up informers for Cloud")
nodeInformer := informerFactory.Core().V1().Nodes().Informer()
nodeInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{
AddFunc: func(obj interface{}) {
node := obj.(*v1.Node)
g.updateNodeZones(nil, node)
},
UpdateFunc: func(prev, obj interface{}) {
prevNode := prev.(*v1.Node)
newNode := obj.(*v1.Node)
if newNode.Labels[v1.LabelFailureDomainBetaZone] ==
prevNode.Labels[v1.LabelFailureDomainBetaZone] {
return
}
g.updateNodeZones(prevNode, newNode)
},
DeleteFunc: func(obj interface{}) {
node, isNode := obj.(*v1.Node)
// We can get DeletedFinalStateUnknown instead of *v1.Node here
// and we need to handle that correctly.
if !isNode {
deletedState, ok := obj.(cache.DeletedFinalStateUnknown)
if !ok {
klog.Errorf("Received unexpected object: %v", obj)
return
}
node, ok = deletedState.Obj.(*v1.Node)
if !ok {
klog.Errorf("DeletedFinalStateUnknown contained non-Node object: %v", deletedState.Obj)
return
}
}
g.updateNodeZones(node, nil)
},
})
g.nodeInformerSynced = nodeInformer.HasSynced
}
func (g *Cloud) updateNodeZones(prevNode, newNode *v1.Node) {
g.nodeZonesLock.Lock()
defer g.nodeZonesLock.Unlock()
if prevNode != nil {
prevZone, ok := prevNode.ObjectMeta.Labels[v1.LabelFailureDomainBetaZone]
if ok {
g.nodeZones[prevZone].Delete(prevNode.ObjectMeta.Name)
if g.nodeZones[prevZone].Len() == 0 {
g.nodeZones[prevZone] = nil
}
}
}
if newNode != nil {
newZone, ok := newNode.ObjectMeta.Labels[v1.LabelFailureDomainBetaZone]
if ok {
if g.nodeZones[newZone] == nil {
g.nodeZones[newZone] = sets.NewString()
}
g.nodeZones[newZone].Insert(newNode.ObjectMeta.Name)
}
}
}
// HasClusterID returns true if the cluster has a clusterID
func (g *Cloud) HasClusterID() bool {
return true
}
// getProjectsBasePath returns the compute API endpoint with the `projects/` element.
// The suffix must be added when generating compute resource urls.
func getProjectsBasePath(basePath string) string {
if !strings.HasSuffix(basePath, "/") {
basePath += "/"
}
if !strings.HasSuffix(basePath, "/projects/") {
basePath += "projects/"
}
return basePath
}
// Project IDs cannot have a digit for the first characeter. If the id contains a digit,
// then it must be a project number.
func isProjectNumber(idOrNumber string) bool {
_, err := strconv.ParseUint(idOrNumber, 10, 64)
return err == nil
}
func gceNetworkURL(apiEndpoint, project, network string) string {
if apiEndpoint == "" {
apiEndpoint = gceComputeAPIEndpoint
}
return apiEndpoint + strings.Join([]string{"projects", project, "global", "networks", network}, "/")
}
func gceSubnetworkURL(apiEndpoint, project, region, subnetwork string) string {
if apiEndpoint == "" {
apiEndpoint = gceComputeAPIEndpoint
}
return apiEndpoint + strings.Join([]string{"projects", project, "regions", region, "subnetworks", subnetwork}, "/")
}
// getRegionInURL parses full resource URLS and shorter URLS
// https://www.googleapis.com/compute/v1/projects/myproject/regions/us-central1/subnetworks/a
// projects/myproject/regions/us-central1/subnetworks/a
// All return "us-central1"
func getRegionInURL(urlStr string) string {
fields := strings.Split(urlStr, "/")
for i, v := range fields {
if v == "regions" && i < len(fields)-1 {
return fields[i+1]
}
}
return ""
}
func getNetworkNameViaMetadata() (string, error) {
result, err := metadata.Get("instance/network-interfaces/0/network")
if err != nil {
return "", err
}
parts := strings.Split(result, "/")
if len(parts) != 4 {
return "", fmt.Errorf("unexpected response: %s", result)
}
return parts[3], nil
}
// getNetwork returns a GCP network
func getNetwork(svc *compute.Service, networkProjectID, networkID string) (*compute.Network, error) {
return svc.Networks.Get(networkProjectID, networkID).Do()
}
// listSubnetworksOfNetwork returns a list of subnetworks for a particular region of a network.
func listSubnetworksOfNetwork(svc *compute.Service, networkProjectID, networkID, region string) ([]*compute.Subnetwork, error) {
var subnets []*compute.Subnetwork
err := svc.Subnetworks.List(networkProjectID, region).Filter(fmt.Sprintf("network eq .*/%v$", networkID)).Pages(context.Background(), func(res *compute.SubnetworkList) error {
subnets = append(subnets, res.Items...)
return nil
})
return subnets, err
}
// getProjectID returns the project's string ID given a project number or string
func getProjectID(svc *compute.Service, projectNumberOrID string) (string, error) {
proj, err := svc.Projects.Get(projectNumberOrID).Do()
if err != nil {
return "", err
}
return proj.Name, nil
}
func getZonesForRegion(svc *compute.Service, projectID, region string) ([]string, error) {
// TODO: use PageToken to list all not just the first 500
listCall := svc.Zones.List(projectID)
// Filtering by region doesn't seem to work
// (tested in https://cloud.google.com/compute/docs/reference/latest/zones/list)
// listCall = listCall.Filter("region eq " + region)
var zones []string
var accumulator = func(response *compute.ZoneList) error {
for _, zone := range response.Items {
regionName := lastComponent(zone.Region)
if regionName == region {
zones = append(zones, zone.Name)
}
}
return nil
}
err := listCall.Pages(context.TODO(), accumulator)
if err != nil {
return nil, fmt.Errorf("unexpected response listing zones: %v", err)
}
return zones, nil
}
func findSubnetForRegion(subnetURLs []string, region string) string {
for _, url := range subnetURLs {
if thisRegion := getRegionInURL(url); thisRegion == region {
return url
}
}
return ""
}
func newOauthClient(tokenSource oauth2.TokenSource) (*http.Client, error) {
if tokenSource == nil {
var err error
tokenSource, err = google.DefaultTokenSource(
context.Background(),
compute.CloudPlatformScope,
compute.ComputeScope)
klog.Infof("Using DefaultTokenSource %#v", tokenSource)
if err != nil {
return nil, err
}
} else {
klog.Infof("Using existing Token Source %#v", tokenSource)
}
backoff := wait.Backoff{
// These values will add up to about a minute. See #56293 for background.
Duration: time.Second,
Factor: 1.4,
Steps: 10,
}
if err := wait.ExponentialBackoff(backoff, func() (bool, error) {
if _, err := tokenSource.Token(); err != nil {
klog.Errorf("error fetching initial token: %v", err)
return false, nil
}
return true, nil
}); err != nil {
return nil, err
}
return oauth2.NewClient(context.Background(), tokenSource), nil
}
func (manager *gceServiceManager) getProjectsAPIEndpoint() string {
projectsAPIEndpoint := gceComputeAPIEndpoint + "projects/"
if manager.gce.service != nil {
projectsAPIEndpoint = manager.gce.projectsBasePath
}
return projectsAPIEndpoint
}

202
staging/src/k8s.io/legacy-cloud-providers/gce/gce_address_manager.go
View File

@@ -1,202 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
"fmt"
"net/http"
compute "google.golang.org/api/compute/v1"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud"
"k8s.io/klog/v2"
)
type addressManager struct {
logPrefix string
svc CloudAddressService
name string
serviceName string
targetIP string
addressType cloud.LbScheme
region string
subnetURL string
tryRelease bool
}
func newAddressManager(svc CloudAddressService, serviceName, region, subnetURL, name, targetIP string, addressType cloud.LbScheme) *addressManager {
return &addressManager{
svc: svc,
logPrefix: fmt.Sprintf("AddressManager(%q)", name),
region: region,
serviceName: serviceName,
name: name,
targetIP: targetIP,
addressType: addressType,
tryRelease: true,
subnetURL: subnetURL,
}
}
// HoldAddress will ensure that the IP is reserved with an address - either owned by the controller
// or by a user. If the address is not the addressManager.name, then it's assumed to be a user's address.
// The string returned is the reserved IP address.
func (am *addressManager) HoldAddress() (string, error) {
// HoldAddress starts with retrieving the address that we use for this load balancer (by name).
// Retrieving an address by IP will indicate if the IP is reserved and if reserved by the user
// or the controller, but won't tell us the current state of the controller's IP. The address
// could be reserving another address; therefore, it would need to be deleted. In the normal
// case of using a controller address, retrieving the address by name results in the fewest API
// calls since it indicates whether a Delete is necessary before Reserve.
klog.V(4).Infof("%v: attempting hold of IP %q Type %q", am.logPrefix, am.targetIP, am.addressType)
// Get the address in case it was orphaned earlier
addr, err := am.svc.GetRegionAddress(am.name, am.region)
if err != nil && !isNotFound(err) {
return "", err
}
if addr != nil {
// If address exists, check if the address had the expected attributes.
validationError := am.validateAddress(addr)
if validationError == nil {
klog.V(4).Infof("%v: address %q already reserves IP %q Type %q. No further action required.", am.logPrefix, addr.Name, addr.Address, addr.AddressType)
return addr.Address, nil
}
klog.V(2).Infof("%v: deleting existing address because %v", am.logPrefix, validationError)
err := am.svc.DeleteRegionAddress(addr.Name, am.region)
if err != nil {
if isNotFound(err) {
klog.V(4).Infof("%v: address %q was not found. Ignoring.", am.logPrefix, addr.Name)
} else {
return "", err
}
} else {
klog.V(4).Infof("%v: successfully deleted previous address %q", am.logPrefix, addr.Name)
}
}
return am.ensureAddressReservation()
}
// ReleaseAddress will release the address if it's owned by the controller.
func (am *addressManager) ReleaseAddress() error {
if !am.tryRelease {
klog.V(4).Infof("%v: not attempting release of address %q.", am.logPrefix, am.targetIP)
return nil
}
klog.V(4).Infof("%v: releasing address %q named %q", am.logPrefix, am.targetIP, am.name)
// Controller only ever tries to unreserve the address named with the load balancer's name.
err := am.svc.DeleteRegionAddress(am.name, am.region)
if err != nil {
if isNotFound(err) {
klog.Warningf("%v: address %q was not found. Ignoring.", am.logPrefix, am.name)
return nil
}
return err
}
klog.V(4).Infof("%v: successfully released IP %q named %q", am.logPrefix, am.targetIP, am.name)
return nil
}
func (am *addressManager) ensureAddressReservation() (string, error) {
// Try reserving the IP with controller-owned address name
// If am.targetIP is an empty string, a new IP will be created.
newAddr := &compute.Address{
Name: am.name,
Description: fmt.Sprintf(`{"kubernetes.io/service-name":"%s"}`, am.serviceName),
Address: am.targetIP,
AddressType: string(am.addressType),
Subnetwork: am.subnetURL,
}
reserveErr := am.svc.ReserveRegionAddress(newAddr, am.region)
if reserveErr == nil {
if newAddr.Address != "" {
klog.V(4).Infof("%v: successfully reserved IP %q with name %q", am.logPrefix, newAddr.Address, newAddr.Name)
return newAddr.Address, nil
}
addr, err := am.svc.GetRegionAddress(newAddr.Name, am.region)
if err != nil {
return "", err
}
klog.V(4).Infof("%v: successfully created address %q which reserved IP %q", am.logPrefix, addr.Name, addr.Address)
return addr.Address, nil
} else if !isHTTPErrorCode(reserveErr, http.StatusConflict) && !isHTTPErrorCode(reserveErr, http.StatusBadRequest) {
// If the IP is already reserved:
// by an internal address: a StatusConflict is returned
// by an external address: a BadRequest is returned
return "", reserveErr
}
// If the target IP was empty, we cannot try to find which IP caused a conflict.
// If the name was already used, then the next sync will attempt deletion of that address.
if am.targetIP == "" {
return "", fmt.Errorf("failed to reserve address %q with no specific IP, err: %v", am.name, reserveErr)
}
// Reserving the address failed due to a conflict or bad request. The address manager just checked that no address
// exists with the name, so it may belong to the user.
addr, err := am.svc.GetRegionAddressByIP(am.region, am.targetIP)
if err != nil {
return "", fmt.Errorf("failed to get address by IP %q after reservation attempt, err: %q, reservation err: %q", am.targetIP, err, reserveErr)
}
// Check that the address attributes are as required.
if err := am.validateAddress(addr); err != nil {
return "", err
}
if am.isManagedAddress(addr) {
// The address with this name is checked at the beginning of 'HoldAddress()', but for some reason
// it was re-created by this point. May be possible that two controllers are running.
klog.Warningf("%v: address %q unexpectedly existed with IP %q.", am.logPrefix, addr.Name, am.targetIP)
} else {
// If the retrieved address is not named with the loadbalancer name, then the controller does not own it, but will allow use of it.
klog.V(4).Infof("%v: address %q was already reserved with name: %q, description: %q", am.logPrefix, am.targetIP, addr.Name, addr.Description)
am.tryRelease = false
}
return addr.Address, nil
}
func (am *addressManager) validateAddress(addr *compute.Address) error {
if am.targetIP != "" && am.targetIP != addr.Address {
return fmt.Errorf("address %q does not have the expected IP %q, actual: %q", addr.Name, am.targetIP, addr.Address)
}
if addr.AddressType != string(am.addressType) {
return fmt.Errorf("address %q does not have the expected address type %q, actual: %q", addr.Name, am.addressType, addr.AddressType)
}
return nil
}
func (am *addressManager) isManagedAddress(addr *compute.Address) bool {
return addr.Name == am.name
}
func ensureAddressDeleted(svc CloudAddressService, name, region string) error {
return ignoreNotFound(svc.DeleteRegionAddress(name, region))
}

150
staging/src/k8s.io/legacy-cloud-providers/gce/gce_address_manager_test.go
View File

@@ -1,150 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
"testing"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
compute "google.golang.org/api/compute/v1"
)
const testSvcName = "my-service"
const testSubnet = "/projects/x/testRegions/us-central1/testSubnetworks/customsub"
const testLBName = "a111111111111111"
var vals = DefaultTestClusterValues()
// TestAddressManagerNoRequestedIP tests the typical case of passing in no requested IP
func TestAddressManagerNoRequestedIP(t *testing.T) {
svc, err := fakeGCECloud(vals)
require.NoError(t, err)
targetIP := ""
mgr := newAddressManager(svc, testSvcName, vals.Region, testSubnet, testLBName, targetIP, cloud.SchemeInternal)
testHoldAddress(t, mgr, svc, testLBName, vals.Region, targetIP, string(cloud.SchemeInternal))
testReleaseAddress(t, mgr, svc, testLBName, vals.Region)
}
// TestAddressManagerBasic tests the typical case of reserving and unreserving an address.
func TestAddressManagerBasic(t *testing.T) {
svc, err := fakeGCECloud(vals)
require.NoError(t, err)
targetIP := "1.1.1.1"
mgr := newAddressManager(svc, testSvcName, vals.Region, testSubnet, testLBName, targetIP, cloud.SchemeInternal)
testHoldAddress(t, mgr, svc, testLBName, vals.Region, targetIP, string(cloud.SchemeInternal))
testReleaseAddress(t, mgr, svc, testLBName, vals.Region)
}
// TestAddressManagerOrphaned tests the case where the address exists with the IP being equal
// to the requested address (forwarding rule or loadbalancer IP).
func TestAddressManagerOrphaned(t *testing.T) {
svc, err := fakeGCECloud(vals)
require.NoError(t, err)
targetIP := "1.1.1.1"
addr := &compute.Address{Name: testLBName, Address: targetIP, AddressType: string(cloud.SchemeInternal)}
err = svc.ReserveRegionAddress(addr, vals.Region)
require.NoError(t, err)
mgr := newAddressManager(svc, testSvcName, vals.Region, testSubnet, testLBName, targetIP, cloud.SchemeInternal)
testHoldAddress(t, mgr, svc, testLBName, vals.Region, targetIP, string(cloud.SchemeInternal))
testReleaseAddress(t, mgr, svc, testLBName, vals.Region)
}
// TestAddressManagerOutdatedOrphan tests the case where an address exists but points to
// an IP other than the forwarding rule or loadbalancer IP.
func TestAddressManagerOutdatedOrphan(t *testing.T) {
svc, err := fakeGCECloud(vals)
require.NoError(t, err)
previousAddress := "1.1.0.0"
targetIP := "1.1.1.1"
addr := &compute.Address{Name: testLBName, Address: previousAddress, AddressType: string(cloud.SchemeExternal)}
err = svc.ReserveRegionAddress(addr, vals.Region)
require.NoError(t, err)
mgr := newAddressManager(svc, testSvcName, vals.Region, testSubnet, testLBName, targetIP, cloud.SchemeInternal)
testHoldAddress(t, mgr, svc, testLBName, vals.Region, targetIP, string(cloud.SchemeInternal))
testReleaseAddress(t, mgr, svc, testLBName, vals.Region)
}
// TestAddressManagerExternallyOwned tests the case where the address exists but isn't
// owned by the controller.
func TestAddressManagerExternallyOwned(t *testing.T) {
svc, err := fakeGCECloud(vals)
require.NoError(t, err)
targetIP := "1.1.1.1"
addr := &compute.Address{Name: "my-important-address", Address: targetIP, AddressType: string(cloud.SchemeInternal)}
err = svc.ReserveRegionAddress(addr, vals.Region)
require.NoError(t, err)
mgr := newAddressManager(svc, testSvcName, vals.Region, testSubnet, testLBName, targetIP, cloud.SchemeInternal)
ipToUse, err := mgr.HoldAddress()
require.NoError(t, err)
assert.NotEmpty(t, ipToUse)
ad, err := svc.GetRegionAddress(testLBName, vals.Region)
assert.True(t, isNotFound(err))
require.Nil(t, ad)
testReleaseAddress(t, mgr, svc, testLBName, vals.Region)
}
// TestAddressManagerExternallyOwned tests the case where the address exists but isn't
// owned by the controller. However, this address has the wrong type.
func TestAddressManagerBadExternallyOwned(t *testing.T) {
svc, err := fakeGCECloud(vals)
require.NoError(t, err)
targetIP := "1.1.1.1"
addr := &compute.Address{Name: "my-important-address", Address: targetIP, AddressType: string(cloud.SchemeExternal)}
err = svc.ReserveRegionAddress(addr, vals.Region)
require.NoError(t, err)
mgr := newAddressManager(svc, testSvcName, vals.Region, testSubnet, testLBName, targetIP, cloud.SchemeInternal)
ad, err := mgr.HoldAddress()
assert.Error(t, err) // FIXME
require.Equal(t, ad, "")
}
func testHoldAddress(t *testing.T, mgr *addressManager, svc CloudAddressService, name, region, targetIP, scheme string) {
ipToUse, err := mgr.HoldAddress()
require.NoError(t, err)
assert.NotEmpty(t, ipToUse)
addr, err := svc.GetRegionAddress(name, region)
require.NoError(t, err)
if targetIP != "" {
assert.EqualValues(t, targetIP, addr.Address)
}
assert.EqualValues(t, scheme, addr.AddressType)
}
func testReleaseAddress(t *testing.T, mgr *addressManager, svc CloudAddressService, name, region string) {
err := mgr.ReleaseAddress()
require.NoError(t, err)
_, err = svc.GetRegionAddress(name, region)
assert.True(t, isNotFound(err))
}

192
staging/src/k8s.io/legacy-cloud-providers/gce/gce_addresses.go
View File

@@ -1,192 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
"fmt"
"k8s.io/klog/v2"
computebeta "google.golang.org/api/compute/v0.beta"
compute "google.golang.org/api/compute/v1"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud/filter"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud/meta"
)
func newAddressMetricContext(request, region string) *metricContext {
return newAddressMetricContextWithVersion(request, region, computeV1Version)
}
func newAddressMetricContextWithVersion(request, region, version string) *metricContext {
return newGenericMetricContext("address", request, region, unusedMetricLabel, version)
}
// ReserveGlobalAddress creates a global address.
// Caller is allocated a random IP if they do not specify an ipAddress. If an
// ipAddress is specified, it must belong to the current project, eg: an
// ephemeral IP associated with a global forwarding rule.
func (g *Cloud) ReserveGlobalAddress(addr *compute.Address) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newAddressMetricContext("reserve", "")
return mc.Observe(g.c.GlobalAddresses().Insert(ctx, meta.GlobalKey(addr.Name), addr))
}
// DeleteGlobalAddress deletes a global address by name.
func (g *Cloud) DeleteGlobalAddress(name string) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newAddressMetricContext("delete", "")
return mc.Observe(g.c.GlobalAddresses().Delete(ctx, meta.GlobalKey(name)))
}
// GetGlobalAddress returns the global address by name.
func (g *Cloud) GetGlobalAddress(name string) (*compute.Address, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newAddressMetricContext("get", "")
v, err := g.c.GlobalAddresses().Get(ctx, meta.GlobalKey(name))
return v, mc.Observe(err)
}
// ReserveRegionAddress creates a region address
func (g *Cloud) ReserveRegionAddress(addr *compute.Address, region string) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newAddressMetricContext("reserve", region)
return mc.Observe(g.c.Addresses().Insert(ctx, meta.RegionalKey(addr.Name, region), addr))
}
// ReserveBetaRegionAddress creates a beta region address
func (g *Cloud) ReserveBetaRegionAddress(addr *computebeta.Address, region string) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newAddressMetricContext("reserve", region)
return mc.Observe(g.c.BetaAddresses().Insert(ctx, meta.RegionalKey(addr.Name, region), addr))
}
// DeleteRegionAddress deletes a region address by name.
func (g *Cloud) DeleteRegionAddress(name, region string) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newAddressMetricContext("delete", region)
return mc.Observe(g.c.Addresses().Delete(ctx, meta.RegionalKey(name, region)))
}
// GetRegionAddress returns the region address by name
func (g *Cloud) GetRegionAddress(name, region string) (*compute.Address, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newAddressMetricContext("get", region)
v, err := g.c.Addresses().Get(ctx, meta.RegionalKey(name, region))
return v, mc.Observe(err)
}
// GetBetaRegionAddress returns the beta region address by name
func (g *Cloud) GetBetaRegionAddress(name, region string) (*computebeta.Address, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newAddressMetricContext("get", region)
v, err := g.c.BetaAddresses().Get(ctx, meta.RegionalKey(name, region))
return v, mc.Observe(err)
}
// GetRegionAddressByIP returns the regional address matching the given IP address.
func (g *Cloud) GetRegionAddressByIP(region, ipAddress string) (*compute.Address, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newAddressMetricContext("list", region)
addrs, err := g.c.Addresses().List(ctx, region, filter.Regexp("address", ipAddress))
mc.Observe(err)
if err != nil {
return nil, err
}
if len(addrs) > 1 {
klog.Warningf("More than one addresses matching the IP %q: %v", ipAddress, addrNames(addrs))
}
for _, addr := range addrs {
if addr.Address == ipAddress {
return addr, nil
}
}
return nil, makeGoogleAPINotFoundError(fmt.Sprintf("Address with IP %q was not found in region %q", ipAddress, region))
}
// GetBetaRegionAddressByIP returns the beta regional address matching the given IP address.
func (g *Cloud) GetBetaRegionAddressByIP(region, ipAddress string) (*computebeta.Address, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newAddressMetricContext("list", region)
addrs, err := g.c.BetaAddresses().List(ctx, region, filter.Regexp("address", ipAddress))
mc.Observe(err)
if err != nil {
return nil, err
}
if len(addrs) > 1 {
klog.Warningf("More than one addresses matching the IP %q: %v", ipAddress, addrNames(addrs))
}
for _, addr := range addrs {
if addr.Address == ipAddress {
return addr, nil
}
}
return nil, makeGoogleAPINotFoundError(fmt.Sprintf("Address with IP %q was not found in region %q", ipAddress, region))
}
func (g *Cloud) getNetworkTierFromAddress(name, region string) (string, error) {
addr, err := g.GetRegionAddress(name, region)
if err != nil {
// Can't get the network tier, just return an error.
return "", err
}
return addr.NetworkTier, nil
}
func addrNames(items interface{}) []string {
var ret []string
switch items := items.(type) {
case []compute.Address:
for _, a := range items {
ret = append(ret, a.Name)
}
case []computebeta.Address:
for _, a := range items {
ret = append(ret, a.Name)
}
}
return ret
}

52
staging/src/k8s.io/legacy-cloud-providers/gce/gce_alpha.go
View File

@@ -1,52 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
const (
// AlphaFeatureILBSubsets allows InternalLoadBalancer services to include a subset
// of cluster nodes as backends instead of all nodes.
AlphaFeatureILBSubsets = "ILBSubsets"
// AlphaFeatureSkipIGsManagement enabled L4 Regional Backend Services and
// disables instance group management in service controller
AlphaFeatureSkipIGsManagement = "SkipIGsManagement"
)
// AlphaFeatureGate contains a mapping of alpha features to whether they are enabled
type AlphaFeatureGate struct {
features map[string]bool
}
// Enabled returns true if the provided alpha feature is enabled
func (af *AlphaFeatureGate) Enabled(key string) bool {
if af == nil || af.features == nil {
return false
}
return af.features[key]
}
// NewAlphaFeatureGate marks the provided alpha features as enabled
func NewAlphaFeatureGate(features []string) *AlphaFeatureGate {
featureMap := make(map[string]bool)
for _, name := range features {
featureMap[name] = true
}
return &AlphaFeatureGate{featureMap}
}

164
staging/src/k8s.io/legacy-cloud-providers/gce/gce_annotations.go
View File

@@ -1,164 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
"fmt"
"k8s.io/klog/v2"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud"
"k8s.io/api/core/v1"
)
// LoadBalancerType defines a specific type for holding load balancer types (eg. Internal)
type LoadBalancerType string
const (
// ServiceAnnotationLoadBalancerType is annotated on a service with type LoadBalancer
// dictates what specific kind of GCP LB should be assembled.
// Currently, only "Internal" is supported.
ServiceAnnotationLoadBalancerType = "networking.gke.io/load-balancer-type"
// Deprecating the old-style naming of LoadBalancerType annotation
deprecatedServiceAnnotationLoadBalancerType = "cloud.google.com/load-balancer-type"
// LBTypeInternal is the constant for the official internal type.
LBTypeInternal LoadBalancerType = "Internal"
// Deprecating the lowercase spelling of Internal.
deprecatedTypeInternalLowerCase LoadBalancerType = "internal"
// ServiceAnnotationILBBackendShare is annotated on a service with "true" when users
// want to share GCP Backend Services for a set of internal load balancers.
// ALPHA feature - this may be removed in a future release.
ServiceAnnotationILBBackendShare = "alpha.cloud.google.com/load-balancer-backend-share"
// This annotation did not correctly specify "alpha", so both annotations will be checked.
deprecatedServiceAnnotationILBBackendShare = "cloud.google.com/load-balancer-backend-share"
// ServiceAnnotationILBAllowGlobalAccess is annotated on a service with "true" when users
// want to access the Internal LoadBalancer globally, and not restricted to the region it is
// created in.
ServiceAnnotationILBAllowGlobalAccess = "networking.gke.io/internal-load-balancer-allow-global-access"
// ServiceAnnotationILBSubnet is annotated on a service with the name of the subnetwork
// the ILB IP Address should be assigned from. By default, this is the subnetwork that the
// cluster is created in.
ServiceAnnotationILBSubnet = "networking.gke.io/internal-load-balancer-subnet"
// NetworkTierAnnotationKey is annotated on a Service object to indicate which
// network tier a GCP LB should use. The valid values are "Standard" and
// "Premium" (default).
NetworkTierAnnotationKey = "cloud.google.com/network-tier"
// NetworkTierAnnotationStandard is an annotation to indicate the Service is on the Standard network tier
NetworkTierAnnotationStandard = cloud.NetworkTierStandard
// NetworkTierAnnotationPremium is an annotation to indicate the Service is on the Premium network tier
NetworkTierAnnotationPremium = cloud.NetworkTierPremium
// RBSAnnotationKey is annotated on a Service object to indicate
// opt-in mode for RBS NetLB
RBSAnnotationKey = "cloud.google.com/l4-rbs"
// RBSEnabled is an annotation to indicate the Service is opt-in for RBS
RBSEnabled = "enabled"
)
// GetLoadBalancerAnnotationType returns the type of GCP load balancer which should be assembled.
func GetLoadBalancerAnnotationType(service *v1.Service) LoadBalancerType {
var lbType LoadBalancerType
for _, ann := range []string{
ServiceAnnotationLoadBalancerType,
deprecatedServiceAnnotationLoadBalancerType,
} {
if v, ok := service.Annotations[ann]; ok {
lbType = LoadBalancerType(v)
break
}
}
switch lbType {
case LBTypeInternal, deprecatedTypeInternalLowerCase:
return LBTypeInternal
default:
return lbType
}
}
// GetLoadBalancerAnnotationBackendShare returns whether this service's backend service should be
// shared with other load balancers. Health checks and the healthcheck firewall will be shared regardless.
func GetLoadBalancerAnnotationBackendShare(service *v1.Service) bool {
if l, exists := service.Annotations[ServiceAnnotationILBBackendShare]; exists && l == "true" {
return true
}
// Check for deprecated annotation key
if l, exists := service.Annotations[deprecatedServiceAnnotationILBBackendShare]; exists && l == "true" {
klog.Warningf("Annotation %q is deprecated and replaced with an alpha-specific key: %q", deprecatedServiceAnnotationILBBackendShare, ServiceAnnotationILBBackendShare)
return true
}
return false
}
// GetServiceNetworkTier returns the network tier of GCP load balancer
// which should be assembled, and an error if the specified tier is not
// supported.
func GetServiceNetworkTier(service *v1.Service) (cloud.NetworkTier, error) {
l, ok := service.Annotations[NetworkTierAnnotationKey]
if !ok {
return cloud.NetworkTierDefault, nil
}
v := cloud.NetworkTier(l)
switch v {
case cloud.NetworkTierStandard:
fallthrough
case cloud.NetworkTierPremium:
return v, nil
default:
return cloud.NetworkTierDefault, fmt.Errorf("unsupported network tier: %q", v)
}
}
// ILBOptions represents the extra options specified when creating a
// load balancer.
type ILBOptions struct {
// AllowGlobalAccess Indicates whether global access is allowed for the LoadBalancer
AllowGlobalAccess bool
// SubnetName indicates which subnet the LoadBalancer VIPs should be assigned from
SubnetName string
}
// GetLoadBalancerAnnotationAllowGlobalAccess returns if global access is enabled
// for the given loadbalancer service.
func GetLoadBalancerAnnotationAllowGlobalAccess(service *v1.Service) bool {
return service.Annotations[ServiceAnnotationILBAllowGlobalAccess] == "true"
}
// GetLoadBalancerAnnotationSubnet returns the configured subnet to assign LoadBalancer IP from.
func GetLoadBalancerAnnotationSubnet(service *v1.Service) string {
if val, exists := service.Annotations[ServiceAnnotationILBSubnet]; exists {
return val
}
return ""
}

74
staging/src/k8s.io/legacy-cloud-providers/gce/gce_annotations_test.go
View File

@@ -1,74 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
"testing"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud"
"k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"github.com/stretchr/testify/assert"
)
func TestServiceNetworkTierAnnotationKey(t *testing.T) {
createTestService := func() *v1.Service {
return &v1.Service{
ObjectMeta: metav1.ObjectMeta{
UID: "randome-uid",
Name: "test-svc",
Namespace: "test-ns",
},
}
}
for testName, testCase := range map[string]struct {
annotations map[string]string
expectedTier cloud.NetworkTier
expectErr bool
}{
"Use the default when the annotation does not exist": {
annotations: nil,
expectedTier: cloud.NetworkTierDefault,
},
"Standard tier": {
annotations: map[string]string{NetworkTierAnnotationKey: "Standard"},
expectedTier: cloud.NetworkTierStandard,
},
"Premium tier": {
annotations: map[string]string{NetworkTierAnnotationKey: "Premium"},
expectedTier: cloud.NetworkTierPremium,
},
"Report an error on invalid network tier value": {
annotations: map[string]string{NetworkTierAnnotationKey: "Unknown-tier"},
expectedTier: cloud.NetworkTierPremium,
expectErr: true,
},
} {
t.Run(testName, func(t *testing.T) {
svc := createTestService()
svc.Annotations = testCase.annotations
actualTier, err := GetServiceNetworkTier(svc)
assert.Equal(t, testCase.expectedTier, actualTier)
assert.Equal(t, testCase.expectErr, err != nil)
})
}
}

238
staging/src/k8s.io/legacy-cloud-providers/gce/gce_backendservice.go
View File

@@ -1,238 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
computealpha "google.golang.org/api/compute/v0.alpha"
computebeta "google.golang.org/api/compute/v0.beta"
compute "google.golang.org/api/compute/v1"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud/filter"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud/meta"
)
func newBackendServiceMetricContext(request, region string) *metricContext {
return newBackendServiceMetricContextWithVersion(request, region, computeV1Version)
}
func newBackendServiceMetricContextWithVersion(request, region, version string) *metricContext {
return newGenericMetricContext("backendservice", request, region, unusedMetricLabel, version)
}
// GetGlobalBackendService retrieves a backend by name.
func (g *Cloud) GetGlobalBackendService(name string) (*compute.BackendService, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newBackendServiceMetricContext("get", "")
v, err := g.c.BackendServices().Get(ctx, meta.GlobalKey(name))
return v, mc.Observe(err)
}
// GetBetaGlobalBackendService retrieves beta backend by name.
func (g *Cloud) GetBetaGlobalBackendService(name string) (*computebeta.BackendService, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newBackendServiceMetricContextWithVersion("get", "", computeBetaVersion)
v, err := g.c.BetaBackendServices().Get(ctx, meta.GlobalKey(name))
return v, mc.Observe(err)
}
// GetAlphaGlobalBackendService retrieves alpha backend by name.
func (g *Cloud) GetAlphaGlobalBackendService(name string) (*computealpha.BackendService, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newBackendServiceMetricContextWithVersion("get", "", computeAlphaVersion)
v, err := g.c.AlphaBackendServices().Get(ctx, meta.GlobalKey(name))
return v, mc.Observe(err)
}
// UpdateGlobalBackendService applies the given BackendService as an update to
// an existing service.
func (g *Cloud) UpdateGlobalBackendService(bg *compute.BackendService) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newBackendServiceMetricContext("update", "")
return mc.Observe(g.c.BackendServices().Update(ctx, meta.GlobalKey(bg.Name), bg))
}
// UpdateBetaGlobalBackendService applies the given beta BackendService as an
// update to an existing service.
func (g *Cloud) UpdateBetaGlobalBackendService(bg *computebeta.BackendService) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newBackendServiceMetricContextWithVersion("update", "", computeBetaVersion)
return mc.Observe(g.c.BetaBackendServices().Update(ctx, meta.GlobalKey(bg.Name), bg))
}
// UpdateAlphaGlobalBackendService applies the given alpha BackendService as an
// update to an existing service.
func (g *Cloud) UpdateAlphaGlobalBackendService(bg *computealpha.BackendService) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newBackendServiceMetricContextWithVersion("update", "", computeAlphaVersion)
return mc.Observe(g.c.AlphaBackendServices().Update(ctx, meta.GlobalKey(bg.Name), bg))
}
// DeleteGlobalBackendService deletes the given BackendService by name.
func (g *Cloud) DeleteGlobalBackendService(name string) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newBackendServiceMetricContext("delete", "")
return mc.Observe(g.c.BackendServices().Delete(ctx, meta.GlobalKey(name)))
}
// CreateGlobalBackendService creates the given BackendService.
func (g *Cloud) CreateGlobalBackendService(bg *compute.BackendService) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newBackendServiceMetricContext("create", "")
return mc.Observe(g.c.BackendServices().Insert(ctx, meta.GlobalKey(bg.Name), bg))
}
// CreateBetaGlobalBackendService creates the given beta BackendService.
func (g *Cloud) CreateBetaGlobalBackendService(bg *computebeta.BackendService) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newBackendServiceMetricContextWithVersion("create", "", computeBetaVersion)
return mc.Observe(g.c.BetaBackendServices().Insert(ctx, meta.GlobalKey(bg.Name), bg))
}
// CreateAlphaGlobalBackendService creates the given alpha BackendService.
func (g *Cloud) CreateAlphaGlobalBackendService(bg *computealpha.BackendService) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newBackendServiceMetricContextWithVersion("create", "", computeAlphaVersion)
return mc.Observe(g.c.AlphaBackendServices().Insert(ctx, meta.GlobalKey(bg.Name), bg))
}
// ListGlobalBackendServices lists all backend services in the project.
func (g *Cloud) ListGlobalBackendServices() ([]*compute.BackendService, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newBackendServiceMetricContext("list", "")
v, err := g.c.BackendServices().List(ctx, filter.None)
return v, mc.Observe(err)
}
// GetGlobalBackendServiceHealth returns the health of the BackendService
// identified by the given name, in the given instanceGroup. The
// instanceGroupLink is the fully qualified self link of an instance group.
func (g *Cloud) GetGlobalBackendServiceHealth(name string, instanceGroupLink string) (*compute.BackendServiceGroupHealth, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newBackendServiceMetricContext("get_health", "")
groupRef := &compute.ResourceGroupReference{Group: instanceGroupLink}
v, err := g.c.BackendServices().GetHealth(ctx, meta.GlobalKey(name), groupRef)
return v, mc.Observe(err)
}
// GetRegionBackendService retrieves a backend by name.
func (g *Cloud) GetRegionBackendService(name, region string) (*compute.BackendService, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newBackendServiceMetricContext("get", region)
v, err := g.c.RegionBackendServices().Get(ctx, meta.RegionalKey(name, region))
return v, mc.Observe(err)
}
// UpdateRegionBackendService applies the given BackendService as an update to
// an existing service.
func (g *Cloud) UpdateRegionBackendService(bg *compute.BackendService, region string) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newBackendServiceMetricContext("update", region)
return mc.Observe(g.c.RegionBackendServices().Update(ctx, meta.RegionalKey(bg.Name, region), bg))
}
// DeleteRegionBackendService deletes the given BackendService by name.
func (g *Cloud) DeleteRegionBackendService(name, region string) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newBackendServiceMetricContext("delete", region)
return mc.Observe(g.c.RegionBackendServices().Delete(ctx, meta.RegionalKey(name, region)))
}
// CreateRegionBackendService creates the given BackendService.
func (g *Cloud) CreateRegionBackendService(bg *compute.BackendService, region string) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newBackendServiceMetricContext("create", region)
return mc.Observe(g.c.RegionBackendServices().Insert(ctx, meta.RegionalKey(bg.Name, region), bg))
}
// ListRegionBackendServices lists all backend services in the project.
func (g *Cloud) ListRegionBackendServices(region string) ([]*compute.BackendService, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newBackendServiceMetricContext("list", region)
v, err := g.c.RegionBackendServices().List(ctx, region, filter.None)
return v, mc.Observe(err)
}
// GetRegionalBackendServiceHealth returns the health of the BackendService
// identified by the given name, in the given instanceGroup. The
// instanceGroupLink is the fully qualified self link of an instance group.
func (g *Cloud) GetRegionalBackendServiceHealth(name, region string, instanceGroupLink string) (*compute.BackendServiceGroupHealth, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newBackendServiceMetricContext("get_health", region)
ref := &compute.ResourceGroupReference{Group: instanceGroupLink}
v, err := g.c.RegionBackendServices().GetHealth(ctx, meta.RegionalKey(name, region), ref)
return v, mc.Observe(err)
}
// SetSecurityPolicyForBetaGlobalBackendService sets the given
// SecurityPolicyReference for the BackendService identified by the given name.
func (g *Cloud) SetSecurityPolicyForBetaGlobalBackendService(backendServiceName string, securityPolicyReference *computebeta.SecurityPolicyReference) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newBackendServiceMetricContextWithVersion("set_security_policy", "", computeBetaVersion)
return mc.Observe(g.c.BetaBackendServices().SetSecurityPolicy(ctx, meta.GlobalKey(backendServiceName), securityPolicyReference))
}
// SetSecurityPolicyForAlphaGlobalBackendService sets the given
// SecurityPolicyReference for the BackendService identified by the given name.
func (g *Cloud) SetSecurityPolicyForAlphaGlobalBackendService(backendServiceName string, securityPolicyReference *computealpha.SecurityPolicyReference) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newBackendServiceMetricContextWithVersion("set_security_policy", "", computeAlphaVersion)
return mc.Observe(g.c.AlphaBackendServices().SetSecurityPolicy(ctx, meta.GlobalKey(backendServiceName), securityPolicyReference))
}

74
staging/src/k8s.io/legacy-cloud-providers/gce/gce_cert.go
View File

@@ -1,74 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
compute "google.golang.org/api/compute/v1"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud/filter"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud/meta"
)
func newCertMetricContext(request string) *metricContext {
return newGenericMetricContext("cert", request, unusedMetricLabel, unusedMetricLabel, computeV1Version)
}
// GetSslCertificate returns the SslCertificate by name.
func (g *Cloud) GetSslCertificate(name string) (*compute.SslCertificate, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newCertMetricContext("get")
v, err := g.c.SslCertificates().Get(ctx, meta.GlobalKey(name))
return v, mc.Observe(err)
}
// CreateSslCertificate creates and returns a SslCertificate.
func (g *Cloud) CreateSslCertificate(sslCerts *compute.SslCertificate) (*compute.SslCertificate, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newCertMetricContext("create")
err := g.c.SslCertificates().Insert(ctx, meta.GlobalKey(sslCerts.Name), sslCerts)
if err != nil {
return nil, mc.Observe(err)
}
return g.GetSslCertificate(sslCerts.Name)
}
// DeleteSslCertificate deletes the SslCertificate by name.
func (g *Cloud) DeleteSslCertificate(name string) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newCertMetricContext("delete")
return mc.Observe(g.c.SslCertificates().Delete(ctx, meta.GlobalKey(name)))
}
// ListSslCertificates lists all SslCertificates in the project.
func (g *Cloud) ListSslCertificates() ([]*compute.SslCertificate, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newCertMetricContext("list")
v, err := g.c.SslCertificates().List(ctx, filter.None)
return v, mc.Observe(err)
}

270
staging/src/k8s.io/legacy-cloud-providers/gce/gce_clusterid.go
View File

@@ -1,270 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
"context"
"crypto/rand"
"encoding/hex"
"errors"
"fmt"
"reflect"
"sync"
"time"
"k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/fields"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/watch"
clientset "k8s.io/client-go/kubernetes"
"k8s.io/client-go/tools/cache"
"k8s.io/klog/v2"
)
const (
// UIDConfigMapName is the Key used to persist UIDs to configmaps.
UIDConfigMapName = "ingress-uid"
// UIDNamespace is the namespace which contains the above config map
UIDNamespace = metav1.NamespaceSystem
// UIDCluster is the data keys for looking up the clusters UID
UIDCluster = "uid"
// UIDProvider is the data keys for looking up the providers UID
UIDProvider = "provider-uid"
// UIDLengthBytes is the length of a UID
UIDLengthBytes = 8
// Frequency of the updateFunc event handler being called
// This does not actually query the apiserver for current state - the local cache value is used.
updateFuncFrequency = 10 * time.Minute
)
// ClusterID is the struct for maintaining information about this cluster's ID
type ClusterID struct {
idLock sync.RWMutex
client clientset.Interface
cfgMapKey string
store cache.Store
providerID *string
clusterID *string
}
// Continually watches for changes to the cluster id config map
func (g *Cloud) watchClusterID(stop <-chan struct{}) {
g.ClusterID = ClusterID{
cfgMapKey: fmt.Sprintf("%v/%v", UIDNamespace, UIDConfigMapName),
client: g.client,
}
mapEventHandler := cache.ResourceEventHandlerFuncs{
AddFunc: func(obj interface{}) {
m, ok := obj.(*v1.ConfigMap)
if !ok || m == nil {
klog.Errorf("Expected v1.ConfigMap, item=%+v, typeIsOk=%v", obj, ok)
return
}
if m.Namespace != UIDNamespace ||
m.Name != UIDConfigMapName {
return
}
klog.V(4).Infof("Observed new configmap for clusteriD: %v, %v; setting local values", m.Name, m.Data)
g.ClusterID.update(m)
},
UpdateFunc: func(old, cur interface{}) {
m, ok := cur.(*v1.ConfigMap)
if !ok || m == nil {
klog.Errorf("Expected v1.ConfigMap, item=%+v, typeIsOk=%v", cur, ok)
return
}
if m.Namespace != UIDNamespace ||
m.Name != UIDConfigMapName {
return
}
if reflect.DeepEqual(old, cur) {
return
}
klog.V(4).Infof("Observed updated configmap for clusteriD %v, %v; setting local values", m.Name, m.Data)
g.ClusterID.update(m)
},
}
listerWatcher := cache.NewListWatchFromClient(g.ClusterID.client.CoreV1().RESTClient(), "configmaps", UIDNamespace, fields.Everything())
var controller cache.Controller
g.ClusterID.store, controller = cache.NewInformer(newSingleObjectListerWatcher(listerWatcher, UIDConfigMapName), &v1.ConfigMap{}, updateFuncFrequency, mapEventHandler)
controller.Run(stop)
}
// GetID returns the id which is unique to this cluster
// if federated, return the provider id (unique to the cluster)
// if not federated, return the cluster id
func (ci *ClusterID) GetID() (string, error) {
if err := ci.getOrInitialize(); err != nil {
return "", err
}
ci.idLock.RLock()
defer ci.idLock.RUnlock()
if ci.clusterID == nil {
return "", errors.New("Could not retrieve cluster id")
}
// If provider ID is set, (Federation is enabled) use this field
if ci.providerID != nil {
return *ci.providerID, nil
}
// providerID is not set, use the cluster id
return *ci.clusterID, nil
}
// GetFederationID returns the id which could represent the entire Federation
// or just the cluster if not federated.
func (ci *ClusterID) GetFederationID() (string, bool, error) {
if err := ci.getOrInitialize(); err != nil {
return "", false, err
}
ci.idLock.RLock()
defer ci.idLock.RUnlock()
if ci.clusterID == nil {
return "", false, errors.New("could not retrieve cluster id")
}
// If provider ID is not set, return false
if ci.providerID == nil || *ci.clusterID == *ci.providerID {
return "", false, nil
}
return *ci.clusterID, true, nil
}
// getOrInitialize either grabs the configmaps current value or defines the value
// and sets the configmap. This is for the case of the user calling GetClusterID()
// before the watch has begun.
func (ci *ClusterID) getOrInitialize() error {
if ci.store == nil {
return errors.New("Cloud.ClusterID is not ready. Call Initialize() before using")
}
if ci.clusterID != nil {
return nil
}
exists, err := ci.getConfigMap()
if err != nil {
return err
} else if exists {
return nil
}
// The configmap does not exist - let's try creating one.
newID, err := makeUID()
if err != nil {
return err
}
klog.V(4).Infof("Creating clusteriD: %v", newID)
cfg := &v1.ConfigMap{
ObjectMeta: metav1.ObjectMeta{
Name: UIDConfigMapName,
Namespace: UIDNamespace,
},
}
cfg.Data = map[string]string{
UIDCluster: newID,
UIDProvider: newID,
}
if _, err := ci.client.CoreV1().ConfigMaps(UIDNamespace).Create(context.TODO(), cfg, metav1.CreateOptions{}); err != nil {
klog.Errorf("GCE cloud provider failed to create %v config map to store cluster id: %v", ci.cfgMapKey, err)
return err
}
klog.V(2).Infof("Created a config map containing clusteriD: %v", newID)
ci.update(cfg)
return nil
}
func (ci *ClusterID) getConfigMap() (bool, error) {
item, exists, err := ci.store.GetByKey(ci.cfgMapKey)
if err != nil {
return false, err
}
if !exists {
return false, nil
}
m, ok := item.(*v1.ConfigMap)
if !ok || m == nil {
err = fmt.Errorf("Expected v1.ConfigMap, item=%+v, typeIsOk=%v", item, ok)
klog.Error(err)
return false, err
}
ci.update(m)
return true, nil
}
func (ci *ClusterID) update(m *v1.ConfigMap) {
ci.idLock.Lock()
defer ci.idLock.Unlock()
if clusterID, exists := m.Data[UIDCluster]; exists {
ci.clusterID = &clusterID
}
if provID, exists := m.Data[UIDProvider]; exists {
ci.providerID = &provID
}
}
func makeUID() (string, error) {
b := make([]byte, UIDLengthBytes)
_, err := rand.Read(b)
if err != nil {
return "", err
}
return hex.EncodeToString(b), nil
}
func newSingleObjectListerWatcher(lw cache.ListerWatcher, objectName string) *singleObjListerWatcher {
return &singleObjListerWatcher{lw: lw, objectName: objectName}
}
type singleObjListerWatcher struct {
lw cache.ListerWatcher
objectName string
}
func (sow *singleObjListerWatcher) List(options metav1.ListOptions) (runtime.Object, error) {
options.FieldSelector = "metadata.name=" + sow.objectName
return sow.lw.List(options)
}
func (sow *singleObjListerWatcher) Watch(options metav1.ListOptions) (watch.Interface, error) {
options.FieldSelector = "metadata.name=" + sow.objectName
return sow.lw.Watch(options)
}

107
staging/src/k8s.io/legacy-cloud-providers/gce/gce_clusters.go
View File

@@ -1,107 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
"context"
"fmt"
"google.golang.org/api/container/v1"
"k8s.io/klog/v2"
)
func newClustersMetricContext(request, zone string) *metricContext {
return newGenericMetricContext("clusters", request, unusedMetricLabel, zone, computeV1Version)
}
// ListClusters will return a list of cluster names for the associated project
func (g *Cloud) ListClusters(ctx context.Context) ([]string, error) {
allClusters := []string{}
for _, zone := range g.managedZones {
clusters, err := g.listClustersInZone(zone)
if err != nil {
return nil, err
}
// TODO: Scoping? Do we need to qualify the cluster name?
allClusters = append(allClusters, clusters...)
}
return allClusters, nil
}
// GetManagedClusters will return the cluster objects associated to this project
func (g *Cloud) GetManagedClusters(ctx context.Context) ([]*container.Cluster, error) {
managedClusters := []*container.Cluster{}
if g.regional {
var err error
managedClusters, err = g.getClustersInLocation(g.region)
if err != nil {
return nil, err
}
} else if len(g.managedZones) >= 1 {
for _, zone := range g.managedZones {
clusters, err := g.getClustersInLocation(zone)
if err != nil {
return nil, err
}
managedClusters = append(managedClusters, clusters...)
}
} else {
return nil, fmt.Errorf("no zones associated with this cluster(%s)", g.ProjectID())
}
return managedClusters, nil
}
// Master returned the dns address of the master
func (g *Cloud) Master(ctx context.Context, clusterName string) (string, error) {
return "k8s-" + clusterName + "-master.internal", nil
}
func (g *Cloud) listClustersInZone(zone string) ([]string, error) {
clusters, err := g.getClustersInLocation(zone)
if err != nil {
return nil, err
}
result := []string{}
for _, cluster := range clusters {
result = append(result, cluster.Name)
}
return result, nil
}
func (g *Cloud) getClustersInLocation(zoneOrRegion string) ([]*container.Cluster, error) {
// TODO: Issue/68913 migrate metric to list_location instead of list_zone.
mc := newClustersMetricContext("list_zone", zoneOrRegion)
// TODO: use PageToken to list all not just the first 500
location := getLocationName(g.projectID, zoneOrRegion)
list, err := g.containerService.Projects.Locations.Clusters.List(location).Do()
if err != nil {
return nil, mc.Observe(err)
}
if list.Header.Get("nextPageToken") != "" {
klog.Errorf("Failed to get all clusters for request, received next page token %s", list.Header.Get("nextPageToken"))
}
return list.Clusters, mc.Observe(nil)
}

1054
staging/src/k8s.io/legacy-cloud-providers/gce/gce_disks.go
View File

File diff suppressed because it is too large Load Diff

1021
staging/src/k8s.io/legacy-cloud-providers/gce/gce_disks_test.go
View File

File diff suppressed because it is too large Load Diff

91
staging/src/k8s.io/legacy-cloud-providers/gce/gce_fake.go
View File

@@ -1,91 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2018 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
"context"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud"
compute "google.golang.org/api/compute/v1"
option "google.golang.org/api/option"
"k8s.io/client-go/tools/cache"
)
// TestClusterValues holds the config values for the fake/test gce cloud object.
type TestClusterValues struct {
ProjectID string
Region string
ZoneName string
SecondaryZoneName string
ClusterID string
ClusterName string
OnXPN bool
}
// DefaultTestClusterValues Creates a reasonable set of default cluster values
// for generating a new test fake GCE cloud instance.
func DefaultTestClusterValues() TestClusterValues {
return TestClusterValues{
ProjectID: "test-project",
Region: "us-central1",
ZoneName: "us-central1-b",
SecondaryZoneName: "us-central1-c",
ClusterID: "test-cluster-id",
ClusterName: "Test Cluster Name",
}
}
// Stubs ClusterID so that ClusterID.getOrInitialize() does not require calling
// gce.Initialize()
func fakeClusterID(clusterID string) ClusterID {
return ClusterID{
clusterID: &clusterID,
store: cache.NewStore(func(obj interface{}) (string, error) {
return "", nil
}),
}
}
// NewFakeGCECloud constructs a fake GCE Cloud from the cluster values.
func NewFakeGCECloud(vals TestClusterValues) *Cloud {
service, err := compute.NewService(context.Background(), option.WithoutAuthentication())
if err != nil {
panic(err)
}
gce := &Cloud{
region: vals.Region,
service: service,
managedZones: []string{vals.ZoneName},
projectID: vals.ProjectID,
networkProjectID: vals.ProjectID,
ClusterID: fakeClusterID(vals.ClusterID),
onXPN: vals.OnXPN,
metricsCollector: newLoadBalancerMetrics(),
projectsBasePath: getProjectsBasePath(service.BasePath),
}
c := cloud.NewMockGCE(&gceProjectRouter{gce})
gce.c = c
return gce
}
// UpdateFakeGCECloud updates the fake GCE cloud with the specified values. Currently only the onXPN value is updated.
func UpdateFakeGCECloud(g *Cloud, vals TestClusterValues) {
g.onXPN = vals.OnXPN
}

77
staging/src/k8s.io/legacy-cloud-providers/gce/gce_firewall.go
View File

@@ -1,77 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
compute "google.golang.org/api/compute/v1"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud/meta"
)
func newFirewallMetricContext(request string) *metricContext {
return newGenericMetricContext("firewall", request, unusedMetricLabel, unusedMetricLabel, computeV1Version)
}
// GetFirewall returns the Firewall by name.
func (g *Cloud) GetFirewall(name string) (*compute.Firewall, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newFirewallMetricContext("get")
v, err := g.c.Firewalls().Get(ctx, meta.GlobalKey(name))
return v, mc.Observe(err)
}
// CreateFirewall creates the passed firewall
func (g *Cloud) CreateFirewall(f *compute.Firewall) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newFirewallMetricContext("create")
return mc.Observe(g.c.Firewalls().Insert(ctx, meta.GlobalKey(f.Name), f))
}
// DeleteFirewall deletes the given firewall rule.
func (g *Cloud) DeleteFirewall(name string) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newFirewallMetricContext("delete")
return mc.Observe(g.c.Firewalls().Delete(ctx, meta.GlobalKey(name)))
}
// UpdateFirewall applies the given firewall as an update to an existing service.
func (g *Cloud) UpdateFirewall(f *compute.Firewall) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newFirewallMetricContext("update")
return mc.Observe(g.c.Firewalls().Update(ctx, meta.GlobalKey(f.Name), f))
}
// PatchFirewall applies the given firewall as an update to an existing service.
func (g *Cloud) PatchFirewall(f *compute.Firewall) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newFirewallMetricContext("Patch")
return mc.Observe(g.c.Firewalls().Patch(ctx, meta.GlobalKey(f.Name), f))
}

193
staging/src/k8s.io/legacy-cloud-providers/gce/gce_forwardingrule.go
View File

@@ -1,193 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud/filter"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud/meta"
computealpha "google.golang.org/api/compute/v0.alpha"
computebeta "google.golang.org/api/compute/v0.beta"
compute "google.golang.org/api/compute/v1"
)
func newForwardingRuleMetricContext(request, region string) *metricContext {
return newForwardingRuleMetricContextWithVersion(request, region, computeV1Version)
}
func newForwardingRuleMetricContextWithVersion(request, region, version string) *metricContext {
return newGenericMetricContext("forwardingrule", request, region, unusedMetricLabel, version)
}
// CreateGlobalForwardingRule creates the passed GlobalForwardingRule
func (g *Cloud) CreateGlobalForwardingRule(rule *compute.ForwardingRule) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newForwardingRuleMetricContext("create", "")
return mc.Observe(g.c.GlobalForwardingRules().Insert(ctx, meta.GlobalKey(rule.Name), rule))
}
// SetProxyForGlobalForwardingRule links the given TargetHttp(s)Proxy with the given GlobalForwardingRule.
// targetProxyLink is the SelfLink of a TargetHttp(s)Proxy.
func (g *Cloud) SetProxyForGlobalForwardingRule(forwardingRuleName, targetProxyLink string) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newForwardingRuleMetricContext("set_proxy", "")
target := &compute.TargetReference{Target: targetProxyLink}
return mc.Observe(g.c.GlobalForwardingRules().SetTarget(ctx, meta.GlobalKey(forwardingRuleName), target))
}
// DeleteGlobalForwardingRule deletes the GlobalForwardingRule by name.
func (g *Cloud) DeleteGlobalForwardingRule(name string) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newForwardingRuleMetricContext("delete", "")
return mc.Observe(g.c.GlobalForwardingRules().Delete(ctx, meta.GlobalKey(name)))
}
// GetGlobalForwardingRule returns the GlobalForwardingRule by name.
func (g *Cloud) GetGlobalForwardingRule(name string) (*compute.ForwardingRule, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newForwardingRuleMetricContext("get", "")
v, err := g.c.GlobalForwardingRules().Get(ctx, meta.GlobalKey(name))
return v, mc.Observe(err)
}
// ListGlobalForwardingRules lists all GlobalForwardingRules in the project.
func (g *Cloud) ListGlobalForwardingRules() ([]*compute.ForwardingRule, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newForwardingRuleMetricContext("list", "")
v, err := g.c.GlobalForwardingRules().List(ctx, filter.None)
return v, mc.Observe(err)
}
// GetRegionForwardingRule returns the RegionalForwardingRule by name & region.
func (g *Cloud) GetRegionForwardingRule(name, region string) (*compute.ForwardingRule, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newForwardingRuleMetricContext("get", region)
v, err := g.c.ForwardingRules().Get(ctx, meta.RegionalKey(name, region))
return v, mc.Observe(err)
}
// GetAlphaRegionForwardingRule returns the Alpha forwarding rule by name & region.
func (g *Cloud) GetAlphaRegionForwardingRule(name, region string) (*computealpha.ForwardingRule, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newForwardingRuleMetricContextWithVersion("get", region, computeAlphaVersion)
v, err := g.c.AlphaForwardingRules().Get(ctx, meta.RegionalKey(name, region))
return v, mc.Observe(err)
}
// GetBetaRegionForwardingRule returns the Beta forwarding rule by name & region.
func (g *Cloud) GetBetaRegionForwardingRule(name, region string) (*computebeta.ForwardingRule, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newForwardingRuleMetricContextWithVersion("get", region, computeBetaVersion)
v, err := g.c.BetaForwardingRules().Get(ctx, meta.RegionalKey(name, region))
return v, mc.Observe(err)
}
// ListRegionForwardingRules lists all RegionalForwardingRules in the project & region.
func (g *Cloud) ListRegionForwardingRules(region string) ([]*compute.ForwardingRule, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newForwardingRuleMetricContext("list", region)
v, err := g.c.ForwardingRules().List(ctx, region, filter.None)
return v, mc.Observe(err)
}
// ListAlphaRegionForwardingRules lists all RegionalForwardingRules in the project & region.
func (g *Cloud) ListAlphaRegionForwardingRules(region string) ([]*computealpha.ForwardingRule, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newForwardingRuleMetricContextWithVersion("list", region, computeAlphaVersion)
v, err := g.c.AlphaForwardingRules().List(ctx, region, filter.None)
return v, mc.Observe(err)
}
// ListBetaRegionForwardingRules lists all RegionalForwardingRules in the project & region.
func (g *Cloud) ListBetaRegionForwardingRules(region string) ([]*computebeta.ForwardingRule, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newForwardingRuleMetricContextWithVersion("list", region, computeBetaVersion)
v, err := g.c.BetaForwardingRules().List(ctx, region, filter.None)
return v, mc.Observe(err)
}
// CreateRegionForwardingRule creates and returns a
// RegionalForwardingRule that points to the given BackendService
func (g *Cloud) CreateRegionForwardingRule(rule *compute.ForwardingRule, region string) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newForwardingRuleMetricContext("create", region)
return mc.Observe(g.c.ForwardingRules().Insert(ctx, meta.RegionalKey(rule.Name, region), rule))
}
// CreateAlphaRegionForwardingRule creates and returns an Alpha
// forwarding rule in the given region.
func (g *Cloud) CreateAlphaRegionForwardingRule(rule *computealpha.ForwardingRule, region string) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newForwardingRuleMetricContextWithVersion("create", region, computeAlphaVersion)
return mc.Observe(g.c.AlphaForwardingRules().Insert(ctx, meta.RegionalKey(rule.Name, region), rule))
}
// CreateBetaRegionForwardingRule creates and returns a Beta
// forwarding rule in the given region.
func (g *Cloud) CreateBetaRegionForwardingRule(rule *computebeta.ForwardingRule, region string) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newForwardingRuleMetricContextWithVersion("create", region, computeBetaVersion)
return mc.Observe(g.c.BetaForwardingRules().Insert(ctx, meta.RegionalKey(rule.Name, region), rule))
}
// DeleteRegionForwardingRule deletes the RegionalForwardingRule by name & region.
func (g *Cloud) DeleteRegionForwardingRule(name, region string) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newForwardingRuleMetricContext("delete", region)
return mc.Observe(g.c.ForwardingRules().Delete(ctx, meta.RegionalKey(name, region)))
}
func (g *Cloud) getNetworkTierFromForwardingRule(name, region string) (string, error) {
fwdRule, err := g.GetRegionForwardingRule(name, region)
if err != nil {
// Can't get the network tier, just return an error.
return "", err
}
return fwdRule.NetworkTier, nil
}

260
staging/src/k8s.io/legacy-cloud-providers/gce/gce_healthchecks.go
View File

@@ -1,260 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
computealpha "google.golang.org/api/compute/v0.alpha"
computebeta "google.golang.org/api/compute/v0.beta"
compute "google.golang.org/api/compute/v1"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud/filter"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud/meta"
)
const (
nodesHealthCheckPath = "/healthz"
// NOTE: Please keep the following port in sync with ProxyHealthzPort in pkg/cluster/ports/ports.go
// ports.ProxyHealthzPort was not used here to avoid dependencies to k8s.io/kubernetes in the
// GCE cloud provider which is required as part of the out-of-tree cloud provider efforts.
// TODO: use a shared constant once ports in pkg/cluster/ports are in a common external repo.
lbNodesHealthCheckPort = 10256
)
func newHealthcheckMetricContext(request string) *metricContext {
return newHealthcheckMetricContextWithVersion(request, computeV1Version)
}
func newHealthcheckMetricContextWithVersion(request, version string) *metricContext {
return newGenericMetricContext("healthcheck", request, unusedMetricLabel, unusedMetricLabel, version)
}
// GetHTTPHealthCheck returns the given HttpHealthCheck by name.
func (g *Cloud) GetHTTPHealthCheck(name string) (*compute.HttpHealthCheck, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newHealthcheckMetricContext("get_legacy")
v, err := g.c.HttpHealthChecks().Get(ctx, meta.GlobalKey(name))
return v, mc.Observe(err)
}
// UpdateHTTPHealthCheck applies the given HttpHealthCheck as an update.
func (g *Cloud) UpdateHTTPHealthCheck(hc *compute.HttpHealthCheck) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newHealthcheckMetricContext("update_legacy")
return mc.Observe(g.c.HttpHealthChecks().Update(ctx, meta.GlobalKey(hc.Name), hc))
}
// DeleteHTTPHealthCheck deletes the given HttpHealthCheck by name.
func (g *Cloud) DeleteHTTPHealthCheck(name string) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newHealthcheckMetricContext("delete_legacy")
return mc.Observe(g.c.HttpHealthChecks().Delete(ctx, meta.GlobalKey(name)))
}
// CreateHTTPHealthCheck creates the given HttpHealthCheck.
func (g *Cloud) CreateHTTPHealthCheck(hc *compute.HttpHealthCheck) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newHealthcheckMetricContext("create_legacy")
return mc.Observe(g.c.HttpHealthChecks().Insert(ctx, meta.GlobalKey(hc.Name), hc))
}
// ListHTTPHealthChecks lists all HttpHealthChecks in the project.
func (g *Cloud) ListHTTPHealthChecks() ([]*compute.HttpHealthCheck, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newHealthcheckMetricContext("list_legacy")
v, err := g.c.HttpHealthChecks().List(ctx, filter.None)
return v, mc.Observe(err)
}
// Legacy HTTPS Health Checks
// GetHTTPSHealthCheck returns the given HttpsHealthCheck by name.
func (g *Cloud) GetHTTPSHealthCheck(name string) (*compute.HttpsHealthCheck, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newHealthcheckMetricContext("get_legacy")
v, err := g.c.HttpsHealthChecks().Get(ctx, meta.GlobalKey(name))
return v, mc.Observe(err)
}
// UpdateHTTPSHealthCheck applies the given HttpsHealthCheck as an update.
func (g *Cloud) UpdateHTTPSHealthCheck(hc *compute.HttpsHealthCheck) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newHealthcheckMetricContext("update_legacy")
return mc.Observe(g.c.HttpsHealthChecks().Update(ctx, meta.GlobalKey(hc.Name), hc))
}
// DeleteHTTPSHealthCheck deletes the given HttpsHealthCheck by name.
func (g *Cloud) DeleteHTTPSHealthCheck(name string) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newHealthcheckMetricContext("delete_legacy")
return mc.Observe(g.c.HttpsHealthChecks().Delete(ctx, meta.GlobalKey(name)))
}
// CreateHTTPSHealthCheck creates the given HttpsHealthCheck.
func (g *Cloud) CreateHTTPSHealthCheck(hc *compute.HttpsHealthCheck) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newHealthcheckMetricContext("create_legacy")
return mc.Observe(g.c.HttpsHealthChecks().Insert(ctx, meta.GlobalKey(hc.Name), hc))
}
// ListHTTPSHealthChecks lists all HttpsHealthChecks in the project.
func (g *Cloud) ListHTTPSHealthChecks() ([]*compute.HttpsHealthCheck, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newHealthcheckMetricContext("list_legacy")
v, err := g.c.HttpsHealthChecks().List(ctx, filter.None)
return v, mc.Observe(err)
}
// Generic HealthCheck
// GetHealthCheck returns the given HealthCheck by name.
func (g *Cloud) GetHealthCheck(name string) (*compute.HealthCheck, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newHealthcheckMetricContext("get")
v, err := g.c.HealthChecks().Get(ctx, meta.GlobalKey(name))
return v, mc.Observe(err)
}
// GetAlphaHealthCheck returns the given alpha HealthCheck by name.
func (g *Cloud) GetAlphaHealthCheck(name string) (*computealpha.HealthCheck, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newHealthcheckMetricContextWithVersion("get", computeAlphaVersion)
v, err := g.c.AlphaHealthChecks().Get(ctx, meta.GlobalKey(name))
return v, mc.Observe(err)
}
// GetBetaHealthCheck returns the given beta HealthCheck by name.
func (g *Cloud) GetBetaHealthCheck(name string) (*computebeta.HealthCheck, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newHealthcheckMetricContextWithVersion("get", computeBetaVersion)
v, err := g.c.BetaHealthChecks().Get(ctx, meta.GlobalKey(name))
return v, mc.Observe(err)
}
// UpdateHealthCheck applies the given HealthCheck as an update.
func (g *Cloud) UpdateHealthCheck(hc *compute.HealthCheck) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newHealthcheckMetricContext("update")
return mc.Observe(g.c.HealthChecks().Update(ctx, meta.GlobalKey(hc.Name), hc))
}
// UpdateAlphaHealthCheck applies the given alpha HealthCheck as an update.
func (g *Cloud) UpdateAlphaHealthCheck(hc *computealpha.HealthCheck) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newHealthcheckMetricContextWithVersion("update", computeAlphaVersion)
return mc.Observe(g.c.AlphaHealthChecks().Update(ctx, meta.GlobalKey(hc.Name), hc))
}
// UpdateBetaHealthCheck applies the given beta HealthCheck as an update.
func (g *Cloud) UpdateBetaHealthCheck(hc *computebeta.HealthCheck) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newHealthcheckMetricContextWithVersion("update", computeBetaVersion)
return mc.Observe(g.c.BetaHealthChecks().Update(ctx, meta.GlobalKey(hc.Name), hc))
}
// DeleteHealthCheck deletes the given HealthCheck by name.
func (g *Cloud) DeleteHealthCheck(name string) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newHealthcheckMetricContext("delete")
return mc.Observe(g.c.HealthChecks().Delete(ctx, meta.GlobalKey(name)))
}
// CreateHealthCheck creates the given HealthCheck.
func (g *Cloud) CreateHealthCheck(hc *compute.HealthCheck) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newHealthcheckMetricContext("create")
return mc.Observe(g.c.HealthChecks().Insert(ctx, meta.GlobalKey(hc.Name), hc))
}
// CreateAlphaHealthCheck creates the given alpha HealthCheck.
func (g *Cloud) CreateAlphaHealthCheck(hc *computealpha.HealthCheck) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newHealthcheckMetricContextWithVersion("create", computeAlphaVersion)
return mc.Observe(g.c.AlphaHealthChecks().Insert(ctx, meta.GlobalKey(hc.Name), hc))
}
// CreateBetaHealthCheck creates the given beta HealthCheck.
func (g *Cloud) CreateBetaHealthCheck(hc *computebeta.HealthCheck) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newHealthcheckMetricContextWithVersion("create", computeBetaVersion)
return mc.Observe(g.c.BetaHealthChecks().Insert(ctx, meta.GlobalKey(hc.Name), hc))
}
// ListHealthChecks lists all HealthCheck in the project.
func (g *Cloud) ListHealthChecks() ([]*compute.HealthCheck, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newHealthcheckMetricContext("list")
v, err := g.c.HealthChecks().List(ctx, filter.None)
return v, mc.Observe(err)
}
// GetNodesHealthCheckPort returns the health check port used by the GCE load
// balancers (l4) for performing health checks on nodes.
func GetNodesHealthCheckPort() int32 {
return lbNodesHealthCheckPort
}
// GetNodesHealthCheckPath returns the health check path used by the GCE load
// balancers (l4) for performing health checks on nodes.
func GetNodesHealthCheckPath() string {
return nodesHealthCheckPath
}

138
staging/src/k8s.io/legacy-cloud-providers/gce/gce_instancegroup.go
View File

@@ -1,138 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
compute "google.golang.org/api/compute/v1"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud/filter"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud/meta"
)
func newInstanceGroupMetricContext(request string, zone string) *metricContext {
return newGenericMetricContext("instancegroup", request, unusedMetricLabel, zone, computeV1Version)
}
// CreateInstanceGroup creates an instance group with the given
// instances. It is the callers responsibility to add named ports.
func (g *Cloud) CreateInstanceGroup(ig *compute.InstanceGroup, zone string) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newInstanceGroupMetricContext("create", zone)
return mc.Observe(g.c.InstanceGroups().Insert(ctx, meta.ZonalKey(ig.Name, zone), ig))
}
// DeleteInstanceGroup deletes an instance group.
func (g *Cloud) DeleteInstanceGroup(name string, zone string) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newInstanceGroupMetricContext("delete", zone)
return mc.Observe(g.c.InstanceGroups().Delete(ctx, meta.ZonalKey(name, zone)))
}
// FilterInstanceGroupsByName lists all InstanceGroups in the project and
// zone that match the name regexp.
func (g *Cloud) FilterInstanceGroupsByNamePrefix(namePrefix, zone string) ([]*compute.InstanceGroup, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newInstanceGroupMetricContext("filter", zone)
v, err := g.c.InstanceGroups().List(ctx, zone, filter.Regexp("name", namePrefix+".*"))
return v, mc.Observe(err)
}
// ListInstanceGroups lists all InstanceGroups in the project and
// zone.
func (g *Cloud) ListInstanceGroups(zone string) ([]*compute.InstanceGroup, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newInstanceGroupMetricContext("list", zone)
v, err := g.c.InstanceGroups().List(ctx, zone, filter.None)
return v, mc.Observe(err)
}
// ListInstancesInInstanceGroup lists all the instances in a given
// instance group and state.
func (g *Cloud) ListInstancesInInstanceGroup(name string, zone string, state string) ([]*compute.InstanceWithNamedPorts, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newInstanceGroupMetricContext("list_instances", zone)
req := &compute.InstanceGroupsListInstancesRequest{InstanceState: state}
v, err := g.c.InstanceGroups().ListInstances(ctx, meta.ZonalKey(name, zone), req, filter.None)
return v, mc.Observe(err)
}
// AddInstancesToInstanceGroup adds the given instances to the given
// instance group.
func (g *Cloud) AddInstancesToInstanceGroup(name string, zone string, instanceRefs []*compute.InstanceReference) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newInstanceGroupMetricContext("add_instances", zone)
// TODO: should cull operation above this layer.
if len(instanceRefs) == 0 {
return nil
}
req := &compute.InstanceGroupsAddInstancesRequest{
Instances: instanceRefs,
}
return mc.Observe(g.c.InstanceGroups().AddInstances(ctx, meta.ZonalKey(name, zone), req))
}
// RemoveInstancesFromInstanceGroup removes the given instances from
// the instance group.
func (g *Cloud) RemoveInstancesFromInstanceGroup(name string, zone string, instanceRefs []*compute.InstanceReference) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newInstanceGroupMetricContext("remove_instances", zone)
// TODO: should cull operation above this layer.
if len(instanceRefs) == 0 {
return nil
}
req := &compute.InstanceGroupsRemoveInstancesRequest{
Instances: instanceRefs,
}
return mc.Observe(g.c.InstanceGroups().RemoveInstances(ctx, meta.ZonalKey(name, zone), req))
}
// SetNamedPortsOfInstanceGroup sets the list of named ports on a given instance group
func (g *Cloud) SetNamedPortsOfInstanceGroup(igName, zone string, namedPorts []*compute.NamedPort) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newInstanceGroupMetricContext("set_namedports", zone)
req := &compute.InstanceGroupsSetNamedPortsRequest{NamedPorts: namedPorts}
return mc.Observe(g.c.InstanceGroups().SetNamedPorts(ctx, meta.ZonalKey(igName, zone), req))
}
// GetInstanceGroup returns an instance group by name.
func (g *Cloud) GetInstanceGroup(name string, zone string) (*compute.InstanceGroup, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newInstanceGroupMetricContext("get", zone)
v, err := g.c.InstanceGroups().Get(ctx, meta.ZonalKey(name, zone))
return v, mc.Observe(err)
}

810
staging/src/k8s.io/legacy-cloud-providers/gce/gce_instances.go
View File

@@ -1,810 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
"context"
"fmt"
"net"
"net/http"
"strings"
"time"
"cloud.google.com/go/compute/metadata"
computebeta "google.golang.org/api/compute/v0.beta"
compute "google.golang.org/api/compute/v1"
"k8s.io/klog/v2"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud/filter"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud/meta"
"k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/types"
"k8s.io/apimachinery/pkg/util/sets"
"k8s.io/apimachinery/pkg/util/wait"
cloudprovider "k8s.io/cloud-provider"
)
const (
defaultZone = ""
networkInterfaceIP = "instance/network-interfaces/%s/ip"
networkInterfaceAccessConfigs = "instance/network-interfaces/%s/access-configs"
networkInterfaceExternalIP = "instance/network-interfaces/%s/access-configs/%s/external-ip"
)
func newInstancesMetricContext(request, zone string) *metricContext {
return newGenericMetricContext("instances", request, unusedMetricLabel, zone, computeV1Version)
}
func splitNodesByZone(nodes []*v1.Node) map[string][]*v1.Node {
zones := make(map[string][]*v1.Node)
for _, n := range nodes {
z := getZone(n)
if z != defaultZone {
zones[z] = append(zones[z], n)
}
}
return zones
}
func getZone(n *v1.Node) string {
zone, ok := n.Labels[v1.LabelFailureDomainBetaZone]
if !ok {
return defaultZone
}
return zone
}
func makeHostURL(projectsAPIEndpoint, projectID, zone, host string) string {
host = canonicalizeInstanceName(host)
return projectsAPIEndpoint + strings.Join([]string{projectID, "zones", zone, "instances", host}, "/")
}
// ToInstanceReferences returns instance references by links
func (g *Cloud) ToInstanceReferences(zone string, instanceNames []string) (refs []*compute.InstanceReference) {
for _, ins := range instanceNames {
instanceLink := makeHostURL(g.projectsBasePath, g.projectID, zone, ins)
refs = append(refs, &compute.InstanceReference{Instance: instanceLink})
}
return refs
}
// NodeAddresses is an implementation of Instances.NodeAddresses.
func (g *Cloud) NodeAddresses(ctx context.Context, nodeName types.NodeName) ([]v1.NodeAddress, error) {
timeoutCtx, cancel := context.WithTimeout(ctx, 1*time.Hour)
defer cancel()
instanceName := string(nodeName)
if g.useMetadataServer {
// Use metadata server if possible
if g.isCurrentInstance(instanceName) {
nics, err := metadata.Get("instance/network-interfaces/")
if err != nil {
return nil, fmt.Errorf("couldn't get network interfaces: %v", err)
}
nicsArr := strings.Split(nics, "/\n")
nodeAddresses := []v1.NodeAddress{}
for _, nic := range nicsArr {
if nic == "" {
continue
}
internalIP, err := metadata.Get(fmt.Sprintf(networkInterfaceIP, nic))
if err != nil {
return nil, fmt.Errorf("couldn't get internal IP: %v", err)
}
nodeAddresses = append(nodeAddresses, v1.NodeAddress{Type: v1.NodeInternalIP, Address: internalIP})
acs, err := metadata.Get(fmt.Sprintf(networkInterfaceAccessConfigs, nic))
if err != nil {
return nil, fmt.Errorf("couldn't get access configs: %v", err)
}
acsArr := strings.Split(acs, "/\n")
for _, ac := range acsArr {
if ac == "" {
continue
}
externalIP, err := metadata.Get(fmt.Sprintf(networkInterfaceExternalIP, nic, ac))
if err != nil {
return nil, fmt.Errorf("couldn't get external IP: %v", err)
}
if externalIP != "" {
nodeAddresses = append(nodeAddresses, v1.NodeAddress{Type: v1.NodeExternalIP, Address: externalIP})
}
}
}
internalDNSFull, err := metadata.Get("instance/hostname")
if err != nil {
klog.Warningf("couldn't get full internal DNS name: %v", err)
} else {
nodeAddresses = append(nodeAddresses,
v1.NodeAddress{Type: v1.NodeInternalDNS, Address: internalDNSFull},
v1.NodeAddress{Type: v1.NodeHostName, Address: internalDNSFull},
)
}
return nodeAddresses, nil
}
}
// Use GCE API
instanceObj, err := g.getInstanceByName(instanceName)
if err != nil {
return nil, fmt.Errorf("couldn't get instance details: %v", err)
}
instance, err := g.c.Instances().Get(timeoutCtx, meta.ZonalKey(canonicalizeInstanceName(instanceObj.Name), instanceObj.Zone))
if err != nil {
return []v1.NodeAddress{}, fmt.Errorf("error while querying for instance: %v", err)
}
return nodeAddressesFromInstance(instance)
}
// NodeAddressesByProviderID will not be called from the node that is requesting this ID.
// i.e. metadata service and other local methods cannot be used here
func (g *Cloud) NodeAddressesByProviderID(ctx context.Context, providerID string) ([]v1.NodeAddress, error) {
timeoutCtx, cancel := context.WithTimeout(ctx, 1*time.Hour)
defer cancel()
_, zone, name, err := splitProviderID(providerID)
if err != nil {
return []v1.NodeAddress{}, err
}
instance, err := g.c.Instances().Get(timeoutCtx, meta.ZonalKey(canonicalizeInstanceName(name), zone))
if err != nil {
return []v1.NodeAddress{}, fmt.Errorf("error while querying for providerID %q: %v", providerID, err)
}
return nodeAddressesFromInstance(instance)
}
// instanceByProviderID returns the cloudprovider instance of the node
// with the specified unique providerID
func (g *Cloud) instanceByProviderID(providerID string) (*gceInstance, error) {
project, zone, name, err := splitProviderID(providerID)
if err != nil {
return nil, err
}
instance, err := g.getInstanceFromProjectInZoneByName(project, zone, name)
if err != nil {
if isHTTPErrorCode(err, http.StatusNotFound) {
return nil, cloudprovider.InstanceNotFound
}
return nil, err
}
return instance, nil
}
// InstanceShutdownByProviderID returns true if the instance is in safe state to detach volumes
func (g *Cloud) InstanceShutdownByProviderID(ctx context.Context, providerID string) (bool, error) {
return false, cloudprovider.NotImplemented
}
// InstanceShutdown returns true if the instance is in safe state to detach volumes
func (g *Cloud) InstanceShutdown(ctx context.Context, node *v1.Node) (bool, error) {
return false, cloudprovider.NotImplemented
}
func nodeAddressesFromInstance(instance *compute.Instance) ([]v1.NodeAddress, error) {
if len(instance.NetworkInterfaces) < 1 {
return nil, fmt.Errorf("could not find network interfaces for instanceID %q", instance.Id)
}
nodeAddresses := []v1.NodeAddress{}
for _, nic := range instance.NetworkInterfaces {
nodeAddresses = append(nodeAddresses, v1.NodeAddress{Type: v1.NodeInternalIP, Address: nic.NetworkIP})
for _, config := range nic.AccessConfigs {
nodeAddresses = append(nodeAddresses, v1.NodeAddress{Type: v1.NodeExternalIP, Address: config.NatIP})
}
}
return nodeAddresses, nil
}
// InstanceTypeByProviderID returns the cloudprovider instance type of the node
// with the specified unique providerID This method will not be called from the
// node that is requesting this ID. i.e. metadata service and other local
// methods cannot be used here
func (g *Cloud) InstanceTypeByProviderID(ctx context.Context, providerID string) (string, error) {
instance, err := g.instanceByProviderID(providerID)
if err != nil {
return "", err
}
return instance.Type, nil
}
// InstanceExistsByProviderID returns true if the instance with the given provider id still exists and is running.
// If false is returned with no error, the instance will be immediately deleted by the cloud controller manager.
func (g *Cloud) InstanceExistsByProviderID(ctx context.Context, providerID string) (bool, error) {
_, err := g.instanceByProviderID(providerID)
if err != nil {
if err == cloudprovider.InstanceNotFound {
return false, nil
}
return false, err
}
return true, nil
}
// InstanceExists returns true if the instance with the given provider id still exists and is running.
// If false is returned with no error, the instance will be immediately deleted by the cloud controller manager.
func (g *Cloud) InstanceExists(ctx context.Context, node *v1.Node) (bool, error) {
providerID := node.Spec.ProviderID
if providerID == "" {
var err error
if providerID, err = cloudprovider.GetInstanceProviderID(ctx, g, types.NodeName(node.Name)); err != nil {
if err == cloudprovider.InstanceNotFound {
return false, nil
}
return false, err
}
}
return g.InstanceExistsByProviderID(ctx, providerID)
}
// InstanceMetadata returns metadata of the specified instance.
func (g *Cloud) InstanceMetadata(ctx context.Context, node *v1.Node) (*cloudprovider.InstanceMetadata, error) {
timeoutCtx, cancel := context.WithTimeout(ctx, 1*time.Hour)
defer cancel()
providerID := node.Spec.ProviderID
if providerID == "" {
var err error
if providerID, err = cloudprovider.GetInstanceProviderID(ctx, g, types.NodeName(node.Name)); err != nil {
return nil, err
}
}
_, zone, name, err := splitProviderID(providerID)
if err != nil {
return nil, err
}
region, err := GetGCERegion(zone)
if err != nil {
return nil, err
}
instance, err := g.c.Instances().Get(timeoutCtx, meta.ZonalKey(canonicalizeInstanceName(name), zone))
if err != nil {
return nil, fmt.Errorf("error while querying for providerID %q: %v", providerID, err)
}
addresses, err := nodeAddressesFromInstance(instance)
if err != nil {
return nil, err
}
return &cloudprovider.InstanceMetadata{
ProviderID: providerID,
InstanceType: lastComponent(instance.MachineType),
NodeAddresses: addresses,
Zone: zone,
Region: region,
}, nil
}
// InstanceID returns the cloud provider ID of the node with the specified NodeName.
func (g *Cloud) InstanceID(ctx context.Context, nodeName types.NodeName) (string, error) {
instanceName := mapNodeNameToInstanceName(nodeName)
if g.useMetadataServer {
// Use metadata, if possible, to fetch ID. See issue #12000
if g.isCurrentInstance(instanceName) {
projectID, zone, err := getProjectAndZone()
if err == nil {
return projectID + "/" + zone + "/" + canonicalizeInstanceName(instanceName), nil
}
}
}
instance, err := g.getInstanceByName(instanceName)
if err != nil {
return "", err
}
return g.projectID + "/" + instance.Zone + "/" + instance.Name, nil
}
// InstanceType returns the type of the specified node with the specified NodeName.
func (g *Cloud) InstanceType(ctx context.Context, nodeName types.NodeName) (string, error) {
instanceName := mapNodeNameToInstanceName(nodeName)
if g.useMetadataServer {
// Use metadata, if possible, to fetch ID. See issue #12000
if g.isCurrentInstance(instanceName) {
mType, err := getCurrentMachineTypeViaMetadata()
if err == nil {
return mType, nil
}
}
}
instance, err := g.getInstanceByName(instanceName)
if err != nil {
return "", err
}
return instance.Type, nil
}
// AddSSHKeyToAllInstances adds an SSH public key as a legal identity for all instances
// expected format for the key is standard ssh-keygen format: <protocol> <blob>
func (g *Cloud) AddSSHKeyToAllInstances(ctx context.Context, user string, keyData []byte) error {
ctx, cancel := context.WithTimeout(ctx, 1*time.Hour)
defer cancel()
return wait.Poll(2*time.Second, 30*time.Second, func() (bool, error) {
project, err := g.c.Projects().Get(ctx, g.projectID)
if err != nil {
klog.Errorf("Could not get project: %v", err)
return false, nil
}
keyString := fmt.Sprintf("%s:%s %s@%s", user, strings.TrimSpace(string(keyData)), user, user)
found := false
for _, item := range project.CommonInstanceMetadata.Items {
if item.Key == "sshKeys" {
if strings.Contains(*item.Value, keyString) {
// We've already added the key
klog.Info("SSHKey already in project metadata")
return true, nil
}
value := *item.Value + "\n" + keyString
item.Value = &value
found = true
break
}
}
if !found {
// This is super unlikely, so log.
klog.Infof("Failed to find sshKeys metadata, creating a new item")
project.CommonInstanceMetadata.Items = append(project.CommonInstanceMetadata.Items,
&compute.MetadataItems{
Key: "sshKeys",
Value: &keyString,
})
}
mc := newInstancesMetricContext("add_ssh_key", "")
err = g.c.Projects().SetCommonInstanceMetadata(ctx, g.projectID, project.CommonInstanceMetadata)
mc.Observe(err)
if err != nil {
klog.Errorf("Could not Set Metadata: %v", err)
return false, nil
}
klog.Infof("Successfully added sshKey to project metadata")
return true, nil
})
}
// GetAllCurrentZones returns all the zones in which k8s nodes are currently running
func (g *Cloud) GetAllCurrentZones() (sets.String, error) {
if g.nodeInformerSynced == nil {
klog.Warning("Cloud object does not have informers set, should only happen in E2E binary.")
return g.GetAllZonesFromCloudProvider()
}
g.nodeZonesLock.Lock()
defer g.nodeZonesLock.Unlock()
if !g.nodeInformerSynced() {
return nil, fmt.Errorf("node informer is not synced when trying to GetAllCurrentZones")
}
zones := sets.NewString()
for zone, nodes := range g.nodeZones {
if len(nodes) > 0 {
zones.Insert(zone)
}
}
return zones, nil
}
// GetAllZonesFromCloudProvider returns all the zones in which nodes are running
// Only use this in E2E tests to get zones, on real clusters this will
// get all zones with compute instances in them even if not k8s instances!!!
// ex. I have k8s nodes in us-central1-c and us-central1-b. I also have
// a non-k8s compute in us-central1-a. This func will return a,b, and c.
//
// TODO: this should be removed from the cloud provider.
func (g *Cloud) GetAllZonesFromCloudProvider() (sets.String, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
zones := sets.NewString()
for _, zone := range g.managedZones {
instances, err := g.c.Instances().List(ctx, zone, filter.None)
if err != nil {
return sets.NewString(), err
}
if len(instances) > 0 {
zones.Insert(zone)
}
}
return zones, nil
}
// InsertInstance creates a new instance on GCP
func (g *Cloud) InsertInstance(project string, zone string, i *compute.Instance) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newInstancesMetricContext("create", zone)
return mc.Observe(g.c.Instances().Insert(ctx, meta.ZonalKey(i.Name, zone), i))
}
// ListInstanceNames returns a string of instance names separated by spaces.
// This method should only be used for e2e testing.
// TODO: remove this method.
func (g *Cloud) ListInstanceNames(project, zone string) (string, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
l, err := g.c.Instances().List(ctx, zone, filter.None)
if err != nil {
return "", err
}
var names []string
for _, i := range l {
names = append(names, i.Name)
}
return strings.Join(names, " "), nil
}
// DeleteInstance deletes an instance specified by project, zone, and name
func (g *Cloud) DeleteInstance(project, zone, name string) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
return g.c.Instances().Delete(ctx, meta.ZonalKey(name, zone))
}
// CurrentNodeName returns the name of the node we are currently running on
// On most clouds (e.g. GCE) this is the hostname, so we provide the hostname
func (g *Cloud) CurrentNodeName(ctx context.Context, hostname string) (types.NodeName, error) {
return types.NodeName(hostname), nil
}
// AliasRangesByProviderID returns a list of CIDR ranges that are assigned to the
// `node` for allocation to pods. Returns a list of the form
// "<ip>/<netmask>".
func (g *Cloud) AliasRangesByProviderID(providerID string) (cidrs []string, err error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
_, zone, name, err := splitProviderID(providerID)
if err != nil {
return nil, err
}
var res *computebeta.Instance
res, err = g.c.BetaInstances().Get(ctx, meta.ZonalKey(canonicalizeInstanceName(name), zone))
if err != nil {
return
}
for _, networkInterface := range res.NetworkInterfaces {
for _, r := range networkInterface.AliasIpRanges {
cidrs = append(cidrs, r.IpCidrRange)
}
}
return
}
// AddAliasToInstanceByProviderID adds an alias to the given instance from the named
// secondary range.
func (g *Cloud) AddAliasToInstanceByProviderID(providerID string, alias *net.IPNet) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
_, zone, name, err := splitProviderID(providerID)
if err != nil {
return err
}
instance, err := g.c.BetaInstances().Get(ctx, meta.ZonalKey(canonicalizeInstanceName(name), zone))
if err != nil {
return err
}
switch len(instance.NetworkInterfaces) {
case 0:
return fmt.Errorf("instance %q has no network interfaces", providerID)
case 1:
default:
klog.Warningf("Instance %q has more than one network interface, using only the first (%v)",
providerID, instance.NetworkInterfaces)
}
iface := &computebeta.NetworkInterface{}
iface.Name = instance.NetworkInterfaces[0].Name
iface.Fingerprint = instance.NetworkInterfaces[0].Fingerprint
iface.AliasIpRanges = append(iface.AliasIpRanges, &computebeta.AliasIpRange{
IpCidrRange: alias.String(),
SubnetworkRangeName: g.secondaryRangeName,
})
mc := newInstancesMetricContext("add_alias", zone)
err = g.c.BetaInstances().UpdateNetworkInterface(ctx, meta.ZonalKey(instance.Name, lastComponent(instance.Zone)), iface.Name, iface)
return mc.Observe(err)
}
// Gets the named instances, returning cloudprovider.InstanceNotFound if any
// instance is not found
func (g *Cloud) getInstancesByNames(names []string) ([]*gceInstance, error) {
foundInstances, err := g.getFoundInstanceByNames(names)
if err != nil {
return nil, err
}
if len(foundInstances) != len(names) {
if len(foundInstances) == 0 {
// return error so the TargetPool nodecount does not drop to 0 unexpectedly.
return nil, cloudprovider.InstanceNotFound
}
klog.Warningf("getFoundInstanceByNames - input instances %d, found %d. Continuing LoadBalancer Update", len(names), len(foundInstances))
}
return foundInstances, nil
}
// Gets the named instances, returning a list of gceInstances it was able to find from the provided
// list of names.
func (g *Cloud) getFoundInstanceByNames(names []string) ([]*gceInstance, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
found := map[string]*gceInstance{}
remaining := len(names)
nodeInstancePrefix := g.nodeInstancePrefix
for _, name := range names {
name = canonicalizeInstanceName(name)
if !strings.HasPrefix(name, g.nodeInstancePrefix) {
klog.Warningf("Instance %q does not conform to prefix %q, removing filter", name, g.nodeInstancePrefix)
nodeInstancePrefix = ""
}
found[name] = nil
}
for _, zone := range g.managedZones {
if remaining == 0 {
break
}
instances, err := g.c.Instances().List(ctx, zone, filter.Regexp("name", nodeInstancePrefix+".*"))
if err != nil {
return nil, err
}
for _, inst := range instances {
if remaining == 0 {
break
}
if _, ok := found[inst.Name]; !ok {
continue
}
if found[inst.Name] != nil {
klog.Errorf("Instance name %q was duplicated (in zone %q and %q)", inst.Name, zone, found[inst.Name].Zone)
continue
}
found[inst.Name] = &gceInstance{
Zone: zone,
Name: inst.Name,
ID: inst.Id,
Disks: inst.Disks,
Type: lastComponent(inst.MachineType),
}
remaining--
}
}
var ret []*gceInstance
var failed []string
for name, instance := range found {
if instance != nil {
ret = append(ret, instance)
} else {
failed = append(failed, name)
}
}
if len(failed) > 0 {
klog.Errorf("Failed to retrieve instances: %v", failed)
}
return ret, nil
}
// Gets the named instance, returning cloudprovider.InstanceNotFound if the instance is not found
func (g *Cloud) getInstanceByName(name string) (*gceInstance, error) {
// Avoid changing behaviour when not managing multiple zones
for _, zone := range g.managedZones {
instance, err := g.getInstanceFromProjectInZoneByName(g.projectID, zone, name)
if err != nil {
if isHTTPErrorCode(err, http.StatusNotFound) {
continue
}
klog.Errorf("getInstanceByName: failed to get instance %s in zone %s; err: %v", name, zone, err)
return nil, err
}
return instance, nil
}
return nil, cloudprovider.InstanceNotFound
}
func (g *Cloud) getInstanceFromProjectInZoneByName(project, zone, name string) (*gceInstance, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
name = canonicalizeInstanceName(name)
mc := newInstancesMetricContext("get", zone)
res, err := g.c.Instances().Get(ctx, meta.ZonalKey(name, zone))
mc.Observe(err)
if err != nil {
return nil, err
}
return &gceInstance{
Zone: lastComponent(res.Zone),
Name: res.Name,
ID: res.Id,
Disks: res.Disks,
Type: lastComponent(res.MachineType),
}, nil
}
func getInstanceIDViaMetadata() (string, error) {
result, err := metadata.Get("instance/hostname")
if err != nil {
return "", err
}
parts := strings.Split(result, ".")
if len(parts) == 0 {
return "", fmt.Errorf("unexpected response: %s", result)
}
return parts[0], nil
}
func getCurrentMachineTypeViaMetadata() (string, error) {
mType, err := metadata.Get("instance/machine-type")
if err != nil {
return "", fmt.Errorf("couldn't get machine type: %v", err)
}
parts := strings.Split(mType, "/")
if len(parts) != 4 {
return "", fmt.Errorf("unexpected response for machine type: %s", mType)
}
return parts[3], nil
}
// isCurrentInstance uses metadata server to check if specified
// instanceID matches current machine's instanceID
func (g *Cloud) isCurrentInstance(instanceID string) bool {
currentInstanceID, err := getInstanceIDViaMetadata()
if err != nil {
// Log and swallow error
klog.Errorf("Failed to fetch instanceID via Metadata: %v", err)
return false
}
return currentInstanceID == canonicalizeInstanceName(instanceID)
}
// ComputeHostTags grabs all tags from all instances being added to the pool.
// * The longest tag that is a prefix of the instance name is used
// * If any instance has no matching prefix tag, return error
// Invoking this method to get host tags is risky since it depends on the
// format of the host names in the cluster. Only use it as a fallback if
// gce.nodeTags is unspecified
func (g *Cloud) computeHostTags(hosts []*gceInstance) ([]string, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
// TODO: We could store the tags in gceInstance, so we could have already fetched it
hostNamesByZone := make(map[string]map[string]bool) // map of zones -> map of names -> bool (for easy lookup)
nodeInstancePrefix := g.nodeInstancePrefix
for _, host := range hosts {
if !strings.HasPrefix(host.Name, g.nodeInstancePrefix) {
klog.Warningf("instance %v does not conform to prefix '%s', ignoring filter", host, g.nodeInstancePrefix)
nodeInstancePrefix = ""
}
z, ok := hostNamesByZone[host.Zone]
if !ok {
z = make(map[string]bool)
hostNamesByZone[host.Zone] = z
}
z[host.Name] = true
}
tags := sets.NewString()
filt := filter.None
if nodeInstancePrefix != "" {
filt = filter.Regexp("name", nodeInstancePrefix+".*")
}
for zone, hostNames := range hostNamesByZone {
instances, err := g.c.Instances().List(ctx, zone, filt)
if err != nil {
return nil, err
}
for _, instance := range instances {
if !hostNames[instance.Name] {
continue
}
longestTag := ""
for _, tag := range instance.Tags.Items {
if strings.HasPrefix(instance.Name, tag) && len(tag) > len(longestTag) {
longestTag = tag
}
}
if len(longestTag) > 0 {
tags.Insert(longestTag)
} else {
return nil, fmt.Errorf("could not find any tag that is a prefix of instance name for instance %s", instance.Name)
}
}
}
if len(tags) == 0 {
return nil, fmt.Errorf("no instances found")
}
return tags.List(), nil
}
// GetNodeTags will first try returning the list of tags specified in GCE cloud Configuration.
// If they weren't provided, it'll compute the host tags with the given hostnames. If the list
// of hostnames has not changed, a cached set of nodetags are returned.
func (g *Cloud) GetNodeTags(nodeNames []string) ([]string, error) {
// If nodeTags were specified through configuration, use them
if len(g.nodeTags) > 0 {
return g.nodeTags, nil
}
g.computeNodeTagLock.Lock()
defer g.computeNodeTagLock.Unlock()
// Early return if hosts have not changed
hosts := sets.NewString(nodeNames...)
if hosts.Equal(g.lastKnownNodeNames) {
return g.lastComputedNodeTags, nil
}
// Get GCE instance data by hostname
instances, err := g.getInstancesByNames(nodeNames)
if err != nil {
return nil, err
}
// Determine list of host tags
tags, err := g.computeHostTags(instances)
if err != nil {
return nil, err
}
// Save the list of tags
g.lastKnownNodeNames = hosts
g.lastComputedNodeTags = tags
return tags, nil
}

69
staging/src/k8s.io/legacy-cloud-providers/gce/gce_instances_test.go
View File

@@ -1,69 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2020 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
"context"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
func TestInstanceExists(t *testing.T) {
gce, err := fakeGCECloud(DefaultTestClusterValues())
require.NoError(t, err)
nodeNames := []string{"test-node-1"}
_, err = createAndInsertNodes(gce, nodeNames, vals.ZoneName)
require.NoError(t, err)
testcases := []struct {
name string
nodeName string
exist bool
expectedErr error
}{
{
name: "node exist",
nodeName: "test-node-1",
exist: true,
expectedErr: nil,
},
{
name: "node not exist",
nodeName: "test-node-2",
exist: false,
expectedErr: nil,
},
}
for _, test := range testcases {
t.Run(test.name, func(t *testing.T) {
node := &v1.Node{ObjectMeta: metav1.ObjectMeta{Name: test.nodeName}}
exist, err := gce.InstanceExists(context.TODO(), node)
assert.Equal(t, test.expectedErr, err, test.name)
assert.Equal(t, test.exist, exist, test.name)
})
}
}

54
staging/src/k8s.io/legacy-cloud-providers/gce/gce_interfaces.go
View File

@@ -1,54 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
computebeta "google.golang.org/api/compute/v0.beta"
compute "google.golang.org/api/compute/v1"
)
// These interfaces are added for testability.
// CloudAddressService is an interface for managing addresses
type CloudAddressService interface {
ReserveRegionAddress(address *compute.Address, region string) error
GetRegionAddress(name string, region string) (*compute.Address, error)
GetRegionAddressByIP(region, ipAddress string) (*compute.Address, error)
DeleteRegionAddress(name, region string) error
// TODO: Mock Global endpoints
// Beta API
ReserveBetaRegionAddress(address *computebeta.Address, region string) error
GetBetaRegionAddress(name string, region string) (*computebeta.Address, error)
GetBetaRegionAddressByIP(region, ipAddress string) (*computebeta.Address, error)
getNetworkTierFromAddress(name, region string) (string, error)
}
// CloudForwardingRuleService is an interface for managing forwarding rules.
// TODO: Expand the interface to include more methods.
type CloudForwardingRuleService interface {
GetRegionForwardingRule(name, region string) (*compute.ForwardingRule, error)
CreateRegionForwardingRule(rule *compute.ForwardingRule, region string) error
DeleteRegionForwardingRule(name, region string) error
// Needed for the "Network Tiers" feature.
getNetworkTierFromForwardingRule(name, region string) (string, error)
}

302
staging/src/k8s.io/legacy-cloud-providers/gce/gce_loadbalancer.go
View File

@@ -1,302 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
"context"
"flag"
"fmt"
"sort"
"strings"
v1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
corev1apply "k8s.io/client-go/applyconfigurations/core/v1"
metav1apply "k8s.io/client-go/applyconfigurations/meta/v1"
"k8s.io/klog/v2"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud"
cloudprovider "k8s.io/cloud-provider"
netutils "k8s.io/utils/net"
)
type cidrs struct {
ipn netutils.IPNetSet
isSet bool
}
var (
l4LbSrcRngsFlag cidrs
l7lbSrcRngsFlag cidrs
)
func init() {
var err error
// L3/4 health checkers have client addresses within these known CIDRs.
l4LbSrcRngsFlag.ipn, err = netutils.ParseIPNets([]string{"130.211.0.0/22", "35.191.0.0/16", "209.85.152.0/22", "209.85.204.0/22"}...)
if err != nil {
panic("Incorrect default GCE L3/4 source ranges")
}
// L7 health checkers have client addresses within these known CIDRs.
l7lbSrcRngsFlag.ipn, err = netutils.ParseIPNets([]string{"130.211.0.0/22", "35.191.0.0/16"}...)
if err != nil {
panic("Incorrect default GCE L7 source ranges")
}
flag.Var(&l4LbSrcRngsFlag, "cloud-provider-gce-lb-src-cidrs", "CIDRs opened in GCE firewall for L4 LB traffic proxy & health checks")
flag.Var(&l7lbSrcRngsFlag, "cloud-provider-gce-l7lb-src-cidrs", "CIDRs opened in GCE firewall for L7 LB traffic proxy & health checks")
}
// String is the method to format the flag's value, part of the flag.Value interface.
func (c *cidrs) String() string {
s := c.ipn.StringSlice()
sort.Strings(s)
return strings.Join(s, ",")
}
// Set supports a value of CSV or the flag repeated multiple times
func (c *cidrs) Set(value string) error {
// On first Set(), clear the original defaults
if !c.isSet {
c.isSet = true
c.ipn = make(netutils.IPNetSet)
} else {
return fmt.Errorf("GCE LB CIDRs have already been set")
}
for _, cidr := range strings.Split(value, ",") {
_, ipnet, err := netutils.ParseCIDRSloppy(cidr)
if err != nil {
return err
}
c.ipn.Insert(ipnet)
}
return nil
}
// L4LoadBalancerSrcRanges contains the ranges of ips used by the L3/L4 GCE load balancers
// for proxying client requests and performing health checks.
func L4LoadBalancerSrcRanges() []string {
return l4LbSrcRngsFlag.ipn.StringSlice()
}
// L7LoadBalancerSrcRanges contains the ranges of ips used by the GCE load balancers L7
// for proxying client requests and performing health checks.
func L7LoadBalancerSrcRanges() []string {
return l7lbSrcRngsFlag.ipn.StringSlice()
}
// GetLoadBalancer is an implementation of LoadBalancer.GetLoadBalancer
func (g *Cloud) GetLoadBalancer(ctx context.Context, clusterName string, svc *v1.Service) (*v1.LoadBalancerStatus, bool, error) {
loadBalancerName := g.GetLoadBalancerName(ctx, clusterName, svc)
fwd, err := g.GetRegionForwardingRule(loadBalancerName, g.region)
if err == nil {
status := &v1.LoadBalancerStatus{}
status.Ingress = []v1.LoadBalancerIngress{{IP: fwd.IPAddress}}
return status, true, nil
}
// Checking for finalizer is more accurate because controller restart could happen in the middle of resource
// deletion. So even though forwarding rule was deleted, cleanup might not have been complete.
if hasFinalizer(svc, ILBFinalizerV1) {
return &v1.LoadBalancerStatus{}, true, nil
}
return nil, false, ignoreNotFound(err)
}
// GetLoadBalancerName is an implementation of LoadBalancer.GetLoadBalancerName.
func (g *Cloud) GetLoadBalancerName(ctx context.Context, clusterName string, svc *v1.Service) string {
// TODO: replace DefaultLoadBalancerName to generate more meaningful loadbalancer names.
return cloudprovider.DefaultLoadBalancerName(svc)
}
// EnsureLoadBalancer is an implementation of LoadBalancer.EnsureLoadBalancer.
func (g *Cloud) EnsureLoadBalancer(ctx context.Context, clusterName string, svc *v1.Service, nodes []*v1.Node) (*v1.LoadBalancerStatus, error) {
loadBalancerName := g.GetLoadBalancerName(ctx, clusterName, svc)
desiredScheme := getSvcScheme(svc)
clusterID, err := g.ClusterID.GetID()
if err != nil {
return nil, err
}
// Services with multiples protocols are not supported by this controller, warn the users and sets
// the corresponding Service Status Condition.
// https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/1435-mixed-protocol-lb
if err := checkMixedProtocol(svc.Spec.Ports); err != nil {
if hasLoadBalancerPortsError(svc) {
return nil, err
}
klog.Warningf("Ignoring service %s/%s using different ports protocols", svc.Namespace, svc.Name)
g.eventRecorder.Event(svc, v1.EventTypeWarning, v1.LoadBalancerPortsErrorReason, "LoadBalancers with multiple protocols are not supported.")
svcApplyStatus := corev1apply.ServiceStatus().WithConditions(
metav1apply.Condition().
WithType(v1.LoadBalancerPortsError).
WithStatus(metav1.ConditionTrue).
WithReason(v1.LoadBalancerPortsErrorReason).
WithMessage("LoadBalancer with multiple protocols are not supported"))
svcApply := corev1apply.Service(svc.Name, svc.Namespace).WithStatus(svcApplyStatus)
if _, errApply := g.client.CoreV1().Services(svc.Namespace).ApplyStatus(ctx, svcApply, metav1.ApplyOptions{FieldManager: "gce-legacy-cloud-controller", Force: true}); errApply != nil {
return nil, errApply
}
return nil, err
}
klog.V(4).Infof("EnsureLoadBalancer(%v, %v, %v, %v, %v): ensure %v loadbalancer", clusterName, svc.Namespace, svc.Name, loadBalancerName, g.region, desiredScheme)
existingFwdRule, err := g.GetRegionForwardingRule(loadBalancerName, g.region)
if err != nil && !isNotFound(err) {
return nil, err
}
if existingFwdRule != nil {
existingScheme := cloud.LbScheme(strings.ToUpper(existingFwdRule.LoadBalancingScheme))
// If the loadbalancer type changes between INTERNAL and EXTERNAL, the old load balancer should be deleted.
if existingScheme != desiredScheme {
klog.V(4).Infof("EnsureLoadBalancer(%v, %v, %v, %v, %v): deleting existing %v loadbalancer", clusterName, svc.Namespace, svc.Name, loadBalancerName, g.region, existingScheme)
switch existingScheme {
case cloud.SchemeInternal:
err = g.ensureInternalLoadBalancerDeleted(clusterName, clusterID, svc)
default:
err = g.ensureExternalLoadBalancerDeleted(clusterName, clusterID, svc)
}
klog.V(4).Infof("EnsureLoadBalancer(%v, %v, %v, %v, %v): done deleting existing %v loadbalancer. err: %v", clusterName, svc.Namespace, svc.Name, loadBalancerName, g.region, existingScheme, err)
if err != nil {
return nil, err
}
// Assume the ensureDeleted function successfully deleted the forwarding rule.
existingFwdRule = nil
}
}
var status *v1.LoadBalancerStatus
switch desiredScheme {
case cloud.SchemeInternal:
status, err = g.ensureInternalLoadBalancer(clusterName, clusterID, svc, existingFwdRule, nodes)
default:
status, err = g.ensureExternalLoadBalancer(clusterName, clusterID, svc, existingFwdRule, nodes)
}
if err != nil {
klog.Errorf("Failed to EnsureLoadBalancer(%s, %s, %s, %s, %s), err: %v", clusterName, svc.Namespace, svc.Name, loadBalancerName, g.region, err)
return status, err
}
klog.V(4).Infof("EnsureLoadBalancer(%s, %s, %s, %s, %s): done ensuring loadbalancer.", clusterName, svc.Namespace, svc.Name, loadBalancerName, g.region)
return status, err
}
// UpdateLoadBalancer is an implementation of LoadBalancer.UpdateLoadBalancer.
func (g *Cloud) UpdateLoadBalancer(ctx context.Context, clusterName string, svc *v1.Service, nodes []*v1.Node) error {
loadBalancerName := g.GetLoadBalancerName(ctx, clusterName, svc)
scheme := getSvcScheme(svc)
clusterID, err := g.ClusterID.GetID()
if err != nil {
return err
}
// Services with multiples protocols are not supported by this controller, warn the users and sets
// the corresponding Service Status Condition, but keep processing the Update to not break upgrades.
// https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/1435-mixed-protocol-lb
if err := checkMixedProtocol(svc.Spec.Ports); err != nil && !hasLoadBalancerPortsError(svc) {
klog.Warningf("Ignoring update for service %s/%s using different ports protocols", svc.Namespace, svc.Name)
g.eventRecorder.Event(svc, v1.EventTypeWarning, v1.LoadBalancerPortsErrorReason, "LoadBalancer with multiple protocols are not supported.")
svcApplyStatus := corev1apply.ServiceStatus().WithConditions(
metav1apply.Condition().
WithType(v1.LoadBalancerPortsError).
WithStatus(metav1.ConditionTrue).
WithReason(v1.LoadBalancerPortsErrorReason).
WithMessage("LoadBalancer with multiple protocols are not supported"))
svcApply := corev1apply.Service(svc.Name, svc.Namespace).WithStatus(svcApplyStatus)
if _, errApply := g.client.CoreV1().Services(svc.Namespace).ApplyStatus(ctx, svcApply, metav1.ApplyOptions{FieldManager: "gce-legacy-cloud-controller", Force: true}); errApply != nil {
// the error is retried by the controller loop
return errApply
}
}
klog.V(4).Infof("UpdateLoadBalancer(%v, %v, %v, %v, %v): updating with %d nodes", clusterName, svc.Namespace, svc.Name, loadBalancerName, g.region, len(nodes))
switch scheme {
case cloud.SchemeInternal:
err = g.updateInternalLoadBalancer(clusterName, clusterID, svc, nodes)
default:
err = g.updateExternalLoadBalancer(clusterName, svc, nodes)
}
klog.V(4).Infof("UpdateLoadBalancer(%v, %v, %v, %v, %v): done updating. err: %v", clusterName, svc.Namespace, svc.Name, loadBalancerName, g.region, err)
return err
}
// EnsureLoadBalancerDeleted is an implementation of LoadBalancer.EnsureLoadBalancerDeleted.
func (g *Cloud) EnsureLoadBalancerDeleted(ctx context.Context, clusterName string, svc *v1.Service) error {
loadBalancerName := g.GetLoadBalancerName(ctx, clusterName, svc)
scheme := getSvcScheme(svc)
clusterID, err := g.ClusterID.GetID()
if err != nil {
return err
}
klog.V(4).Infof("EnsureLoadBalancerDeleted(%v, %v, %v, %v, %v): deleting loadbalancer", clusterName, svc.Namespace, svc.Name, loadBalancerName, g.region)
switch scheme {
case cloud.SchemeInternal:
err = g.ensureInternalLoadBalancerDeleted(clusterName, clusterID, svc)
default:
err = g.ensureExternalLoadBalancerDeleted(clusterName, clusterID, svc)
}
klog.V(4).Infof("EnsureLoadBalancerDeleted(%v, %v, %v, %v, %v): done deleting loadbalancer. err: %v", clusterName, svc.Namespace, svc.Name, loadBalancerName, g.region, err)
return err
}
func getSvcScheme(svc *v1.Service) cloud.LbScheme {
if t := GetLoadBalancerAnnotationType(svc); t == LBTypeInternal {
return cloud.SchemeInternal
}
return cloud.SchemeExternal
}
// checkMixedProtocol checks if the Service Ports uses different protocols,
// per examples, TCP and UDP.
func checkMixedProtocol(ports []v1.ServicePort) error {
if len(ports) == 0 {
return nil
}
firstProtocol := ports[0].Protocol
for _, port := range ports[1:] {
if port.Protocol != firstProtocol {
return fmt.Errorf("mixed protocol is not supported for LoadBalancer")
}
}
return nil
}
// hasLoadBalancerPortsError checks if the Service has the LoadBalancerPortsError set to True
func hasLoadBalancerPortsError(service *v1.Service) bool {
if service == nil {
return false
}
for _, cond := range service.Status.Conditions {
if cond.Type == v1.LoadBalancerPortsError {
return cond.Status == metav1.ConditionTrue
}
}
return false
}

1167
staging/src/k8s.io/legacy-cloud-providers/gce/gce_loadbalancer_external.go
View File

File diff suppressed because it is too large Load Diff

1748
staging/src/k8s.io/legacy-cloud-providers/gce/gce_loadbalancer_external_test.go
View File

File diff suppressed because it is too large Load Diff

1018
staging/src/k8s.io/legacy-cloud-providers/gce/gce_loadbalancer_internal.go
View File

File diff suppressed because it is too large Load Diff

2019
staging/src/k8s.io/legacy-cloud-providers/gce/gce_loadbalancer_internal_test.go
View File

File diff suppressed because it is too large Load Diff

182
staging/src/k8s.io/legacy-cloud-providers/gce/gce_loadbalancer_metrics.go
View File

@@ -1,182 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2020 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
"sync"
"time"
"k8s.io/apimachinery/pkg/util/wait"
"k8s.io/component-base/metrics"
"k8s.io/component-base/metrics/legacyregistry"
"k8s.io/klog/v2"
)
const (
label = "feature"
)
var (
metricsInterval = 10 * time.Minute
l4ILBCount = metrics.NewGaugeVec(
&metrics.GaugeOpts{
Name: "number_of_l4_ilbs",
Help: "Number of L4 ILBs",
StabilityLevel: metrics.ALPHA,
},
[]string{label},
)
)
// init registers L4 internal loadbalancer usage metrics.
func init() {
klog.V(3).Infof("Registering Service Controller loadbalancer usage metrics %v", l4ILBCount)
legacyregistry.MustRegister(l4ILBCount)
}
// LoadBalancerMetrics is a cache that contains loadbalancer service resource
// states for computing usage metrics.
type LoadBalancerMetrics struct {
// l4ILBServiceMap is a map of service key and L4 ILB service state.
l4ILBServiceMap map[string]L4ILBServiceState
sync.Mutex
}
type feature string
func (f feature) String() string {
return string(f)
}
const (
l4ILBService = feature("L4ILBService")
l4ILBGlobalAccess = feature("L4ILBGlobalAccess")
l4ILBCustomSubnet = feature("L4ILBCustomSubnet")
// l4ILBInSuccess feature specifies that ILB VIP is configured.
l4ILBInSuccess = feature("L4ILBInSuccess")
// l4ILBInInError feature specifies that an error had occurred for this service
// in ensureInternalLoadbalancer method.
l4ILBInError = feature("L4ILBInError")
)
// L4ILBServiceState contains Internal Loadbalancer feature states as specified
// in k8s Service.
type L4ILBServiceState struct {
// EnabledGlobalAccess specifies if Global Access is enabled.
EnabledGlobalAccess bool
// EnabledCustomSubNet specifies if Custom Subnet is enabled.
EnabledCustomSubnet bool
// InSuccess specifies if the ILB service VIP is configured.
InSuccess bool
}
// loadbalancerMetricsCollector is an interface to update/delete L4 loadbalancer
// states in the cache that is used for computing L4 Loadbalancer usage metrics.
type loadbalancerMetricsCollector interface {
// Run starts a goroutine to compute and export metrics a periodic interval.
Run(stopCh <-chan struct{})
// SetL4ILBService adds/updates L4 ILB service state for given service key.
SetL4ILBService(svcKey string, state L4ILBServiceState)
// DeleteL4ILBService removes the given L4 ILB service key.
DeleteL4ILBService(svcKey string)
}
// newLoadBalancerMetrics initializes LoadBalancerMetrics and starts a goroutine
// to compute and export metrics periodically.
func newLoadBalancerMetrics() loadbalancerMetricsCollector {
return &LoadBalancerMetrics{
l4ILBServiceMap: make(map[string]L4ILBServiceState),
}
}
// Run implements loadbalancerMetricsCollector.
func (lm *LoadBalancerMetrics) Run(stopCh <-chan struct{}) {
klog.V(3).Infof("Loadbalancer Metrics initialized. Metrics will be exported at an interval of %v", metricsInterval)
// Compute and export metrics periodically.
select {
case <-stopCh:
return
case <-time.After(metricsInterval): // Wait for service states to be populated in the cache before computing metrics.
wait.Until(lm.export, metricsInterval, stopCh)
}
}
// SetL4ILBService implements loadbalancerMetricsCollector.
func (lm *LoadBalancerMetrics) SetL4ILBService(svcKey string, state L4ILBServiceState) {
lm.Lock()
defer lm.Unlock()
if lm.l4ILBServiceMap == nil {
klog.Fatalf("Loadbalancer Metrics failed to initialize correctly.")
}
lm.l4ILBServiceMap[svcKey] = state
}
// DeleteL4ILBService implements loadbalancerMetricsCollector.
func (lm *LoadBalancerMetrics) DeleteL4ILBService(svcKey string) {
lm.Lock()
defer lm.Unlock()
delete(lm.l4ILBServiceMap, svcKey)
}
// export computes and exports loadbalancer usage metrics.
func (lm *LoadBalancerMetrics) export() {
ilbCount := lm.computeL4ILBMetrics()
klog.V(5).Infof("Exporting L4 ILB usage metrics: %#v", ilbCount)
for feature, count := range ilbCount {
l4ILBCount.With(map[string]string{label: feature.String()}).Set(float64(count))
}
klog.V(5).Infof("L4 ILB usage metrics exported.")
}
// computeL4ILBMetrics aggregates L4 ILB metrics in the cache.
func (lm *LoadBalancerMetrics) computeL4ILBMetrics() map[feature]int {
lm.Lock()
defer lm.Unlock()
klog.V(4).Infof("Computing L4 ILB usage metrics from service state map: %#v", lm.l4ILBServiceMap)
counts := map[feature]int{
l4ILBService: 0,
l4ILBGlobalAccess: 0,
l4ILBCustomSubnet: 0,
l4ILBInSuccess: 0,
l4ILBInError: 0,
}
for key, state := range lm.l4ILBServiceMap {
klog.V(6).Infof("ILB Service %s has EnabledGlobalAccess: %t, EnabledCustomSubnet: %t, InSuccess: %t", key, state.EnabledGlobalAccess, state.EnabledCustomSubnet, state.InSuccess)
counts[l4ILBService]++
if !state.InSuccess {
counts[l4ILBInError]++
// Skip counting other features if the service is in error state.
continue
}
counts[l4ILBInSuccess]++
if state.EnabledGlobalAccess {
counts[l4ILBGlobalAccess]++
}
if state.EnabledCustomSubnet {
counts[l4ILBCustomSubnet]++
}
}
klog.V(4).Info("L4 ILB usage metrics computed.")
return counts
}

170
staging/src/k8s.io/legacy-cloud-providers/gce/gce_loadbalancer_metrics_test.go
View File

@@ -1,170 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2020 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
"strconv"
"testing"
"github.com/google/go-cmp/cmp"
)
func TestComputeL4ILBMetrics(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
desc string
serviceStates []L4ILBServiceState
expectL4ILBCount map[feature]int
}{
{
desc: "empty input",
serviceStates: []L4ILBServiceState{},
expectL4ILBCount: map[feature]int{
l4ILBService: 0,
l4ILBGlobalAccess: 0,
l4ILBCustomSubnet: 0,
l4ILBInSuccess: 0,
l4ILBInError: 0,
},
},
{
desc: "one l4 ilb service",
serviceStates: []L4ILBServiceState{
newL4ILBServiceState(false, false, true),
},
expectL4ILBCount: map[feature]int{
l4ILBService: 1,
l4ILBGlobalAccess: 0,
l4ILBCustomSubnet: 0,
l4ILBInSuccess: 1,
l4ILBInError: 0,
},
},
{
desc: "l4 ilb service in error state",
serviceStates: []L4ILBServiceState{
newL4ILBServiceState(false, true, false),
},
expectL4ILBCount: map[feature]int{
l4ILBService: 1,
l4ILBGlobalAccess: 0,
l4ILBCustomSubnet: 0,
l4ILBInSuccess: 0,
l4ILBInError: 1,
},
},
{
desc: "global access for l4 ilb service enabled",
serviceStates: []L4ILBServiceState{
newL4ILBServiceState(true, false, true),
},
expectL4ILBCount: map[feature]int{
l4ILBService: 1,
l4ILBGlobalAccess: 1,
l4ILBCustomSubnet: 0,
l4ILBInSuccess: 1,
l4ILBInError: 0,
},
},
{
desc: "custom subnet for l4 ilb service enabled",
serviceStates: []L4ILBServiceState{
newL4ILBServiceState(false, true, true),
},
expectL4ILBCount: map[feature]int{
l4ILBService: 1,
l4ILBGlobalAccess: 0,
l4ILBCustomSubnet: 1,
l4ILBInSuccess: 1,
l4ILBInError: 0,
},
},
{
desc: "both global access and custom subnet for l4 ilb service enabled",
serviceStates: []L4ILBServiceState{
newL4ILBServiceState(true, true, true),
},
expectL4ILBCount: map[feature]int{
l4ILBService: 1,
l4ILBGlobalAccess: 1,
l4ILBCustomSubnet: 1,
l4ILBInSuccess: 1,
l4ILBInError: 0,
},
},
{
desc: "many l4 ilb services",
serviceStates: []L4ILBServiceState{
newL4ILBServiceState(false, false, true),
newL4ILBServiceState(false, true, true),
newL4ILBServiceState(true, false, true),
newL4ILBServiceState(true, true, true),
},
expectL4ILBCount: map[feature]int{
l4ILBService: 4,
l4ILBGlobalAccess: 2,
l4ILBCustomSubnet: 2,
l4ILBInSuccess: 4,
l4ILBInError: 0,
},
},
{
desc: "many l4 ilb services with some in error state",
serviceStates: []L4ILBServiceState{
newL4ILBServiceState(false, false, true),
newL4ILBServiceState(false, true, false),
newL4ILBServiceState(false, true, true),
newL4ILBServiceState(true, false, true),
newL4ILBServiceState(true, false, false),
newL4ILBServiceState(true, true, true),
},
expectL4ILBCount: map[feature]int{
l4ILBService: 6,
l4ILBGlobalAccess: 2,
l4ILBCustomSubnet: 2,
l4ILBInSuccess: 4,
l4ILBInError: 2,
},
},
} {
tc := tc
t.Run(tc.desc, func(t *testing.T) {
t.Parallel()
newMetrics := LoadBalancerMetrics{
l4ILBServiceMap: make(map[string]L4ILBServiceState),
}
for i, serviceState := range tc.serviceStates {
newMetrics.SetL4ILBService(strconv.Itoa(i), serviceState)
}
got := newMetrics.computeL4ILBMetrics()
if diff := cmp.Diff(tc.expectL4ILBCount, got); diff != "" {
t.Fatalf("Got diff for L4 ILB service counts (-want +got):\n%s", diff)
}
})
}
}
func newL4ILBServiceState(globalAccess, customSubnet, inSuccess bool) L4ILBServiceState {
return L4ILBServiceState{
EnabledGlobalAccess: globalAccess,
EnabledCustomSubnet: customSubnet,
InSuccess: inSuccess,
}
}

129
staging/src/k8s.io/legacy-cloud-providers/gce/gce_loadbalancer_naming.go
View File

@@ -1,129 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
"crypto/sha1"
"encoding/hex"
"fmt"
"strings"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud"
"k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/types"
)
// Internal Load Balancer
// Instance groups remain legacy named to stay consistent with ingress
func makeInstanceGroupName(clusterID string) string {
prefix := "k8s-ig"
// clusterID might be empty for legacy clusters
if clusterID == "" {
return prefix
}
return fmt.Sprintf("%s--%s", prefix, clusterID)
}
func makeBackendServiceName(loadBalancerName, clusterID string, shared bool, scheme cloud.LbScheme, protocol v1.Protocol, svcAffinity v1.ServiceAffinity) string {
if shared {
hash := sha1.New()
// For every non-nil option, hash its value. Currently, only service affinity is relevant.
hash.Write([]byte(string(svcAffinity)))
hashed := hex.EncodeToString(hash.Sum(nil))
hashed = hashed[:16]
// k8s- 4
// {clusterid}- 17
// {scheme}- 9 (internal/external)
// {protocol}- 4 (tcp/udp)
// nmv1- 5 (naming convention version)
// {suffix} 16 (hash of settings)
// -----------------
// 55 characters used
return fmt.Sprintf("k8s-%s-%s-%s-nmv1-%s", clusterID, strings.ToLower(string(scheme)), strings.ToLower(string(protocol)), hashed)
}
return loadBalancerName
}
func makeHealthCheckName(loadBalancerName, clusterID string, shared bool) string {
if shared {
return fmt.Sprintf("k8s-%s-node", clusterID)
}
return loadBalancerName
}
func makeHealthCheckFirewallNameFromHC(healthCheckName string) string {
return healthCheckName + "-hc"
}
func makeHealthCheckFirewallName(loadBalancerName, clusterID string, shared bool) string {
if shared {
return fmt.Sprintf("k8s-%s-node-hc", clusterID)
}
return loadBalancerName + "-hc"
}
func makeBackendServiceDescription(nm types.NamespacedName, shared bool) string {
if shared {
return ""
}
return fmt.Sprintf(`{"kubernetes.io/service-name":"%s"}`, nm.String())
}
// External Load Balancer
// makeServiceDescription is used to generate descriptions for forwarding rules and addresses.
func makeServiceDescription(serviceName string) string {
return fmt.Sprintf(`{"kubernetes.io/service-name":"%s"}`, serviceName)
}
// MakeNodesHealthCheckName returns name of the health check resource used by
// the GCE load balancers (l4) for performing health checks on nodes.
func MakeNodesHealthCheckName(clusterID string) string {
return fmt.Sprintf("k8s-%v-node", clusterID)
}
func makeHealthCheckDescription(serviceName string) string {
return fmt.Sprintf(`{"kubernetes.io/service-name":"%s"}`, serviceName)
}
// MakeHealthCheckFirewallName returns the firewall name used by the GCE load
// balancers (l4) for performing health checks.
func MakeHealthCheckFirewallName(clusterID, hcName string, isNodesHealthCheck bool) string {
if isNodesHealthCheck {
return MakeNodesHealthCheckName(clusterID) + "-http-hc"
}
return "k8s-" + hcName + "-http-hc"
}
// MakeFirewallName returns the firewall name used by the GCE load
// balancers (l4) for serving traffic.
func MakeFirewallName(name string) string {
return fmt.Sprintf("k8s-fw-%s", name)
}
func makeFirewallDescription(serviceName, ipAddress string) string {
return fmt.Sprintf(`{"kubernetes.io/service-name":"%s", "kubernetes.io/service-ip":"%s"}`,
serviceName, ipAddress)
}

450
staging/src/k8s.io/legacy-cloud-providers/gce/gce_loadbalancer_test.go
View File

@@ -1,450 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
"context"
"fmt"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
v1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
func TestGetLoadBalancer(t *testing.T) {
t.Parallel()
vals := DefaultTestClusterValues()
gce, err := fakeGCECloud(vals)
require.NoError(t, err)
apiService := fakeLoadbalancerService("")
// When a loadbalancer has not been created
status, found, err := gce.GetLoadBalancer(context.Background(), vals.ClusterName, apiService)
assert.Nil(t, status)
assert.False(t, found)
assert.NoError(t, err)
nodeNames := []string{"test-node-1"}
nodes, err := createAndInsertNodes(gce, nodeNames, vals.ZoneName)
require.NoError(t, err)
expectedStatus, err := gce.EnsureLoadBalancer(context.Background(), vals.ClusterName, apiService, nodes)
require.NoError(t, err)
status, found, err = gce.GetLoadBalancer(context.Background(), vals.ClusterName, apiService)
assert.Equal(t, expectedStatus, status)
assert.True(t, found)
assert.NoError(t, err)
}
func TestEnsureLoadBalancerCreatesExternalLb(t *testing.T) {
t.Parallel()
vals := DefaultTestClusterValues()
gce, err := fakeGCECloud(vals)
require.NoError(t, err)
nodeNames := []string{"test-node-1"}
nodes, err := createAndInsertNodes(gce, nodeNames, vals.ZoneName)
require.NoError(t, err)
apiService := fakeLoadbalancerService("")
status, err := gce.EnsureLoadBalancer(context.Background(), vals.ClusterName, apiService, nodes)
assert.NoError(t, err)
assert.NotEmpty(t, status.Ingress)
assertExternalLbResources(t, gce, apiService, vals, nodeNames)
}
func TestEnsureLoadBalancerCreatesInternalLb(t *testing.T) {
t.Parallel()
vals := DefaultTestClusterValues()
gce, err := fakeGCECloud(vals)
require.NoError(t, err)
nodeNames := []string{"test-node-1"}
nodes, err := createAndInsertNodes(gce, nodeNames, vals.ZoneName)
require.NoError(t, err)
apiService := fakeLoadbalancerService(string(LBTypeInternal))
apiService, err = gce.client.CoreV1().Services(apiService.Namespace).Create(context.TODO(), apiService, metav1.CreateOptions{})
require.NoError(t, err)
status, err := gce.EnsureLoadBalancer(context.Background(), vals.ClusterName, apiService, nodes)
assert.NoError(t, err)
assert.NotEmpty(t, status.Ingress)
assertInternalLbResources(t, gce, apiService, vals, nodeNames)
}
func TestEnsureLoadBalancerDeletesExistingInternalLb(t *testing.T) {
t.Parallel()
vals := DefaultTestClusterValues()
gce, err := fakeGCECloud(vals)
require.NoError(t, err)
nodeNames := []string{"test-node-1"}
nodes, err := createAndInsertNodes(gce, nodeNames, vals.ZoneName)
require.NoError(t, err)
apiService := fakeLoadbalancerService("")
createInternalLoadBalancer(gce, apiService, nil, nodeNames, vals.ClusterName, vals.ClusterID, vals.ZoneName)
status, err := gce.EnsureLoadBalancer(context.Background(), vals.ClusterName, apiService, nodes)
assert.NoError(t, err)
assert.NotEmpty(t, status.Ingress)
assertExternalLbResources(t, gce, apiService, vals, nodeNames)
assertInternalLbResourcesDeleted(t, gce, apiService, vals, false)
}
func TestEnsureLoadBalancerDeletesExistingExternalLb(t *testing.T) {
t.Parallel()
vals := DefaultTestClusterValues()
gce, err := fakeGCECloud(vals)
require.NoError(t, err)
nodeNames := []string{"test-node-1"}
nodes, err := createAndInsertNodes(gce, nodeNames, vals.ZoneName)
require.NoError(t, err)
apiService := fakeLoadbalancerService("")
createExternalLoadBalancer(gce, apiService, nodeNames, vals.ClusterName, vals.ClusterID, vals.ZoneName)
apiService = fakeLoadbalancerService(string(LBTypeInternal))
apiService, err = gce.client.CoreV1().Services(apiService.Namespace).Create(context.TODO(), apiService, metav1.CreateOptions{})
require.NoError(t, err)
status, err := gce.EnsureLoadBalancer(context.Background(), vals.ClusterName, apiService, nodes)
assert.NoError(t, err)
assert.NotEmpty(t, status.Ingress)
assertInternalLbResources(t, gce, apiService, vals, nodeNames)
assertExternalLbResourcesDeleted(t, gce, apiService, vals, false)
}
func TestEnsureLoadBalancerDeletedDeletesExternalLb(t *testing.T) {
t.Parallel()
vals := DefaultTestClusterValues()
gce, err := fakeGCECloud(vals)
require.NoError(t, err)
nodeNames := []string{"test-node-1"}
_, err = createAndInsertNodes(gce, nodeNames, vals.ZoneName)
require.NoError(t, err)
apiService := fakeLoadbalancerService("")
createExternalLoadBalancer(gce, apiService, nodeNames, vals.ClusterName, vals.ClusterID, vals.ZoneName)
err = gce.EnsureLoadBalancerDeleted(context.Background(), vals.ClusterName, apiService)
assert.NoError(t, err)
assertExternalLbResourcesDeleted(t, gce, apiService, vals, true)
}
func TestEnsureLoadBalancerDeletedDeletesInternalLb(t *testing.T) {
t.Parallel()
vals := DefaultTestClusterValues()
gce, err := fakeGCECloud(vals)
require.NoError(t, err)
nodeNames := []string{"test-node-1"}
_, err = createAndInsertNodes(gce, nodeNames, vals.ZoneName)
require.NoError(t, err)
apiService := fakeLoadbalancerService(string(LBTypeInternal))
apiService, err = gce.client.CoreV1().Services(apiService.Namespace).Create(context.TODO(), apiService, metav1.CreateOptions{})
require.NoError(t, err)
createInternalLoadBalancer(gce, apiService, nil, nodeNames, vals.ClusterName, vals.ClusterID, vals.ZoneName)
err = gce.EnsureLoadBalancerDeleted(context.Background(), vals.ClusterName, apiService)
assert.NoError(t, err)
assertInternalLbResourcesDeleted(t, gce, apiService, vals, true)
}
func TestProjectsBasePath(t *testing.T) {
t.Parallel()
vals := DefaultTestClusterValues()
gce, err := fakeGCECloud(vals)
// Loadbalancer controller code expects basepath to contain the projects string.
expectProjectsBasePath := "https://compute.googleapis.com/compute/v1/projects/"
// See https://github.com/kubernetes/kubernetes/issues/102757, the endpoint can have mtls in some cases.
expectMtlsProjectsBasePath := "https://compute.mtls.googleapis.com/compute/v1/projects/"
require.NoError(t, err)
if gce.projectsBasePath != expectProjectsBasePath && gce.projectsBasePath != expectMtlsProjectsBasePath {
t.Errorf("Compute projectsBasePath has changed. Got %q, want %q or %q", gce.projectsBasePath, expectProjectsBasePath, expectMtlsProjectsBasePath)
}
}
func TestEnsureLoadBalancerMixedProtocols(t *testing.T) {
t.Parallel()
vals := DefaultTestClusterValues()
gce, err := fakeGCECloud(vals)
require.NoError(t, err)
nodeNames := []string{"test-node-1"}
nodes, err := createAndInsertNodes(gce, nodeNames, vals.ZoneName)
require.NoError(t, err)
apiService := fakeLoadbalancerService("")
apiService.Spec.Ports = append(apiService.Spec.Ports, v1.ServicePort{
Protocol: v1.ProtocolUDP,
Port: int32(8080),
})
apiService, err = gce.client.CoreV1().Services(apiService.Namespace).Create(context.TODO(), apiService, metav1.CreateOptions{})
require.NoError(t, err)
_, err = gce.EnsureLoadBalancer(context.Background(), vals.ClusterName, apiService, nodes)
if err == nil {
t.Errorf("Expected error ensuring loadbalancer for Service with multiple ports")
}
if err.Error() != "mixed protocol is not supported for LoadBalancer" {
t.Fatalf("unexpected error, got: %s wanted \"mixed protocol is not supported for LoadBalancer\"", err.Error())
}
apiService, err = gce.client.CoreV1().Services(apiService.Namespace).Get(context.TODO(), apiService.Name, metav1.GetOptions{})
if err != nil {
t.Fatalf("Unexpected error: %v", err)
}
if !hasLoadBalancerPortsError(apiService) {
t.Fatalf("Expected condition %v to be True, got %v", v1.LoadBalancerPortsError, apiService.Status.Conditions)
}
}
func TestUpdateLoadBalancerMixedProtocols(t *testing.T) {
t.Parallel()
vals := DefaultTestClusterValues()
gce, err := fakeGCECloud(vals)
require.NoError(t, err)
nodeNames := []string{"test-node-1"}
nodes, err := createAndInsertNodes(gce, nodeNames, vals.ZoneName)
require.NoError(t, err)
apiService := fakeLoadbalancerService("")
apiService.Spec.Ports = append(apiService.Spec.Ports, v1.ServicePort{
Protocol: v1.ProtocolUDP,
Port: int32(8080),
})
apiService, err = gce.client.CoreV1().Services(apiService.Namespace).Create(context.TODO(), apiService, metav1.CreateOptions{})
require.NoError(t, err)
// create an external loadbalancer to simulate an upgrade scenario where the loadbalancer exists
// before the new controller is running and later the Service is updated
_, err = createExternalLoadBalancer(gce, apiService, nodeNames, vals.ClusterName, vals.ClusterID, vals.ZoneName)
assert.NoError(t, err)
err = gce.UpdateLoadBalancer(context.Background(), vals.ClusterName, apiService, nodes)
if err != nil {
t.Fatalf("Unexpected error: %v", err)
}
apiService, err = gce.client.CoreV1().Services(apiService.Namespace).Get(context.TODO(), apiService.Name, metav1.GetOptions{})
if err != nil {
t.Fatalf("Unexpected error: %v", err)
}
if !hasLoadBalancerPortsError(apiService) {
t.Fatalf("Expected condition %v to be True, got %v", v1.LoadBalancerPortsError, apiService.Status.Conditions)
}
}
func TestCheckMixedProtocol(t *testing.T) {
tests := []struct {
name string
annotations map[string]string
ports []v1.ServicePort
wantErr error
}{
{
name: "TCP",
annotations: make(map[string]string),
ports: []v1.ServicePort{
{
Protocol: v1.ProtocolTCP,
Port: int32(8080),
},
},
wantErr: nil,
},
{
name: "UDP",
annotations: map[string]string{ServiceAnnotationLoadBalancerType: "nlb"},
ports: []v1.ServicePort{
{
Protocol: v1.ProtocolUDP,
Port: int32(8080),
},
},
wantErr: nil,
},
{
name: "TCP",
annotations: make(map[string]string),
ports: []v1.ServicePort{
{
Name: "port80",
Protocol: v1.ProtocolTCP,
Port: int32(80),
},
{
Name: "port8080",
Protocol: v1.ProtocolTCP,
Port: int32(8080),
},
},
wantErr: nil,
},
{
name: "UDP",
annotations: map[string]string{ServiceAnnotationLoadBalancerType: "nlb"},
ports: []v1.ServicePort{
{
Name: "port80",
Protocol: v1.ProtocolUDP,
Port: int32(80),
},
{
Name: "port8080",
Protocol: v1.ProtocolUDP,
Port: int32(8080),
},
},
wantErr: nil,
},
{
name: "TCP and UDP",
annotations: map[string]string{ServiceAnnotationLoadBalancerType: "nlb"},
ports: []v1.ServicePort{
{
Protocol: v1.ProtocolUDP,
Port: int32(53),
},
{
Protocol: v1.ProtocolTCP,
Port: int32(53),
},
},
wantErr: fmt.Errorf("mixed protocol is not supported for LoadBalancer"),
},
}
for _, test := range tests {
tt := test
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
err := checkMixedProtocol(tt.ports)
if tt.wantErr != nil {
assert.EqualError(t, err, tt.wantErr.Error())
} else {
assert.Equal(t, err, nil)
}
})
}
}
func Test_hasLoadBalancerPortsError(t *testing.T) {
tests := []struct {
name string
service *v1.Service
want bool
}{
{
name: "no status",
service: &v1.Service{},
},
{
name: "condition set to true",
service: &v1.Service{
ObjectMeta: metav1.ObjectMeta{Name: "service1"},
Spec: v1.ServiceSpec{
ClusterIPs: []string{"1.2.3.4"},
Type: "LoadBalancer",
Ports: []v1.ServicePort{{Port: 80, Protocol: "TCP"}},
},
Status: v1.ServiceStatus{
LoadBalancer: v1.LoadBalancerStatus{
Ingress: []v1.LoadBalancerIngress{{IP: "2.3.4.5"}, {IP: "3.4.5.6"}}},
Conditions: []metav1.Condition{
{
Type: v1.LoadBalancerPortsError,
Status: metav1.ConditionTrue,
},
},
},
},
want: true,
},
{
name: "condition set false",
service: &v1.Service{
ObjectMeta: metav1.ObjectMeta{Name: "service1"},
Spec: v1.ServiceSpec{
ClusterIPs: []string{"1.2.3.4"},
Type: "LoadBalancer",
Ports: []v1.ServicePort{{Port: 80, Protocol: "TCP"}},
},
Status: v1.ServiceStatus{
LoadBalancer: v1.LoadBalancerStatus{
Ingress: []v1.LoadBalancerIngress{{IP: "2.3.4.5"}, {IP: "3.4.5.6"}}},
Conditions: []metav1.Condition{
{
Type: v1.LoadBalancerPortsError,
Status: metav1.ConditionFalse,
},
},
},
},
},
{
name: "multiple conditions unrelated",
service: &v1.Service{
ObjectMeta: metav1.ObjectMeta{Name: "service1"},
Spec: v1.ServiceSpec{
ClusterIPs: []string{"1.2.3.4"},
Type: "LoadBalancer",
Ports: []v1.ServicePort{{Port: 80, Protocol: "TCP"}},
},
Status: v1.ServiceStatus{
LoadBalancer: v1.LoadBalancerStatus{
Ingress: []v1.LoadBalancerIngress{{IP: "2.3.4.5"}, {IP: "3.4.5.6"}}},
Conditions: []metav1.Condition{
{
Type: "condition1",
Status: metav1.ConditionFalse,
},
{
Type: "condition2",
Status: metav1.ConditionTrue,
},
},
},
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := hasLoadBalancerPortsError(tt.service); got != tt.want {
t.Errorf("hasLoadBalancerPortsError() = %v, want %v", got, tt.want)
}
})
}
}

316
staging/src/k8s.io/legacy-cloud-providers/gce/gce_loadbalancer_utils_test.go
View File

@@ -1,316 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// This file contains shared functions and variables to set up for tests for
// ExternalLoadBalancer and InternalLoadBalancers. It currently cannot live in a
// separate package from GCE because then it would cause a circular import.
package gce
import (
"context"
"fmt"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
compute "google.golang.org/api/compute/v1"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud"
v1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/tools/record"
servicehelpers "k8s.io/cloud-provider/service/helpers"
)
// TODO(yankaiz): Create shared error types for both test/non-test codes.
const (
eventReasonManualChange = "LoadBalancerManualChange"
errPrefixGetTargetPool = "error getting load balancer's target pool:"
wrongTier = "SupremeLuxury"
errStrUnsupportedTier = "unsupported network tier: \"" + wrongTier + "\""
fakeSvcName = "fakesvc"
)
func fakeLoadbalancerService(lbType string) *v1.Service {
return fakeLoadbalancerServiceHelper(lbType, ServiceAnnotationLoadBalancerType)
}
func fakeLoadBalancerServiceDeprecatedAnnotation(lbType string) *v1.Service {
return fakeLoadbalancerServiceHelper(lbType, deprecatedServiceAnnotationLoadBalancerType)
}
func fakeLoadbalancerServiceHelper(lbType string, annotationKey string) *v1.Service {
return &v1.Service{
ObjectMeta: metav1.ObjectMeta{
Name: fakeSvcName,
Annotations: map[string]string{annotationKey: lbType},
},
Spec: v1.ServiceSpec{
SessionAffinity: v1.ServiceAffinityClientIP,
Type: v1.ServiceTypeLoadBalancer,
Ports: []v1.ServicePort{{Protocol: v1.ProtocolTCP, Port: int32(123)}},
},
}
}
var (
FirewallChangeMsg = fmt.Sprintf("%s %s %s", v1.EventTypeNormal, eventReasonManualChange, eventMsgFirewallChange)
)
func createAndInsertNodes(gce *Cloud, nodeNames []string, zoneName string) ([]*v1.Node, error) {
nodes := []*v1.Node{}
for _, name := range nodeNames {
// Inserting the same node name twice causes an error - here we check if
// the instance exists already before insertion.
// TestUpdateExternalLoadBalancer inserts a new node, and relies on an older
// node to already have been inserted.
instance, _ := gce.getInstanceByName(name)
if instance == nil {
err := gce.InsertInstance(
gce.ProjectID(),
zoneName,
&compute.Instance{
Name: name,
Tags: &compute.Tags{
Items: []string{name},
},
// add Instance.Zone, otherwise InstanceID() won't return a right instanceID.
Zone: zoneName,
},
)
if err != nil {
return nodes, err
}
}
nodes = append(
nodes,
&v1.Node{
ObjectMeta: metav1.ObjectMeta{
Name: name,
Labels: map[string]string{
v1.LabelHostname: name,
v1.LabelFailureDomainBetaZone: zoneName,
},
},
},
)
}
return nodes, nil
}
func assertExternalLbResources(t *testing.T, gce *Cloud, apiService *v1.Service, vals TestClusterValues, nodeNames []string) {
lbName := gce.GetLoadBalancerName(context.TODO(), "", apiService)
hcName := MakeNodesHealthCheckName(vals.ClusterID)
// Check that Firewalls are created for the LoadBalancer and the HealthCheck
fwNames := []string{
MakeFirewallName(lbName), // Firewalls for external LBs are prefixed with k8s-fw-
MakeHealthCheckFirewallName(vals.ClusterID, hcName, true),
}
for _, fwName := range fwNames {
firewall, err := gce.GetFirewall(fwName)
require.NoError(t, err)
assert.Equal(t, nodeNames, firewall.TargetTags)
assert.NotEmpty(t, firewall.SourceRanges)
}
// Check that TargetPool is Created
pool, err := gce.GetTargetPool(lbName, gce.region)
require.NoError(t, err)
assert.Equal(t, lbName, pool.Name)
assert.NotEmpty(t, pool.HealthChecks)
assert.Equal(t, 1, len(pool.Instances))
// Check that HealthCheck is created
healthcheck, err := gce.GetHTTPHealthCheck(hcName)
require.NoError(t, err)
assert.Equal(t, hcName, healthcheck.Name)
// Check that ForwardingRule is created
fwdRule, err := gce.GetRegionForwardingRule(lbName, gce.region)
require.NoError(t, err)
assert.Equal(t, lbName, fwdRule.Name)
assert.Equal(t, "TCP", fwdRule.IPProtocol)
assert.Equal(t, "123-123", fwdRule.PortRange)
}
func assertExternalLbResourcesDeleted(t *testing.T, gce *Cloud, apiService *v1.Service, vals TestClusterValues, firewallsDeleted bool) {
lbName := gce.GetLoadBalancerName(context.TODO(), "", apiService)
hcName := MakeNodesHealthCheckName(vals.ClusterID)
if firewallsDeleted {
// Check that Firewalls are deleted for the LoadBalancer and the HealthCheck
fwNames := []string{
MakeFirewallName(lbName),
MakeHealthCheckFirewallName(vals.ClusterID, hcName, true),
}
for _, fwName := range fwNames {
firewall, err := gce.GetFirewall(fwName)
require.Error(t, err)
assert.Nil(t, firewall)
}
// Check forwarding rule is deleted
fwdRule, err := gce.GetRegionForwardingRule(lbName, gce.region)
require.Error(t, err)
assert.Nil(t, fwdRule)
}
// Check that TargetPool is deleted
pool, err := gce.GetTargetPool(lbName, gce.region)
require.Error(t, err)
assert.Nil(t, pool)
// Check that HealthCheck is deleted
healthcheck, err := gce.GetHTTPHealthCheck(hcName)
require.Error(t, err)
assert.Nil(t, healthcheck)
}
func assertInternalLbResources(t *testing.T, gce *Cloud, apiService *v1.Service, vals TestClusterValues, nodeNames []string) {
lbName := gce.GetLoadBalancerName(context.TODO(), "", apiService)
// Check that Instance Group is created
igName := makeInstanceGroupName(vals.ClusterID)
ig, err := gce.GetInstanceGroup(igName, vals.ZoneName)
assert.NoError(t, err)
assert.Equal(t, igName, ig.Name)
// Check that Firewalls are created for the LoadBalancer and the HealthCheck
fwNames := []string{
MakeFirewallName(lbName),
makeHealthCheckFirewallName(lbName, vals.ClusterID, true),
}
for _, fwName := range fwNames {
firewall, err := gce.GetFirewall(fwName)
require.NoError(t, err)
assert.Equal(t, nodeNames, firewall.TargetTags)
assert.NotEmpty(t, firewall.SourceRanges)
}
// Check that HealthCheck is created
sharedHealthCheck := !servicehelpers.RequestsOnlyLocalTraffic(apiService)
hcName := makeHealthCheckName(lbName, vals.ClusterID, sharedHealthCheck)
healthcheck, err := gce.GetHealthCheck(hcName)
require.NoError(t, err)
assert.Equal(t, hcName, healthcheck.Name)
// Check that BackendService exists
sharedBackend := shareBackendService(apiService)
backendServiceName := makeBackendServiceName(lbName, vals.ClusterID, sharedBackend, cloud.SchemeInternal, "TCP", apiService.Spec.SessionAffinity)
backendServiceLink := gce.getBackendServiceLink(backendServiceName)
bs, err := gce.GetRegionBackendService(backendServiceName, gce.region)
require.NoError(t, err)
assert.Equal(t, "TCP", bs.Protocol)
assert.Equal(
t,
[]string{healthcheck.SelfLink},
bs.HealthChecks,
)
// Check that ForwardingRule is created
fwdRule, err := gce.GetRegionForwardingRule(lbName, gce.region)
require.NoError(t, err)
assert.Equal(t, lbName, fwdRule.Name)
assert.Equal(t, "TCP", fwdRule.IPProtocol)
assert.Equal(t, backendServiceLink, fwdRule.BackendService)
// if no Subnetwork specified, defaults to the GCE NetworkURL
assert.Equal(t, gce.NetworkURL(), fwdRule.Subnetwork)
// Check that the IP address has been released. IP is only reserved until ensure function exits.
ip, err := gce.GetRegionAddress(lbName, gce.region)
require.Error(t, err)
assert.Nil(t, ip)
}
func assertInternalLbResourcesDeleted(t *testing.T, gce *Cloud, apiService *v1.Service, vals TestClusterValues, firewallsDeleted bool) {
lbName := gce.GetLoadBalancerName(context.TODO(), "", apiService)
sharedHealthCheck := !servicehelpers.RequestsOnlyLocalTraffic(apiService)
hcName := makeHealthCheckName(lbName, vals.ClusterID, sharedHealthCheck)
// ensureExternalLoadBalancer and ensureInternalLoadBalancer both create
// Firewalls with the same name.
if firewallsDeleted {
// Check that Firewalls are deleted for the LoadBalancer and the HealthCheck
fwNames := []string{
MakeFirewallName(lbName),
MakeHealthCheckFirewallName(vals.ClusterID, hcName, true),
}
for _, fwName := range fwNames {
firewall, err := gce.GetFirewall(fwName)
require.Error(t, err)
assert.Nil(t, firewall)
}
// Check forwarding rule is deleted
fwdRule, err := gce.GetRegionForwardingRule(lbName, gce.region)
require.Error(t, err)
assert.Nil(t, fwdRule)
}
// Check that Instance Group is deleted
igName := makeInstanceGroupName(vals.ClusterID)
ig, err := gce.GetInstanceGroup(igName, vals.ZoneName)
assert.Error(t, err)
assert.Nil(t, ig)
// Check that HealthCheck is deleted
healthcheck, err := gce.GetHealthCheck(hcName)
require.Error(t, err)
assert.Nil(t, healthcheck)
// Check that the IP address has been released
ip, err := gce.GetRegionAddress(lbName, gce.region)
require.Error(t, err)
assert.Nil(t, ip)
}
func checkEvent(t *testing.T, recorder *record.FakeRecorder, expected string, shouldMatch bool) bool {
select {
case received := <-recorder.Events:
if strings.HasPrefix(received, expected) != shouldMatch {
t.Errorf(received)
if shouldMatch {
t.Errorf("Should receive message \"%v\" but got \"%v\".", expected, received)
} else {
t.Errorf("Unexpected event \"%v\".", received)
}
}
return false
case <-time.After(2 * time.Second):
if shouldMatch {
t.Errorf("Should receive message \"%v\" but got timed out.", expected)
}
return true
}
}

138
staging/src/k8s.io/legacy-cloud-providers/gce/gce_networkendpointgroup.go
View File

@@ -1,138 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
"fmt"
"strings"
computebeta "google.golang.org/api/compute/v0.beta"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud/filter"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud/meta"
)
func newNetworkEndpointGroupMetricContext(request string, zone string) *metricContext {
return newGenericMetricContext("networkendpointgroup_", request, unusedMetricLabel, zone, computeBetaVersion)
}
// GetNetworkEndpointGroup returns the collection of network endpoints for the name in zone
func (g *Cloud) GetNetworkEndpointGroup(name string, zone string) (*computebeta.NetworkEndpointGroup, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newNetworkEndpointGroupMetricContext("get", zone)
v, err := g.c.BetaNetworkEndpointGroups().Get(ctx, meta.ZonalKey(name, zone))
return v, mc.Observe(err)
}
// ListNetworkEndpointGroup returns the collection of network endpoints for the zone
func (g *Cloud) ListNetworkEndpointGroup(zone string) ([]*computebeta.NetworkEndpointGroup, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newNetworkEndpointGroupMetricContext("list", zone)
negs, err := g.c.BetaNetworkEndpointGroups().List(ctx, zone, filter.None)
return negs, mc.Observe(err)
}
// AggregatedListNetworkEndpointGroup returns a map of zone -> endpoint group.
func (g *Cloud) AggregatedListNetworkEndpointGroup() (map[string][]*computebeta.NetworkEndpointGroup, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newNetworkEndpointGroupMetricContext("aggregated_list", "")
// TODO: filter for the region the cluster is in.
all, err := g.c.BetaNetworkEndpointGroups().AggregatedList(ctx, filter.None)
if err != nil {
return nil, mc.Observe(err)
}
ret := map[string][]*computebeta.NetworkEndpointGroup{}
for key, byZone := range all {
// key is "zones/<zone name>"
parts := strings.Split(key, "/")
if len(parts) != 2 {
return nil, mc.Observe(fmt.Errorf("invalid key for AggregatedListNetworkEndpointGroup: %q", key))
}
zone := parts[1]
ret[zone] = append(ret[zone], byZone...)
}
return ret, mc.Observe(nil)
}
// CreateNetworkEndpointGroup creates an endpoint group in the zone
func (g *Cloud) CreateNetworkEndpointGroup(neg *computebeta.NetworkEndpointGroup, zone string) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newNetworkEndpointGroupMetricContext("create", zone)
return mc.Observe(g.c.BetaNetworkEndpointGroups().Insert(ctx, meta.ZonalKey(neg.Name, zone), neg))
}
// DeleteNetworkEndpointGroup deletes the name endpoint group from the zone
func (g *Cloud) DeleteNetworkEndpointGroup(name string, zone string) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newNetworkEndpointGroupMetricContext("delete", zone)
return mc.Observe(g.c.BetaNetworkEndpointGroups().Delete(ctx, meta.ZonalKey(name, zone)))
}
// AttachNetworkEndpoints associates the referenced endpoints with the named endpoint group in the zone
func (g *Cloud) AttachNetworkEndpoints(name, zone string, endpoints []*computebeta.NetworkEndpoint) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newNetworkEndpointGroupMetricContext("attach", zone)
req := &computebeta.NetworkEndpointGroupsAttachEndpointsRequest{
NetworkEndpoints: endpoints,
}
return mc.Observe(g.c.BetaNetworkEndpointGroups().AttachNetworkEndpoints(ctx, meta.ZonalKey(name, zone), req))
}
// DetachNetworkEndpoints breaks the association between the referenced endpoints and the named endpoint group in the zone
func (g *Cloud) DetachNetworkEndpoints(name, zone string, endpoints []*computebeta.NetworkEndpoint) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newNetworkEndpointGroupMetricContext("detach", zone)
req := &computebeta.NetworkEndpointGroupsDetachEndpointsRequest{
NetworkEndpoints: endpoints,
}
return mc.Observe(g.c.BetaNetworkEndpointGroups().DetachNetworkEndpoints(ctx, meta.ZonalKey(name, zone), req))
}
// ListNetworkEndpoints returns all the endpoints associated with the endpoint group in zone and optionally their status.
func (g *Cloud) ListNetworkEndpoints(name, zone string, showHealthStatus bool) ([]*computebeta.NetworkEndpointWithHealthStatus, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newNetworkEndpointGroupMetricContext("list_networkendpoints", zone)
healthStatus := "SKIP"
if showHealthStatus {
healthStatus = "SHOW"
}
req := &computebeta.NetworkEndpointGroupsListEndpointsRequest{
HealthStatus: healthStatus,
}
l, err := g.c.BetaNetworkEndpointGroups().ListNetworkEndpoints(ctx, meta.ZonalKey(name, zone), req, filter.None)
return l, mc.Observe(err)
}

110
staging/src/k8s.io/legacy-cloud-providers/gce/gce_routes.go
View File

@@ -1,110 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
"context"
"fmt"
"net/http"
"path"
"time"
"google.golang.org/api/compute/v1"
"k8s.io/apimachinery/pkg/types"
"k8s.io/klog/v2"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud/filter"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud/meta"
cloudprovider "k8s.io/cloud-provider"
)
func newRoutesMetricContext(request string) *metricContext {
return newGenericMetricContext("routes", request, unusedMetricLabel, unusedMetricLabel, computeV1Version)
}
// ListRoutes in the cloud environment.
func (g *Cloud) ListRoutes(ctx context.Context, clusterName string) ([]*cloudprovider.Route, error) {
timeoutCtx, cancel := context.WithTimeout(ctx, 1*time.Hour)
defer cancel()
mc := newRoutesMetricContext("list")
prefix := truncateClusterName(clusterName)
f := filter.Regexp("name", prefix+"-.*").AndRegexp("network", g.NetworkURL()).AndRegexp("description", k8sNodeRouteTag)
routes, err := g.c.Routes().List(timeoutCtx, f)
if err != nil {
return nil, mc.Observe(err)
}
var croutes []*cloudprovider.Route
for _, r := range routes {
target := path.Base(r.NextHopInstance)
// TODO: Should we lastComponent(target) this?
targetNodeName := types.NodeName(target) // NodeName == Instance Name on GCE
croutes = append(croutes, &cloudprovider.Route{
Name: r.Name,
TargetNode: targetNodeName,
DestinationCIDR: r.DestRange,
})
}
return croutes, mc.Observe(nil)
}
// CreateRoute in the cloud environment.
func (g *Cloud) CreateRoute(ctx context.Context, clusterName string, nameHint string, route *cloudprovider.Route) error {
timeoutCtx, cancel := context.WithTimeout(ctx, 1*time.Hour)
defer cancel()
mc := newRoutesMetricContext("create")
targetInstance, err := g.getInstanceByName(mapNodeNameToInstanceName(route.TargetNode))
if err != nil {
return mc.Observe(err)
}
cr := &compute.Route{
// TODO(thockin): generate a unique name for node + route cidr. Don't depend on name hints.
Name: truncateClusterName(clusterName) + "-" + nameHint,
DestRange: route.DestinationCIDR,
NextHopInstance: fmt.Sprintf("zones/%s/instances/%s", targetInstance.Zone, targetInstance.Name),
Network: g.NetworkURL(),
Priority: 1000,
Description: k8sNodeRouteTag,
}
err = g.c.Routes().Insert(timeoutCtx, meta.GlobalKey(cr.Name), cr)
if isHTTPErrorCode(err, http.StatusConflict) {
klog.Infof("Route %q already exists.", cr.Name)
err = nil
}
return mc.Observe(err)
}
// DeleteRoute from the cloud environment.
func (g *Cloud) DeleteRoute(ctx context.Context, clusterName string, route *cloudprovider.Route) error {
timeoutCtx, cancel := context.WithTimeout(ctx, 1*time.Hour)
defer cancel()
mc := newRoutesMetricContext("delete")
return mc.Observe(g.c.Routes().Delete(timeoutCtx, meta.GlobalKey(route.Name)))
}
func truncateClusterName(clusterName string) string {
if len(clusterName) > 26 {
return clusterName[:26]
}
return clusterName
}

119
staging/src/k8s.io/legacy-cloud-providers/gce/gce_securitypolicy.go
View File

@@ -1,119 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2018 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
computebeta "google.golang.org/api/compute/v0.beta"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud/filter"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud/meta"
)
func newSecurityPolicyMetricContextWithVersion(request, version string) *metricContext {
return newGenericMetricContext("securitypolicy", request, "", unusedMetricLabel, version)
}
// GetBetaSecurityPolicy retrieves a security policy.
func (g *Cloud) GetBetaSecurityPolicy(name string) (*computebeta.SecurityPolicy, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newSecurityPolicyMetricContextWithVersion("get", computeBetaVersion)
v, err := g.c.BetaSecurityPolicies().Get(ctx, meta.GlobalKey(name))
return v, mc.Observe(err)
}
// ListBetaSecurityPolicy lists all security policies in the project.
func (g *Cloud) ListBetaSecurityPolicy() ([]*computebeta.SecurityPolicy, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newSecurityPolicyMetricContextWithVersion("list", computeBetaVersion)
v, err := g.c.BetaSecurityPolicies().List(ctx, filter.None)
return v, mc.Observe(err)
}
// CreateBetaSecurityPolicy creates the given security policy.
func (g *Cloud) CreateBetaSecurityPolicy(sp *computebeta.SecurityPolicy) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newSecurityPolicyMetricContextWithVersion("create", computeBetaVersion)
return mc.Observe(g.c.BetaSecurityPolicies().Insert(ctx, meta.GlobalKey(sp.Name), sp))
}
// DeleteBetaSecurityPolicy deletes the given security policy.
func (g *Cloud) DeleteBetaSecurityPolicy(name string) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newSecurityPolicyMetricContextWithVersion("delete", computeBetaVersion)
return mc.Observe(g.c.BetaSecurityPolicies().Delete(ctx, meta.GlobalKey(name)))
}
// PatchBetaSecurityPolicy applies the given security policy as a
// patch to an existing security policy.
func (g *Cloud) PatchBetaSecurityPolicy(sp *computebeta.SecurityPolicy) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newSecurityPolicyMetricContextWithVersion("patch", computeBetaVersion)
return mc.Observe(g.c.BetaSecurityPolicies().Patch(ctx, meta.GlobalKey(sp.Name), sp))
}
// GetRuleForBetaSecurityPolicy gets rule from a security policy.
func (g *Cloud) GetRuleForBetaSecurityPolicy(name string) (*computebeta.SecurityPolicyRule, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newSecurityPolicyMetricContextWithVersion("get_rule", computeBetaVersion)
v, err := g.c.BetaSecurityPolicies().GetRule(ctx, meta.GlobalKey(name))
return v, mc.Observe(err)
}
// AddRuletoBetaSecurityPolicy adds the given security policy rule to
// a security policy.
func (g *Cloud) AddRuletoBetaSecurityPolicy(name string, spr *computebeta.SecurityPolicyRule) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newSecurityPolicyMetricContextWithVersion("add_rule", computeBetaVersion)
return mc.Observe(g.c.BetaSecurityPolicies().AddRule(ctx, meta.GlobalKey(name), spr))
}
// PatchRuleForBetaSecurityPolicy patches the given security policy
// rule to a security policy.
func (g *Cloud) PatchRuleForBetaSecurityPolicy(name string, spr *computebeta.SecurityPolicyRule) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newSecurityPolicyMetricContextWithVersion("patch_rule", computeBetaVersion)
return mc.Observe(g.c.BetaSecurityPolicies().PatchRule(ctx, meta.GlobalKey(name), spr))
}
// RemoveRuleFromBetaSecurityPolicy removes rule from a security policy.
func (g *Cloud) RemoveRuleFromBetaSecurityPolicy(name string) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newSecurityPolicyMetricContextWithVersion("remove_rule", computeBetaVersion)
return mc.Observe(g.c.BetaSecurityPolicies().RemoveRule(ctx, meta.GlobalKey(name)))
}

83
staging/src/k8s.io/legacy-cloud-providers/gce/gce_targetpool.go
View File

@@ -1,83 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
compute "google.golang.org/api/compute/v1"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud/meta"
)
func newTargetPoolMetricContext(request, region string) *metricContext {
return newGenericMetricContext("targetpool", request, region, unusedMetricLabel, computeV1Version)
}
// GetTargetPool returns the TargetPool by name.
func (g *Cloud) GetTargetPool(name, region string) (*compute.TargetPool, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newTargetPoolMetricContext("get", region)
v, err := g.c.TargetPools().Get(ctx, meta.RegionalKey(name, region))
return v, mc.Observe(err)
}
// CreateTargetPool creates the passed TargetPool
func (g *Cloud) CreateTargetPool(tp *compute.TargetPool, region string) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newTargetPoolMetricContext("create", region)
return mc.Observe(g.c.TargetPools().Insert(ctx, meta.RegionalKey(tp.Name, region), tp))
}
// DeleteTargetPool deletes TargetPool by name.
func (g *Cloud) DeleteTargetPool(name, region string) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newTargetPoolMetricContext("delete", region)
return mc.Observe(g.c.TargetPools().Delete(ctx, meta.RegionalKey(name, region)))
}
// AddInstancesToTargetPool adds instances by link to the TargetPool
func (g *Cloud) AddInstancesToTargetPool(name, region string, instanceRefs []*compute.InstanceReference) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
req := &compute.TargetPoolsAddInstanceRequest{
Instances: instanceRefs,
}
mc := newTargetPoolMetricContext("add_instances", region)
return mc.Observe(g.c.TargetPools().AddInstance(ctx, meta.RegionalKey(name, region), req))
}
// RemoveInstancesFromTargetPool removes instances by link to the TargetPool
func (g *Cloud) RemoveInstancesFromTargetPool(name, region string, instanceRefs []*compute.InstanceReference) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
req := &compute.TargetPoolsRemoveInstanceRequest{
Instances: instanceRefs,
}
mc := newTargetPoolMetricContext("remove_instances", region)
return mc.Observe(g.c.TargetPools().RemoveInstance(ctx, meta.RegionalKey(name, region), req))
}

142
staging/src/k8s.io/legacy-cloud-providers/gce/gce_targetproxy.go
View File

@@ -1,142 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
compute "google.golang.org/api/compute/v1"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud/filter"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud/meta"
)
func newTargetProxyMetricContext(request string) *metricContext {
return newGenericMetricContext("targetproxy", request, unusedMetricLabel, unusedMetricLabel, computeV1Version)
}
// GetTargetHTTPProxy returns the UrlMap by name.
func (g *Cloud) GetTargetHTTPProxy(name string) (*compute.TargetHttpProxy, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newTargetProxyMetricContext("get")
v, err := g.c.TargetHttpProxies().Get(ctx, meta.GlobalKey(name))
return v, mc.Observe(err)
}
// CreateTargetHTTPProxy creates a TargetHttpProxy
func (g *Cloud) CreateTargetHTTPProxy(proxy *compute.TargetHttpProxy) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newTargetProxyMetricContext("create")
return mc.Observe(g.c.TargetHttpProxies().Insert(ctx, meta.GlobalKey(proxy.Name), proxy))
}
// SetURLMapForTargetHTTPProxy sets the given UrlMap for the given TargetHttpProxy.
func (g *Cloud) SetURLMapForTargetHTTPProxy(proxy *compute.TargetHttpProxy, urlMapLink string) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
ref := &compute.UrlMapReference{UrlMap: urlMapLink}
mc := newTargetProxyMetricContext("set_url_map")
return mc.Observe(g.c.TargetHttpProxies().SetUrlMap(ctx, meta.GlobalKey(proxy.Name), ref))
}
// DeleteTargetHTTPProxy deletes the TargetHttpProxy by name.
func (g *Cloud) DeleteTargetHTTPProxy(name string) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newTargetProxyMetricContext("delete")
return mc.Observe(g.c.TargetHttpProxies().Delete(ctx, meta.GlobalKey(name)))
}
// ListTargetHTTPProxies lists all TargetHttpProxies in the project.
func (g *Cloud) ListTargetHTTPProxies() ([]*compute.TargetHttpProxy, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newTargetProxyMetricContext("list")
v, err := g.c.TargetHttpProxies().List(ctx, filter.None)
return v, mc.Observe(err)
}
// TargetHttpsProxy management
// GetTargetHTTPSProxy returns the UrlMap by name.
func (g *Cloud) GetTargetHTTPSProxy(name string) (*compute.TargetHttpsProxy, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newTargetProxyMetricContext("get")
v, err := g.c.TargetHttpsProxies().Get(ctx, meta.GlobalKey(name))
return v, mc.Observe(err)
}
// CreateTargetHTTPSProxy creates a TargetHttpsProxy
func (g *Cloud) CreateTargetHTTPSProxy(proxy *compute.TargetHttpsProxy) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newTargetProxyMetricContext("create")
return mc.Observe(g.c.TargetHttpsProxies().Insert(ctx, meta.GlobalKey(proxy.Name), proxy))
}
// SetURLMapForTargetHTTPSProxy sets the given UrlMap for the given TargetHttpsProxy.
func (g *Cloud) SetURLMapForTargetHTTPSProxy(proxy *compute.TargetHttpsProxy, urlMapLink string) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newTargetProxyMetricContext("set_url_map")
ref := &compute.UrlMapReference{UrlMap: urlMapLink}
return mc.Observe(g.c.TargetHttpsProxies().SetUrlMap(ctx, meta.GlobalKey(proxy.Name), ref))
}
// SetSslCertificateForTargetHTTPSProxy sets the given SslCertificate for the given TargetHttpsProxy.
func (g *Cloud) SetSslCertificateForTargetHTTPSProxy(proxy *compute.TargetHttpsProxy, sslCertURLs []string) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newTargetProxyMetricContext("set_ssl_cert")
req := &compute.TargetHttpsProxiesSetSslCertificatesRequest{
SslCertificates: sslCertURLs,
}
return mc.Observe(g.c.TargetHttpsProxies().SetSslCertificates(ctx, meta.GlobalKey(proxy.Name), req))
}
// DeleteTargetHTTPSProxy deletes the TargetHttpsProxy by name.
func (g *Cloud) DeleteTargetHTTPSProxy(name string) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newTargetProxyMetricContext("delete")
return mc.Observe(g.c.TargetHttpsProxies().Delete(ctx, meta.GlobalKey(name)))
}
// ListTargetHTTPSProxies lists all TargetHttpsProxies in the project.
func (g *Cloud) ListTargetHTTPSProxies() ([]*compute.TargetHttpsProxy, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newTargetProxyMetricContext("list")
v, err := g.c.TargetHttpsProxies().List(ctx, filter.None)
return v, mc.Observe(err)
}

640
staging/src/k8s.io/legacy-cloud-providers/gce/gce_test.go
View File

@@ -1,640 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2014 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
"context"
"reflect"
"strings"
"testing"
"golang.org/x/oauth2/google"
cloudprovider "k8s.io/cloud-provider"
)
func TestReadConfigFile(t *testing.T) {
const s = `[Global]
token-url = my-token-url
token-body = my-token-body
project-id = my-project
network-project-id = my-network-project
network-name = my-network
subnetwork-name = my-subnetwork
secondary-range-name = my-secondary-range
node-tags = my-node-tag1
node-instance-prefix = my-prefix
multizone = true
regional = true
`
reader := strings.NewReader(s)
config, err := readConfig(reader)
if err != nil {
t.Fatalf("Unexpected config parsing error %v", err)
}
expected := &ConfigFile{Global: ConfigGlobal{
TokenURL: "my-token-url",
TokenBody: "my-token-body",
ProjectID: "my-project",
NetworkProjectID: "my-network-project",
NetworkName: "my-network",
SubnetworkName: "my-subnetwork",
SecondaryRangeName: "my-secondary-range",
NodeTags: []string{"my-node-tag1"},
NodeInstancePrefix: "my-prefix",
Multizone: true,
Regional: true,
}}
if !reflect.DeepEqual(expected, config) {
t.Fatalf("Expected config file values to be read into ConfigFile struct. \nExpected:\n%+v\nActual:\n%+v", expected, config)
}
}
func TestExtraKeyInConfig(t *testing.T) {
const s = `[Global]
project-id = my-project
unknown-key = abc
network-name = my-network
`
reader := strings.NewReader(s)
config, err := readConfig(reader)
if err != nil {
t.Fatalf("Unexpected config parsing error %v", err)
}
if config.Global.ProjectID != "my-project" || config.Global.NetworkName != "my-network" {
t.Fatalf("Expected config values to continue to be read despite extra key-value pair.")
}
}
func TestGetRegion(t *testing.T) {
zoneName := "us-central1-b"
regionName, err := GetGCERegion(zoneName)
if err != nil {
t.Fatalf("unexpected error from GetGCERegion: %v", err)
}
if regionName != "us-central1" {
t.Errorf("Unexpected region from GetGCERegion: %s", regionName)
}
gce := &Cloud{
localZone: zoneName,
region: regionName,
}
zones, ok := gce.Zones()
if !ok {
t.Fatalf("Unexpected missing zones impl")
}
zone, err := zones.GetZone(context.TODO())
if err != nil {
t.Fatalf("unexpected error %v", err)
}
if zone.Region != "us-central1" {
t.Errorf("Unexpected region: %s", zone.Region)
}
}
func TestComparingHostURLs(t *testing.T) {
tests := []struct {
host1 string
zone string
name string
expectEqual bool
}{
{
host1: "https://www.googleapis.com/compute/v1/projects/1234567/zones/us-central1-f/instances/kubernetes-node-fhx1",
zone: "us-central1-f",
name: "kubernetes-node-fhx1",
expectEqual: true,
},
{
host1: "https://www.googleapis.com/compute/v1/projects/cool-project/zones/us-central1-f/instances/kubernetes-node-fhx1",
zone: "us-central1-f",
name: "kubernetes-node-fhx1",
expectEqual: true,
},
{
host1: "https://www.googleapis.com/compute/v23/projects/1234567/zones/us-central1-f/instances/kubernetes-node-fhx1",
zone: "us-central1-f",
name: "kubernetes-node-fhx1",
expectEqual: true,
},
{
host1: "https://www.googleapis.com/compute/v24/projects/1234567/regions/us-central1/zones/us-central1-f/instances/kubernetes-node-fhx1",
zone: "us-central1-f",
name: "kubernetes-node-fhx1",
expectEqual: true,
},
{
host1: "https://www.googleapis.com/compute/v1/projects/1234567/zones/us-central1-f/instances/kubernetes-node-fhx1",
zone: "us-central1-c",
name: "kubernetes-node-fhx1",
expectEqual: false,
},
{
host1: "https://www.googleapis.com/compute/v1/projects/1234567/zones/us-central1-f/instances/kubernetes-node-fhx",
zone: "us-central1-f",
name: "kubernetes-node-fhx1",
expectEqual: false,
},
{
host1: "https://www.googleapis.com/compute/v1/projects/1234567/zones/us-central1-f/instances/kubernetes-node-fhx1",
zone: "us-central1-f",
name: "kubernetes-node-fhx",
expectEqual: false,
},
}
for _, test := range tests {
link1 := hostURLToComparablePath(test.host1)
testInstance := &gceInstance{
Name: canonicalizeInstanceName(test.name),
Zone: test.zone,
}
link2 := testInstance.makeComparableHostPath()
if test.expectEqual && link1 != link2 {
t.Errorf("expected link1 and link2 to be equal, got %s and %s", link1, link2)
} else if !test.expectEqual && link1 == link2 {
t.Errorf("expected link1 and link2 not to be equal, got %s and %s", link1, link2)
}
}
}
func TestSplitProviderID(t *testing.T) {
providers := []struct {
providerID string
project string
zone string
instance string
fail bool
}{
{
providerID: ProviderName + "://project-example-164317/us-central1-f/kubernetes-node-fhx1",
project: "project-example-164317",
zone: "us-central1-f",
instance: "kubernetes-node-fhx1",
fail: false,
},
{
providerID: ProviderName + "://project-example.164317/us-central1-f/kubernetes-node-fhx1",
project: "project-example.164317",
zone: "us-central1-f",
instance: "kubernetes-node-fhx1",
fail: false,
},
{
providerID: ProviderName + "://project-example-164317/us-central1-fkubernetes-node-fhx1",
project: "",
zone: "",
instance: "",
fail: true,
},
{
providerID: ProviderName + ":/project-example-164317/us-central1-f/kubernetes-node-fhx1",
project: "",
zone: "",
instance: "",
fail: true,
},
{
providerID: "aws://project-example-164317/us-central1-f/kubernetes-node-fhx1",
project: "",
zone: "",
instance: "",
fail: true,
},
{
providerID: ProviderName + "://project-example-164317/us-central1-f/kubernetes-node-fhx1/",
project: "",
zone: "",
instance: "",
fail: true,
},
{
providerID: ProviderName + "://project-example.164317//kubernetes-node-fhx1",
project: "",
zone: "",
instance: "",
fail: true,
},
{
providerID: ProviderName + "://project-example.164317/kubernetes-node-fhx1",
project: "",
zone: "",
instance: "",
fail: true,
},
}
for _, test := range providers {
project, zone, instance, err := splitProviderID(test.providerID)
if (err != nil) != test.fail {
t.Errorf("Expected to fail=%t, with pattern %v", test.fail, test)
}
if test.fail {
continue
}
if project != test.project {
t.Errorf("Expected %v, but got %v", test.project, project)
}
if zone != test.zone {
t.Errorf("Expected %v, but got %v", test.zone, zone)
}
if instance != test.instance {
t.Errorf("Expected %v, but got %v", test.instance, instance)
}
}
}
func TestGetZoneByProviderID(t *testing.T) {
tests := []struct {
providerID string
expectedZone cloudprovider.Zone
fail bool
description string
}{
{
providerID: ProviderName + "://project-example-164317/us-central1-f/kubernetes-node-fhx1",
expectedZone: cloudprovider.Zone{FailureDomain: "us-central1-f", Region: "us-central1"},
fail: false,
description: "standard gce providerID",
},
{
providerID: ProviderName + "://project-example-164317/us-central1-f/kubernetes-node-fhx1/",
expectedZone: cloudprovider.Zone{},
fail: true,
description: "too many slashes('/') trailing",
},
{
providerID: ProviderName + "://project-example.164317//kubernetes-node-fhx1",
expectedZone: cloudprovider.Zone{},
fail: true,
description: "too many slashes('/') embedded",
},
{
providerID: ProviderName + "://project-example-164317/uscentral1f/kubernetes-node-fhx1",
expectedZone: cloudprovider.Zone{},
fail: true,
description: "invalid name of the GCE zone",
},
}
gce := &Cloud{
localZone: "us-central1-f",
region: "us-central1",
}
for _, test := range tests {
zone, err := gce.GetZoneByProviderID(context.TODO(), test.providerID)
if (err != nil) != test.fail {
t.Errorf("Expected to fail=%t, provider ID %v, tests %s", test.fail, test, test.description)
}
if test.fail {
continue
}
if zone != test.expectedZone {
t.Errorf("Expected %v, but got %v", test.expectedZone, zone)
}
}
}
func TestGenerateCloudConfigs(t *testing.T) {
configBoilerplate := ConfigGlobal{
TokenURL: "",
TokenBody: "",
ProjectID: "project-id",
NetworkName: "network-name",
SubnetworkName: "",
SecondaryRangeName: "",
NodeTags: []string{"node-tag"},
NodeInstancePrefix: "node-prefix",
Multizone: false,
Regional: false,
APIEndpoint: "",
LocalZone: "us-central1-a",
AlphaFeatures: []string{},
}
cloudBoilerplate := CloudConfig{
APIEndpoint: "",
ProjectID: "project-id",
NetworkProjectID: "",
Region: "us-central1",
Zone: "us-central1-a",
ManagedZones: []string{"us-central1-a"},
NetworkName: "network-name",
SubnetworkName: "",
NetworkURL: "",
SubnetworkURL: "",
SecondaryRangeName: "",
NodeTags: []string{"node-tag"},
TokenSource: google.ComputeTokenSource(""),
NodeInstancePrefix: "node-prefix",
UseMetadataServer: true,
AlphaFeatureGate: &AlphaFeatureGate{map[string]bool{}},
}
testCases := []struct {
name string
config func() ConfigGlobal
cloud func() CloudConfig
}{
{
name: "Empty Config",
config: func() ConfigGlobal { return configBoilerplate },
cloud: func() CloudConfig { return cloudBoilerplate },
},
{
name: "Nil token URL",
config: func() ConfigGlobal {
v := configBoilerplate
v.TokenURL = "nil"
return v
},
cloud: func() CloudConfig {
v := cloudBoilerplate
v.TokenSource = nil
return v
},
},
{
name: "Network Project ID",
config: func() ConfigGlobal {
v := configBoilerplate
v.NetworkProjectID = "my-awesome-project"
return v
},
cloud: func() CloudConfig {
v := cloudBoilerplate
v.NetworkProjectID = "my-awesome-project"
return v
},
},
{
name: "Specified API Endpint",
config: func() ConfigGlobal {
v := configBoilerplate
v.APIEndpoint = "https://www.googleapis.com/compute/staging_v1/"
return v
},
cloud: func() CloudConfig {
v := cloudBoilerplate
v.APIEndpoint = "https://www.googleapis.com/compute/staging_v1/"
return v
},
},
{
name: "Network & Subnetwork names",
config: func() ConfigGlobal {
v := configBoilerplate
v.NetworkName = "my-network"
v.SubnetworkName = "my-subnetwork"
return v
},
cloud: func() CloudConfig {
v := cloudBoilerplate
v.NetworkName = "my-network"
v.SubnetworkName = "my-subnetwork"
return v
},
},
{
name: "Network & Subnetwork URLs",
config: func() ConfigGlobal {
v := configBoilerplate
v.NetworkName = "https://www.googleapis.com/compute/v1/projects/project-id/global/networks/my-network"
v.SubnetworkName = "https://www.googleapis.com/compute/v1/projects/project-id/regions/us-central1/subnetworks/my-subnetwork"
return v
},
cloud: func() CloudConfig {
v := cloudBoilerplate
v.NetworkName = ""
v.SubnetworkName = ""
v.NetworkURL = "https://www.googleapis.com/compute/v1/projects/project-id/global/networks/my-network"
v.SubnetworkURL = "https://www.googleapis.com/compute/v1/projects/project-id/regions/us-central1/subnetworks/my-subnetwork"
return v
},
},
{
name: "Multizone",
config: func() ConfigGlobal {
v := configBoilerplate
v.Multizone = true
return v
},
cloud: func() CloudConfig {
v := cloudBoilerplate
v.ManagedZones = nil
return v
},
},
{
name: "Regional",
config: func() ConfigGlobal {
v := configBoilerplate
v.Regional = true
return v
},
cloud: func() CloudConfig {
v := cloudBoilerplate
v.Regional = true
v.ManagedZones = nil
return v
},
},
{
name: "Secondary Range Name",
config: func() ConfigGlobal {
v := configBoilerplate
v.SecondaryRangeName = "my-secondary"
return v
},
cloud: func() CloudConfig {
v := cloudBoilerplate
v.SecondaryRangeName = "my-secondary"
return v
},
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
resultCloud, err := generateCloudConfig(&ConfigFile{Global: tc.config()})
if err != nil {
t.Fatalf("Unexpect error: %v", err)
}
v := tc.cloud()
if !reflect.DeepEqual(*resultCloud, v) {
t.Errorf("Got: \n%v\nWant\n%v\n", v, *resultCloud)
}
})
}
}
func TestNewAlphaFeatureGate(t *testing.T) {
testCases := []struct {
alphaFeatures []string
expectEnabled []string
expectDisabled []string
}{
// enable foo bar
{
alphaFeatures: []string{"foo", "bar"},
expectEnabled: []string{"foo", "bar"},
expectDisabled: []string{"aaa"},
},
// no alpha feature
{
alphaFeatures: []string{},
expectEnabled: []string{},
expectDisabled: []string{"foo", "bar"},
},
// unsupported alpha feature
{
alphaFeatures: []string{"aaa", "foo"},
expectEnabled: []string{"foo"},
expectDisabled: []string{},
},
// enable foo
{
alphaFeatures: []string{"foo"},
expectEnabled: []string{"foo"},
expectDisabled: []string{"bar"},
},
}
for _, tc := range testCases {
featureGate := NewAlphaFeatureGate(tc.alphaFeatures)
for _, key := range tc.expectEnabled {
if !featureGate.Enabled(key) {
t.Errorf("Expect %q to be enabled.", key)
}
}
for _, key := range tc.expectDisabled {
if featureGate.Enabled(key) {
t.Errorf("Expect %q to be disabled.", key)
}
}
}
}
func TestGetRegionInURL(t *testing.T) {
cases := map[string]string{
"https://www.googleapis.com/compute/v1/projects/my-project/regions/us-central1/subnetworks/a": "us-central1",
"https://www.googleapis.com/compute/v1/projects/my-project/regions/us-west2/subnetworks/b": "us-west2",
"projects/my-project/regions/asia-central1/subnetworks/c": "asia-central1",
"regions/europe-north2": "europe-north2",
"my-url": "",
"": "",
}
for input, output := range cases {
result := getRegionInURL(input)
if result != output {
t.Errorf("Actual result %q does not match expected result %q for input: %q", result, output, input)
}
}
}
func TestFindSubnetForRegion(t *testing.T) {
s := []string{
"https://www.googleapis.com/compute/v1/projects/my-project/regions/us-central1/subnetworks/default-38b01f54907a15a7",
"https://www.googleapis.com/compute/v1/projects/my-project/regions/us-west1/subnetworks/default",
"https://www.googleapis.com/compute/v1/projects/my-project/regions/us-east1/subnetworks/default-277eec3815f742b6",
"https://www.googleapis.com/compute/v1/projects/my-project/regions/us-east4/subnetworks/default",
"https://www.googleapis.com/compute/v1/projects/my-project/regions/asia-northeast1/subnetworks/default",
"https://www.googleapis.com/compute/v1/projects/my-project/regions/asia-east1/subnetworks/default-8e020b4b8b244809",
"https://www.googleapis.com/compute/v1/projects/my-project/regions/australia-southeast1/subnetworks/default",
"https://www.googleapis.com/compute/v1/projects/my-project/regions/southamerica-east1/subnetworks/default",
"https://www.googleapis.com/compute/v1/projects/my-project/regions/europe-west3/subnetworks/default",
"https://www.googleapis.com/compute/v1/projects/my-project/regions/asia-southeast1/subnetworks/default",
"",
}
actual := findSubnetForRegion(s, "asia-east1")
expectedResult := "https://www.googleapis.com/compute/v1/projects/my-project/regions/asia-east1/subnetworks/default-8e020b4b8b244809"
if actual != expectedResult {
t.Errorf("Actual result %q does not match expected result %q", actual, expectedResult)
}
var nilSlice []string
res := findSubnetForRegion(nilSlice, "us-central1")
if res != "" {
t.Errorf("expected an empty result, got %v", res)
}
}
func TestLastComponent(t *testing.T) {
cases := map[string]string{
"https://www.googleapis.com/compute/v1/projects/my-project/regions/us-central1/subnetworks/a": "a",
"https://www.googleapis.com/compute/v1/projects/my-project/regions/us-central1/subnetworks/b": "b",
"projects/my-project/regions/us-central1/subnetworks/c": "c",
"d": "d",
"": "",
}
for input, output := range cases {
result := lastComponent(input)
if result != output {
t.Errorf("Actual result %q does not match expected result %q for input: %q", result, output, input)
}
}
}
func TestGetProjectsBasePath(t *testing.T) {
testCases := []struct {
basePath string
expectProjectsBasePath string
}{
// basepath ends in `/projects/`
{
basePath: "path/to/api/projects/",
expectProjectsBasePath: "path/to/api/projects/",
},
// basepath ends in `/projects`, without trailing /
{
basePath: "path/to/api/projects",
expectProjectsBasePath: "path/to/api/projects/",
},
// basepath does not end in `/projects/`
{
basePath: "path/to/api/",
expectProjectsBasePath: "path/to/api/projects/",
},
// basepath does not end in `/projects/` and does not have trailing /
{
basePath: "path/to/api",
expectProjectsBasePath: "path/to/api/projects/",
},
}
for _, tc := range testCases {
projectsBasePath := getProjectsBasePath(tc.basePath)
if projectsBasePath != tc.expectProjectsBasePath {
t.Errorf("Expected projects base path %s; but got %s", tc.expectProjectsBasePath, projectsBasePath)
}
}
}

220
staging/src/k8s.io/legacy-cloud-providers/gce/gce_tpu.go
View File

@@ -1,220 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2018 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
"context"
"encoding/json"
"fmt"
"net/http"
"time"
"google.golang.org/api/googleapi"
tpuapi "google.golang.org/api/tpu/v1"
"k8s.io/klog/v2"
"k8s.io/apimachinery/pkg/util/wait"
)
// newTPUService returns a new tpuService using the client to communicate with
// the Cloud TPU APIs.
func newTPUService(client *http.Client) (*tpuService, error) {
s, err := tpuapi.NewService(context.Background())
if err != nil {
return nil, err
}
return &tpuService{
projects: tpuapi.NewProjectsService(s),
}, nil
}
// tpuService encapsulates the TPU services on nodes and the operations on the
// nodes.
type tpuService struct {
projects *tpuapi.ProjectsService
}
// CreateTPU creates the Cloud TPU node with the specified name in the
// specified zone.
func (g *Cloud) CreateTPU(ctx context.Context, name, zone string, node *tpuapi.Node) (*tpuapi.Node, error) {
var err error
mc := newTPUMetricContext("create", zone)
defer mc.Observe(err)
var op *tpuapi.Operation
parent := getTPUParentName(g.projectID, zone)
op, err = g.tpuService.projects.Locations.Nodes.Create(parent, node).NodeId(name).Do()
if err != nil {
return nil, err
}
klog.V(2).Infof("Creating Cloud TPU %q in zone %q with operation %q", name, zone, op.Name)
op, err = g.waitForTPUOp(ctx, op)
if err != nil {
return nil, err
}
err = getErrorFromTPUOp(op)
if err != nil {
return nil, err
}
output := new(tpuapi.Node)
err = json.Unmarshal(op.Response, output)
if err != nil {
err = fmt.Errorf("failed to unmarshal response from operation %q: response = %v, err = %v", op.Name, op.Response, err)
return nil, err
}
return output, nil
}
// DeleteTPU deletes the Cloud TPU with the specified name in the specified
// zone.
func (g *Cloud) DeleteTPU(ctx context.Context, name, zone string) error {
var err error
mc := newTPUMetricContext("delete", zone)
defer mc.Observe(err)
var op *tpuapi.Operation
name = getTPUName(g.projectID, zone, name)
op, err = g.tpuService.projects.Locations.Nodes.Delete(name).Do()
if err != nil {
return err
}
klog.V(2).Infof("Deleting Cloud TPU %q in zone %q with operation %q", name, zone, op.Name)
op, err = g.waitForTPUOp(ctx, op)
if err != nil {
return err
}
err = getErrorFromTPUOp(op)
return err
}
// GetTPU returns the Cloud TPU with the specified name in the specified zone.
func (g *Cloud) GetTPU(ctx context.Context, name, zone string) (*tpuapi.Node, error) {
mc := newTPUMetricContext("get", zone)
name = getTPUName(g.projectID, zone, name)
node, err := g.tpuService.projects.Locations.Nodes.Get(name).Do()
if err != nil {
return nil, mc.Observe(err)
}
return node, mc.Observe(nil)
}
// ListTPUs returns Cloud TPUs in the specified zone.
func (g *Cloud) ListTPUs(ctx context.Context, zone string) ([]*tpuapi.Node, error) {
mc := newTPUMetricContext("list", zone)
parent := getTPUParentName(g.projectID, zone)
var nodes []*tpuapi.Node
var accumulator = func(response *tpuapi.ListNodesResponse) error {
nodes = append(nodes, response.Nodes...)
return nil
}
err := g.tpuService.projects.Locations.Nodes.List(parent).Pages(ctx, accumulator)
if err != nil {
return nil, mc.Observe(err)
}
return nodes, mc.Observe(nil)
}
// ListLocations returns the zones where Cloud TPUs are available.
func (g *Cloud) ListLocations(ctx context.Context) ([]*tpuapi.Location, error) {
mc := newTPUMetricContext("list_locations", "")
parent := getTPUProjectURL(g.projectID)
var locations []*tpuapi.Location
var accumulator = func(response *tpuapi.ListLocationsResponse) error {
locations = append(locations, response.Locations...)
return nil
}
err := g.tpuService.projects.Locations.List(parent).Pages(ctx, accumulator)
if err != nil {
return nil, mc.Observe(err)
}
return locations, mc.Observe(nil)
}
// waitForTPUOp checks whether the op is done every 30 seconds before the ctx
// is cancelled.
func (g *Cloud) waitForTPUOp(ctx context.Context, op *tpuapi.Operation) (*tpuapi.Operation, error) {
if err := wait.PollInfinite(30*time.Second, func() (bool, error) {
// Check if context has been cancelled.
select {
case <-ctx.Done():
klog.V(3).Infof("Context for operation %q has been cancelled: %s", op.Name, ctx.Err())
return true, ctx.Err()
default:
}
klog.V(3).Infof("Waiting for operation %q to complete...", op.Name)
start := time.Now()
g.operationPollRateLimiter.Accept()
duration := time.Since(start)
if duration > 5*time.Second {
klog.V(2).Infof("Getting operation %q throttled for %v", op.Name, duration)
}
var err error
op, err = g.tpuService.projects.Locations.Operations.Get(op.Name).Do()
if err != nil {
return true, err
}
if op.Done {
klog.V(3).Infof("Operation %q has completed", op.Name)
return true, nil
}
return false, nil
}); err != nil {
return nil, fmt.Errorf("failed to wait for operation %q: %s", op.Name, err)
}
return op, nil
}
// newTPUMetricContext returns a new metricContext used for recording metrics
// of Cloud TPU API calls.
func newTPUMetricContext(request, zone string) *metricContext {
return newGenericMetricContext("tpus", request, unusedMetricLabel, zone, "v1")
}
// getErrorFromTPUOp returns the error in the failed op, or nil if the op
// succeed.
func getErrorFromTPUOp(op *tpuapi.Operation) error {
if op != nil && op.Error != nil {
return &googleapi.Error{
Code: op.ServerResponse.HTTPStatusCode,
Message: op.Error.Message,
}
}
return nil
}
func getTPUProjectURL(project string) string {
return fmt.Sprintf("projects/%s", project)
}
func getTPUParentName(project, zone string) string {
return fmt.Sprintf("projects/%s/locations/%s", project, zone)
}
func getTPUName(project, zone, name string) string {
return fmt.Sprintf("projects/%s/locations/%s/nodes/%s", project, zone, name)
}

79
staging/src/k8s.io/legacy-cloud-providers/gce/gce_urlmap.go
View File

@@ -1,79 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
compute "google.golang.org/api/compute/v1"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud/filter"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud/meta"
)
func newURLMapMetricContext(request string) *metricContext {
return newGenericMetricContext("urlmap", request, unusedMetricLabel, unusedMetricLabel, computeV1Version)
}
// GetURLMap returns the UrlMap by name.
func (g *Cloud) GetURLMap(name string) (*compute.UrlMap, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newURLMapMetricContext("get")
v, err := g.c.UrlMaps().Get(ctx, meta.GlobalKey(name))
return v, mc.Observe(err)
}
// CreateURLMap creates a url map
func (g *Cloud) CreateURLMap(urlMap *compute.UrlMap) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newURLMapMetricContext("create")
return mc.Observe(g.c.UrlMaps().Insert(ctx, meta.GlobalKey(urlMap.Name), urlMap))
}
// UpdateURLMap applies the given UrlMap as an update
func (g *Cloud) UpdateURLMap(urlMap *compute.UrlMap) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newURLMapMetricContext("update")
return mc.Observe(g.c.UrlMaps().Update(ctx, meta.GlobalKey(urlMap.Name), urlMap))
}
// DeleteURLMap deletes a url map by name.
func (g *Cloud) DeleteURLMap(name string) error {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newURLMapMetricContext("delete")
return mc.Observe(g.c.UrlMaps().Delete(ctx, meta.GlobalKey(name)))
}
// ListURLMaps lists all UrlMaps in the project.
func (g *Cloud) ListURLMaps() ([]*compute.UrlMap, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newURLMapMetricContext("list")
v, err := g.c.UrlMaps().List(ctx, filter.None)
return v, mc.Observe(err)
}

418
staging/src/k8s.io/legacy-cloud-providers/gce/gce_util.go
View File

@@ -1,418 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
"context"
"errors"
"fmt"
"net"
"net/http"
"regexp"
"sort"
"strings"
"sync"
"cloud.google.com/go/compute/metadata"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud/meta"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud/mock"
compute "google.golang.org/api/compute/v1"
"google.golang.org/api/googleapi"
v1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/types"
"k8s.io/apimachinery/pkg/util/sets"
"k8s.io/client-go/kubernetes/fake"
v1core "k8s.io/client-go/kubernetes/typed/core/v1"
"k8s.io/client-go/tools/record"
servicehelper "k8s.io/cloud-provider/service/helpers"
netutils "k8s.io/utils/net"
)
const (
// NetLBFinalizerV2 is the finalizer used by newer controllers that manage L4 External LoadBalancer services.
NetLBFinalizerV2 = "gke.networking.io/l4-netlb-v2"
)
func fakeGCECloud(vals TestClusterValues) (*Cloud, error) {
gce := NewFakeGCECloud(vals)
gce.AlphaFeatureGate = NewAlphaFeatureGate([]string{})
gce.nodeInformerSynced = func() bool { return true }
gce.client = fake.NewSimpleClientset()
gce.eventRecorder = &record.FakeRecorder{}
mockGCE := gce.c.(*cloud.MockGCE)
mockGCE.MockTargetPools.AddInstanceHook = mock.AddInstanceHook
mockGCE.MockTargetPools.RemoveInstanceHook = mock.RemoveInstanceHook
mockGCE.MockForwardingRules.InsertHook = mock.InsertFwdRuleHook
mockGCE.MockAddresses.InsertHook = mock.InsertAddressHook
mockGCE.MockAlphaAddresses.InsertHook = mock.InsertAlphaAddressHook
mockGCE.MockAlphaAddresses.X = mock.AddressAttributes{}
mockGCE.MockAddresses.X = mock.AddressAttributes{}
mockGCE.MockInstanceGroups.X = mock.InstanceGroupAttributes{
InstanceMap: make(map[meta.Key]map[string]*compute.InstanceWithNamedPorts),
Lock: &sync.Mutex{},
}
mockGCE.MockInstanceGroups.AddInstancesHook = mock.AddInstancesHook
mockGCE.MockInstanceGroups.RemoveInstancesHook = mock.RemoveInstancesHook
mockGCE.MockInstanceGroups.ListInstancesHook = mock.ListInstancesHook
mockGCE.MockRegionBackendServices.UpdateHook = mock.UpdateRegionBackendServiceHook
mockGCE.MockHealthChecks.UpdateHook = mock.UpdateHealthCheckHook
mockGCE.MockFirewalls.UpdateHook = mock.UpdateFirewallHook
mockGCE.MockFirewalls.PatchHook = mock.UpdateFirewallHook
keyGA := meta.GlobalKey("key-ga")
mockGCE.MockZones.Objects[*keyGA] = &cloud.MockZonesObj{
Obj: &compute.Zone{Name: vals.ZoneName, Region: gce.getRegionLink(vals.Region)},
}
return gce, nil
}
func registerTargetPoolAddInstanceHook(gce *Cloud, callback func(*compute.TargetPoolsAddInstanceRequest)) error {
mockGCE, ok := gce.c.(*cloud.MockGCE)
if !ok {
return fmt.Errorf("couldn't cast cloud to mockGCE: %#v", gce)
}
existingHandler := mockGCE.MockTargetPools.AddInstanceHook
hook := func(ctx context.Context, key *meta.Key, req *compute.TargetPoolsAddInstanceRequest, m *cloud.MockTargetPools) error {
callback(req)
return existingHandler(ctx, key, req, m)
}
mockGCE.MockTargetPools.AddInstanceHook = hook
return nil
}
func registerTargetPoolRemoveInstanceHook(gce *Cloud, callback func(*compute.TargetPoolsRemoveInstanceRequest)) error {
mockGCE, ok := gce.c.(*cloud.MockGCE)
if !ok {
return fmt.Errorf("couldn't cast cloud to mockGCE: %#v", gce)
}
existingHandler := mockGCE.MockTargetPools.RemoveInstanceHook
hook := func(ctx context.Context, key *meta.Key, req *compute.TargetPoolsRemoveInstanceRequest, m *cloud.MockTargetPools) error {
callback(req)
return existingHandler(ctx, key, req, m)
}
mockGCE.MockTargetPools.RemoveInstanceHook = hook
return nil
}
type gceInstance struct {
Zone string
Name string
ID uint64
Disks []*compute.AttachedDisk
Type string
}
var (
autoSubnetIPRange = &net.IPNet{
IP: netutils.ParseIPSloppy("10.128.0.0"),
Mask: net.CIDRMask(9, 32),
}
)
var providerIDRE = regexp.MustCompile(`^` + ProviderName + `://([^/]+)/([^/]+)/([^/]+)$`)
func getProjectAndZone() (string, string, error) {
result, err := metadata.Get("instance/zone")
if err != nil {
return "", "", err
}
parts := strings.Split(result, "/")
if len(parts) != 4 {
return "", "", fmt.Errorf("unexpected response: %s", result)
}
zone := parts[3]
projectID, err := metadata.ProjectID()
if err != nil {
return "", "", err
}
return projectID, zone, nil
}
func (g *Cloud) raiseFirewallChangeNeededEvent(svc *v1.Service, cmd string) {
msg := fmt.Sprintf("Firewall change required by security admin: `%v`", cmd)
if g.eventRecorder != nil && svc != nil {
g.eventRecorder.Event(svc, v1.EventTypeNormal, "LoadBalancerManualChange", msg)
}
}
// FirewallToGCloudCreateCmd generates a gcloud command to create a firewall with specified params
func FirewallToGCloudCreateCmd(fw *compute.Firewall, projectID string) string {
args := firewallToGcloudArgs(fw, projectID)
return fmt.Sprintf("gcloud compute firewall-rules create %v --network %v %v", fw.Name, getNameFromLink(fw.Network), args)
}
// FirewallToGCloudUpdateCmd generates a gcloud command to update a firewall to specified params
func FirewallToGCloudUpdateCmd(fw *compute.Firewall, projectID string) string {
args := firewallToGcloudArgs(fw, projectID)
return fmt.Sprintf("gcloud compute firewall-rules update %v %v", fw.Name, args)
}
// FirewallToGCloudDeleteCmd generates a gcloud command to delete a firewall to specified params
func FirewallToGCloudDeleteCmd(fwName, projectID string) string {
return fmt.Sprintf("gcloud compute firewall-rules delete %v --project %v", fwName, projectID)
}
func firewallToGcloudArgs(fw *compute.Firewall, projectID string) string {
var allPorts []string
for _, a := range fw.Allowed {
for _, p := range a.Ports {
allPorts = append(allPorts, fmt.Sprintf("%v:%v", a.IPProtocol, p))
}
}
// Sort all slices to prevent the event from being duped
sort.Strings(allPorts)
allow := strings.Join(allPorts, ",")
sort.Strings(fw.SourceRanges)
srcRngs := strings.Join(fw.SourceRanges, ",")
sort.Strings(fw.TargetTags)
targets := strings.Join(fw.TargetTags, ",")
return fmt.Sprintf("--description %q --allow %v --source-ranges %v --target-tags %v --project %v", fw.Description, allow, srcRngs, targets, projectID)
}
// Take a GCE instance 'hostname' and break it down to something that can be fed
// to the GCE API client library. Basically this means reducing 'kubernetes-
// node-2.c.my-proj.internal' to 'kubernetes-node-2' if necessary.
func canonicalizeInstanceName(name string) string {
ix := strings.Index(name, ".")
if ix != -1 {
name = name[:ix]
}
return name
}
// Returns the last component of a URL, i.e. anything after the last slash
// If there is no slash, returns the whole string
func lastComponent(s string) string {
lastSlash := strings.LastIndex(s, "/")
if lastSlash != -1 {
s = s[lastSlash+1:]
}
return s
}
// mapNodeNameToInstanceName maps a k8s NodeName to a GCE Instance Name
// This is a simple string cast.
func mapNodeNameToInstanceName(nodeName types.NodeName) string {
return string(nodeName)
}
// GetGCERegion returns region of the gce zone. Zone names
// are of the form: ${region-name}-${ix}.
// For example, "us-central1-b" has a region of "us-central1".
// So we look for the last '-' and trim to just before that.
func GetGCERegion(zone string) (string, error) {
ix := strings.LastIndex(zone, "-")
if ix == -1 {
return "", fmt.Errorf("unexpected zone: %s", zone)
}
return zone[:ix], nil
}
func isHTTPErrorCode(err error, code int) bool {
apiErr, ok := err.(*googleapi.Error)
return ok && apiErr.Code == code
}
func isInUsedByError(err error) bool {
apiErr, ok := err.(*googleapi.Error)
if !ok || apiErr.Code != http.StatusBadRequest {
return false
}
return strings.Contains(apiErr.Message, "being used by")
}
// splitProviderID splits a provider's id into core components.
// A providerID is build out of '${ProviderName}://${project-id}/${zone}/${instance-name}'
// See cloudprovider.GetInstanceProviderID.
func splitProviderID(providerID string) (project, zone, instance string, err error) {
matches := providerIDRE.FindStringSubmatch(providerID)
if len(matches) != 4 {
return "", "", "", errors.New("error splitting providerID")
}
return matches[1], matches[2], matches[3], nil
}
func equalStringSets(x, y []string) bool {
if len(x) != len(y) {
return false
}
xString := sets.NewString(x...)
yString := sets.NewString(y...)
return xString.Equal(yString)
}
func isNotFound(err error) bool {
return isHTTPErrorCode(err, http.StatusNotFound)
}
func ignoreNotFound(err error) error {
if err == nil || isNotFound(err) {
return nil
}
return err
}
func isNotFoundOrInUse(err error) bool {
return isNotFound(err) || isInUsedByError(err)
}
func isForbidden(err error) bool {
return isHTTPErrorCode(err, http.StatusForbidden)
}
func makeGoogleAPINotFoundError(message string) error {
return &googleapi.Error{Code: http.StatusNotFound, Message: message}
}
// containsCIDR returns true if outer contains inner.
func containsCIDR(outer, inner *net.IPNet) bool {
return outer.Contains(firstIPInRange(inner)) && outer.Contains(lastIPInRange(inner))
}
// firstIPInRange returns the first IP in a given IP range.
func firstIPInRange(ipNet *net.IPNet) net.IP {
return ipNet.IP.Mask(ipNet.Mask)
}
// lastIPInRange returns the last IP in a given IP range.
func lastIPInRange(cidr *net.IPNet) net.IP {
ip := append([]byte{}, cidr.IP...)
for i, b := range cidr.Mask {
ip[i] |= ^b
}
return ip
}
// subnetsInCIDR takes a list of subnets for a single region and
// returns subnets which exists in the specified CIDR range.
func subnetsInCIDR(subnets []*compute.Subnetwork, cidr *net.IPNet) ([]*compute.Subnetwork, error) {
var res []*compute.Subnetwork
for _, subnet := range subnets {
_, subnetRange, err := netutils.ParseCIDRSloppy(subnet.IpCidrRange)
if err != nil {
return nil, fmt.Errorf("unable to parse CIDR %q for subnet %q: %v", subnet.IpCidrRange, subnet.Name, err)
}
if containsCIDR(cidr, subnetRange) {
res = append(res, subnet)
}
}
return res, nil
}
type netType string
const (
netTypeLegacy netType = "LEGACY"
netTypeAuto netType = "AUTO"
netTypeCustom netType = "CUSTOM"
)
func typeOfNetwork(network *compute.Network) netType {
if network.IPv4Range != "" {
return netTypeLegacy
}
if network.AutoCreateSubnetworks {
return netTypeAuto
}
return netTypeCustom
}
func getLocationName(project, zoneOrRegion string) string {
return fmt.Sprintf("projects/%s/locations/%s", project, zoneOrRegion)
}
func addFinalizer(service *v1.Service, kubeClient v1core.CoreV1Interface, key string) error {
if hasFinalizer(service, key) {
return nil
}
// Make a copy so we don't mutate the shared informer cache.
updated := service.DeepCopy()
updated.ObjectMeta.Finalizers = append(updated.ObjectMeta.Finalizers, key)
_, err := servicehelper.PatchService(kubeClient, service, updated)
return err
}
// removeFinalizer patches the service to remove finalizer.
func removeFinalizer(service *v1.Service, kubeClient v1core.CoreV1Interface, key string) error {
if !hasFinalizer(service, key) {
return nil
}
// Make a copy so we don't mutate the shared informer cache.
updated := service.DeepCopy()
updated.ObjectMeta.Finalizers = removeString(updated.ObjectMeta.Finalizers, key)
_, err := servicehelper.PatchService(kubeClient, service, updated)
return err
}
// hasFinalizer returns if the given service has the specified key in its list of finalizers.
func hasFinalizer(service *v1.Service, key string) bool {
for _, finalizer := range service.ObjectMeta.Finalizers {
if finalizer == key {
return true
}
}
return false
}
// removeString returns a newly created []string that contains all items from slice that
// are not equal to s.
func removeString(slice []string, s string) []string {
var newSlice []string
for _, item := range slice {
if item != s {
newSlice = append(newSlice, item)
}
}
return newSlice
}
// usesL4RBS checks if service uses Regional Backend Service as a Backend.
// Such services implemented in other controllers and
// should not be handled by Service Controller.
func usesL4RBS(service *v1.Service, forwardingRule *compute.ForwardingRule) bool {
// Detect RBS by annotation
if val, ok := service.Annotations[RBSAnnotationKey]; ok && val == RBSEnabled {
return true
}
// Detect RBS by finalizer
if hasFinalizer(service, NetLBFinalizerV2) {
return true
}
// Detect RBS by existing forwarding rule with Backend Service attached
if forwardingRule != nil && forwardingRule.BackendService != "" {
return true
}
return false
}

159
staging/src/k8s.io/legacy-cloud-providers/gce/gce_util_test.go
View File

@@ -1,159 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
"context"
"reflect"
"testing"
compute "google.golang.org/api/compute/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
netutils "k8s.io/utils/net"
)
func TestLastIPInRange(t *testing.T) {
for _, tc := range []struct {
cidr string
want string
}{
{"10.1.2.3/32", "10.1.2.3"},
{"10.1.2.0/31", "10.1.2.1"},
{"10.1.0.0/30", "10.1.0.3"},
{"10.0.0.0/29", "10.0.0.7"},
{"::0/128", "::"},
{"::0/127", "::1"},
{"::0/126", "::3"},
{"::0/120", "::ff"},
} {
_, c, err := netutils.ParseCIDRSloppy(tc.cidr)
if err != nil {
t.Errorf("can't parse CIDR %v = _, %v, %v; want nil", tc.cidr, c, err)
continue
}
if lastIP := lastIPInRange(c); lastIP.String() != tc.want {
t.Errorf("LastIPInRange(%v) = %v; want %v", tc.cidr, lastIP, tc.want)
}
}
}
func TestSubnetsInCIDR(t *testing.T) {
subnets := []*compute.Subnetwork{
{
Name: "A",
IpCidrRange: "10.0.0.0/20",
},
{
Name: "B",
IpCidrRange: "10.0.16.0/20",
},
{
Name: "C",
IpCidrRange: "10.132.0.0/20",
},
{
Name: "D",
IpCidrRange: "10.0.32.0/20",
},
{
Name: "E",
IpCidrRange: "10.134.0.0/20",
},
}
expectedNames := []string{"C", "E"}
gotSubs, err := subnetsInCIDR(subnets, autoSubnetIPRange)
if err != nil {
t.Errorf("autoSubnetInList() = _, %v", err)
}
var gotNames []string
for _, v := range gotSubs {
gotNames = append(gotNames, v.Name)
}
if !reflect.DeepEqual(gotNames, expectedNames) {
t.Errorf("autoSubnetInList() = %v, expected: %v", gotNames, expectedNames)
}
}
func TestFirewallToGcloudArgs(t *testing.T) {
firewall := compute.Firewall{
Description: "Last Line of Defense",
TargetTags: []string{"jock-nodes", "band-nodes"},
SourceRanges: []string{"3.3.3.3/20", "1.1.1.1/20", "2.2.2.2/20"},
Allowed: []*compute.FirewallAllowed{
{
IPProtocol: "udp",
Ports: []string{"321", "123-456", "123"},
},
{
IPProtocol: "tcp",
Ports: []string{"321", "123-456", "123"},
},
{
IPProtocol: "sctp",
Ports: []string{"321", "123-456", "123"},
},
},
}
got := firewallToGcloudArgs(&firewall, "my-project")
var e = `--description "Last Line of Defense" --allow sctp:123,sctp:123-456,sctp:321,tcp:123,tcp:123-456,tcp:321,udp:123,udp:123-456,udp:321 --source-ranges 1.1.1.1/20,2.2.2.2/20,3.3.3.3/20 --target-tags band-nodes,jock-nodes --project my-project`
if got != e {
t.Errorf("%q does not equal %q", got, e)
}
}
// TestAddRemoveFinalizer tests the add/remove and hasFinalizer methods.
func TestAddRemoveFinalizer(t *testing.T) {
svc := fakeLoadbalancerService(string(LBTypeInternal))
gce, err := fakeGCECloud(vals)
if err != nil {
t.Fatalf("Failed to get GCE client, err %v", err)
}
svc, err = gce.client.CoreV1().Services(svc.Namespace).Create(context.TODO(), svc, metav1.CreateOptions{})
if err != nil {
t.Errorf("Failed to create service %s, err %v", svc.Name, err)
}
err = addFinalizer(svc, gce.client.CoreV1(), ILBFinalizerV1)
if err != nil {
t.Fatalf("Failed to add finalizer, err %v", err)
}
svc, err = gce.client.CoreV1().Services(svc.Namespace).Get(context.TODO(), svc.Name, metav1.GetOptions{})
if err != nil {
t.Errorf("Failed to get service, err %v", err)
}
if !hasFinalizer(svc, ILBFinalizerV1) {
t.Errorf("Unable to find finalizer '%s' in service %s", ILBFinalizerV1, svc.Name)
}
err = removeFinalizer(svc, gce.client.CoreV1(), ILBFinalizerV1)
if err != nil {
t.Fatalf("Failed to remove finalizer, err %v", err)
}
svc, err = gce.client.CoreV1().Services(svc.Namespace).Get(context.TODO(), svc.Name, metav1.GetOptions{})
if err != nil {
t.Errorf("Failed to get service, err %v", err)
}
if hasFinalizer(svc, ILBFinalizerV1) {
t.Errorf("Failed to remove finalizer '%s' in service %s", ILBFinalizerV1, svc.Name)
}
}

95
staging/src/k8s.io/legacy-cloud-providers/gce/gce_zones.go
View File

@@ -1,95 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
"context"
"fmt"
"strings"
compute "google.golang.org/api/compute/v1"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud/filter"
"k8s.io/apimachinery/pkg/types"
cloudprovider "k8s.io/cloud-provider"
)
func newZonesMetricContext(request, region string) *metricContext {
return newGenericMetricContext("zones", request, region, unusedMetricLabel, computeV1Version)
}
// GetZone creates a cloudprovider.Zone of the current zone and region
func (g *Cloud) GetZone(ctx context.Context) (cloudprovider.Zone, error) {
return cloudprovider.Zone{
FailureDomain: g.localZone,
Region: g.region,
}, nil
}
// GetZoneByProviderID implements Zones.GetZoneByProviderID
// This is particularly useful in external cloud providers where the kubelet
// does not initialize node data.
func (g *Cloud) GetZoneByProviderID(ctx context.Context, providerID string) (cloudprovider.Zone, error) {
_, zone, _, err := splitProviderID(providerID)
if err != nil {
return cloudprovider.Zone{}, err
}
region, err := GetGCERegion(zone)
if err != nil {
return cloudprovider.Zone{}, err
}
return cloudprovider.Zone{FailureDomain: zone, Region: region}, nil
}
// GetZoneByNodeName implements Zones.GetZoneByNodeName
// This is particularly useful in external cloud providers where the kubelet
// does not initialize node data.
func (g *Cloud) GetZoneByNodeName(ctx context.Context, nodeName types.NodeName) (cloudprovider.Zone, error) {
instanceName := mapNodeNameToInstanceName(nodeName)
instance, err := g.getInstanceByName(instanceName)
if err != nil {
return cloudprovider.Zone{}, err
}
region, err := GetGCERegion(instance.Zone)
if err != nil {
return cloudprovider.Zone{}, err
}
return cloudprovider.Zone{FailureDomain: instance.Zone, Region: region}, nil
}
// ListZonesInRegion returns all zones in a GCP region
func (g *Cloud) ListZonesInRegion(region string) ([]*compute.Zone, error) {
ctx, cancel := cloud.ContextWithCallTimeout()
defer cancel()
mc := newZonesMetricContext("list", region)
// Use regex match instead of an exact regional link constructed from getRegionalLink below.
// See comments in issue kubernetes/kubernetes#87905
list, err := g.c.Zones().List(ctx, filter.Regexp("region", fmt.Sprintf(".*/regions/%s", region)))
if err != nil {
return nil, mc.Observe(err)
}
return list, mc.Observe(err)
}
func (g *Cloud) getRegionLink(region string) string {
return g.projectsBasePath + strings.Join([]string{g.projectID, "regions", region}, "/")
}

113
staging/src/k8s.io/legacy-cloud-providers/gce/gcpcredential/credentialutil.go
View File

@@ -1,113 +0,0 @@
/*
Copyright 2021 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gcpcredential
import (
"encoding/json"
"errors"
"fmt"
"io"
"io/ioutil"
"net/http"
"k8s.io/cloud-provider/credentialconfig"
"k8s.io/klog/v2"
)
const (
maxReadLength = 10 * 1 << 20 // 10MB
)
// HTTPError wraps a non-StatusOK error code as an error.
type HTTPError struct {
StatusCode int
URL string
}
// Error implements error
func (he *HTTPError) Error() string {
return fmt.Sprintf("http status code: %d while fetching url %s",
he.StatusCode, he.URL)
}
// ReadURL read contents from given url
func ReadURL(url string, client *http.Client, header *http.Header) (body []byte, err error) {
req, err := http.NewRequest("GET", url, nil)
if err != nil {
return nil, err
}
if header != nil {
req.Header = *header
}
resp, err := client.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
klog.V(2).InfoS("Failed to read URL", "statusCode", resp.StatusCode, "URL", url)
return nil, &HTTPError{
StatusCode: resp.StatusCode,
URL: url,
}
}
limitedReader := &io.LimitedReader{R: resp.Body, N: maxReadLength}
contents, err := ioutil.ReadAll(limitedReader)
if err != nil {
return nil, err
}
if limitedReader.N <= 0 {
return nil, errors.New("the read limit is reached")
}
return contents, nil
}
// ReadDockerConfigFileFromURL read a docker config file from the given url
func ReadDockerConfigFileFromURL(url string, client *http.Client, header *http.Header) (cfg credentialconfig.RegistryConfig, err error) {
if contents, err := ReadURL(url, client, header); err == nil {
return ReadDockerConfigFileFromBytes(contents)
}
return nil, err
}
type internalRegistryConfig map[string]RegistryConfigEntry
// ReadDockerConfigFileFromBytes read a docker config file from the given bytes
func ReadDockerConfigFileFromBytes(contents []byte) (cfg credentialconfig.RegistryConfig, err error) {
serializableCfg := internalRegistryConfig{}
if err = json.Unmarshal(contents, &serializableCfg); err != nil {
return nil, errors.New("error occurred while trying to unmarshal json")
}
return convertToExternalConfig(serializableCfg), nil
}
func convertToExternalConfig(in internalRegistryConfig) (cfg credentialconfig.RegistryConfig) {
configMap := credentialconfig.RegistryConfig{}
for k, v := range in {
configMap[k] = credentialconfig.RegistryConfigEntry{
Username: v.Username,
Password: v.Password,
Email: v.Email,
}
}
return configMap
}

130
staging/src/k8s.io/legacy-cloud-providers/gce/gcpcredential/gcpcredential.go
View File

@@ -1,130 +0,0 @@
/*
Copyright 2020 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gcpcredential
import (
"encoding/json"
"net/http"
"strings"
"k8s.io/cloud-provider/credentialconfig"
"k8s.io/klog/v2"
)
const (
metadataURL = "http://metadata.google.internal./computeMetadata/v1/"
metadataAttributes = metadataURL + "instance/attributes/"
// DockerConfigKey is the URL of the dockercfg metadata key used by DockerConfigKeyProvider.
DockerConfigKey = metadataAttributes + "google-dockercfg"
// DockerConfigURLKey is the URL of the dockercfg metadata key used by DockerConfigURLKeyProvider.
DockerConfigURLKey = metadataAttributes + "google-dockercfg-url"
serviceAccounts = metadataURL + "instance/service-accounts/"
metadataScopes = metadataURL + "instance/service-accounts/default/scopes"
metadataToken = metadataURL + "instance/service-accounts/default/token"
metadataEmail = metadataURL + "instance/service-accounts/default/email"
// StorageScopePrefix is the prefix checked by ContainerRegistryProvider.Enabled.
StorageScopePrefix = "https://www.googleapis.com/auth/devstorage"
cloudPlatformScopePrefix = "https://www.googleapis.com/auth/cloud-platform"
defaultServiceAccount = "default/"
)
// GCEProductNameFile is the product file path that contains the cloud service name.
// This is a variable instead of a const to enable testing.
var GCEProductNameFile = "/sys/class/dmi/id/product_name"
// For these urls, the parts of the host name can be glob, for example '*.gcr.io" will match
// "foo.gcr.io" and "bar.gcr.io".
var containerRegistryUrls = []string{"container.cloud.google.com", "gcr.io", "*.gcr.io", "*.pkg.dev"}
var metadataHeader = &http.Header{
"Metadata-Flavor": []string{"Google"},
}
// ProvideConfigKey implements a dockercfg-based authentication flow.
func ProvideConfigKey(client *http.Client, image string) credentialconfig.RegistryConfig {
// Read the contents of the google-dockercfg metadata key and
// parse them as an alternate .dockercfg
if cfg, err := ReadDockerConfigFileFromURL(DockerConfigKey, client, metadataHeader); err != nil {
klog.Errorf("while reading 'google-dockercfg' metadata: %v", err)
} else {
return cfg
}
return credentialconfig.RegistryConfig{}
}
// ProvideURLKey implements a dockercfg-url-based authentication flow.
func ProvideURLKey(client *http.Client, image string) credentialconfig.RegistryConfig {
// Read the contents of the google-dockercfg-url key and load a .dockercfg from there
if url, err := ReadURL(DockerConfigURLKey, client, metadataHeader); err != nil {
klog.Errorf("while reading 'google-dockercfg-url' metadata: %v", err)
} else {
if strings.HasPrefix(string(url), "http") {
if cfg, err := ReadDockerConfigFileFromURL(string(url), client, nil); err != nil {
klog.Errorf("while reading 'google-dockercfg-url'-specified url: %s, %v", string(url), err)
} else {
return cfg
}
} else {
// TODO(mattmoor): support reading alternate scheme URLs (e.g. gs:// or s3://)
klog.Errorf("Unsupported URL scheme: %s", string(url))
}
}
return credentialconfig.RegistryConfig{}
}
// TokenBlob is used to decode the JSON blob containing an access token
// that is returned by GCE metadata.
type TokenBlob struct {
AccessToken string `json:"access_token"`
}
// ProvideContainerRegistry implements a gcr.io-based authentication flow.
func ProvideContainerRegistry(client *http.Client, image string) credentialconfig.RegistryConfig {
cfg := credentialconfig.RegistryConfig{}
tokenJSONBlob, err := ReadURL(metadataToken, client, metadataHeader)
if err != nil {
klog.Errorf("while reading access token endpoint: %v", err)
return cfg
}
email, err := ReadURL(metadataEmail, client, metadataHeader)
if err != nil {
klog.Errorf("while reading email endpoint: %v", err)
return cfg
}
var parsedBlob TokenBlob
if err := json.Unmarshal([]byte(tokenJSONBlob), &parsedBlob); err != nil {
klog.Errorf("error while parsing json blob of length %d", len(tokenJSONBlob))
return cfg
}
entry := credentialconfig.RegistryConfigEntry{
Username: "_token",
Password: parsedBlob.AccessToken,
Email: string(email),
}
// Add our entry for each of the supported container registry URLs
for _, k := range containerRegistryUrls {
cfg[k] = entry
}
return cfg
}

110
staging/src/k8s.io/legacy-cloud-providers/gce/gcpcredential/registry_marshal.go
View File

@@ -1,110 +0,0 @@
/*
Copyright 2021 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gcpcredential
import (
"encoding/base64"
"encoding/json"
"fmt"
"strings"
"k8s.io/cloud-provider/credentialconfig"
)
// registryConfigEntryWithAuth is used solely for deserializing the Auth field
// into a dockerConfigEntry during JSON deserialization.
type registryConfigEntryWithAuth struct {
// +optional
Username string `json:"username,omitempty"`
// +optional
Password string `json:"password,omitempty"`
// +optional
Email string `json:"email,omitempty"`
// +optional
Auth string `json:"auth,omitempty"`
}
// RegistryConfigEntry is a serializable wrapper around credentialconfig.RegistryConfigEntry.
type RegistryConfigEntry struct {
credentialconfig.RegistryConfigEntry
}
// UnmarshalJSON implements the json.Unmarshaler interface.
func (ident *RegistryConfigEntry) UnmarshalJSON(data []byte) error {
var tmp registryConfigEntryWithAuth
err := json.Unmarshal(data, &tmp)
if err != nil {
return err
}
ident.Username = tmp.Username
ident.Password = tmp.Password
ident.Email = tmp.Email
if len(tmp.Auth) == 0 {
return nil
}
ident.Username, ident.Password, err = decodeRegistryConfigFieldAuth(tmp.Auth)
return err
}
// MarshalJSON implements the json.Marshaler interface.
func (ident RegistryConfigEntry) MarshalJSON() ([]byte, error) {
toEncode := registryConfigEntryWithAuth{ident.Username, ident.Password, ident.Email, ""}
toEncode.Auth = encodeRegistryConfigFieldAuth(ident.Username, ident.Password)
return json.Marshal(toEncode)
}
// decodeRegistryConfigFieldAuth deserializes the "auth" field from dockercfg into a
// username and a password. The format of the auth field is base64(<username>:<password>).
func decodeRegistryConfigFieldAuth(field string) (username, password string, err error) {
var decoded []byte
// StdEncoding can only decode padded string
// RawStdEncoding can only decode unpadded string
if strings.HasSuffix(strings.TrimSpace(field), "=") {
// decode padded data
decoded, err = base64.StdEncoding.DecodeString(field)
} else {
// decode unpadded data
decoded, err = base64.RawStdEncoding.DecodeString(field)
}
if err != nil {
return
}
parts := strings.SplitN(string(decoded), ":", 2)
if len(parts) != 2 {
err = fmt.Errorf("unable to parse auth field, must be formatted as base64(username:password)")
return
}
username = parts[0]
password = parts[1]
return
}
func encodeRegistryConfigFieldAuth(username, password string) string {
fieldValue := username + ":" + password
return base64.StdEncoding.EncodeToString([]byte(fieldValue))
}

231
staging/src/k8s.io/legacy-cloud-providers/gce/gcpcredential/registry_marshal_test.go
View File

@@ -1,231 +0,0 @@
/*
Copyright 2021 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gcpcredential
import (
"encoding/base64"
"encoding/json"
"k8s.io/cloud-provider/credentialconfig"
"reflect"
"testing"
)
// Code copied (and edited to replace DockerConfig* with RegistryConfig*) from:
// pkg/credentialprovider/config_test.go.
func TestRegistryConfigEntryJSONDecode(t *testing.T) {
tests := []struct {
input []byte
expect RegistryConfigEntry
fail bool
}{
// simple case, just decode the fields
{
// Fake values for testing.
input: []byte(`{"username": "foo", "password": "bar", "email": "foo@example.com"}`),
expect: RegistryConfigEntry{
credentialconfig.RegistryConfigEntry{
Username: "foo",
Password: "bar",
Email: "foo@example.com",
},
},
fail: false,
},
// auth field decodes to username & password
{
input: []byte(`{"auth": "Zm9vOmJhcg==", "email": "foo@example.com"}`),
expect: RegistryConfigEntry{
credentialconfig.RegistryConfigEntry{
Username: "foo",
Password: "bar",
Email: "foo@example.com",
},
},
fail: false,
},
// auth field overrides username & password
{
// Fake values for testing.
input: []byte(`{"username": "foo", "password": "bar", "auth": "cGluZzpwb25n", "email": "foo@example.com"}`),
expect: RegistryConfigEntry{
credentialconfig.RegistryConfigEntry{
Username: "ping",
Password: "pong",
Email: "foo@example.com",
},
},
fail: false,
},
// poorly-formatted auth causes failure
{
input: []byte(`{"auth": "pants", "email": "foo@example.com"}`),
expect: RegistryConfigEntry{
credentialconfig.RegistryConfigEntry{
Username: "",
Password: "",
Email: "foo@example.com",
},
},
fail: true,
},
// invalid JSON causes failure
{
input: []byte(`{"email": false}`),
expect: RegistryConfigEntry{
credentialconfig.RegistryConfigEntry{
Username: "",
Password: "",
Email: "",
},
},
fail: true,
},
}
for i, tt := range tests {
var output RegistryConfigEntry
err := json.Unmarshal(tt.input, &output)
if (err != nil) != tt.fail {
t.Errorf("case %d: expected fail=%t, got err=%v", i, tt.fail, err)
}
if !reflect.DeepEqual(tt.expect, output) {
t.Errorf("case %d: expected output %#v, got %#v", i, tt.expect, output)
}
}
}
func TestDecodeRegistryConfigFieldAuth(t *testing.T) {
tests := []struct {
input string
username string
password string
fail bool
}{
// auth field decodes to username & password
{
input: "Zm9vOmJhcg==",
username: "foo",
password: "bar",
},
// some test as before but with field not well padded
{
input: "Zm9vOmJhcg",
username: "foo",
password: "bar",
},
// some test as before but with new line characters
{
input: "Zm9vOm\nJhcg==\n",
username: "foo",
password: "bar",
},
// standard encoding (with padding)
{
input: base64.StdEncoding.EncodeToString([]byte("foo:bar")),
username: "foo",
password: "bar",
},
// raw encoding (without padding)
{
input: base64.RawStdEncoding.EncodeToString([]byte("foo:bar")),
username: "foo",
password: "bar",
},
// the input is encoded with encodeRegistryConfigFieldAuth (standard encoding)
{
input: encodeRegistryConfigFieldAuth("foo", "bar"),
username: "foo",
password: "bar",
},
// good base64 data, but no colon separating username & password
{
input: "cGFudHM=",
fail: true,
},
// only new line characters are ignored
{
input: "Zm9vOmJhcg== ",
fail: true,
},
// bad base64 data
{
input: "pants",
fail: true,
},
}
for i, tt := range tests {
username, password, err := decodeRegistryConfigFieldAuth(tt.input)
if (err != nil) != tt.fail {
t.Errorf("case %d: expected fail=%t, got err=%v", i, tt.fail, err)
}
if tt.username != username {
t.Errorf("case %d: expected username %q, got %q", i, tt.username, username)
}
if tt.password != password {
t.Errorf("case %d: expected password %q, got %q", i, tt.password, password)
}
}
}
func TestRegistryConfigEntryJSONCompatibleEncode(t *testing.T) {
tests := []struct {
input RegistryConfigEntry
expect []byte
}{
// simple case, just decode the fields
{
// Fake values for testing.
expect: []byte(`{"username":"foo","password":"bar","email":"foo@example.com","auth":"Zm9vOmJhcg=="}`),
input: RegistryConfigEntry{
credentialconfig.RegistryConfigEntry{
Username: "foo",
Password: "bar",
Email: "foo@example.com",
},
},
},
}
for i, tt := range tests {
actual, err := json.Marshal(tt.input)
if err != nil {
t.Errorf("case %d: unexpected error: %v", i, err)
}
if string(tt.expect) != string(actual) {
t.Errorf("case %d: expected %v, got %v", i, string(tt.expect), string(actual))
}
}
}

111
staging/src/k8s.io/legacy-cloud-providers/gce/metrics.go
View File

@@ -1,111 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2014 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
"time"
"k8s.io/component-base/metrics"
"k8s.io/component-base/metrics/legacyregistry"
)
const (
// Version strings for recording metrics.
computeV1Version = "v1"
computeAlphaVersion = "alpha"
computeBetaVersion = "beta"
)
type apiCallMetrics struct {
latency *metrics.HistogramVec
errors *metrics.CounterVec
}
var (
metricLabels = []string{
"request", // API function that is begin invoked.
"region", // region (optional).
"zone", // zone (optional).
"version", // API version.
}
apiMetrics = registerAPIMetrics()
)
type metricContext struct {
start time.Time
// The cardinalities of attributes and metricLabels (defined above) must
// match, or prometheus will panic.
attributes []string
}
// Value for an unused label in the metric dimension.
const unusedMetricLabel = "<n/a>"
// Observe the result of a API call.
func (mc *metricContext) Observe(err error) error {
apiMetrics.latency.WithLabelValues(mc.attributes...).Observe(
time.Since(mc.start).Seconds())
if err != nil {
apiMetrics.errors.WithLabelValues(mc.attributes...).Inc()
}
return err
}
func newGenericMetricContext(prefix, request, region, zone, version string) *metricContext {
if len(zone) == 0 {
zone = unusedMetricLabel
}
if len(region) == 0 {
region = unusedMetricLabel
}
return &metricContext{
start: time.Now(),
attributes: []string{prefix + "_" + request, region, zone, version},
}
}
// registerApiMetrics adds metrics definitions for a category of API calls.
func registerAPIMetrics() *apiCallMetrics {
metrics := &apiCallMetrics{
latency: metrics.NewHistogramVec(
&metrics.HistogramOpts{
Name: "cloudprovider_gce_api_request_duration_seconds",
Help: "Latency of a GCE API call",
StabilityLevel: metrics.ALPHA,
},
metricLabels,
),
errors: metrics.NewCounterVec(
&metrics.CounterOpts{
Name: "cloudprovider_gce_api_request_errors",
Help: "Number of errors for an API call",
StabilityLevel: metrics.ALPHA,
},
metricLabels,
),
}
legacyregistry.MustRegister(metrics.latency)
legacyregistry.MustRegister(metrics.errors)
return metrics
}

31
staging/src/k8s.io/legacy-cloud-providers/gce/metrics_test.go
View File

@@ -1,31 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
"testing"
"github.com/stretchr/testify/assert"
)
func TestVerifyMetricLabelCardinality(t *testing.T) {
mc := newGenericMetricContext("foo", "get", "us-central1", "<n/a>", "alpha")
assert.Len(t, mc.attributes, len(metricLabels), "cardinalities of labels and values must match")
}

78
staging/src/k8s.io/legacy-cloud-providers/gce/support.go
View File

@@ -1,78 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
"context"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud"
"github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud/meta"
)
// gceProjectRouter sends requests to the appropriate project ID.
type gceProjectRouter struct {
gce *Cloud
}
// ProjectID returns the project ID to be used for the given operation.
func (r *gceProjectRouter) ProjectID(ctx context.Context, version meta.Version, service string) string {
switch service {
case "Firewalls", "Routes":
return r.gce.NetworkProjectID()
default:
return r.gce.projectID
}
}
// gceRateLimiter implements cloud.RateLimiter.
type gceRateLimiter struct {
gce *Cloud
}
// Accept blocks until the operation can be performed.
//
// TODO: the current cloud provider policy doesn't seem to be correct as it
// only rate limits the polling operations, but not the /submission/ of
// operations.
func (l *gceRateLimiter) Accept(ctx context.Context, key *cloud.RateLimitKey) error {
if key.Operation == "Get" && key.Service == "Operations" {
// Wait a minimum amount of time regardless of rate limiter.
rl := &cloud.MinimumRateLimiter{
// Convert flowcontrol.RateLimiter into cloud.RateLimiter
RateLimiter: &cloud.AcceptRateLimiter{
Acceptor: l.gce.operationPollRateLimiter,
},
Minimum: operationPollInterval,
}
return rl.Accept(ctx, key)
}
return nil
}
// CreateGCECloudWithCloud is a helper function to create an instance of Cloud with the
// given Cloud interface implementation. Typical usage is to use cloud.NewMockGCE to get a
// handle to a mock Cloud instance and then use that for testing.
func CreateGCECloudWithCloud(config *CloudConfig, c cloud.Cloud) (*Cloud, error) {
gceCloud, err := CreateGCECloud(config)
if err == nil {
gceCloud.c = c
}
return gceCloud, err
}

130
staging/src/k8s.io/legacy-cloud-providers/gce/token_source.go
View File

@@ -1,130 +0,0 @@
//go:build !providerless
// +build !providerless
/*
Copyright 2015 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package gce
import (
"context"
"encoding/json"
"net/http"
"strings"
"time"
"golang.org/x/oauth2"
"golang.org/x/oauth2/google"
"google.golang.org/api/googleapi"
"k8s.io/client-go/util/flowcontrol"
"k8s.io/component-base/metrics"
"k8s.io/component-base/metrics/legacyregistry"
)
const (
// Max QPS to allow through to the token URL.
tokenURLQPS = .05 // back off to once every 20 seconds when failing
// Maximum burst of requests to token URL before limiting.
tokenURLBurst = 3
)
/*
* By default, all the following metrics are defined as falling under
* ALPHA stability level https://github.com/kubernetes/enhancements/blob/master/keps/sig-instrumentation/1209-metrics-stability/kubernetes-control-plane-metrics-stability.md#stability-classes)
*
* Promoting the stability level of the metric is a responsibility of the component owner, since it
* involves explicitly acknowledging support for the metric across multiple releases, in accordance with
* the metric stability policy.
*/
var (
getTokenCounter = metrics.NewCounter(
&metrics.CounterOpts{
Name: "get_token_count",
Help: "Counter of total Token() requests to the alternate token source",
StabilityLevel: metrics.ALPHA,
},
)
getTokenFailCounter = metrics.NewCounter(
&metrics.CounterOpts{
Name: "get_token_fail_count",
Help: "Counter of failed Token() requests to the alternate token source",
StabilityLevel: metrics.ALPHA,
},
)
)
func init() {
legacyregistry.MustRegister(getTokenCounter)
legacyregistry.MustRegister(getTokenFailCounter)
}
// AltTokenSource is the structure holding the data for the functionality needed to generates tokens
type AltTokenSource struct {
oauthClient *http.Client
tokenURL string
tokenBody string `datapolicy:"token"`
throttle flowcontrol.RateLimiter
}
// Token returns a token which may be used for authentication
func (a *AltTokenSource) Token() (*oauth2.Token, error) {
a.throttle.Accept()
getTokenCounter.Inc()
t, err := a.token()
if err != nil {
getTokenFailCounter.Inc()
}
return t, err
}
func (a *AltTokenSource) token() (*oauth2.Token, error) {
req, err := http.NewRequest("POST", a.tokenURL, strings.NewReader(a.tokenBody))
if err != nil {
return nil, err
}
res, err := a.oauthClient.Do(req)
if err != nil {
return nil, err
}
defer res.Body.Close()
if err := googleapi.CheckResponse(res); err != nil {
return nil, err
}
var tok struct {
AccessToken string `json:"accessToken" datapolicy:"token"`
ExpireTime time.Time `json:"expireTime"`
}
if err := json.NewDecoder(res.Body).Decode(&tok); err != nil {
return nil, err
}
return &oauth2.Token{
AccessToken: tok.AccessToken,
Expiry: tok.ExpireTime,
}, nil
}
// NewAltTokenSource constructs a new alternate token source for generating tokens.
func NewAltTokenSource(tokenURL, tokenBody string) oauth2.TokenSource {
client := oauth2.NewClient(context.Background(), google.ComputeTokenSource(""))
a := &AltTokenSource{
oauthClient: client,
tokenURL: tokenURL,
tokenBody: tokenBody,
throttle: flowcontrol.NewTokenBucketRateLimiter(tokenURLQPS, tokenURLBurst),
}
return oauth2.ReuseTokenSource(nil, a)
}

654
test/e2e/autoscaling/custom_metrics_stackdriver_autoscaling.go
View File

@@ -1,654 +0,0 @@
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package autoscaling
import (
"context"
"fmt"
"math"
"strings"
"time"
gcm "google.golang.org/api/monitoring/v3"
"google.golang.org/api/option"
appsv1 "k8s.io/api/apps/v1"
autoscalingv2 "k8s.io/api/autoscaling/v2"
v1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/resource"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/util/wait"
clientset "k8s.io/client-go/kubernetes"
"k8s.io/kubernetes/test/e2e/feature"
"k8s.io/kubernetes/test/e2e/framework"
e2edeployment "k8s.io/kubernetes/test/e2e/framework/deployment"
e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper"
"k8s.io/kubernetes/test/e2e/instrumentation/monitoring"
admissionapi "k8s.io/pod-security-admission/api"
"github.com/onsi/ginkgo/v2"
"golang.org/x/oauth2/google"
)
const (
stackdriverExporterDeployment = "stackdriver-exporter-deployment"
dummyDeploymentName = "dummy-deployment"
stackdriverExporterPod = "stackdriver-exporter-pod"
externalMetricValue = int64(85)
)
type externalMetricTarget struct {
value int64
isAverage bool
}
var _ = SIGDescribe("[HPA]", feature.CustomMetricsAutoscaling, "Horizontal pod autoscaling (scale resource: Custom Metrics from Stackdriver)", func() {
ginkgo.BeforeEach(func() {
e2eskipper.SkipUnlessProviderIs("gce", "gke")
})
f := framework.NewDefaultFramework("horizontal-pod-autoscaling")
f.NamespacePodSecurityLevel = admissionapi.LevelPrivileged
ginkgo.Describe("with Custom Metric of type Pod from Stackdriver", func() {
ginkgo.It("should scale down", func(ctx context.Context) {
initialReplicas := 2
// metric should cause scale down
metricValue := int64(100)
metricTarget := 2 * metricValue
metricSpecs := []autoscalingv2.MetricSpec{
podMetricSpecWithAverageValueTarget(monitoring.CustomMetricName, metricTarget),
}
tc := CustomMetricTestCase{
framework: f,
kubeClient: f.ClientSet,
initialReplicas: initialReplicas,
scaledReplicas: 1,
deployment: monitoring.SimpleStackdriverExporterDeployment(stackdriverExporterDeployment, f.Namespace.ObjectMeta.Name, int32(initialReplicas), metricValue),
hpa: hpa("custom-metrics-pods-hpa", f.Namespace.ObjectMeta.Name, stackdriverExporterDeployment, 1, 3, metricSpecs),
}
tc.Run(ctx)
})
ginkgo.It("should scale up with two metrics", func(ctx context.Context) {
initialReplicas := 1
// metric 1 would cause a scale down, if not for metric 2
metric1Value := int64(100)
metric1Target := 2 * metric1Value
// metric2 should cause a scale up
metric2Value := int64(200)
metric2Target := int64(0.5 * float64(metric2Value))
metricSpecs := []autoscalingv2.MetricSpec{
podMetricSpecWithAverageValueTarget("metric1", metric1Target),
podMetricSpecWithAverageValueTarget("metric2", metric2Target),
}
containers := []monitoring.CustomMetricContainerSpec{
{
Name: "stackdriver-exporter-metric1",
MetricName: "metric1",
MetricValue: metric1Value,
},
{
Name: "stackdriver-exporter-metric2",
MetricName: "metric2",
MetricValue: metric2Value,
},
}
tc := CustomMetricTestCase{
framework: f,
kubeClient: f.ClientSet,
initialReplicas: initialReplicas,
scaledReplicas: 3,
deployment: monitoring.StackdriverExporterDeployment(stackdriverExporterDeployment, f.Namespace.ObjectMeta.Name, int32(initialReplicas), containers),
hpa: hpa("custom-metrics-pods-hpa", f.Namespace.ObjectMeta.Name, stackdriverExporterDeployment, 1, 3, metricSpecs),
}
tc.Run(ctx)
})
ginkgo.It("should scale down with Prometheus", func(ctx context.Context) {
initialReplicas := 2
// metric should cause scale down
metricValue := int64(100)
metricTarget := 2 * metricValue
metricSpecs := []autoscalingv2.MetricSpec{
podMetricSpecWithAverageValueTarget(monitoring.CustomMetricName, metricTarget),
}
tc := CustomMetricTestCase{
framework: f,
kubeClient: f.ClientSet,
initialReplicas: initialReplicas,
scaledReplicas: 1,
deployment: monitoring.PrometheusExporterDeployment(stackdriverExporterDeployment, f.Namespace.ObjectMeta.Name, int32(initialReplicas), metricValue),
hpa: hpa("custom-metrics-pods-hpa", f.Namespace.ObjectMeta.Name, stackdriverExporterDeployment, 1, 3, metricSpecs),
}
tc.Run(ctx)
})
})
ginkgo.Describe("with Custom Metric of type Object from Stackdriver", func() {
ginkgo.It("should scale down", func(ctx context.Context) {
initialReplicas := 2
// metric should cause scale down
metricValue := int64(100)
metricTarget := 2 * metricValue
metricSpecs := []autoscalingv2.MetricSpec{
objectMetricSpecWithValueTarget(metricTarget),
}
tc := CustomMetricTestCase{
framework: f,
kubeClient: f.ClientSet,
initialReplicas: initialReplicas,
scaledReplicas: 1,
deployment: noExporterDeployment(dummyDeploymentName, f.Namespace.ObjectMeta.Name, int32(initialReplicas)),
pod: monitoring.StackdriverExporterPod(stackdriverExporterPod, f.Namespace.Name, stackdriverExporterPod, monitoring.CustomMetricName, metricValue),
hpa: hpa("custom-metrics-objects-hpa", f.Namespace.ObjectMeta.Name, dummyDeploymentName, 1, 3, metricSpecs),
}
tc.Run(ctx)
})
ginkgo.It("should scale down to 0", func(ctx context.Context) {
initialReplicas := 2
// metric should cause scale down
metricValue := int64(0)
metricTarget := int64(200)
metricSpecs := []autoscalingv2.MetricSpec{
objectMetricSpecWithValueTarget(metricTarget),
}
tc := CustomMetricTestCase{
framework: f,
kubeClient: f.ClientSet,
initialReplicas: initialReplicas,
scaledReplicas: 0,
deployment: noExporterDeployment(dummyDeploymentName, f.Namespace.ObjectMeta.Name, int32(initialReplicas)),
pod: monitoring.StackdriverExporterPod(stackdriverExporterPod, f.Namespace.Name, stackdriverExporterPod, monitoring.CustomMetricName, metricValue),
hpa: hpa("custom-metrics-objects-hpa", f.Namespace.ObjectMeta.Name, dummyDeploymentName, 0, 3, metricSpecs),
}
tc.Run(ctx)
})
})
ginkgo.Describe("with External Metric from Stackdriver", func() {
ginkgo.It("should scale down with target value", func(ctx context.Context) {
initialReplicas := 2
// metric should cause scale down
metricValue := externalMetricValue
metricTarget := 3 * metricValue
metricSpecs := []autoscalingv2.MetricSpec{
externalMetricSpecWithTarget("target", f.Namespace.ObjectMeta.Name, externalMetricTarget{
value: metricTarget,
isAverage: false,
}),
}
tc := CustomMetricTestCase{
framework: f,
kubeClient: f.ClientSet,
initialReplicas: initialReplicas,
scaledReplicas: 1,
deployment: noExporterDeployment(dummyDeploymentName, f.Namespace.ObjectMeta.Name, int32(initialReplicas)),
pod: monitoring.StackdriverExporterPod(stackdriverExporterPod, f.Namespace.Name, stackdriverExporterPod, "target", metricValue),
hpa: hpa("custom-metrics-external-hpa", f.Namespace.ObjectMeta.Name, dummyDeploymentName, 1, 3, metricSpecs),
}
tc.Run(ctx)
})
ginkgo.It("should scale down with target average value", func(ctx context.Context) {
initialReplicas := 2
// metric should cause scale down
metricValue := externalMetricValue
metricAverageTarget := 3 * metricValue
metricSpecs := []autoscalingv2.MetricSpec{
externalMetricSpecWithTarget("target_average", f.Namespace.ObjectMeta.Name, externalMetricTarget{
value: metricAverageTarget,
isAverage: true,
}),
}
tc := CustomMetricTestCase{
framework: f,
kubeClient: f.ClientSet,
initialReplicas: initialReplicas,
scaledReplicas: 1,
deployment: noExporterDeployment(dummyDeploymentName, f.Namespace.ObjectMeta.Name, int32(initialReplicas)),
pod: monitoring.StackdriverExporterPod(stackdriverExporterPod, f.Namespace.Name, stackdriverExporterPod, "target_average", externalMetricValue),
hpa: hpa("custom-metrics-external-hpa", f.Namespace.ObjectMeta.Name, dummyDeploymentName, 1, 3, metricSpecs),
}
tc.Run(ctx)
})
ginkgo.It("should scale up with two metrics", func(ctx context.Context) {
initialReplicas := 1
// metric 1 would cause a scale down, if not for metric 2
metric1Value := externalMetricValue
metric1Target := 2 * metric1Value
// metric2 should cause a scale up
metric2Value := externalMetricValue
metric2Target := int64(math.Ceil(0.5 * float64(metric2Value)))
metricSpecs := []autoscalingv2.MetricSpec{
externalMetricSpecWithTarget("external_metric_1", f.Namespace.ObjectMeta.Name, externalMetricTarget{
value: metric1Target,
isAverage: true,
}),
externalMetricSpecWithTarget("external_metric_2", f.Namespace.ObjectMeta.Name, externalMetricTarget{
value: metric2Target,
isAverage: true,
}),
}
containers := []monitoring.CustomMetricContainerSpec{
{
Name: "stackdriver-exporter-metric1",
MetricName: "external_metric_1",
MetricValue: metric1Value,
},
{
Name: "stackdriver-exporter-metric2",
MetricName: "external_metric_2",
MetricValue: metric2Value,
},
}
tc := CustomMetricTestCase{
framework: f,
kubeClient: f.ClientSet,
initialReplicas: initialReplicas,
scaledReplicas: 3,
deployment: monitoring.StackdriverExporterDeployment(dummyDeploymentName, f.Namespace.ObjectMeta.Name, int32(initialReplicas), containers),
hpa: hpa("custom-metrics-external-hpa", f.Namespace.ObjectMeta.Name, dummyDeploymentName, 1, 3, metricSpecs),
}
tc.Run(ctx)
})
})
ginkgo.Describe("with multiple metrics of different types", func() {
ginkgo.It("should scale up when one metric is missing (Pod and External metrics)", func(ctx context.Context) {
initialReplicas := 1
// First metric a pod metric which is missing.
// Second metric is external metric which is present, it should cause scale up.
metricSpecs := []autoscalingv2.MetricSpec{
podMetricSpecWithAverageValueTarget(monitoring.CustomMetricName, 2*externalMetricValue),
externalMetricSpecWithTarget("external_metric", f.Namespace.ObjectMeta.Name, externalMetricTarget{
value: int64(math.Ceil(0.5 * float64(externalMetricValue))),
isAverage: true,
}),
}
containers := []monitoring.CustomMetricContainerSpec{
{
Name: "stackdriver-exporter-metric",
MetricName: "external_metric",
MetricValue: externalMetricValue,
},
// Pod Resource metric is missing from here.
}
tc := CustomMetricTestCase{
framework: f,
kubeClient: f.ClientSet,
initialReplicas: initialReplicas,
scaledReplicas: 3,
deployment: monitoring.StackdriverExporterDeployment(dummyDeploymentName, f.Namespace.ObjectMeta.Name, int32(initialReplicas), containers),
hpa: hpa("multiple-metrics", f.Namespace.ObjectMeta.Name, dummyDeploymentName, 1, 3, metricSpecs)}
tc.Run(ctx)
})
ginkgo.It("should scale up when one metric is missing (Resource and Object metrics)", func(ctx context.Context) {
initialReplicas := 1
metricValue := int64(100)
// First metric a resource metric which is missing (no consumption).
// Second metric is object metric which is present, it should cause scale up.
metricSpecs := []autoscalingv2.MetricSpec{
resourceMetricSpecWithAverageUtilizationTarget(50),
objectMetricSpecWithValueTarget(int64(math.Ceil(0.5 * float64(metricValue)))),
}
tc := CustomMetricTestCase{
framework: f,
kubeClient: f.ClientSet,
initialReplicas: initialReplicas,
scaledReplicas: 3,
deployment: monitoring.SimpleStackdriverExporterDeployment(dummyDeploymentName, f.Namespace.ObjectMeta.Name, int32(initialReplicas), 0),
pod: monitoring.StackdriverExporterPod(stackdriverExporterPod, f.Namespace.Name, stackdriverExporterPod, monitoring.CustomMetricName, metricValue),
hpa: hpa("multiple-metrics", f.Namespace.ObjectMeta.Name, dummyDeploymentName, 1, 3, metricSpecs)}
tc.Run(ctx)
})
ginkgo.It("should not scale down when one metric is missing (Container Resource and External Metrics)", func(ctx context.Context) {
initialReplicas := 2
// First metric a container resource metric which is missing.
// Second metric is external metric which is present, it should cause scale down if the first metric wasn't missing.
metricSpecs := []autoscalingv2.MetricSpec{
containerResourceMetricSpecWithAverageUtilizationTarget("container-resource-metric", 50),
externalMetricSpecWithTarget("external_metric", f.Namespace.ObjectMeta.Name, externalMetricTarget{
value: 2 * externalMetricValue,
isAverage: true,
}),
}
containers := []monitoring.CustomMetricContainerSpec{
{
Name: "stackdriver-exporter-metric",
MetricName: "external_metric",
MetricValue: externalMetricValue,
},
// Container Resource metric is missing from here.
}
tc := CustomMetricTestCase{
framework: f,
kubeClient: f.ClientSet,
initialReplicas: initialReplicas,
scaledReplicas: initialReplicas,
verifyStability: true,
deployment: monitoring.StackdriverExporterDeployment(dummyDeploymentName, f.Namespace.ObjectMeta.Name, int32(initialReplicas), containers),
hpa: hpa("multiple-metrics", f.Namespace.ObjectMeta.Name, dummyDeploymentName, 1, 3, metricSpecs)}
tc.Run(ctx)
})
ginkgo.It("should not scale down when one metric is missing (Pod and Object Metrics)", func(ctx context.Context) {
initialReplicas := 2
metricValue := int64(100)
// First metric an object metric which is missing.
// Second metric is pod metric which is present, it should cause scale down if the first metric wasn't missing.
metricSpecs := []autoscalingv2.MetricSpec{
objectMetricSpecWithValueTarget(int64(math.Ceil(0.5 * float64(metricValue)))),
podMetricSpecWithAverageValueTarget("pod_metric", 2*metricValue),
}
containers := []monitoring.CustomMetricContainerSpec{
{
Name: "stackdriver-exporter-metric",
MetricName: "pod_metric",
MetricValue: metricValue,
},
}
tc := CustomMetricTestCase{
framework: f,
kubeClient: f.ClientSet,
initialReplicas: initialReplicas,
scaledReplicas: initialReplicas,
verifyStability: true,
deployment: monitoring.StackdriverExporterDeployment(dummyDeploymentName, f.Namespace.ObjectMeta.Name, int32(initialReplicas), containers),
hpa: hpa("multiple-metrics", f.Namespace.ObjectMeta.Name, dummyDeploymentName, 1, 3, metricSpecs)}
tc.Run(ctx)
})
})
})
// CustomMetricTestCase is a struct for test cases.
type CustomMetricTestCase struct {
framework *framework.Framework
hpa *autoscalingv2.HorizontalPodAutoscaler
kubeClient clientset.Interface
deployment *appsv1.Deployment
pod *v1.Pod
initialReplicas int
scaledReplicas int
verifyStability bool
}
// Run starts test case.
func (tc *CustomMetricTestCase) Run(ctx context.Context) {
projectID := framework.TestContext.CloudConfig.ProjectID
client, err := google.DefaultClient(ctx, gcm.CloudPlatformScope)
if err != nil {
framework.Failf("Failed to initialize gcm default client, %v", err)
}
// Hack for running tests locally, needed to authenticate in Stackdriver
// If this is your use case, create application default credentials:
// $ gcloud auth application-default login
// and uncomment following lines:
// ts, err := google.DefaultTokenSource(oauth2.NoContext)
// framework.Logf("Couldn't get application default credentials, %v", err)
// if err != nil {
// framework.Failf("Error accessing application default credentials, %v", err)
// }
// client = oauth2.NewClient(oauth2.NoContext, ts)
gcmService, err := gcm.NewService(ctx, option.WithHTTPClient(client))
if err != nil {
framework.Failf("Failed to create gcm service, %v", err)
}
// Set up a cluster: create a custom metric and set up k8s-sd adapter
err = monitoring.CreateDescriptors(gcmService, projectID)
if err != nil {
if strings.Contains(err.Error(), "Request throttled") {
e2eskipper.Skipf("Skipping...hitting rate limits on creating and updating metrics/labels")
}
framework.Failf("Failed to create metric descriptor: %v", err)
}
defer monitoring.CleanupDescriptors(gcmService, projectID)
err = monitoring.CreateAdapter(monitoring.AdapterDefault)
defer monitoring.CleanupAdapter(monitoring.AdapterDefault)
if err != nil {
framework.Failf("Failed to set up: %v", err)
}
// Run application that exports the metric
err = createDeploymentToScale(ctx, tc.framework, tc.kubeClient, tc.deployment, tc.pod)
if err != nil {
framework.Failf("Failed to create stackdriver-exporter pod: %v", err)
}
ginkgo.DeferCleanup(cleanupDeploymentsToScale, tc.framework, tc.kubeClient, tc.deployment, tc.pod)
// Wait for the deployment to run
waitForReplicas(ctx, tc.deployment.ObjectMeta.Name, tc.framework.Namespace.ObjectMeta.Name, tc.kubeClient, 15*time.Minute, tc.initialReplicas)
// Autoscale the deployment
_, err = tc.kubeClient.AutoscalingV2().HorizontalPodAutoscalers(tc.framework.Namespace.ObjectMeta.Name).Create(ctx, tc.hpa, metav1.CreateOptions{})
if err != nil {
framework.Failf("Failed to create HPA: %v", err)
}
ginkgo.DeferCleanup(framework.IgnoreNotFound(tc.kubeClient.AutoscalingV2().HorizontalPodAutoscalers(tc.framework.Namespace.ObjectMeta.Name).Delete), tc.hpa.ObjectMeta.Name, metav1.DeleteOptions{})
waitForReplicas(ctx, tc.deployment.ObjectMeta.Name, tc.framework.Namespace.ObjectMeta.Name, tc.kubeClient, 15*time.Minute, tc.scaledReplicas)
if tc.verifyStability {
ensureDesiredReplicasInRange(ctx, tc.deployment.ObjectMeta.Name, tc.framework.Namespace.ObjectMeta.Name, tc.kubeClient, tc.scaledReplicas, tc.scaledReplicas, 10*time.Minute)
}
}
func createDeploymentToScale(ctx context.Context, f *framework.Framework, cs clientset.Interface, deployment *appsv1.Deployment, pod *v1.Pod) error {
if deployment != nil {
_, err := cs.AppsV1().Deployments(f.Namespace.ObjectMeta.Name).Create(ctx, deployment, metav1.CreateOptions{})
if err != nil {
return err
}
}
if pod != nil {
_, err := cs.CoreV1().Pods(f.Namespace.ObjectMeta.Name).Create(ctx, pod, metav1.CreateOptions{})
if err != nil {
return err
}
}
return nil
}
func cleanupDeploymentsToScale(ctx context.Context, f *framework.Framework, cs clientset.Interface, deployment *appsv1.Deployment, pod *v1.Pod) {
if deployment != nil {
_ = cs.AppsV1().Deployments(f.Namespace.ObjectMeta.Name).Delete(ctx, deployment.ObjectMeta.Name, metav1.DeleteOptions{})
}
if pod != nil {
_ = cs.CoreV1().Pods(f.Namespace.ObjectMeta.Name).Delete(ctx, pod.ObjectMeta.Name, metav1.DeleteOptions{})
}
}
func podMetricSpecWithAverageValueTarget(metric string, targetValue int64) autoscalingv2.MetricSpec {
return autoscalingv2.MetricSpec{
Type: autoscalingv2.PodsMetricSourceType,
Pods: &autoscalingv2.PodsMetricSource{
Metric: autoscalingv2.MetricIdentifier{
Name: metric,
},
Target: autoscalingv2.MetricTarget{
Type: autoscalingv2.AverageValueMetricType,
AverageValue: resource.NewQuantity(targetValue, resource.DecimalSI),
},
},
}
}
func objectMetricSpecWithValueTarget(targetValue int64) autoscalingv2.MetricSpec {
return autoscalingv2.MetricSpec{
Type: autoscalingv2.ObjectMetricSourceType,
Object: &autoscalingv2.ObjectMetricSource{
Metric: autoscalingv2.MetricIdentifier{
Name: monitoring.CustomMetricName,
},
DescribedObject: autoscalingv2.CrossVersionObjectReference{
Kind: "Pod",
Name: stackdriverExporterPod,
},
Target: autoscalingv2.MetricTarget{
Type: autoscalingv2.ValueMetricType,
Value: resource.NewQuantity(targetValue, resource.DecimalSI),
},
},
}
}
func resourceMetricSpecWithAverageUtilizationTarget(targetValue int32) autoscalingv2.MetricSpec {
return autoscalingv2.MetricSpec{
Type: autoscalingv2.ResourceMetricSourceType,
Resource: &autoscalingv2.ResourceMetricSource{
Name: v1.ResourceCPU,
Target: autoscalingv2.MetricTarget{
Type: autoscalingv2.UtilizationMetricType,
AverageUtilization: &targetValue,
},
},
}
}
func containerResourceMetricSpecWithAverageUtilizationTarget(containerName string, targetValue int32) autoscalingv2.MetricSpec {
return autoscalingv2.MetricSpec{
Type: autoscalingv2.ContainerResourceMetricSourceType,
ContainerResource: &autoscalingv2.ContainerResourceMetricSource{
Name: v1.ResourceCPU,
Container: containerName,
Target: autoscalingv2.MetricTarget{
Type: autoscalingv2.UtilizationMetricType,
AverageUtilization: &targetValue,
},
},
}
}
func externalMetricSpecWithTarget(metric string, namespace string, target externalMetricTarget) autoscalingv2.MetricSpec {
selector := &metav1.LabelSelector{
MatchLabels: map[string]string{"resource.type": "k8s_pod"},
MatchExpressions: []metav1.LabelSelectorRequirement{
{
Key: "resource.labels.namespace_name",
Operator: metav1.LabelSelectorOpIn,
Values: []string{namespace},
},
{
Key: "resource.labels.pod_name",
Operator: metav1.LabelSelectorOpExists,
Values: []string{},
},
},
}
metricSpec := autoscalingv2.MetricSpec{
Type: autoscalingv2.ExternalMetricSourceType,
External: &autoscalingv2.ExternalMetricSource{
Metric: autoscalingv2.MetricIdentifier{
Name: "custom.googleapis.com|" + metric,
Selector: selector,
},
},
}
if target.isAverage {
metricSpec.External.Target.Type = autoscalingv2.AverageValueMetricType
metricSpec.External.Target.AverageValue = resource.NewQuantity(target.value, resource.DecimalSI)
} else {
metricSpec.External.Target.Type = autoscalingv2.ValueMetricType
metricSpec.External.Target.Value = resource.NewQuantity(target.value, resource.DecimalSI)
}
return metricSpec
}
func hpa(name, namespace, deploymentName string, minReplicas, maxReplicas int32, metricSpecs []autoscalingv2.MetricSpec) *autoscalingv2.HorizontalPodAutoscaler {
return &autoscalingv2.HorizontalPodAutoscaler{
ObjectMeta: metav1.ObjectMeta{
Name: name,
Namespace: namespace,
},
Spec: autoscalingv2.HorizontalPodAutoscalerSpec{
Metrics: metricSpecs,
MinReplicas: &minReplicas,
MaxReplicas: maxReplicas,
ScaleTargetRef: autoscalingv2.CrossVersionObjectReference{
APIVersion: "apps/v1",
Kind: "Deployment",
Name: deploymentName,
},
},
}
}
func waitForReplicas(ctx context.Context, deploymentName, namespace string, cs clientset.Interface, timeout time.Duration, desiredReplicas int) {
interval := 20 * time.Second
err := wait.PollUntilContextTimeout(ctx, interval, timeout, true, func(ctx context.Context) (bool, error) {
deployment, err := cs.AppsV1().Deployments(namespace).Get(ctx, deploymentName, metav1.GetOptions{})
if err != nil {
framework.Failf("Failed to get replication controller %s: %v", deployment, err)
}
replicas := int(deployment.Status.ReadyReplicas)
framework.Logf("waiting for %d replicas (current: %d)", desiredReplicas, replicas)
return replicas == desiredReplicas, nil // Expected number of replicas found. Exit.
})
if err != nil {
framework.Failf("Timeout waiting %v for %v replicas", timeout, desiredReplicas)
}
}
func ensureDesiredReplicasInRange(ctx context.Context, deploymentName, namespace string, cs clientset.Interface, minDesiredReplicas, maxDesiredReplicas int, timeout time.Duration) {
interval := 60 * time.Second
err := wait.PollUntilContextTimeout(ctx, interval, timeout, true, func(ctx context.Context) (bool, error) {
deployment, err := cs.AppsV1().Deployments(namespace).Get(ctx, deploymentName, metav1.GetOptions{})
if err != nil {
return true, err
}
replicas := int(deployment.Status.ReadyReplicas)
framework.Logf("expecting there to be in [%d, %d] replicas (are: %d)", minDesiredReplicas, maxDesiredReplicas, replicas)
if replicas < minDesiredReplicas {
return false, fmt.Errorf("number of replicas below target")
} else if replicas > maxDesiredReplicas {
return false, fmt.Errorf("number of replicas above target")
} else {
return false, nil // Expected number of replicas found. Continue polling until timeout.
}
})
// The call above always returns an error, but if it is timeout, it's OK (condition satisfied all the time).
if wait.Interrupted(err) || strings.Contains(err.Error(), "would exceed context deadline") {
framework.Logf("Number of replicas was stable over %v", timeout)
return
}
framework.ExpectNoErrorWithOffset(1, err)
}
func noExporterDeployment(name, namespace string, replicas int32) *appsv1.Deployment {
d := e2edeployment.NewDeployment(name, replicas, map[string]string{"name": name}, "", "", appsv1.RollingUpdateDeploymentStrategyType)
d.ObjectMeta.Namespace = namespace
d.Spec.Template.Spec = v1.PodSpec{Containers: []v1.Container{
{
Name: "sleeper",
Image: "registry.k8s.io/e2e-test-images/agnhost:2.40",
ImagePullPolicy: v1.PullAlways,
Command: []string{"/agnhost"},
Args: []string{"pause"}, // do nothing forever
},
}}
return d
}

Some files were not shown because too many files have changed in this diff Show More