Commit Graph

252 Commits

Author SHA1 Message Date
Justin Santa Barbara
2958ea253a GCE: Allow for reuse of master
This is for internal use at the moment, for testing Ubernetes Lite, but
arguably makes the code a little cleaner.

Also rename KUBE_SHARE_MASTER -> KUBE_USE_EXISTING_MASTER
2016-01-20 15:37:00 -05:00
Josh Ellithorpe
76e8a8b08d Fix issues with Python3 and bring up a dev cluster 2016-01-12 11:08:42 -08:00
Justin Santa Barbara
d91ad0bb6f AWS: Create a cloudwatch alarm to reboot the master on failure
This is an easier alternative to keep a master running than trying to
dynamically find & attach master volumes.

To actually work, it requires that users create the EC2ActionsAccess IAM
role as required by CloudWatch, see e.g.
http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/UsingIAM.html
2016-01-09 14:53:29 -05:00
Justin Santa Barbara
57265a4c74 AWS: Pass NUM_NODES in instance-data
The bootstrap scripts already assume it is set.
2016-01-05 23:10:43 -05:00
Carles Lopez
c8beaac51d Add root device detection to boot correctly 2015-12-17 13:33:00 +01:00
Michael Nikitochkin
2272de1f67 AWS: Fixed compacting script of cloud init for MacOS
In MacOS there is error during setup a new cluster:

```
+ sed -i -e 's/^[[:blank:]]*#.*$//' -e '/^[[:blank:]]*$/d' /sometmpfile
sed: -e: No such file or directory
```

Because sed version of MacOS does not support modern features.
2015-12-16 09:20:57 +01:00
Thomas Ploch
189f2436e3 [AWS] Wrong assignment of KUBE_MASTER_IP with Elastic IP
Currently when using a custom elastic IP, the ENV var `KUBE_MASTER_IP` gets
the output of `$(assign-elastic-ip $ip $master_id)` assigned.

This is wrong since the command returns a string:
`Attaching IP 99.999.999.999 to instance i-9999999`

This patch fixes the assignment by calling `get_instance_public_ip` again.
2015-12-12 13:53:01 +01:00
Justin Santa Barbara
428e4156e2 AWS: Strip more comments from instance user-data
To stay under the 16KB limit
2015-12-07 22:27:33 -05:00
k8s-merge-robot
b69018eab2 Merge pull request #17844 from eosrei/kube-util-minion-node
Auto commit by PR queue bot
2015-12-02 03:06:27 -08:00
k8s-merge-robot
bb86936765 Merge pull request #17803 from davidsiefert/master
Auto commit by PR queue bot
2015-12-01 02:15:11 -08:00
k8s-merge-robot
7bcd7759fa Merge pull request #17814 from themez/master
Auto commit by PR queue bot
2015-12-01 00:22:51 -08:00
Brad Erickson
0bd16e0437 Minion->Node rename: kube-util hosting provider helper functions 2015-11-26 09:31:46 -08:00
ThemeZ
3342e4e1d3 Correct s3 url base for region "cn-north-1" 2015-11-26 11:37:28 +08:00
David Siefert
94d32588f4 Extracting more functions for reuse out of kube-up 2015-11-25 17:29:30 -06:00
Brad Erickson
53172a5356 Minion->Node rename: NUM_NODES 2015-11-25 00:45:10 -08:00
Brad Erickson
e67be19a5b Minion->Node rename: OLD_NODE_TAG, NODE_TAG 2015-11-25 00:45:09 -08:00
Brad Erickson
bd06c19aa8 Minion->Node rename: NODE_SIZE 2015-11-25 00:45:09 -08:00
Brad Erickson
8431993a44 Minion->Node rename: NODE_SG_ID, NODE_SG_NAME 2015-11-25 00:45:09 -08:00
Brad Erickson
1846cfc129 Minion->Node rename: NODE_ROOT_DISK_SIZE, NODE_ROOT_DISK_TYPE, etc
NODE_SCOPES
2015-11-25 00:45:09 -08:00
Brad Erickson
fc04b55088 Minion->Node rename: NODE_NAMES, NODE_NAME, NODE_PORT 2015-11-25 00:45:09 -08:00
Brad Erickson
83ed2fa22e Minion->Node rename: NODE_DISK_SIZE, NODE_DISK_TYPE, NODE_HOSTNAME, etc
NODE_IDS
NODE_ID
NODE_IMAGE_PROJECT
NODE_IMAGE
2015-11-25 00:45:09 -08:00
Brad Erickson
a36d3390bf Minion->Node rename: KUBERNETES_NODE_MEMORY, VAGRANT_NODE_NAMES, etc
ENABLE_NODE_PUBLIC_IP
NODE_ADDRESS
NODE_BLOCK_DEVICE_MAPPINGS
NODE_CONTAINER_ADDRS
NODE_CONTAINER_NETMASKS
NODE_CONTAINER_SUBNET_BASE
NODE_CONTAINER_SUBNETS
NODE_CPU
2015-11-25 00:43:52 -08:00
Brad Erickson
ae314ad246 Minion->Node rename: KUBE_NODE_IP_ADDRESSES, KUBE_NODE_IP_ADDRESS 2015-11-25 00:43:52 -08:00
Brad Erickson
68539ae8a4 Minion->Node rename: KUBE_NODE_IMAGE 2015-11-25 00:43:51 -08:00
Brad Erickson
6b91b45eff Minion->Node rename: IAM_PROFILE_NODE, KUBE_ENABLE_NODE_PUBLIC_IP, etc
KUBE_GCE_NODE_IMAGE, KUBE_GCE_NODE_PROJECT, KUBEMARK_NUM_NODES
2015-11-25 00:43:51 -08:00
k8s-merge-robot
440bdd27d5 Merge pull request #17087 from justinsb/aws_delete_ebs
Auto commit by PR queue bot
2015-11-12 01:17:40 -08:00
Justin Santa Barbara
d5f62ca67b AWS: Don't use JSON parsing in kube-up
We use the AWS CLI support for --query and --filter instead; should be
more reliable and clearer.

Also set the output format to text, so we don't have to set it every
time and don't risk problems if we forget to set it.

Fixes #16747

We do still have to use JSON parsing in one place: ELB does not support
--filter, so we have to use Python there.
2015-11-10 23:30:42 -05:00
Christian Stewart
d61d57adec
Remove ENABLE_EXPERIMENTAL_API and similar in favor of KUBE_RUNTIME_CONFIG.
Addresses #15968

This patch removes KUBE_ENABLE_EXPERIMENTAL_API and similar calls in
favor of specifying desired features in KUBE_RUNTIME_CONFIG. Changes
have also been made to e2e scripts to re-enable using
KUBE_RUNTIME_CONFIG rather than EXPERIMENTAL_API env vars.

This also introduces KUBE_ENABLE_DAEMONSETS and KUBE_ENABLE_DEPLOYMENTS.

Signed-off-by: Christian Stewart <christian@paral.in>
2015-11-10 21:36:34 -05:00
k8s-merge-robot
6fbae33eb1 Merge pull request #16918 from justinsb/aws_fix_kubedown_no_instances
Auto commit by PR queue bot
2015-11-10 11:11:56 -08:00
k8s-merge-robot
157f663f5b Merge pull request #16867 from justinsb/aws_log_upload_urls
Auto commit by PR queue bot
2015-11-07 00:00:28 -08:00
k8s-merge-robot
b719d7a45e Merge pull request #15070 from justinsb/aws_print_region_if_not_matching
Auto commit by PR queue bot
2015-11-06 10:31:29 -08:00
Justin Santa Barbara
84bab0dae5 AWS: kube-down was failing with no instances
The new delete-ASG logic wasn't correct if there were no instances
2015-11-06 10:43:13 -05:00
Justin Santa Barbara
5bde574eca AWS: Log the download URLs after we upload to S3
This is helpful if you want to manually update a cluster.
2015-11-05 14:39:18 -05:00
Justin Santa Barbara
2d5671aec2 AWS: Fix kube-up comments per code review 2015-11-03 13:12:17 -05:00
Justin Santa Barbara
41743ce87a AWS: Delete ASGs by following instances
We can't tag ASGs, but we can see what instances are running in an ASG,
and we can match those by our tags.

So look for our running instances, and look for the ASGs that created
them, and delete those.

This can be defeated (most notably if users change the ASG size to 0),
but it is safer that other deletion methods.
2015-11-03 11:58:05 -05:00
Justin Santa Barbara
cf86ecc604 AWS: Only match minions in the current ASG
i.e. don't assume there is only one ASG any more
2015-11-03 11:58:05 -05:00
Justin Santa Barbara
d64643fe26 AWS: Experimental support for multiple subnets/AZs in kube-up
By setting KUBE_SHARE_MASTER=true we reuse an existing master, rather
than creating a new one.

By setting KUBE_SUBNET_CIDR=172.20.1.0/24 you can specify the CIDR for a
new subnet, avoiding conflicts.

Both these options are documented only in kube-up and clearly marked as
'experimental' i.e. likely to change.

By combining these, you can kube-up a cluster normally, and then kube-up
a cluster in a different AZ, and the new nodes will attach to the same
master.

KUBE_SHARE_MASTER is also useful for addding a second node
auto-scaling-group, for example if you wanted to mix spot & on-demand
instances.
2015-11-03 11:57:59 -05:00
Justin Santa Barbara
df3897c4ab Split main kube-up function into sub-functions
Makes it a little easier to read, also lets us reuse a master in the
next commit without a complex diff.
2015-11-03 10:16:32 -05:00
Justin Santa Barbara
1607ac1a7a AWS: use filters in get_igw_id & get_subnet_id
Also remove unused get_route_table_id
2015-11-03 10:09:51 -05:00
Jeff Lowdermilk
de4f29f1f5 Load or generate auth from kubeconfig on kube-up
Allows loading existing auth from kubeconfig on kube-up if a
valid KUBE_CONTEXT is specified, instead of always force
regenerating auth (basic or token) when creating a new cluster.
2015-10-30 10:19:40 -07:00
Jan Safranek
fe0741bffe Configure cluster for e2e tests.
When KUBE_E2E_STORAGE_TEST_ENVIRONMENT is set to 'true', kube-up.sh script
will:

- Install the right packages for all storage volumes.
- Use devicemapper as docker storage backend. 'aufs', the default one on
Debian, does not support extended attibutes required by Ceph RBD and Gluster
server containers.

Tested on GCE and Vagrant, e2e tests for storage volumes passes without any
additional configuration.
2015-10-29 11:03:34 +01:00
Justin Santa Barbara
1ab663e912 AWS: Log the region when we don't recognize it.
This will help users see when they've typoed the region name (#14278)
2015-10-27 10:18:52 -04:00
Justin Santa Barbara
ff44cd113a AWS: Wire-up ENABLE_DEPLOYMENT env var
The AWS equivalent of a558fca24b
2015-10-21 21:57:38 -04:00
Mike Danese
e9ff8e7f97 fix extensions enabling in aws kube-up 2015-10-20 15:59:30 -07:00
Isaac Hollander McCreery
a379e52c57 Factor find-release-tars into cluster/common 2015-10-16 13:09:52 -07:00
Justin Santa Barbara
33d0541a00 AWS: Add configuration for ENABLE_EXPERIMENTAL
We need this for some tests; not all the options are fully plumbed in,
but should enable experimental/v1alpha1, as needed for jobs tests.

In particular, ENABLE_NODE_AUTOSCALER is not yet actually implemented.
2015-10-08 10:15:10 -04:00
Alex Robinson
1efee0b49e Merge pull request #8255 from iterion/default_dhcp_option_set
AWS: Create DHCP Option Set on kube-up
2015-10-05 16:29:48 -07:00
Alex Robinson
0b6bcab091 Merge pull request #15049 from Juniper/opencontrail-provisioning
Add opencontrail networking provisioning support in kubernetes via salt based provisioning
2015-10-05 14:40:11 -07:00
Adam Sunderland
1c77179652 Adding DHCP Option Setup
Fix Spacing
2015-10-05 14:13:40 -05:00
Alex Robinson
8b68089ac4 Merge pull request #15071 from justinsb/aws_include_distro_when_not_recognized
AWS: Log the distro when we don't recognize it
2015-10-05 11:19:38 -07:00
Justin Santa Barbara
d898834037 AWS: Log the distro when we don't recognize it
Similar to #15070, we should log the distro if we're going to tell the
user we can't match it (so the user can see if they have typoed it, and
so it will hopefully be included to us in error reports)
2015-10-05 09:25:59 -04:00
Justin Santa Barbara
f763c59743 AWS: Increase the timeout for S3 bucket creation
The current timeout of 5 seconds is needlessly short, given that we
fail kube-up if the (eventually consistent?) bucket creation takes
longer.

Raise it to 120 seconds.

Possibly related to issue #14278
2015-10-05 09:17:12 -04:00
Ananth Suryanarayana
d50d7763da Add opencontrail networking provisioning support in kubernetes salt based provisioning
OpenContrail is an open-source based networking software which provides virtualization support for the cloud.

This change-set adds ability to install and provision opencontrail software for networking in kubernetes based cloud environment.

There are basically 3 components

o kube-network-manager -- plugin between contrail components and kubernets components
o provision_master.sh -- OpenContrail software installer and provisioner in master node
o provision_minion.sh -- OpenContrail software installer and provisioner in minion node(s)

These are driven via salt configuration files

One can provision opencontrail by just setting "export NETWORK_PROVIDER=opencontrail"
Optionally, OPENCONTRAIL_TAG, and OPENCONTRAIL_KUBERNETES_TAG can be used to
specify opencontrail and contrail-kubernetes software versions to install and provision.

Public-IP Subnet provided by contrail can be configured via OPENCONTRAIL_PUBLIC_SUBNET
environment variable

At this moment, plan is to add support for aws, gce and vagrant based platforms

For more information on contrail-kubernetes, please visit https://github.com/juniper/contrail-kubernetes For more information on opencontrail, please visit http://www.opencontrail.org
2015-10-03 08:03:02 -07:00
Brian Grant
1b8727ada1 Merge pull request #13072 from mikaelh/master
Elastic IP need to be assigned after the instance is in running state
2015-09-24 17:39:26 -07:00
Mikael Hammarin
c82e645f9d Elastic IP need to be assigned after the instance is in running state. Currently always fails with "The pending instance x is not in a valid state for this operation" and defaults to public ip.
Fix: Moved assign-elastic-ip to run after wait-for-instance-running
2015-08-23 13:39:01 +02:00
Jeff Lowdermilk
438216844f Stop reusing basic auth on cluster create 2015-08-22 11:45:46 -07:00
erulabs
5ee816e905 Add config option for AWS EBS root volume size 2015-08-18 11:03:23 -07:00
Veres Lajos
9f77e49109 typofix - https://github.com/vlajos/misspell_fixer 2015-08-08 22:31:48 +01:00
Justin Santa Barbara
230de787be AWS: Query for location of S3 bucket, use that
Previously we would rely on the s3 bucket's region being configured
correctly, at least for the existence check.  By querying for the bucket
region and then going direct to the correct region, we avoid errors and
we avoid potential eventual consistency problems.

May be related to issue: #12109
2015-08-01 14:09:23 -04:00
Piotr Szczesniak
f48543aba5 Made enabling Kube UI configurable 2015-07-27 08:23:04 +02:00
Justin Santa Barbara
355e922b5d AWS: Restore (erroneously) removed get_instance_private_ip
Fix #11201
2015-07-14 13:45:12 -04:00
Victor Marmol
ee8d2857fc Merge pull request #9381 from justinsb/aws_support_jessie
AWS: Add support for Vivid Vervet & Debian Jessie
2015-07-09 15:20:37 -07:00
Justin Santa Barbara
c78b71d399 AWS: Treat ubuntu as an alias for 'latest ubuntu', i.e. 'vivid' 2015-07-08 17:00:44 -04:00
Justin Santa Barbara
bdb2196abe AWS: Initial support for Ubuntu Vivid Vervet 2015-07-08 17:00:43 -04:00
Victor Marmol
7dd0dc1786 Merge pull request #10752 from justinsb/aws_query_by_subnet_id
AWS: Query by subnet-id, if we have it
2015-07-08 13:57:33 -07:00
Victor Marmol
3dd3d6e889 Merge pull request #10751 from justinsb/aws_existing_subnet
AWS: Allow callers to specify VPC_ID and SUBNET_ID
2015-07-08 13:57:13 -07:00
Justin Santa Barbara
8e69495f7f AWS: Add jessie support to cluster script 2015-07-08 16:47:55 -04:00
Justin Santa Barbara
cc68f87a7b AWS: Query by subnet-id, if we have it
If we are matching an existing subnet-id, query using that subnet-id!
2015-07-06 10:14:15 -04:00
Justin Santa Barbara
2429c7edf8 AWS: Allow callers to specify VPC_ID and SUBNET_ID
This is for people that want to run in a shared VPC/Subnet; while this should
work, we don't actively want to support it yet.  So we don't block it,
but we don't document/encourage it either!
2015-07-06 10:10:34 -04:00
Justin Santa Barbara
c676c11189 AWS: Configure SSL certificate alternate-names
GCE does this in its per-provider scripts; this does the same for AWS and lets
other providers do the same; I believe kube2sky requires 10.0.0.1 as a SAN.
2015-07-03 01:18:07 -04:00
Justin Santa Barbara
2a8d46864b AWS: Use the SSH key fingerprint, not the AWS fingerprint
This is unfortunate, because it means we have two fingerprints,
although arguably the OpenSSH key fingerprint is much more common.
However, the OSX Mavericks version of ssh-keygen can't compute
the AWS fingerprint correctly (e.g. https://www.netmeister.org/blog/ssh2pkcs8.html)

So we work on OSX Mavericks, we use the more common OpenSSH fingerprint.
2015-07-01 04:10:48 -07:00
Robert Bailey
e6f14a21ce Merge pull request #10403 from justinsb/ebs-for-master-data-2
AWS: Use persistent disk on master
2015-06-26 21:52:10 -07:00
Justin Santa Barbara
a3cb84fa56 AWS: Wait for disk to be attached in setup-master-pd.sh 2015-06-26 11:52:45 -04:00
Justin Santa Barbara
6e09cd3ffd AWS: Create/reuse master pd directly 2015-06-26 11:52:45 -04:00
Justin Santa Barbara
7059f379a4 AWS: Mount persistent disk as sdb on master 2015-06-26 11:52:44 -04:00
Adam Sunderland
66e0c5432f Use EBS for Master Data
Signed-off-by: Adam Sunderland <iterion@gmail.com>
2015-06-26 11:52:44 -04:00
Eric Paris
58df58f3d7 Remove unused enable_node_monitoring option
Back in 1a7f7245e7 we dropped the one
place this was used, but left all of the variable and definitions and
garbage around cluster/
2015-06-25 20:57:56 -04:00
Jeff Lowdermilk
e2d496e45d Merge pull request #9737 from iterion/use-openssl-rsa
AWS: Use RSA to Generate Fingerprint
2015-06-23 10:14:27 -07:00
Justin Santa Barbara
2a5ed2f086 AWS: Use auto-scaling group to run minions
This uses the dynamic CIDR work, and we set source-dest-check to false
when we configure the route (which kind-of makes sense)
2015-06-19 10:22:15 -04:00
Justin Santa Barbara
a4e15cdf3e AWS: Configure minion routes dynamically
We need to implement the Routes interface, and then enable the functionality in the cluster scripts.
2015-06-18 14:59:37 -07:00
Fabio Yeon
c4d040c720 Merge pull request #9745 from justinsb/aws_manage_route_table
AWS: Create our own route table & tag it for management
2015-06-15 15:18:39 -07:00
Fabio Yeon
f033349bea Merge pull request #9746 from justinsb/aws_faster_tagging
AWS: delay less when creating tags
2015-06-15 13:34:44 -07:00
Justin Santa Barbara
6839ae9a44 AWS: delay less when creating tags
This is important because tag creation isn't atomic, and we want to minimize
the window in which we can leave 'dangling' resources around.
2015-06-12 18:32:55 -04:00
Justin Santa Barbara
1f04284508 AWS: Create our own route table & tag it for management 2015-06-12 18:32:10 -04:00
Adam Sunderland
7760d079ae Update SSH User For Master SSH Check 2015-06-12 15:38:48 -05:00
Adam Sunderland
d82bfffe9d Use RSA to Generate Fingerprint 2015-06-12 15:29:37 -05:00
Abhi Shah
82aa8f9984 Merge pull request #9371 from justinsb/aws_support_wheezy
AWS: Support wheezy, for parity with GCE
2015-06-11 15:08:32 -07:00
Abhi Shah
064e7146e1 Merge pull request #9481 from justinsb/aws_fix_push
AWS: Include (idempotent) ensure-temp-dir in upload-server-tars
2015-06-11 11:24:18 -07:00
Justin Santa Barbara
8aae864784 AWS: Support wheezy, for parity with GCE 2015-06-09 23:45:41 -04:00
krousey
e19e4bcd12 Merge pull request #9365 from justinsb/fix_9246
AWS: Only log "Starting cluster using os distro..." in kube-up
2015-06-09 09:51:22 -07:00
Justin Santa Barbara
d92863523f AWS: Include (idempotent) ensure-temp-dir in upload-server-tars
This way we won't forget it.  Fixes kube-push, where I forgot it.
2015-06-09 11:10:15 -04:00
krousey
e14d9038fe Merge pull request #9367 from justinsb/aws_ssh_check
AWS: Check for SSH connectivity & better logging on failure
2015-06-08 16:48:04 -07:00
Justin Santa Barbara
1c229e5284 Removed extra blank line in aws util.sh 2015-06-08 16:59:03 -04:00
Justin Santa Barbara
2619b6198a AWS: Mount ephemeral devices, even if not specified in the AMI
We mount up to 4; this covers almost all instance types.
2015-06-08 16:59:03 -04:00
Justin Santa Barbara
72496e7368 AWS: Check for SSH connectivity & better logging on failure 2015-06-06 14:27:41 -04:00
Justin Santa Barbara
a9e1e1033b AWS: Only log "Starting cluster using os distro..." in kube-up
Fixes #9246
2015-06-06 12:48:49 -04:00
Justin Santa Barbara
112a013567 AWS: Support different docker storage mechanism by setting DOCKER_STORAGE
For parity with GCE, we really want to support aufs.

But we previously supported btrfs, so we want to expose that.

Most of the work here is required for aufs, and we let advanced users choose
devicemapper/btrfs if they have a setup that works for those configurations.
2015-06-06 12:13:51 -04:00
Justin Santa Barbara
48e8a8b0ec AWS: Set up security groups, to mirror GCE firewalling
Some slightly fussy code to enable load-balancers to talk to
instances, but otherwise relatively simple.
2015-06-05 16:10:08 -04:00
Justin Santa Barbara
33a3d884f2 AWS: Filter by Cluster tag, rationalize EC2 abstraction
Whenever we do a list we now filter on tags so we only see resources relating
to our cluster.

Also, rationalize all the DescribeX calls:
 * They all take a request object (so that we can pass filters)
 * They do paging if that is required (and return the underlying resources)
 * They wrap any error with a "error while listing X: %v" message
2015-06-05 16:09:01 -04:00
Quinton Hoole
710df2b619 Merge pull request #9294 from justinsb/aws_ssh_key_fingerprint
AWS: Support multiple SSH keys (embed the hash in the name)
2015-06-05 09:37:31 -07:00