Bare Metal Service Load Balancers
AKA "how to set up a bank of haproxy for platforms that don't have load balancers".
Disclaimer:
- This is a work in progress.
- A better way to achieve this will probably emerge once discussions on (#260, #561) converge.
- Backends are pluggable, but Haproxy is the only loadbalancer with a working implementation.
- I have never deployed haproxy to production, so contributions are welcome (see wishlist for ideas).
- For fault tolerant load balancing of ingress traffic, you need:
- Multiple hosts running load balancers
- Multiple A records for each hostname in a DNS service.
This module will not help with the latter
Overview
Ingress
There are 2 ways to expose a service to ingress traffic in the current kubernetes service model:
- Create a cloud load balancer.
- Allocate a port (the same port) on every node in your cluster and proxy ingress traffic through that port to the endpoints.
The service-loadbalancer aims to give you 1 on bare metal, making 2 unnecessary for the common case. The replication controller manifest in this directly creates a service-loadbalancer pod on all nodes with the role=loadbalancer label. Each service-loadbalancer pod contains:
- A load balancer controller that watches the kubernetes api for services and endpoints.
- A load balancer manifest. This is used to bootstrap the load balancer. The load balancer itself is pluggable, so you can easily swap haproxy for something like f5 or pound.
- A template used to write load balancer rules. This is tied to the loadbalancer used in the manifest, since each one has a different config format.
L7 load balancing of Http services: The load balancer controller automatically exposes http services to ingress traffic on all nodes with a role=loadbalancer label. It assumes all services are http unless otherwise instructed. Each http service gets a loadbalancer forwarding rule, such that requests received on http://loadbalancer-node/serviceName:port balanced between its endpoints according to the algorithm specified in the loadbalacer.json manifest. You do not need more than a single loadbalancer pod to balance across all your http services (you can scale the rc to increase capacity).
L4 loadbalancing of Tcp services: Since one needs to specify ports at pod creation time (kubernetes doesn't currently support port ranges), a single loadbalancer is tied to a set of preconfigured node ports, and hence a set of TCP services it can expose. The load balancer controller will dynamically add rules for each configured TCP service as it pops into existence. However, each "new" (unspecified in the tcpServices section of the loadbalancer.json) service will need you to open up a new container-host port pair for traffic. You can achieve this by creating a new loadbalancer pod with the targetPort set to the name of your service, and that service specified in the tcpServices map of the new loadbalancer.
Cross-cluster loadbalancing
On cloud providers that offer a private ip range for all instances on a network, you can setup multiple clusters in different availability zones, on the same network, and loadbalancer services across these zones. On GCE for example, every instance is a member of a single network. A network performs the same function that a router does: it defines the network range and gateway IP address, handles communication between instances, and serves as a gateway between instances and other networks. On such networks the endpoints of a service in one cluster are visible in all other clusters in the same network, so you can setup an edge loadbalancer that watches a kubernetes master of another cluster for services. Such a deployment allows you to fallback to a different AZ during times of duress or planned downtime (eg: database update).
Examples
Initial cluster state:
$ kubectl get svc --all-namespaces -o yaml | grep -i "selfLink"
selfLink: /api/v1/namespaces/default/services/kubernetes
selfLink: /api/v1/namespaces/default/services/nginxsvc
selfLink: /api/v1/namespaces/kube-system/services/elasticsearch-logging
selfLink: /api/v1/namespaces/kube-system/services/kibana-logging
selfLink: /api/v1/namespaces/kube-system/services/kube-dns
selfLink: /api/v1/namespaces/kube-system/services/kube-ui
selfLink: /api/v1/namespaces/kube-system/services/monitoring-grafana
selfLink: /api/v1/namespaces/kube-system/services/monitoring-heapster
selfLink: /api/v1/namespaces/kube-system/services/monitoring-influxdb
These are all the cluster addon services in namespace=kube-system.
Create a loadbalancer
- Loadbalancers are created via a ReplicationController.
- Load balancers will only run on nodes with the
role=loadbalancerlabel.
$ kubectl create -f ./rc.yaml
replicationcontrollers/service-loadbalancer
$ kubectl get pods -l app=service-loadbalancer
NAME READY STATUS RESTARTS AGE
service-loadbalancer-dapxv 0/2 Pending 0 1m
$ kubectl describe pods -l app=service-loadbalancer
Events:
FirstSeen From Reason Message
Tue, 21 Jul 2015 11:19:22 -0700 {scheduler } failedScheduling Failed for reason MatchNodeSelector and possibly others
Notice that the pod hasn't started because the scheduler is waiting for you to tell it which nodes to use as a load balancer.
$ kubectl label node e2e-test-beeps-minion-c9up role=loadbalancer
NAME LABELS STATUS
e2e-test-beeps-minion-c9up kubernetes.io/hostname=e2e-test-beeps-minion-c9up,role=loadbalancer Ready
Expose services
Your kube-ui should be publicly accessible once the loadbalancer created in the previous step is in Running (if you're on a cloud provider, you need to create firewall-rules for :80)
$ kubectl get nodes e2e-test-beeps-minion-c9up -o json | grep -i externalip -A 1
"type": "ExternalIP",
"address": "104.197.63.17"
$ curl http://104.197.63.17/kube-ui
HTTP
You can use the https-nginx example to create some new HTTP/HTTPS services.
$ cd ../../examples/https-nginx
$ make keys secret KEY=/tmp/nginx.key CERT=/tmp/nginx.crt SECRET=/tmp/secret.json
$ kubectl create -f /tmp/secret.json
$ kubectl get secrets
NAME TYPE DATA
default-token-vklfs kubernetes.io/service-account-token 2
nginxsecret Opaque 2
Lets introduce a small twist. The nginx-app example exposes the nginx service using NodePort, which means it opens up a random port on every node in your cluster and exposes the service on that. Delete the type: NodePort line before creating it.
$ kubectl create -f nginx-app.yaml
$ kubectl get svc
NAME LABELS SELECTOR IP(S) PORT(S)
kubernetes component=apiserver,provider=kubernetes <none> 10.0.0.1 443/TCP
nginxsvc app=nginx app=nginx 10.0.79.131 80/TCP
443/TCP
$ curl http://104.197.63.17/nginxsvc
HTTPS
HTTPS services are handled at L4 (see wishlist)
$ curl https://104.197.63.17:8080 -k
A couple of points to note:
- The nginxsvc is specified in the tcpServices of the loadbalancer.json manifest.
- The https service is accessible directly on the specified port, which matches the service port.
- You need to take care of ensuring there is no collision between these service ports on the node.
TCP
$ cat mysql-app.yaml
apiVersion: v1
kind: Pod
metadata:
name: mysql
labels:
name: mysql
spec:
containers:
- image: mysql
name: mysql
env:
- name: MYSQL_ROOT_PASSWORD
# Use secrets instead of env for passwords
value: password
ports:
- containerPort: 3306
name: mysql
volumeMounts:
# name must match the volume name below
- name: mysql-storage
# mount path within the container
mountPath: /var/lib/mysql
volumes:
- name: mysql-storage
emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
labels:
name: mysql
name: mysql
spec:
type: NodePort
ports:
# the port that this service should serve on
- port: 3306
# label keys and values that must match in order to receive traffic for this service
selector:
name: mysql
We'll create the service and access mysql from outside the cluster:
$ kubectl create -f mysql-app.yaml
$ kubeclt get svc
NAME LABELS SELECTOR IP(S) PORT(S)
kubernetes component=apiserver,provider=kubernetes <none> 10.0.0.1 443/TCP
nginxsvc app=nginx app=nginx 10.0.79.131 80/TCP
443/TCP
mysql app=mysql app=mysql 10.0.63.72 3306/TCP
$ mysql -u root -ppassword --host 104.197.63.17 --port 3306 -e 'show databases;'
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
+--------------------+
Cross-cluster loadbalancing
This is a slightly advanced example. It only works if both clusters are running on the same network, on a cloud provider that provides a private ip range per network (eg: GCE, GKE, AWS).
Setup
Before creating the cluster, lets have a look at our kubectl config:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: REDACTED
server: https://104.197.84.16
name: <clustername>
...
current-context: <clustername>
...
Now spin up a cluster in europe.
KUBE_GCE_ZONE=europe-west1-b KUBE_GCE_INSTANCE_PREFIX=eu ./cluster/kube-up.sh
Your kubectl config should contain both clusters:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: REDACTED
server: https://146.148.25.221
name: <clustername_eu>
- cluster:
certificate-authority-data: REDACTED
server: https://104.197.84.16
name: <clustername>
...
current-context: kubernetesdev_eu
...
And kubectl get nodes should agree:
$ kubectl get nodes
NAME LABELS STATUS
eu-minion-0n61 kubernetes.io/hostname=eu-minion-0n61 Ready
eu-minion-79ua kubernetes.io/hostname=eu-minion-79ua Ready
eu-minion-7wz7 kubernetes.io/hostname=eu-minion-7wz7 Ready
eu-minion-loh2 kubernetes.io/hostname=eu-minion-loh2 Ready
$ kubectl config use-context <clustername>
$ kubectl get nodes
NAME LABELS STATUS
kubernetes-minion-5jtd kubernetes.io/hostname=kubernetes-minion-5jtd,role=loadbalancer Ready
kubernetes-minion-lqfc kubernetes.io/hostname=kubernetes-minion-lqfc Ready
kubernetes-minion-sjra kubernetes.io/hostname=kubernetes-minion-sjra Ready
kubernetes-minion-wul8 kubernetes.io/hostname=kubernetes-minion-wul8 Ready
Testing reachability
For this test to work we'll need to create a service in europe:
$ kubectl config use-context <clustername_eu>
$ kubectl create -f /tmp/secret.json
$ kubectl create -f nginx-app.yaml
$ kubectl exec -it my-nginx-luiln -- echo "Europe nginx" >> /usr/share/nginx/html/index.html
$ kubectl get ep
NAME ENDPOINTS
kubernetes 10.240.249.92:443
nginxsvc 10.244.0.4:80,10.244.0.4:443
Just to test reachability, we'll try hitting the europe nginx from our initial us-central cluster. Create a basic curl pod:
apiVersion: v1
kind: Pod
metadata:
name: curlpod
spec:
containers:
- image: radial/busyboxplus:curl
command:
- sleep
- "360000000"
imagePullPolicy: IfNotPresent
name: curlcontainer
restartPolicy: Always
And test that you can actually reach the test nginx service across continents.
$ kubectl config use-context kubernetesdev_kubernetes
$ kubectl create -f curlpod.yaml
$ kubectl -it exec curlpod -- /bin/sh
[ root@curlpod:/ ]$ curl http://10.244.0.4:80
Europe nginx
This proves reachability. Now we'll configure a loadbalancer that exposes all the services in the Europe cluster to the US cluster.
Create the kubeconfig secret
We will need to grant whatever pod we run the loadbalancer in, accesss to the remote cluster via a kubeconfig. This is so kubectl works in the pod, just like it did on our local machine in the previous step. First create a secret with the contents of the current kube config:
$ kubectl config use-context <clustername_eu>
$ go run ./make_secret.go --kubeconfig=/home/beeps/.kube/config > /tmp/secret.json
$ kubectl config use-context <clustername>
$ kubectl create -f /tmp/secret.json
Now modify the loadbalancer manifest. We will create this loadbalancer in our first cluster and have it publish the services from the second cluster (eu). This is the entire modified loadbalancer manifest:
apiVersion: v1
kind: ReplicationController
metadata:
name: service-loadbalancer
labels:
app: service-loadbalancer
version: v1
spec:
replicas: 1
selector:
app: service-loadbalancer
version: v1
template:
metadata:
labels:
app: service-loadbalancer
version: v1
spec:
volumes:
# token from the eu cluster, must already exist
# and match the name of the volume using in container
- name: eu-config
secret:
secretName: kubeconfig
nodeSelector:
role: loadbalancer
containers:
- image: gcr.io/google_containers/servicelb:0.1
imagePullPolicy: Always
livenessProbe:
httpGet:
path: /healthz
port: 8081
scheme: HTTP
initialDelaySeconds: 30
timeoutSeconds: 5
name: haproxy
ports:
# All http services
- containerPort: 80
hostPort: 80
protocol: TCP
# nginx https
- containerPort: 443
hostPort: 8080
protocol: TCP
# mysql
- containerPort: 3306
hostPort: 3306
protocol: TCP
# haproxy stats
- containerPort: 1936
hostPort: 1936
protocol: TCP
resources: {}
args:
- --tcp-services=mysql:3306,nginxsvc:443
- --use-kubernetes-cluster-service=false
# use-kubernetes-cluster-service=false in conjunction with the
# kube/config will force the service-loadbalancer to watch for
# services form the eu cluster.
volumeMounts:
- mountPath: /.kube
name: eu-config
env:
- name: KUBECONFIG
value: /.kube/config
Note that it is essentially the same as the rc.yaml checked into the service-loadbalancer directory expect that it consumes the kubeconfig secret create in the last stage and has an extra KUBECONFIG environment variable.
$ kubectl config use-context <clustername>
$ kubectl create -f rc.yaml
$ kubectl get pods -o wide
service-loadbalancer-5o2p4 1/1 Running 0 13m kubernetes-minion-5jtd
$ kubectl get node kubernetes-minion-5jtd -o json | grep -i externalip -A 2
"type": "ExternalIP",
"address": "104.197.81.116"
$ curl http://104.197.81.116/nginxsvc
Europe nginx
Troubleshooting:
- If you can curl or netcat the endpoint from the pod (with kubectl exec) and not from the node, you have not specified hostport and containerport.
- If you can hit the ips from the node but not from your machine outside the cluster, you have not opened firewall rules for the right network.
- If you can't hit the ips from within the container, either haproxy or the service_loadbalacer script is not runing.
- Use ps in the pod
- sudo restart haproxy in the pod
- cat /etc/haproxy/haproxy.cfg in the pod
- try kubectl logs haproxy
- run the service_loadbalancer with --dry
- Check http://<node_ip>:1936 for the stats page. It requires the password used in the template file.
- Try talking to haproxy on the stats socket directly on the container using kubectl exec, eg: echo “show info” | socat unix-connect:/tmp/haproxy stdio
Wishlist:
-
Allow services to specify their url routes (see openshift routes)
-
Scrape :1926 and scale replica count of the loadbalancer rc from a helper pod (this is basically ELB)
-
Scrape :1936/;csv and autoscale services
-
Better https support. 3 options to handle ssl:
- Termination: certificate lives on load balancer. All traffic to load balancer is encrypted, traffic from load balancer to service is not.
- Pass Through: Load balancer drops down to L4 balancing and forwards TCP encrypted packets to destination.
- Redirect: All traffic is https. HTTP connections are encrypted using load balancer certs.
Currently you need to trigger TCP loadbalancing for your https service by specifying it in loadbalancer.json. Support for the other 2 would be nice.
-
Multinamespace support: Currently the controller only watches a single namespace for services.
-
Support for external services (eg: amazon rds)
-
Dynamically modify loadbalancer.json. Will become unnecessary when we have a loadbalancer resource.
-
Headless services: I just didn't think people would care enough about this.