HAProxy for OpenShift/OKD IPI install on vSphere

I am a big fan of OpenShift and it’s open-source upstream OKD. Kubernetes is a bunch of building blocks and API’s that can be messy to assemble and configure and RedHat has done a great job packaging K8S up.

Installing on most IaaS’s is great….except vSphere where UPI (User Provisioned Infrastructure) is quite complex and involved and IPI (Installer Provisioned Infrastructure) is almost perfect save one major issue!

When you install via IPI on vSphere, OKD/OpenShift boots VM’s with DHCP, first a Bootstrap node, which will grab a pair of Virtual IP’s you configure it to, then the three master nodes. The Virtual IP’s are pointed to a pair of “keepalived” load balancers that track nodes as they boot. Unfortunately, the keepalived only checks if the process is running, not if the load balancer is working, so it’s common to see installations fail with timeouts when the keepalived load balancer gets stuck, maybe moving from the bootstrap node to another node. Its traffic just stalls.

The solution I used on my home lab was to set up a HAProxy load balancer on a tiny Debian box, point my DNS to that load balancer, then that load balancer starts with the two VIP addresses. I add the masters nodes as they boot and come online. If the keepalived balancer fails, everything continues to boot happily.  When the workers appear I add them as a backup if the keepalived fails.

global
	log /dev/log	local0
	log /dev/log	local1 notice
	chroot /var/lib/haproxy
	stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
	stats timeout 30s
	user haproxy
	group haproxy
	daemon

	# Default SSL material locations
	ca-base /etc/ssl/certs
	crt-base /etc/ssl/private

	# See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
	log	global
	mode	http
	option	httplog
	option	dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
	errorfile 400 /etc/haproxy/errors/400.http
	errorfile 403 /etc/haproxy/errors/403.http
	errorfile 408 /etc/haproxy/errors/408.http
	errorfile 500 /etc/haproxy/errors/500.http
	errorfile 502 /etc/haproxy/errors/502.http
	errorfile 503 /etc/haproxy/errors/503.http
	errorfile 504 /etc/haproxy/errors/504.http

listen stats
  bind 172.29.0.1:9090
  balance
  mode http
  stats enable
  stats auth admin:admin
  stats uri  /haproxy?stats

# OKD
#
frontend https-bootstrap
  mode tcp
  bind 172.29.0.1:22623
  default_backend bootstrap

frontend https-masters
  mode tcp
  bind 172.29.0.1:6443
  default_backend masters

frontend http-apps
  mode http
  bind 172.29.0.2:80
  default_backend http-pool

frontend https-apps
  mode tcp
  bind 172.29.0.2:443
  default_backend https-pool

backend bootstrap
  mode tcp
  balance roundrobin
  server vip1    172.29.0.3:22623  check  # Will move between nodes
  server master1 172.16.1.55:22623 check
  server master2 172.16.1.56:22623 check
  server master3 172.16.1.57:22623 check
  server boot1   172.16.1.58:22623 check  # Bootstrap node

backend masters
  mode tcp
  balance roundrobin
  server vip1    172.29.0.3:6443  check  # Will move between nodes
  server master1 172.16.1.55:6443 check
  server master2 172.16.1.56:6443 check
  server master3 172.16.1.57:6443 check
  server boot1   172.16.1.58:6443 check  # Bootstrap node

backend http-pool
  mode http
  balance leastconn
  server vip2     172.29.0.4:80  check  # Will move between nodes
  server worker1a 172.16.1.157:80 check
  server worker2a 172.16.1.156:80 check
  server worker3a 172.16.1.61:80 check
  server boot1    172.16.1.58:80 check  # Bootstrap node

backend https-pool
  mode tcp
  balance leastconn
  server vip2     172.29.0.4:443  check  # Will move between nodes
  server worker1a 172.16.1.157:443 check
  server worker2a 172.16.1.156:443 check
  server worker3a 172.16.1.61:443 check
  server boot1    172.16.1.58:443 check  # Bootstrap node

This isn’t a long-term solution because the D in DHCP stands for dynamic and addresses will change over time but for home labs or places where you don’t control the network this is how you can get going.

Spread the word

twitter icon facebook icon linkedin icon