Performing EKS Upgrades with Terraform

Photo by Joel Thorner on Unsplash

In a previous blog post we’ve shown you how to deploy EKS quickly and easily with Terraform. AWS recently release version v1.18 of Kubernetes on EKS so now is the perfect opportunity to see how to upgrade an EKS cluster using Terraform.

For the rest of this blog it is assumed that you’ve used https://github.com/cweibel/example-terraform-eks/tree/main/eks_and_fargate to spin up your EKS cluster and a single managed Node Group.

Performing Upgrades

Upgrades can be done through either the AWS Console UI or via Terraform. We’ll assume that you want to continue to use Terraform to manage EKS after you’ve bootstrapped the environment.

Let’s follow the mama duck!

Upgrades are done in two or more phases. The first phase involves updating the version of Kubernetes on the master/control node. The subsequent phases involve updating the one or more Node Groups you have defined.

Step 1 – Upgrade the master

This is straight forward. In this repo set cluster.tf local variables to the desired version:

locals {  cluster_version = "1.18"   # Assuming you initially deployed 1.17
}

Perform a terraform apply and perform an update in-place:

Resource actions are indicated with the following symbols:
  ~ update in-place
Terraform will perform the following actions:
  # module.eks.aws_eks_cluster.this[0] will be updated in-place
  ~ resource "aws_eks_cluster" "this" {
        arn                       = "arn:aws:eks:us-west-2:123456789012:cluster/eks-cweibel2"
        certificate_authority     = [
            {
                data = "LS0tLREDACTEDtLS0tLQo="
            },
        ]
        created_at                = "2020-09-16 16:07:38.259 +0000 UTC"
        enabled_cluster_log_types = []
        endpoint                  = "https://1234AB1AEB234567FA5EBBAA67ED8BC9.gr7.us-west-2.eks.amazonaws.com"
        id                        = "eks-cweibel2"
        identity                  = [
            {
                oidc = [
                    {
                        issuer = "https://oidc.eks.us-west-2.amazonaws.com/id/1234AB1AEB234567FA5EBBAA67ED8BC9"
                    },
                ]
            },
        ]
        name                      = "eks-cweibel2"
        platform_version          = "eks.3"
        role_arn                  = "arn:aws:iam::123456789012:role/eks-cweibel220200916160718353000000001"
        status                    = "ACTIVE"
        tags                      = {}
      ~ version                   = "1.17" -> "1.18"
        timeouts {
            create = "30m"
            delete = "15m"
        }
        vpc_config {
            cluster_security_group_id = "sg-081a41a4f850bc69b"
            endpoint_private_access   = false
            endpoint_public_access    = true
            public_access_cidrs       = [
                "0.0.0.0/0",
            ]
            security_group_ids        = [
                "sg-084cb02c3a6d6442c",
            ]
            subnet_ids                = [
                "subnet-05fa7d7ec68ecfae8",
                "subnet-07c8750e73172d61d",
                "subnet-0c9b19e2856e8b867",
            ]
            vpc_id                    = "vpc-05f0ba84696234a43"
        }
    }
Plan: 0 to add, 1 to change, 0 to destroy.

This will take a few minutes to run, don’t jump ahead until this step finishes.

Step 2a – Upgrading Node Groups

Once the master node has been upgraded to the newer version, each of the Node Groups can be upgraded, following the mama duck. This is done by tainting the NodeGroup resources:

terraform taint "module.eks.module.node_groups.random_pet.node_groups[\"eks_nodes\"]"
terraform taint "module.eks.module.node_groups.aws_eks_node_group.workers[\"eks_nodes\"]"

This will not do an in-place upgrade. What it will do is:

  • Spin an entirely new NodeGroup set of EC2 instances using the newer AMI.
  • Once all the new worker nodes are healthy, the older nodes will be drained and status will be Ready,SchedulingDisabled.
  • After the pods are moved off of the old workers, the underlying EC2 instances are terminated and the upgrade is complete

While this is happening, to get a “BOSH like experience to know the state of the upgrade” you can run a watch kubectl get nodes command. In the example below the new workers have been successfully added and the old workers are in the process of draining:

$ kubectl get nodes
NAME                                         STATUS                     ROLES    AGE   VERSION
ip-10-20-68-201.us-west-2.compute.internal   Ready                      <none>   17m   v1.18.1-eks-4c6976
ip-10-20-71-150.us-west-2.compute.internal   Ready,SchedulingDisabled   <none>   13m   v1.17.3-eks-2ba888
ip-10-20-76-255.us-west-2.compute.internal   Ready                      <none>   17m   v1.18.1-eks-4c6976
ip-10-20-83-114.us-west-2.compute.internal   Ready                      <none>   17m   v1.18.1-eks-4c6976
ip-10-20-73-121.us-west-2.compute.internal   Ready,SchedulingDisabled   <none>   13m   v1.17.3-eks-2ba888
ip-10-20-75-133.us-west-2.compute.internal   Ready,SchedulingDisabled   <none>   13m   v1.17.3-eks-2ba888

If you have additional Node Groups (managed or unmanaged) you can now loop through and taint the worker groups. To help prevent bumping in AWS EC2 resource quotas you’ll likely only want to perform upgrades to one worker group at a time.

Step 2b – Upgrading Fargate

Once the master node has been upgraded to the newer version, any newly created fargate pods will deploy EC2 instances with AMIs based on the version the master has.

To upgrade existing Fargate pods, there is no terraform to do this. The pods needs to be destroyed and recreated. To do this without downtime for the app itself, Kubernetes Deployments should be leveraged which can do rolling upgrades.

For example, if there is a Deployment called nginx which has 3 replicas:

$ kubectl get pods
NAME                               READY   STATUS    RESTARTS   AGE
nginx-deployment-f8d6d5c66-62s4f   1/1     Running   0          51m
nginx-deployment-f8d6d5c66-7tmc9   1/1     Running   0          52m
nginx-deployment-f8d6d5c66-qlkf5   1/1     Running   0          50m

You can see the 3 Fargate nodes that are associated to these 3 pods:

$ kubectl get nodes
NAME                                                 STATUS   ROLES    AGE   VERSION
fargate-ip-10-20-79-20.us-west-2.compute.internal    Ready    <none>   49m   v1.17.8-eks-e16311
fargate-ip-10-20-80-167.us-west-2.compute.internal   Ready    <none>   50m   v1.17.8-eks-e16311
fargate-ip-10-20-83-190.us-west-2.compute.internal   Ready    <none>   51m   v1.17.8-eks-e16311
ip-10-20-67-247.us-west-2.compute.internal           Ready    <none>   84m   v1.17.3-eks-2ba888
ip-10-20-77-210.us-west-2.compute.internal           Ready    <none>   85m   v1.17.3-eks-2ba888
ip-10-20-86-121.us-west-2.compute.internal           Ready    <none>   85m   v1.17.3-eks-2ba888

Perform upgrade of master from 1.17 to 1.18, any new fargate pods created will use the 1.18 AMI. To rotate the existing Fargate Pods, perform a rolling update:

$ kubectl -n fgnamespace rollout restart deployment nginx-deployment
deployment.apps/nginx-deployment restarted

To see the status of the rollout:

kubectl rollout status deployments/nginx-deployment
Waiting for deployment "nginx-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "nginx-deployment" rollout to finish: 1 old replicas are pending termination...
deployment "nginx-deployment" successfully rolled out

If you look at the list of nodes at this point, the Fargate Pods are now all leveraging the v1.18.1 AMI

$ kubectl get nodes
NAME                                                 STATUS   ROLES    AGE     VERSION
fargate-ip-10-20-69-132.us-west-2.compute.internal   Ready    <none>   2m56s   v1.18.1-eks-a84824
fargate-ip-10-20-69-55.us-west-2.compute.internal    Ready    <none>   4m14s   v1.18.1-eks-a84824
fargate-ip-10-20-86-53.us-west-2.compute.internal    Ready    <none>   97s     v1.18.1-eks-a84824
ip-10-20-67-247.us-west-2.compute.internal           Ready    <none>   90m     v1.17.3-eks-2ba888
ip-10-20-77-210.us-west-2.compute.internal           Ready    <none>   91m     v1.17.3-eks-2ba888
ip-10-20-86-121.us-west-2.compute.internal           Ready    <none>   91m     v1.17.3-eks-2ba888

Opinion: Import Notes for Fargate

Fargate is an odd duck when it comes to an Ops team being responsible for performing upgrades. Unless the Fargate Pods are recreated via rolling upgrades there is no way for Ops personnel to do this work without looping through the namespaces in the Fargate Profile and performing rolling upgrades of the deployment. If pods aren’t associated to a Kubernetes Deployment, such as an app that only has a pod spec, tooling needs to be created to orchestrate the restart.

Otherwise the responsibility of performing upgrades to Fargate Pods falls to the app owner. This could be dangerous from a compliance/security perspective.

What’s Next?

Need a dash of Cloud Foundry deployed on top of your Terraform managed EKS cluster? Checkout Deploying KubeCF to EKS, much of that blog is automated into https://github.com/cweibel/example-terraform-eks/tree/main/eks_for_kubecf_v2. If you have any questions, please feel free to ask in the comments section below.

Enjoy!

Spread the word

twitter icon facebook icon linkedin icon