SMS Blog

AWS EKS Upgrades: Don’t Pay the Price for Outdated Clusters

Rob Stewart/31 January 2025/

Introduction

Amazon makes new EKS versions available shortly after the core Kubernetes release approximately every four months. Each new release of EKS benefits from enhanced security and access to new capabilities and comes with 14 months of standard support and an additional 12 months of extended support. At the end of the extended support period, AWS will automatically update EKS clusters to a supported version.

Unfortunately, keeping your AWS EKS clusters up to date may be disruptive. Every Kubernetes upgrade introduces API changes and component updates which must be tested with your workloads. For busy teams that are already juggling many priorities this can be a challenge. Despite these difficulties, getting too far behind on EKS releases can be costly. Starting in April 2024, AWS now charges an additional $.50 per cluster hour, which amounts to more than $300 a month for each EKS cluster running a release of Kubernetes on extended support. For teams that are already struggling to manage their cloud spend, the extra ongoing expense of multiple EKS clusters running on extended support is unwelcome. Therefore, it is necessary to plan for regular EKS upgrades. This blog post will provide general guidance on how to plan EKS upgrades and provide some additional resources for further reading on this complex topic.

There are several distinct elements of an EKS upgrade:

Upgrading the EKS Control Plane
Upgrading the EKS Node Groups
Upgrading the EKS managed add-ons.
Upgrading any executables used to interact with the cluster such as kubectl
Validating the updates with the workloads running on the cluster

Figure 1: Upgrading the different elements of AWS EKS

Before you can proceed with your EKS upgrade you should first review the current configuration of your EKS cluster and then plan how you will handle each one of these elements.

Review the current EKS Configuration

Before proceeding with any changes, it is necessary to review the current configuration of the EKS cluster you plan to upgrade. Collect the following information:

The current version of the EKS cluster control plane
The current versions of the EKS node groups deployed to the cluster
The current versions of any EKS managed add-ons which are deployed to the cluster

You can obtain this information by reviewing the EKS cluster configuration in the AWS Console.

Determine the Upgrade Version

After you have collected this information, the next step is to identify the version of EKS that you will be upgrading to so that you can review the AWS EKS Change Log for the new release as well as the change log for the Kubernetes release under consideration. Make a note of any issues that may impact the workloads deployed to the cluster by identifying any deprecated and removed API Usage in workloads. If any Kubernetes API changes impact running workloads, then those workloads will have to be updated as part of the deployment of the Kubernetes upgrade.

As you develop your EKS upgrade plans, please keep the following in mind:

Downgrading the Kubernetes version for an Amazon EKS cluster is not supported. If you find that you need to downgrade your cluster, the only way to accomplish the downgrade is to create a new cluster on the desired version and migrate your workloads to it.
EKS upgrades must proceed directly from Kubernetes point release to point release. For example, it is not possible to update an EKS cluster on version 1.28 directly to 1.30; in order to update a cluster on 1.28 to 1.30 one must first update the cluster to 1.29 and then upgrade the cluster to 1.30.

Plan for Upgrading Node Groups

After confirming your approach for upgrading the EKS control plane, the next step is to consider updates to your node groups.

Verify that the EKS Managed Node Groups are on a release version that matches the current EKS version running on the cluster. The release version of each managed node group should have already been upgraded along with the previous cluster version upgrade so that the EKS cluster version and the release version running on the EKS node groups match. However, if you find that a node group is not on the same Kubernetes release as the cluster then you should first update the EKS node groups to the current Kubernetes release prior to performing the Kubernetes version upgrade.
If you are updating the EKS managed node groups via the AWS Console, then you do not need to determine which release version of the node group to use. The latest release version will be used automatically. However, if you are deploying the EKS node group updates via Infrastructure as Code then you need to determine the Target release_version that you will update the AWS Managed EKS Node Groups to after upgrading the EKS cluster. After you finish the EKS Cluster upgrade, the AWS managed node groups must also be updated to the same release as the cluster. Identify the desired release version by consulting the official EKS AMI change log on GitHub. Make sure to make a note of the release_version which corresponds to the Kubernetes release that the cluster will be upgraded to by expanding the section matching the current Kubernetes release.

A few things to keep in mind regarding planning upgrades to your node groups:

If you deployed node groups with custom AMIs then you will need to plan how you will update your AMI and manage the other changes to launch template versions.
When the nodes in a node group are replaced, Kubernetes will start new pods on the new node and then terminate the pods on the node that will be terminated. This process may impact running workloads if the new pods encounter issues when starting. For example, if you are using the EBS CSI driver with Persistent Volumes then the new pod may be unable to bind the persistent volume and will fail to start if the new node running the pod and the EBS volume are in different availability zones. In this scenario, the workload may need to be removed and redeployed to restore it to a running state.

Plan for Upgrading EKS Add-Ons

Next, determine whether the EKS add-ons currently deployed to the cluster will need to be updated after the control plane is updated:

A given EKS add-on is typically compatible with multiple EKS versions. Run the following command for each add-on that is deployed to your cluster specifying the name of the deployed add-on and the version of EKS that you will be upgrading to so that you can determine whether the add-on will need to be updated in order to be compatible with the new release of Kubernetes that you are updating to: aws eks describe-addon-versions --addon-name NAME-OF-EKS-ADD-ON --kubernetes-version 1.29
Document the AWS EKS Managed Add-Ons which need to be updated after the control plane is updated.

Plan for Updating Kubernetes Executables

Next, you will need to plan how you will update any executables including kubectl that you use to interact with your Kubernetes cluster to releases that are compatible with the new release of Kubernetes.

Plan for Testing the Upgrade with your Workloads

Finally, it will be imperative to test the workloads running on the cluster after performing an EKS upgrade to confirm that all of them are operating as expected. In many cases, it may be prudent to carry out the planned EKS upgrade on a test cluster where workload validation can be performed prior to executing an upgrade on a cluster running production workloads. Depending on the complexity of your workloads, more elaborate upgrade strategies such as blue/green deployments might be worth considering; however, they have not been addressed here for the sake of brevity.

Additional resources

Please refer to the following additional resources as you prepare for your next EKS upgrade:

Conclusion

AWS EKS is an amazing platform for running containerized workloads in the cloud. However, it requires ongoing attention. You will always be better off executing EKS upgrades on your schedule and on your terms versus allowing AWS to update your clusters. As with so many other things, it is important to do your research so that you are not taken by surprise when going through an EKS upgrade.