Implementation of AWS EKS for AI applications

AWS EKS Implementation with Karpenter, Amazon EKS managed node group, Amazon managed Prometheus (AMP), Amazon managed Grafana and AWS Devops Agent.

In this article I am sharing my implementation experience of managing AWS EKS for Enterprise AI Applications using Kubernetes Node Autoscaler tool Karpenter along with Amazon EKS Managed Node Group, Amazon managed Prometheus(AMP), Amazon managed Grafana and AWS Devops Agent. The implementation is IAC (Infrastructure As Code) using Terraform.

Amazon EKS

Amazon Elastic Kubernetes Service EKS is a managed Kubernetes Service from AWS. AWS manages the availability and scalability of control plane of the Kuberenetes Cluster. User needs to manage the security of the cluster. User is responsible for scalability and availability of worker nodes of the cluster and applications deployed in the cluster. User has to manage upgradation of EKS cluster version in conformance with AWS.

Amazon EKS Auto Mode

Amazon provides us EKS Auto Mode, which takes care of all the above functionalities with Production ready Implementation using Karpenter. Amazon EKS Auto Mode is best suited for Enterprise and start-ups who do not want to manage EKS with Expert Kubernetes Administrators. However, It has an additional cost of auto mode management for each each Node (EC2 instance) based on the instance type. Also, if our application requires control over the Node provision like instance types and specifications, we can setup a production ready infrastructure as shared in my article below.

Kubernetes node autoscaler — Karpenter

Karpenter is an open-source tool used to provide horizontal node scaling in Kubernetes. Based on cluster workload, Karpenter scales up an a new node in Kubernetes. It can work with AWS, Azure, GCP, etc.

Karpenter Node Scaling

Node Auto Scaling

Auto scaling is a feature in which Kubernetes adjusts its pods and their resources according to workload of the application. However, if there are not sufficient resources for Kubernetes it cannot scale up properly.

This will result in intermittent availability issues for our application and customers utilizing it. To rectify the availability issues, we have implemented Karpenter to scale up the worker nodes (increases EC2 instances for the cluster in AWS).

Common question is, can replica set help us?

No! Replica set can help us scale up the pods however it may not help if there is no sufficient compute resource. In this scenario we need to scale up Kubernetes worker nodes as well (node scaling). This cannot be handled by replica set.

How can we do node scaling for AWS EKS Clusters?

We have achieved it using Karpenter implementation in our AWS EKS along with Amazon EKS Managed Node Group. Node Scaling helps the Kubernetes Cluster to create a new node (new EC2 instance in AWS) to manage cluster workloads. This has helped us with seamless in-place upgrade of AWS EKS Version and achieve Zero downtime upgradation.

Zero Downtime during AWS EKS Version upgrade due to efficient Node Scaling using Karpenter

We have used terraform for AWS EKS (Elastic Kubernetes Service) version upgrade. We have been using automated version upgrade for versions 1.30, 1.31,1.32, 1.33 ,1.34 and 1.35 successfully. One common observation during these upgrades is rolling update process. It ensures zero downtime during this version upgrade activity.

What is rolling update?

Rolling update ensure new version pods are created and starts running. After its successful running in ready status, the old version pods are scaled down. This is ensured for cluster worker nodes as well. New worker nodes are scaled up in new version and pods are provisioned. After the successful running of the new version pods, the old version pods are scaled down followed by scale down of old version nodes.

Amazon EKS managed node group

We have used Amazon EKS managed node group in our cluster. An EKS managed node group is a group of EC2 worker nodes that Amazon EKS helps provision and manage. These nodes run Kubernetes workloads in an EKS cluster.

Sample Terraform code adapted from aws_eks_node_group | Resources | hashicorp/aws | Terraform | Terraform Registry

resource "aws_eks_node_group" "example" {
  cluster_name    = aws_eks_cluster.example.name
  node_group_name = "example"
  node_role_arn   = aws_iam_role.example.arn
  subnet_ids      = aws_subnet.example[*].id

  scaling_config {
    desired_size = 1
    max_size     = 2
    min_size     = 1
  }

  update_config {
    max_unavailable = 1
  }

  # Ensure that IAM Role permissions are created before and deleted after EKS Node Group handling.
  # Otherwise, EKS will not be able to properly delete EC2 Instances and Elastic Network Interfaces.
  depends_on = [
    aws_iam_role_policy_attachment.example-AmazonEKSWorkerNodePolicy,
    aws_iam_role_policy_attachment.example-AmazonEKS_CNI_Policy,
    aws_iam_role_policy_attachment.example-AmazonEC2ContainerRegistryReadOnly,
  ]
}

In the above configuration we can specify launch template for nodes(EC2 instances) which will create EC2 instances based on our preferences.

Get Naveena Ravi’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Sample Launch template adapted from aws_launch_template | Resources | hashicorp/aws | Terraform | Terraform Registry

resource "aws_launch_template" "example" {
  name = "example"

  block_device_mappings {
    device_name = "/dev/sdf"

    ebs {
      volume_size = 20
    }
  }

  capacity_reservation_specification {
    capacity_reservation_preference = "open"
  }

  cpu_options {
    core_count       = 4
    threads_per_core = 2
  }

  credit_specification {
    cpu_credits = "standard"
  }

  disable_api_stop        = true
  disable_api_termination = true

  ebs_optimized = true

  iam_instance_profile {
    name = "test"
  }

  image_id = "ami-test"

  instance_initiated_shutdown_behavior = "terminate"

  instance_market_options {
    market_type = "spot"
  }

  instance_type = "t2.micro"

  kernel_id = "test"

  key_name = "test"

  license_specification {
    license_configuration_arn = "arn:aws:license-manager:eu-west-1:123456789012:license-configuration:lic-0123456789abcdef0123456789abcdef"
  }

  metadata_options {
    http_endpoint               = "enabled"
    http_tokens                 = "required"
    http_put_response_hop_limit = 1
    instance_metadata_tags      = "enabled"
  }

  monitoring {
    enabled = true
  }

  network_performance_options {
    bandwidth_weighting = "vpc-1"
  }

  network_interfaces {
    associate_public_ip_address = true
  }

  placement {
    availability_zone = "us-west-2a"
  }

  ram_disk_id = "test"

  vpc_security_group_ids = ["sg-12345678"]

  tag_specifications {
    resource_type = "instance"

    tags = {
      Name = "test"
    }
  }

  user_data = filebase64("${path.module}/example.sh")
}

EKS Blueprints

We used eks-blueprints-addons to setup Karpenter for AWS EKS Elastic Kubernetes cluster. We used karpenter version 0.37.0 and it is compatible with AWS EKS version 1.31, 1.32, 1.33 ,1.34 and 1.35 as well. We have got terraform code from aws-ia/eks-blueprints-addons/aws | Terraform Registry and GitHub — aws-ia/terraform-aws-eks-blueprints-addons: Terraform module which provisions addons on Amazon EKS clusters · GitHub

module "eks_blueprints_addons" {
  source = "aws-ia/eks-blueprints-addons/aws"
  version = "1.21.1" 

  cluster_name      = module.eks.cluster_name
  cluster_endpoint  = module.eks.cluster_endpoint
  cluster_version   = module.eks.cluster_version
  oidc_provider_arn = module.eks.oidc_provider_arn

  eks_addons = {
    aws-ebs-csi-driver = {
      most_recent = true
    }
    coredns = {
      most_recent = true
    }
    vpc-cni = {
      most_recent = true
    }
    kube-proxy = {
      most_recent = true
    }
  }

  enable_aws_load_balancer_controller    = true
  enable_cluster_proportional_autoscaler = true
  enable_karpenter                       = true
  enable_kube_prometheus_stack           = true
  enable_metrics_server                  = true
  enable_external_dns                    = true
  enable_cert_manager                    = true
  cert_manager_route53_hosted_zone_arns  = ["arn:aws:route53:::hostedzone/XXXXXXXXXXXXX"]

  tags = {
    Environment = "dev"
  }
}

Refer my article for more details on Karpenter Implementation using EKS Blueprints Manage Karpenter using AWS EKS Blueprint addons | AWS Builder Center

Karpenter Nodepool

We need to create a nodepool manifest for Karpenter, which will specify the type of nodes (EC2 instances) to be provisioned.

Nodepool is created along with EC2NodeClass manifest. EC2NodeClass is an AWS-specific Kubernetes Custom Resource Definition (CRD) that defines the configuration how Karpenter provisions the nodes (EC2) in AWS cloud.

Samples shown below adapted from NodePools | Karpenter.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
---
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiFamily: AL2023

  subnetSelectorTerms:
    # Select on any subnet that has the "karpenter.sh/discovery: ${CLUSTER_NAME}"
    # AND the "environment: test" tag OR any subnet with ID "subnet-09fa4a0a8f233a921"
    - tags:
        karpenter.sh/discovery: "${CLUSTER_NAME}"
        environment: test
    - id: subnet-09fa4a0a8f233a921

  # Required, discovers security groups to attach to instances
  # Each term in the array of securityGroupSelectorTerms is ORed together
  # Within a single term, all conditions are ANDed
  securityGroupSelectorTerms:
    # Select on any security group that has both the "karpenter.sh/discovery: ${CLUSTER_NAME}" tag
    # AND the "environment: test" tag OR any security group with the "my-security-group" name
    # OR any security group with ID "sg-063d7acfb4b06c82c"
    - tags:
        karpenter.sh/discovery: "${CLUSTER_NAME}"
        environment: test
    - name: my-security-group
    - id: sg-063d7acfb4b06c82c
    # Optional, IAM role to use for the node identity.
  # Must specify one of "role" or "instanceProfile" for Karpenter to launch nodes
  role: "KarpenterNodeRole-${CLUSTER_NAME}"

  tags:
    cluster: "$cluster_name"
    Name: "${cluster_name}-worker-karpenter-node"

Observability

We have used Amazon managed Prometheus and Amazon managed Grafana for monitoring our application. In addition, we configured AWS Devops Agent.

Amazon managed Prometheus

We implemented Prometheus for observability and monitoring using Amazon managed Prometheus. Sample code from Terraform Registry and GitHub repository

provider "aws" {
 region = "us-west-2"
}
resource "aws_prometheus_workspace" "example" {
 alias = "example-workspace"
}
resource "aws_prometheus_scraper" "example" {
 source {
   eks {
     cluster_arn = data.aws_eks_cluster.example.arn
     subnet_ids = data.aws_eks_cluster.example.vpc_config[0].subnet_ids
   }
 }
 destination {
   amp {
     workspace_arn = aws_prometheus_workspace.example.arn
   }
 }
 scrape_configuration = <<EOT
global:
 scrape_interval: 30s
scrape_configs:
 - job_name: 'kubernetes-apiservers'
   kubernetes_sd_configs:
     - role: endpoints
   relabel_configs:
     - action: keep
       source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
       regex: default;kubernetes;https
EOT
}

Amazon managed Grafana

We implemented Grafana dashboard monitoring using Amazon managed Grafana using Terraform as IAC. Sample code from aws_grafana_workspace | Resources | hashicorp/aws | Terraform | Terraform Registry

resource "aws_grafana_workspace" "example" {
  account_access_type      = "CURRENT_ACCOUNT"
  authentication_providers = ["SAML"]
  permission_type          = "SERVICE_MANAGED"
  role_arn                 = aws_iam_role.assume.arn

  configuration = jsonencode(
    {
      "plugins" = {
        "pluginAdminEnabled" = true
      },
      "unifiedAlerting" = {
        "enabled" = false
      }
    }
  )
}



resource "aws_iam_role" "assume" {
  name = "grafana-assume"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Sid    = ""
        Principal = {
          Service = "grafana.amazonaws.com"
        }
      },
    ]
  })
}

AWS Devops Agent

AWS DevOps Agent is a managed, always‑available “frontier agent” that acts like an autonomous on‑call engineer. It continuously learns your application topology, dependencies, and operational patterns to improve investigations and recommendations over time. (ref: Amazon.com )

We use AWS Devops Agent to analyse our issues by chatting in a natural language about our EKS Cluster. We have been getting good response appropriate to the applications we run. Around 90% of our analysis time is saved.

Refer my article for more details on AWS Devops Agent Implementation and its Applications in managing AWS EKS seamlessly. My Assistant, My Chatbot, My AWS Devops Agent | AWS Builder Center

AWS Devops Agent

AWS EKS Version Upgrade

Kubernetes rapidly evolves with new features, design updates, and bug fixes. The community releases new Kubernetes minor versions (such as 1.37) on average once every four months. Amazon EKS follows the upstream release and deprecation cycle for minor versions. As new Kubernetes versions become available in Amazon EKS, we recommend that you proactively update your clusters to use the latest available version.(ref:Understand the Kubernetes version lifecycle on EKS — Amazon EKS)

We have been doing regular in-place AWS EKS version upgrades to stay in Standard Support Model. Please refer my articles to know about upgrade using Terraform and benefits of EKS version upgrade activity.

AWS EKS version upgrade from 1.33 to 1.34 using Terraform | AWS Builder Center

Benefits of EKS Version Upgrades