Understand EKS costs

DoiT Cloud Navigator: Enhanced and above

Overview

Amazon Elastic Kubernetes Service (Amazon EKS) is a managed service to run Kubernetes on AWS. DoiT Cloud Navigator provides an EKS cost monitoring solution that uses the OpenTelemetry Collector to collect metrics generated by kube-state-metrics (KSM) of EKS clusters running on Amazon EC2 nodes. The metrics are further integrated with AWS billing data, allowing you to break down EKS cost by Kubernetes resources such as pods, nodes, namespaces and other dimensions supported in the DoiT Cloud Navigator.

Kubernetes compatibility

The DoiT EKS cost monitoring solution supports the latest release of kube-state-metrics, which interacts with Kubernetes clusters via client-go. See compatibility matrix for kube-state-metrics and client-go and compatibility matrix for client-go and Kubernetes cluster for details.

Before you begin

• The DoiT EKS cost monitoring solution is not available if you're on a DoiT consolidated billing account (payer account ID 561602220360, 017920819041, and 279843869311).

• DoiT Cloud Navigator relies on the AWS-generated cost allocation tag eks:cluster-name to identify EKS EC2 instances. Make sure the tag has been activated and your clusters are tagged correctly in the AWS billing data. See Activating AWS-generated tags cost allocation tags.

EKS clusters in the DoiT console

View EKS clusters

DoiT Cloud Navigator performs a daily scan on AWS billing data to uncover EKS clusters in your account. It assumes that the AWS-generated cost allocation tag eks:cluster-name has been applied to your EKS EC2 instances.

To view the EKS clusters:

1. Sign in to the DoiT console, select Dashboards from the top navigation bar.

2. Go to the Home page or the Account dashboard, select the Connect your EKS clusters quick Link.

   Identified clusters are listed on the EKS clusters page. (There might be a delay for newly created clusters to appear in the DoiT console.)

   

   You can perform different tasks in accordance with the status of a cluster:

   • Active: The cluster is already connected to the DoiT Cloud Navigator. You can download its deployment configuration file by selecting Download yaml from the three dots menu at the rightmost end of the cluster entry row.

   • Not started: The cluster is not connected to the DoiT Cloud Navigator yet. Select Connect to start the connecting process.

   • Deleted: The CloudFormation stack that allows DoiT to access the cluster has been removed. Select Fix to re-connect the cluster or download the deployment configuration file via the three dots menu at the rightmost end of the cluster entry row.

Connect an EKS cluster

To analyze the usage and cost of resources in an EKS cluster, you need to connect the cluster to the DoiT Cloud Navigator. You can choose either Terraform or CloudFormation (with kubectl or Helm) for the deployment. Be aware that:

• Clusters in the same account or region must be deployed using the same deployment method.

• It may take up to 24 hours before your data appear in DoiT Cloud Analytics.

Required permissions

To connect an EKS cluster to the DoiT Cloud Navigator, you need the following permissions:

• Kubernetes admin permissions to deploy the OpenTelemetry Collector agent on the cluster, and

• AWS permissions to set up the necessary resources, including:

  • Creating a role and IAM policy to export the EKS metrics collected by the OpenTelemetry Collector, creating an S3 bucket to store the metrics, and setting up an OpenID Connect (OIDC) identity provider that authenticates the role to the cluster.
  • Creating a role and IAM policy to give DoiT read-only access to get the metrics from the S3 bucket.
  Details

  Permission	Description
  iam:CreatePolicy	Creates a new managed policy for your AWS account.
  iam:DeletePolicy	Deletes the specified managed policy.
  iam:AttachRolePolicy	Attaches the specified managed policy to an IAM role.
  iam:DetachRolePolicy	Removes the specified managed policy from a role.
  iam:CreateRole	Creates a new role for your AWS account.
  iam:DeleteRole	Deletes the specified role.
  iam:PutRolePolicy	Adds or updates a policy document in the specified IAM role.
  iam:DeleteRolePolicy	Deletes the specified policy in the specified IAM role.
  iam:ListRoles	Lists the IAM roles with the specified path prefix.
  iam:GetRole	Retrieves information about an IAM role, including the role's path, GUID, ARN, and the trust policy that grants permission to assume the role.
  iam:PassRole	Passes an IAM role to an AWS service.
  iam:GetPolicy	Retrieves information about the specified managed policy.
  s3:CreateBucket	Creates a new Amazon S3 bucket.
  s3:DeleteBucket	Deletes a specific S3 bucket.
  s3:PutLifecycleConfiguration	Creates a new lifecycle configuration for the S3 bucket or replaces an existing one.
  s3:GetLifecycleConfiguration	Returns the lifecycle configuration information set on the S3 bucket.
  lambda:CreateFunction	Creates a Lambda function.
  lambda:UpdateFunctionCode	Updates a Lambda function's code.
  lambda:UpdateFunctionConfiguration	Modifies the version-specific settings of a Lambda function.
  lambda:DeleteFunction	Deletes a Lambda function.
  lambda:AddPermission	Grants an AWS service, AWS account, or AWS organization permission to use a function.
  lambda:GetFunction	Returns information about the function or function version, with a link to download the deployment package.
  lambda:InvokeFunction	Invokes a Lambda function.
  cloudformation:CreateStack	Creates a stack as specified in the template.
  cloudformation:DescribeStacks	Returns the description for the specified stack or all the stacks created.
  cloudformation:DescribeStackEvents	Returns all stack related events for a specified stack in reverse chronological order.
  cloudformation:DeleteStack	Deletes a specified stack.
  cloudformation:DescribeStackResource	Returns a description of the specified resource in the specified stack.
  cloudformation:DescribeStackResources	Returns AWS resource descriptions for running and deleted stacks.
  cloudformation:GetTemplate	Returns the template body for a specified stack.
  cloudformation:GetTemplateSummary	Returns information about a new or existing template.
  cloudformation:ListStacks	Returns the summary information for stacks with matching status.
  cloudformation:UpdateStack	Updates a stack as specified in the template.
  SNS:Publish	Gives users permissions to publish to the topic.

See also

Amazon EKS User Guide: Allowing users to access your cluster

Terraform deployment

1. In the DoiT console, select Terraform as the deployment method.

2. Clone the DoiT terraform-eks-lens repository for the account/region.

   git clone https://github.com/doitintl/terraform-eks-lens.git eks-lens-ACCOUNT-REGION
   
   cd eks-lens-ACCOUNT-REGION

3. Sign in to the Amazon EKS console, select your cluster on the Clusters page.

4. In the Details section on the Overview tab, copy the value of the OpenID Connect provider URL and then paste it in the DoiT console to download the Terraform configuration file, CLUSTERNAME.tf, for your cluster. Save the downloaded file in the current Terraform directory.

5. Create a new file named CLUSTERNAME_provider.tf, copy the code snippet in the DoiT console and modify it to set up your Terraform Kubernetes provider.

6. Copy the code snippet in the DoiT console and modify it to set up your AWS provider in the aws_provider.tf file.

7. Run the following Terraform commands in sequence:

   • The terraform init command initializes a working directory containing Terraform configuration files.
   • The terraform plan command creates an execution plan that allows you to preview the changes that Terraform plans to make to your infrastructure.
   • The terraform apply command executes the actions proposed in the Terraform plan.

8. In the DoiT console, select Finish to complete the deployment.

If successful, the status of cluster shows Active on the EKS clusters page.

Troubleshooting

If you've successfully executed the Terraform commands but the state of your cluster still shows Not started, try the following:

1. Open your Terraform configuration file CLUSTERNAME.tf, find the curl command in the section of null_resource and deploy_cluster.

2. Run the curl command to send a request to https://console.doit.com/webhooks/v1/eks-metrics/terraform-validate with the correct parameters.

CloudFormation deployment

The CloudFormation deployment process consists of two steps.

Step 1: Add permission

In this step, you create a CloudFormation stack using the DoiT EKS onboarding template.

1. In the DoiT console, select CloudFormation as the deployment method, click Next, and then select Open CloudFormation Stack.

2. In the AWS CloudFormation console, review the pre-populated fields, and then create a stack using the DoiT template.

3. Select the checkbox at the bottom of the page to acknowledge that AWS CloudFormation might create IAM resources with custom names.

4. Create the stack.

5. Once the stack is created, navigate back to the DoiT console. You should see a confirmation message that says Permission successfully added. Select Next to proceed.

Step 2: Connect and validate

In this step, you use kubectl or Helm to install the required components on your Kubernetes clusters using an auto-generated Kubernetes Deployment file or an EKS Lens Helm chart.

Using kubectl

1. Download the deployment YAML file, if you haven't done so in the previous step.

2. Open AWS CloudShell in the AWS Management Console. Upload the deployment YAML file.

3. In the DoiT console, copy the command kubectl apply -f DEPLOYMENT_YAML_FILE. It is used to update the cluster configuration.

4. Paste the command in the AWS CloudShell and run it to update the cluster configuration.

The deployment file creates two service accounts in the namespace doit-eks-metrics:

• Service account doit-kube-state-metrics: To deploy the kube-state-metrics (KSM) service on the cluster.
• Service account doit-collector: To deploy the OpenTelemetry Collector on the cluster.

1. In the DoiT console, select Check to validate the connection. If successful, the status of cluster shows Active on the EKS clusters page.

Using Helm

1. Install Helm on your local system.

2. Copy the Helm commands shown in the DoiT console and run them in sequence.

   • The helm repo add command adds the chart repository to your local helm installation.
   • The helm template command renders chart template locally.
   • The helm upgrade --install command installs the doit-eks-lens chart with its specific kube-state-metrics deployment.

3. In the DoiT console, select Check to validate the connection. If successful, the status of cluster shows Active on the EKS clusters page.

Multiple EKS clusters

If you have multiple EKS clusters, create a new CloudFormation stack for each cluster as some AWS resources are deployed at the cluster level.

The S3 bucket should be created only once when onboarding the first cluster in each account/region. If the stack creation for a later cluster fails because of the existing S3 bucket, set the CreateBucket parameter to false; you should not change the bucket name.

Analyzing EKS cost

DoiT Cloud Analytics integrates EKS cluster metrics with AWS billing data. It allows you to break down EKS cost by Kubernetes resources and other Dimensions and labels supported in the DoiT Cloud Navigator.

To build and run Cloud Analytics reports, you need the Cloud Analytics permission in the DoiT console.

EKS labels

EKS labels are assigned at the Pods level. You can use EKS labels to track Kubernetes resources usage, analyze project cost, and identify cost optimization opportunities.

In the DoiT Cloud Analytics Reports, EKS labels are grouped under the EKS Labels section, which consists of two categories:

• Kubernetes metadata: Data that helps uniquely identify a Kubernetes object, including EKS billing_source, EKS cluster name, EKS deployment_kind EKS pod, EKS uid, EKS pod_owner_name, and EKS nodegroup name.

• Custom labels: Labels defined and assigned from your side. These labels appear with a prefix label_ in the DoiT console, for example, label_k8s_app.

DoiT Cloud Analytics integrates EKS metrics with AWS billing data, which means you can combine EKS labels with AWS cost allocation tags assigned to other AWS resources in the same report.

Example reports

The DoiT console provides a pre-built EKS Lens dashboard with report widgets that highlight various aspects of your Amazon EKS spend. You can also build your own reports using EKS labels and other dimensions. Below are some examples.

Cost breakdown by namespace and pod owner

This example breaks down the cost of the EKS cluster eks-cluster-1 in the last month by namepace and pod owner.

Cost breakdown by cost type and SKU

This example further breaks down the EKS cluster cost by cost type and SKU.

Cost attribution across services

In this example, the organization uses the label Team to distinguish AWS resources requested by different entities. To display the costs of EKS and other AWS services in the same report, you can include both the standard label Team and the EKS custom label EKS label_team. (Alternatively, you can create an attribution that contains the standard label and its corresponding EKS custom label, and then use attributions in your report.)

EKS clusters offboarding

To offboard an EKS cluster from the DoiT Cloud Navigator:

• Cluster connected via Terraform: Run the terraform destroy command to destroy the full stack based on your CLUSTERNAME.tf file, or use the -target option to destroy single resources, for example, terraform destroy -target RESOURCE_TYPE.NAME.

• Cluster connected via CloudFormation with helm:

  1. Delete the CloudFormation stack of the cluster from your AWS account. See Deleting a stack.

  2. Run the helm uninstall doit-eks-lens command to delete the agent (OpenTelemetry Collector) from Kubernetes.

• Clusters connected via CloudFormation with kubectl:

  1. Delete the CloudFormation stack of the cluster from your AWS account. See Deleting a stack.

  2. Run the kubectl delete -f DEPLOYMENT_YAML_FILE command from the AWS CloudShell to delete the agent (OpenTelemetry Collector) configuration.

To remove multiple clusters, repeat the steps above for each one.

FAQ

What's the difference between EKS Lens, Kubecost, and OpenCost?

While Kubecost and OpenCost provide valuable Kubernetes cost management capabilities, DoiT EKS Lens is specifically optimized for AWS environments, offering a more integrated approach to managing EKS costs and performance:

• EKS Lens is designed with a deep understanding of AWS services and their integration with EKS, providing insights and analytics that are closely aligned with AWS billing and management practices.

• EKS Lens focuses on EKS-specific metrics, offering detailed insights into cluster performance, cost allocation, and optimization opportunities within the AWS ecosystem.

• EKS Lens leverages AWS-native services for enhanced security, compliance, and performance monitoring, ensuring a cohesive and secure environment for your EKS clusters.

• With EKS Lens, you can easily customize dashboards and reports in the DoiT console, enabling effective cost management and optimization strategies.

The table below lists the main differences between DoiT EKS Lens, Amazon EKS optimized Kubecost custom bundle, and OpenCost.

Feature	DoiT EKS Lens	Amazon EKS optimized Kubecost custom bundle	OpenCost
Description	A DoiT Cloud Analytics feature that provides comprehensive visibility into EKS spend.	A customized version of Kubecost by AWS, including a subset of commercial features.	A vendor-neutral open source project for measuring and allocating cloud infrastructure and container costs in real time.
Number of clusters	Unlimited (unified multi-cluster view)	Unlimited (unified multi-cluster view)	Unlimited (no unified view)
Number of labels for pods	Unlimited	Unlimited	Unlimited
EKS on Fargate	Limited support (for Amazon EKS on Fargate, DoiT Cloud Analytics uses metrics from AWS billing data but not metrics generated by kube-state-metrics).	Kubecost tracks costs for Amazon EKS on Fargate with lower accuracy than with EKS on EC2.	No support for Amazon EKS on Fargate.
Deployment	Terraform deployment: using DoiT terraform-eks-lens repository; bundled with OpenTelemetry Collector and kube-state-metrics dependencies. CloudFormation deployment: using CloudFormation stack and kubectl or DoiT EKS Lens Helm chart. Bundled with OpenTelemetry Collector and kube-state-metrics dependencies.	Deployed using Helm. Bundled with Prometheus and Grafana dependencies.	Deployed as a pod. Prometheus and kube-state-metrics dependencies are managed separately. Also provides a community-supported Helm chart.
Metrics retention	Subject to your contract with DoiT.	Unlimited historical data for database retention; 15 days for the Kubecost API retention (ETL).	Limited by Prometheus environment.
Cloud billing integration	DoiT Cloud Advantage: Custom pricing support for AWS (including multiple clusters and multiple accounts). DoiT Cloud Navigator: Core billing integrations with AWS EKS.	Custom pricing support for AWS (including multiple clusters and multiple accounts).	Core billing integrations with AWS EKS.
Support	Subject to your contract with DoiT.	Subject to your AWS support agreements.	Built and supported by community users.

See also

• AWS Documentation: Cost monitoring: Frequently asked questions

• OpenCost Documentation: FAQ: What's the difference between OpenCost and Kubecost?

• Kubecost Documentation: OpenCost Product Comparison

Can I use existing OpenTelemetry Collector and kube-state-metrics on my clusters for DoiT EKS Lens?

Currently, no. You need to deploy the two components following the instructions to Connect an EKS cluster. The components will be deployed in the namespace doit-eks-metrics.

There is no quick link to "connect your EKS clusters".

The DoiT Cloud Navigator uses the AWS-generated cost allocation tag eks:cluster-name to identify EKS EC2 instances. Make sure your clusters are tagged correctly in the AWS billing data. See Activating AWS-generated tags cost allocation tags and AWS cost allocation tags in DoiT Cloud Navigator.

In addition, the DoiT Cloud Navigator scans the billing data on a daily basis. If your clusters are newly created (within 72 hours) or have no spend, then the DoiT Cloud Navigator will not be able to detect them.

Interactive demo

Try out our interactive demo for a hands-on walk-through experience.

If the demo doesn't display properly, try expanding your browser window or opening the demo in a new tab.