Understand EKS costs
Overview
Amazon Elastic Kubernetes Service (Amazon EKS) is a managed service to run Kubernetes on AWS. DoiT Cloud Navigator provides an EKS cost monitoring solution that uses the OpenTelemetry Collector to collect metrics generated by kube-state-metrics (KSM) of EKS clusters running on Amazon EC2 nodes. The metrics are further integrated with AWS billing data, allowing you to break down EKS cost by Kubernetes resources such as pods, nodes, namespaces and other dimensions supported in the DoiT Cloud Navigator.
Kubernetes compatibility
The DoiT EKS cost monitoring solution supports the latest release of kube-state-metrics
, which interacts with Kubernetes clusters via client-go
. See compatibility matrix for kube-state-metrics and client-go and compatibility matrix for client-go and Kubernetes cluster for details.
Before you begin
-
The DoiT EKS cost monitoring solution is not available if you're on a DoiT consolidated billing account (payer account ID
561602220360
,017920819041
, and279843869311
). -
DoiT Cloud Navigator relies on the AWS-generated cost allocation tag
eks:cluster-name
to identify EKS EC2 instances. Make sure the tag has been activated and your clusters are tagged correctly in the AWS billing data. See Activating AWS-generated tags cost allocation tags.
EKS clusters in the DoiT console
View EKS clusters
DoiT Cloud Navigator performs a daily scan on AWS billing data to uncover EKS clusters in your account. It assumes that the AWS-generated cost allocation tag eks:cluster-name
has been applied to your EKS EC2 instances.
To view the EKS clusters:
-
Sign in to the DoiT console, select Dashboards from the top navigation bar.
-
Go to the Home page or the Account dashboard, select the Connect your EKS clusters quick Link.
Identified clusters are listed on the EKS clusters page. (There might be a delay for newly created clusters to appear in the DoiT console.)
You can perform different tasks in accordance with the status of a cluster:
-
Active: The cluster is already connected to the DoiT Cloud Navigator. You can download its deployment configuration file by selecting Download yaml from the three dots menu at the rightmost end of the cluster entry row.
-
Not started: The cluster is not connected to the DoiT Cloud Navigator yet. Select Connect to start the connecting process.
-
Deleted: The CloudFormation stack that allows DoiT to access the cluster has been removed. Select Fix to re-connect the cluster or download the deployment configuration file via the three dots menu at the rightmost end of the cluster entry row.
-
Connect an EKS cluster
To analyze the usage and cost of resources in an EKS cluster, you need to connect the cluster to the DoiT Cloud Navigator. You can choose either Terraform or CloudFormation (with kubectl or Helm) for the deployment. Be aware that:
-
Clusters in the same account or region must be deployed using the same deployment method.
-
It may take up to 24 hours before your data appear in DoiT Cloud Analytics.
Required permissions
To connect an EKS cluster to the DoiT Cloud Navigator, you need the following permissions:
-
Kubernetes admin permissions to deploy the OpenTelemetry Collector agent on the cluster, and
-
AWS permissions to set up the necessary resources, including:
- Creating a role and IAM policy to export the EKS metrics collected by the OpenTelemetry Collector, creating an S3 bucket to store the metrics, and setting up an OpenID Connect (OIDC) identity provider that authenticates the role to the cluster.
- Creating a role and IAM policy to give DoiT read-only access to get the metrics from the S3 bucket.
Details
Permission Description iam:CreatePolicy
Creates a new managed policy for your AWS account. iam:DeletePolicy
Deletes the specified managed policy. iam:AttachRolePolicy
Attaches the specified managed policy to an IAM role. iam:DetachRolePolicy
Removes the specified managed policy from a role. iam:CreateRole
Creates a new role for your AWS account. iam:DeleteRole
Deletes the specified role. iam:PutRolePolicy
Adds or updates a policy document in the specified IAM role. iam:DeleteRolePolicy
Deletes the specified policy in the specified IAM role. iam:ListRoles
Lists the IAM roles with the specified path prefix. iam:GetRole
Retrieves information about an IAM role, including the role's path, GUID, ARN, and the trust policy that grants permission to assume the role. iam:PassRole
Passes an IAM role to an AWS service. iam:GetPolicy
Retrieves information about the specified managed policy. s3:CreateBucket
Creates a new Amazon S3 bucket. s3:DeleteBucket
Deletes a specific S3 bucket. s3:PutLifecycleConfiguration
Creates a new lifecycle configuration for the S3 bucket or replaces an existing one. s3:GetLifecycleConfiguration
Returns the lifecycle configuration information set on the S3 bucket. lambda:CreateFunction
Creates a Lambda function. lambda:UpdateFunctionCode
Updates a Lambda function's code. lambda:UpdateFunctionConfiguration
Modifies the version-specific settings of a Lambda function. lambda:DeleteFunction
Deletes a Lambda function. lambda:AddPermission
Grants an AWS service, AWS account, or AWS organization permission to use a function. lambda:GetFunction
Returns information about the function or function version, with a link to download the deployment package. lambda:InvokeFunction
Invokes a Lambda function. cloudformation:CreateStack
Creates a stack as specified in the template. cloudformation:DescribeStacks
Returns the description for the specified stack or all the stacks created. cloudformation:DescribeStackEvents
Returns all stack related events for a specified stack in reverse chronological order. cloudformation:DeleteStack
Deletes a specified stack. cloudformation:DescribeStackResource
Returns a description of the specified resource in the specified stack. cloudformation:DescribeStackResources
Returns AWS resource descriptions for running and deleted stacks. cloudformation:GetTemplate
Returns the template body for a specified stack. cloudformation:GetTemplateSummary
Returns information about a new or existing template. cloudformation:ListStacks
Returns the summary information for stacks with matching status. cloudformation:UpdateStack
Updates a stack as specified in the template. SNS:Publish
Gives users permissions to publish to the topic.
Amazon EKS User Guide: Allowing users to access your cluster
Terraform deployment
-
In the DoiT console, select Terraform as the deployment method.
-
Clone the DoiT terraform-eks-lens repository for the account/region.
git clone https://github.com/doitintl/terraform-eks-lens.git eks-lens-ACCOUNT-REGION
cd eks-lens-ACCOUNT-REGION -
Sign in to the Amazon EKS console, select your cluster on the Clusters page.
-
In the Details section on the Overview tab, copy the value of the OpenID Connect provider URL and then paste it in the DoiT console to download the Terraform configuration file,
CLUSTERNAME.tf
, for your cluster. Save the downloaded file in the current Terraform directory. -
Create a new file named
CLUSTERNAME_provider.tf
, copy the code snippet in the DoiT console and modify it to set up your Terraform Kubernetes provider. -
Copy the code snippet in the DoiT console and modify it to set up your AWS provider in the
aws_provider.tf
file. -
Run the following Terraform commands in sequence:
- The
terraform init
command initializes a working directory containing Terraform configuration files. - The
terraform plan
command creates an execution plan that allows you to preview the changes that Terraform plans to make to your infrastructure. - The
terraform apply
command executes the actions proposed in the Terraform plan.
- The
-
In the DoiT console, select Finish to complete the deployment.
If successful, the status of cluster shows Active on the EKS clusters page.
Troubleshooting
If you've successfully executed the Terraform commands but the state of your cluster still shows Not started
, try the following:
-
Open your Terraform configuration file
CLUSTERNAME.tf
, find the curl command in the section ofnull_resource
anddeploy_cluster
. -
Run the curl command to send a request to
https://console.doit.com/webhooks/v1/eks-metrics/terraform-validate
with the correct parameters.
CloudFormation deployment
The CloudFormation deployment process consists of two steps.
Step 1: Add permission
In this step, you create a CloudFormation stack using the DoiT EKS onboarding template.
-
In the DoiT console, select CloudFormation as the deployment method, click Next, and then select Open CloudFormation Stack.
-
In the AWS CloudFormation console, review the pre-populated fields, and then create a stack using the DoiT template.
-
Select the checkbox at the bottom of the page to acknowledge that AWS CloudFormation might create IAM resources with custom names.
-
Create the stack.
-
Once the stack is created, navigate back to the DoiT console. You should see a confirmation message that says
Permission successfully added
. Select Next to proceed.
Step 2: Connect and validate
In this step, you use kubectl or Helm to install the required components on your Kubernetes clusters using an auto-generated Kubernetes Deployment file or an EKS Lens Helm chart.
Using kubectl
-
Download the deployment YAML file, if you haven't done so in the previous step.
-
Open AWS CloudShell in the AWS Management Console. Upload the deployment YAML file.
-
In the DoiT console, copy the command
kubectl apply -f DEPLOYMENT_YAML_FILE
. It is used to update the cluster configuration. -
Paste the command in the AWS CloudShell and run it to update the cluster configuration.
The deployment file creates two service accounts in the namespace doit-eks-metrics
:
- Service account
doit-kube-state-metrics
: To deploy the kube-state-metrics (KSM) service on the cluster. - Service account
doit-collector
: To deploy the OpenTelemetry Collector on the cluster.
- In the DoiT console, select Check to validate the connection. If successful, the status of cluster shows Active on the EKS clusters page.
Using Helm
-
Install Helm on your local system.
-
Copy the Helm commands shown in the DoiT console and run them in sequence.
- The
helm repo add
command adds the chart repository to your local helm installation. - The
helm template
command renders chart template locally. - The
helm upgrade --install
command installs thedoit-eks-lens
chart with its specifickube-state-metrics
deployment.
- The
-
In the DoiT console, select Check to validate the connection. If successful, the status of cluster shows Active on the EKS clusters page.
Multiple EKS clusters
If you have multiple EKS clusters, create a new CloudFormation stack for each cluster as some AWS resources are deployed at the cluster level.
The S3 bucket should be created only once when onboarding the first cluster in each account/region. If the stack creation for a later cluster fails because of the existing S3 bucket, set the CreateBucket parameter to false
; you should not change the bucket name.
Analyzing EKS cost
DoiT Cloud Analytics integrates EKS cluster metrics with AWS billing data. It allows you to break down EKS cost by Kubernetes resources and other Dimensions and labels supported in the DoiT Cloud Navigator.
To build and run Cloud Analytics reports, you need the Cloud Analytics permission in the DoiT console.
EKS labels
EKS labels are assigned at the Pods level. You can use EKS labels to track Kubernetes resources usage, analyze project cost, and identify cost optimization opportunities.
In the DoiT Cloud Analytics Reports, EKS labels are grouped under the EKS Labels section, which consists of two categories:
-
Kubernetes metadata: Data that helps uniquely identify a Kubernetes object, including
EKS billing_source
,EKS cluster name
,EKS deployment_kind
EKS pod
,EKS uid
,EKS pod_owner_name
, andEKS nodegroup name
. -
Custom labels: Labels defined and assigned from your side. These labels appear with a prefix
label_
in the DoiT console, for example,label_k8s_app
.
DoiT Cloud Analytics integrates EKS metrics with AWS billing data, which means you can combine EKS labels with AWS cost allocation tags assigned to other AWS resources in the same report.
Example reports
The DoiT console provides a pre-built EKS Lens dashboard with report widgets that highlight various aspects of your Amazon EKS spend. You can also build your own reports using EKS labels and other dimensions. Below are some examples.
Cost breakdown by namespace and pod owner
This example breaks down the cost of the EKS cluster eks-cluster-1
in the last month by namepace and pod owner.
Cost breakdown by cost type and SKU
This example further breaks down the EKS cluster cost by cost type and SKU.
Cost attribution across services
In this example, the organization uses the label Team
to distinguish AWS resources requested by different entities. To display the costs of EKS and other AWS services in the same report, you can include both the standard label Team
and the EKS custom label EKS label_team
. (Alternatively, you can create an attribution that contains the standard label and its corresponding EKS custom label, and then use attributions in your report.)
EKS clusters offboarding
To offboard an EKS cluster from the DoiT Cloud Navigator:
-
Cluster connected via Terraform: Run the terraform destroy command to destroy the full stack based on your
CLUSTERNAME.tf
file, or use the-target
option to destroy single resources, for example,terraform destroy -target RESOURCE_TYPE.NAME
. -
Cluster connected via CloudFormation with helm:
-
Delete the CloudFormation stack of the cluster from your AWS account. See Deleting a stack.
-
Run the
helm uninstall doit-eks-lens
command to delete the agent (OpenTelemetry Collector) from Kubernetes.
-
-
Clusters connected via CloudFormation with kubectl:
-
Delete the CloudFormation stack of the cluster from your AWS account. See Deleting a stack.
-
Run the
kubectl delete -f DEPLOYMENT_YAML_FILE
command from the AWS CloudShell to delete the agent (OpenTelemetry Collector) configuration.
-
To remove multiple clusters, repeat the steps above for each one.
FAQ
What's the difference between EKS Lens, Kubecost, and OpenCost?
While Kubecost and OpenCost provide valuable Kubernetes cost management capabilities, DoiT EKS Lens is specifically optimized for AWS environments, offering a more integrated approach to managing EKS costs and performance:
-
EKS Lens is designed with a deep understanding of AWS services and their integration with EKS, providing insights and analytics that are closely aligned with AWS billing and management practices.
-
EKS Lens focuses on EKS-specific metrics, offering detailed insights into cluster performance, cost allocation, and optimization opportunities within the AWS ecosystem.
-
EKS Lens leverages AWS-native services for enhanced security, compliance, and performance monitoring, ensuring a cohesive and secure environment for your EKS clusters.
-
With EKS Lens, you can easily customize dashboards and reports in the DoiT console, enabling effective cost management and optimization strategies.
The table below lists the main differences between DoiT EKS Lens, Amazon EKS optimized Kubecost custom bundle, and OpenCost.
Feature | DoiT EKS Lens | Amazon EKS optimized Kubecost custom bundle | OpenCost |
---|---|---|---|
Description | A DoiT Cloud Analytics feature that provides comprehensive visibility into EKS spend. | A customized version of Kubecost by AWS, including a subset of commercial features. | A vendor-neutral open source project for measuring and allocating cloud infrastructure and container costs in real time. |
Number of clusters | Unlimited (unified multi-cluster view) | Unlimited (unified multi-cluster view) | Unlimited (no unified view) |
Number of labels for pods | Unlimited | Unlimited | Unlimited |
EKS on Fargate | Limited support (for Amazon EKS on Fargate, DoiT Cloud Analytics uses metrics from AWS billing data but not metrics generated by kube-state-metrics ). | Kubecost tracks costs for Amazon EKS on Fargate with lower accuracy than with EKS on EC2. | No support for Amazon EKS on Fargate. |
Deployment |
| Deployed using Helm. Bundled with Prometheus and Grafana dependencies. | Deployed as a pod. Prometheus and kube-state-metrics dependencies are managed separately.Also provides a community-supported Helm chart. |
Metrics retention | Subject to your contract with DoiT. | Unlimited historical data for database retention; 15 days for the Kubecost API retention (ETL). | Limited by Prometheus environment. |
Cloud billing integration |
| Custom pricing support for AWS (including multiple clusters and multiple accounts). | Core billing integrations with AWS EKS. |
Support | Subject to your contract with DoiT. | Subject to your AWS support agreements. | Built and supported by community users. |
-
AWS Documentation: Cost monitoring: Frequently asked questions
-
OpenCost Documentation: FAQ: What's the difference between OpenCost and Kubecost?
-
Kubecost Documentation: OpenCost Product Comparison
Can I use existing OpenTelemetry Collector and kube-state-metrics on my clusters for DoiT EKS Lens?
Currently, no. You need to deploy the two components following the instructions to Connect an EKS cluster. The components will be deployed in the namespace doit-eks-metrics
.
There is no quick link to "connect your EKS clusters".
The DoiT Cloud Navigator uses the AWS-generated cost allocation tag eks:cluster-name
to identify EKS EC2 instances. Make sure your clusters are tagged correctly in the AWS billing data. See Activating AWS-generated tags cost allocation tags and AWS cost allocation tags in DoiT Cloud Navigator.
In addition, the DoiT Cloud Navigator scans the billing data on a daily basis. If your clusters are newly created (within 72 hours) or have no spend, then the DoiT Cloud Navigator will not be able to detect them.
Interactive demo
Try out our interactive demo for a hands-on walk-through experience.
If the demo doesn't display properly, try expanding your browser window or opening the demo in a new tab.