Activate real-time anomaly detection

DoiT real-time cost anomaly detection monitors cost spikes on cloud workloads in near real time, using estimated on-demand costs based on usage derived from event logs.

Note

Real-time anomaly detection is available with a DoiT Cloud Intelligence Enhanced or Enterprise plan.

Real-time anomaly detection for AWS workloads

Our current implementation supports real-time anomaly detection for:

• Amazon Elastic Compute Cloud (EC2)

• Amazon Relational Database Service (RDS). For provisioned RDS instances, the following components are supported:

  • DB Instance hours: Based on the type of DB instance consumed.

  • Storage (per GB per month): Based on the storage capacity you have provisioned to your DB instance.

  • Provisioned IOPS per month: Provisioned IOPS rate, regardless of IOPS consumed (For Amazon RDS Provisioned IOPS storage only).

  See also Working with storage for Amazon RDS DB instances.

To activate real-time anomaly detection, you need to have AWS CloudTrail in place and enable the real-time anomaly feature on the relevant AWS accounts.

AWS CloudTrail

We recommend creating a multi-Region trail to log events in all AWS Regions.

If you have created an organization in AWS Organizations, you can create a multi-Region organization trail to log events in all regions of all AWS accounts in that organization.

S3 bucket's region

When creating the trail, you must choose an Amazon S3 bucket created in one of the following regions to store the CloudTrail log files:

• us-east-1
• us-east-2
• us-west-2
• eu-west-1
• eu-central-1

Enable real-time anomaly on AWS accounts

You can enable real-time anomaly detection when linking a new account or editing an existing one. Real-time anomaly detection should be enabled only for AWS accounts that host CloudTrail log files.

Enable on a new account

1. Sign in to the DoiT console, select the gear icon () from the top navigation bar, and then select Amazon Web Services under Cloud settings.

2. On the Link Amazon Web Services page, select Link account.

3. Provide information about the S3 bucket that stores the CloudTrail logs. Note that the configuration varies with the way you create the AWS IAM role.

   • Create a role automatically: Select the Real-time anomaly feature and enter the S3 bucket name.

     

     Tip

     The CloudFormation stack created for the IAM role also grants the s3:GetBucketLocation and s3:PutBucketNotification permissions, which are necessary for the DoiT Real-time anomaly feature to locate the bucket and add an AWS CloudTrail bucket notification. You can remove these two permissions after the account has been successfully linked. See Real-time anomaly for more information about the required permissions.

   • Create a role manually: Enter the S3 bucket name and specify the region where the bucket resides.

     

Enable on an existing account

To enable real-time anomaly on an already linked account:

1. Locate the account of interest on the Link Amazon Web Services page.

2. Select the three dots menu (⋮) at the rightmost end of the account entry, and then select Edit account.

3. Select Real-time anomaly as the feature to add, expand the feature, and enter the name of the S3 bucket that contains CloudTrail log files. The bucket must reside in one of the supported regions.

   

4. Select Update account to create a CloudFormation stack in the AWS console that updates the IAM role with the new permissions, or select Prefer CLI to get the command to create the CloudFormation stack via AWS CloudShell.

   Tip

   The CloudFormation stack created for the IAM role also grants the s3:GetBucketLocation and s3:PutBucketNotification permissions, which are necessary for the DoiT Real-time anomaly feature to locate the bucket and add an AWS CloudTrail bucket notification. You can remove these two permissions after the account has been successfully linked. See Real-time anomaly for more information about the required permissions.

AWS multi-account environment

If your organization has an AWS multi-account environment set up with AWS Control Tower, and the KMS policy and CloudTrail S3 bucket are hosted by different accounts, to enable real-time anomaly, you need to perform two separate tasks:

1. Link the account that hosts the CloudTrail S3 bucket to the DoiT platform.

2. Grant the following permissions to the DoiT IAM role created in the previous step:

   • The permission to use the AWS KMS key that encrypts CloudTrail log files. You need to update the KMS policy to add a new statement that includes the ARN of the DoiT IAM role.

   • The ec2:DescribeImages permission to retrieve EC2 instance information in private images owned by the AWS account.

Real-time anomaly detection costs

The real-time anomaly detection feature leverages AWS CloudTrail events stored in an S3 bucket. The feature may incur costs from two aspects:

• Management events delivered to Amazon S3: As the first copy of management events within each region is delivered free of charge, you can avoid this cost by creating a multi-Region organization trail and removing other trails that cause duplicate events recording. See AWS CloudTrail pricing and Managing CloudTrail trail costs for more information.

• S3 Storage and access:

  • The storage cost of CloudTrail management events in S3 buckets is usually low, even for accounts with a high volume of activities.

  • The anomaly detection engine reads the log files from the same region where the S3 bucket resides (thus avoiding the higher fee for data transferred across regions or out to the internet). The cost is charged on the SKU DataTransfer-Out-Bytes and is usually under US$1.00 per month.

Below is an example report that shows the S3 costs of the real-time anomaly detection feature in recent months.

Tip

When filtering resources by S3 bucket name, if the S3 bucket you're looking for doesn't appear in the list, select the plus icon (+) to add the exact name as a new filter. See Filter results.