Activate real-time anomaly detection

DoiT real-time cost anomaly detection monitors cost spikes on cloud workloads in near real time, using estimated on-demand costs based on usage derived from event logs.

Real-time anomaly detection for AWS workloads

Our current implementation supports real-time anomaly detection for Amazon Elastic Compute Cloud (EC2).

To activate real-time anomaly detection, you need to have AWS CloudTrail in place and enable the real-time anomaly feature on the relevant AWS accounts.

AWS CloudTrail

We recommend creating a multi-Region trail to log events for the Amazon EC2 service in all AWS Regions for cost effectiveness. If you have created an organization in AWS Organizations, you can create a multi-Region organization trail to log EC2 events in all regions of all AWS accounts in that organization.

When creating the trail, you must specify an Amazon S3 bucket created in the us-east-1 region to store the CloudTrail log files.

Enable real-time anomaly on AWS accounts

Enable real-time anomaly only on AWS accounts that host CloudTrail log files.

1. Sign in to the DoiT console, select the gear icon () from the top navigation bar, and then select Amazon Web Services.

2. Link your AWS account that hosts the CloudTrail logs to the DoiT platform if you haven't done so.

   Tip

   If you create a role manually, you need to add two IAM policies and an Amazon S3 event notification for the CloudTrail bucket. See Feature permissions for details.

3. Edit the account. Select Real-time anomaly as a feature to add to the account (see Real-time anomaly for explanations about the required permissions).

   

4. Expand the Real-time anomaly feature, enter the name of the S3 bucket that contains CloudTrail log files. The bucket must be in the us-east-1 region.

   

5. Select Update account (or Link account) to create a CloudFormation stack in the AWS console that updates the IAM role with the new permissions, or select Prefer CLI to get the command to create the CloudFormation stack via AWS CloudShell.

AWS multi-account environment

If your organization has an AWS multi-account environment set up with AWS Control Tower, and the KMS policy and CloudTrail S3 bucket are hosted by different accounts, to enable real-time anomaly, you need to perform two separate tasks:

1. Link the account that hosts the CloudTrail S3 bucket to the DoiT platform.

2. Grant the following permissions to the DoiT IAM role created in the previous step:

   • The permission to use the AWS KMS key that encrypts CloudTrail log files. You need to update the KMS policy to add a new statement that includes the ARN of the DoiT IAM role.

   • The ec2:DescribeImages permission to retrieve EC2 instance information in private images owned by the AWS account.

Real-time anomaly detection costs

The real-time anomaly detection feature leverages AWS CloudTrail events stored in an S3 bucket. The feature may incur costs from two aspects:

• Management events delivered to Amazon S3: As the first copy of management events within each region is delivered free of charge, you can avoid this cost by creating a multi-Region organization trail and removing other trails that cause duplicate events recording. See AWS CloudTrail pricing and Managing CloudTrail trail costs for more information.

• S3 Storage and access:

  • The storage cost of CloudTrail management events in S3 buckets is usually low, even for accounts with a high volume of activities.

  • The anomaly detection engine reads the log files from the same region where the S3 bucket resides (thus avoiding the higher fee for data transferred across regions or out to the internet). The cost is charged on the SKU DataTransfer-Out-Bytes and is usually under US$1.00 per month.

Below is an example report that shows the S3 costs of the real-time anomaly detection feature in recent months.

Tip

When filtering resources by S3 bucket name, if the S3 bucket you're looking for doesn't appear in the list, select the plus icon (+) to add the exact name as a new filter. See Filter results.