Backfill historical CUR

Before you begin

We support backfill of historical data with the following constraints:

• You must be on a dedicated payer account (i.e., not a member account of a DoiT consolidated billing management account).

• The CUR files are no older than one year.

Prepare the AWS Cost and Usage Reports

Note

Skip this step if transitioning from a DoiT consolidated billing management account to a dedicated payer account. In such cases, the DoiT Platform automatically handles historical data backfill.

AWS has limitations regarding the backfill of cost data when you change organizations or the structure of your organization. For example, if your Organization structure changed during the onboarding process with DoiT, then you can't get a backfill of data with the previous organization structure and the instructions below won't apply.

To backfill the DoiT Platform with historical data, you need to add the relevant AWS Cost and Usage Reports (AWS CUR) files to an Amazon S3 bucket in your dedicated payer account with DoiT.

To prepare the historical CUR files in case you do not have them yet:

• If you work directly with AWS, open a support case with AWS to request a backfill of your cost data. It may take 24 hours for AWS to push the data to the S3 bucket in your Amazon account. See also Backfill cost data.

• If you're moving from a different AWS reseller to DoiT, please open a support ticket to discuss the possible solutions with our support team.

Once you got the historical CUR files, proceed to the next step to add them to your DoiT S3 bucket.

Add CUR files to the S3 bucket in your DoiT account

When configuring your account in the DoiT Platform, DoiT AWS Ops team will create an Amazon S3 bucket (named doitintl-awsops-{id}) in your dedicated payer account to receive and store the CUR files. The DoiT Platform reads CUR files exclusively from this bucket and automatically discovers new CUR files added in it.

If your historical CUR files do not exist in the S3 bucket yet, you have to manually add them there. DoiT Platform doesn't automate the transfer.

1. Sign in to your AWS Billing & Cost Management console.

2. Select Cost & Usage Reports on the navigation pane, verify that you have enabled cost and usage reports.

3. Find the S3 bucket (named doitintl-awsops-{id}) in your dedicated payer account.

4. Add the manifest file of your historical CUR files to the date range folder. Below is an example:

   <base-CUR-path>/yyyymmdd-yyyymmdd/<manifest-file>-Manifest.json

   • base-CUR-path: The base path of the CUR files. Each payer has its own base folder path. For example: CUR/doitintl-awsops-{id}.

   • yyyymmdd-yyyymmdd: The billing month notation. For example: 20220101-20220201 for the billing month of January 2022.

   • <manifest-file>-Manifest.json: This manifest file is the map of the CUR files related to the billing month. In each folder there must be such a file with its name ending in -Manifest.json.

5. Add all the CUR files to the folder specified in the %-Manifest.json file under reportKeys.

   Below is an example reportKeys section in a manifest file:

   reportKeys:
     0:      "org/CUR/doitintl-awsops-{id}/example-hourly-cur/20220301-20220401/3d4115f5-cbe4-4e76-b480-c116dc632458/example-hourly-cur-1.csv.gz"
     1:      "org/CUR/doitintl-awsops-{id}/example-hourly-cur/20220301-20220401/3d4115f5-cbe4-4e76-b480-c116dc632458/example-hourly-cur-2.csv.gz"

   The CUR files must be CSV files compressed using gzip (see Amazon S3 organization and naming conventions for more information).

   Tip

   If you're moving from a DoiT consolidated account to a dedicated payer account and want a backfill of cost data of one year, you only need to copy the latest CUR file of each month from the CloudHealth S3 bucket (for example, a bucket named cloudhealth-aws-costs) to your DoiT S3 bucket (named doitintl-awsops-{id}).

6. Grant the DoiT Platform access to the S3 bucket.

Once you uploaded the files, they will be discovered by the DoiT Platform in about six hours.