Skip to main content

Import historical data

Before you begin​

Caution

Due to data latency, we suggest that after you've finished transferring Google Cloud projects, wait 72 hours before you start importing historical data.

Once you connect your Google Cloud organization to the DoiT Platform, you can upload historical Google Cloud billing data into the DoiT Platform to analyze your cloud spend and identify optimization opportunities.

  • We import your historical Cloud Billing data from Google BigQuery. To start collecting your Google Cloud Billing data, you must enable Cloud Billing data export to BigQuery.

  • You can only import as much historical Cloud Billing data as you already have in BigQuery.

    For example, suppose you enable Cloud Billing data export to BigQuery for the first time while following this document, you will only have around one month of historical data to import. See Data availability for more information.

  • You can follow the same process to import the historical billing data of a project that has moved from a non-DoiT billing account to a DoiT account.

Identify the service account​

  1. Log in to the DoiT Console, select the gear icon () from the top navigation bar, and then select Google Cloud.

  2. Select your desired service account, and copy the service account email.

    The Google Cloud settings screen

See also

Google Cloud Docs: Listing service accounts

Grant the required permissions​

Note

Your Google Cloud account needs to have the appropriate IAM role/permission to grant the required permissions to the service account.

Grant the BigQuery Data Viewer role​

You can provide the BigQuery Data Viewer role using either the Google Cloud console or the bq command-line tool.

Google Cloud console​

  1. Navigate to the BigQuery Console, and locate the dataset that contains your BQ billing data.

    See also

    Google Cloud Docs: Getting dataset information

  2. In the details panel, select Sharing, and then select Permissions.

  3. Select Add principal.

    1. In the New principals field, paste the email address of the service account that has been linked to the DoiT Platform.

    2. For Select a role, choose BigQuery and the predefined IAM role BigQuery Data Viewer.

  4. Select Done to apply the changes.

bq commands​

Copy the command code snippets below and run them in sequence in the Google Cloud Shell.

gcloud config set project $customer_billing_project_id

bq show --format=prettyjson $DATASET_ID |
  jq '.access = .access + [{"role": "READER", "userByEmail": "$customer_service_account' \
  >permissions.json

bq update --source permissions.json $DATASET_ID

Grant the bigquery.jobs.create permission​

  1. Log in to the Google Cloud console, select your organization.

  2. Navigate to the Roles page.

  3. Select the DoiT Service Account, check if the bigquery.jobs.create permission has been granted.

    • If yes, no further action is needed since the permissions are inherited by the projects.

    • If not, select Edit Role and add Permission: bigquery.jobs.create.

Required Permission
  • Assets Manager

  1. Log in to the DoiT Console, select Billing from the top navigation bar, and then select Assets.

  2. Select Google Cloud from the left-hand menu.

    The Google Cloud assets page

  3. Select Link direct account.

    Enter the following information:

    • Billing Account ID: The ID of your direct billing account. You can find your Cloud Billing accounts on the Manage Billing Accounts page in the Google Cloud console.

    • Source Project ID: The ID of the project that contains your BigQuery billing export data. See Identifying projects.

    • Source Dataset ID: The ID of the dataset that contains your BigQuery billing export data. See Getting dataset information. The dataset name is also referred to as the dataset ID.

  4. Select Import to continue.

Import status​

Status: Success​

Once the backfill process starts, the status of the billing account will show Success. Note that depending on the volume of your historical data, it may take 24 hours or more to complete the backfill.

Direct billing account with a Success status

Status: Error​

If the status of the billing account shows Error after selecting Import, check whether the information provided is accurate.

  • To correct the source project ID and the source dataset ID, select the three dots menu (⋮) at the rightmost end of the billing account entry row, and then select Manage connection to update the information.

    Direct billing account with a Success status

  • If the billing account ID needs an update, please open a support ticket to ask DoiT support team to remove the connection, then follow the instructions on this page to link the correct billing account.

Troubleshooting​

My imported data is incomplete​

Because of Google Cloud's frequency of data loads, after transferring Google Cloud projects to your Cloud Billing accounts with DoiT, data related to the period before the transfer will continue to go to the "old" Billing account.

Importing historical data into the DoiT Platform is a one-time operation. To ensure you have the complete billing data, we suggest you wait 72 hours after transferring projects before you start importing historical data.

If your imported data is incomplete, please open a support ticket to remove the connection. Then you can re-import the data.

Last updated on