BigQuery real-time anomaly detection

DoiT's near real-time anomaly detection for BigQuery monitors on-demand workloads at the project level across all regions and zones. The detection system fetches raw data of query-type jobs every 30 minutes and applies on-demand pricing to estimate costs, which serve as the basis for anomaly detection.

Required permissions

To activate real-time anomaly detection for BigQuery:

• Your DoiT account must have the Manage Settings permission.

• You must have been granted the Role Administrator (roles/iam.RoleAdmin) IAM role on the relevant project.

Enable real-time anomalies for BigQuery

You can enable real-time anomaly detection for BigQuery when connecting a new project or editing an existing one (see Connect Google Cloud resources). To enable the feature on multiple projects, you need to repeat the process for each one.

Tip

We're currently working on real-time anomaly detection for slot reservations. To be ready when the feature is released, we encourage you to enable real-time anomalies for BigQuery on all your reservation administration projects in advance. See Identify administration projects for BigQuery reservations.

Enable when connecting a project

1. Sign in to the DoiT console, select Integrate from the top navigation bar, and then select Google Cloud.

2. From the Connect drop-down, select Project.

3. Select the Real-time Anomalies – BigQuery checkbox. You can expand the feature to view its required permissions. (See Security and data access policy: Feature permissions for details.)

   

4. Select Generate gcloud commands.

5. Follow the instructions displayed in the side panel to configure your service account. See Connect a Google Cloud project for more information.

6. If the connection has been set up successfully, the status of the Real-time Anomalies – BigQuery feature will show Healthy.

Enable on an existing connection

To add real-time anomalies to a connected project:

1. Locate the service account of interest on the Google Cloud access & features page.

2. Select the kebab menu (⋮) next to the project connection, and then select Edit.

3. Select the Real-time Anomalies – BigQuery checkbox to add the feature.

4. Select Generate gcloud commands.

5. Follow the instructions displayed in the side panel to update your custom role.

6. Select Done to enable the feature.

Real-time anomaly detection costs

The real-time anomaly detection feature leverages the BigQuery jobs API to collect raw query data of on-demand workloads in your connected projects. The BigQuery jobs API doesn't incur additional costs.

Disable real-time anomalies for BigQuery

To disable the real-time anomalies feature for BigQuery, run the following command to revoke the permissions granted to the DoiT custom role (make sure to replace the <ROLE_NAME> and <PROJECT_ID> with the correct values):

gcloud iam roles update <ROLE_NAME> --project=<PROJECT_ID> --remove-permissions=bigquery.jobs.get,bigquery.jobs.list,bigquery.jobs.listAll,bigquery.reservations.get,bigquery.reservations.list,monitoring.timeSeries.list,monitoring.metricDescriptors.get

Identify administration projects for BigQuery reservations

BigQuery commitments and reservations are associated with an administration project, which is billed for the committed slots (see Specifying an administration project for more information).

If you are using multiple administration projects, the easiest way to identify them is by running a report in the DoiT console with the following configurations:

• Filter results: Service is BigQuery Reservation API (filtering for a specific service will add the Provider filter automatically, see Select provider)

• Group by: Project/Account ID, SKU

The example report below shows the Google Cloud projects that were billed for BigQuery reservations in the last three months. They are the administration projects for BigQuery Reservations and should have the Real-time Anomalies – BigQuery feature enabled.