Skip to main content

Google Cloud real-time anomaly detection

DoiT supports real-time anomaly detection for Google Compute Engine (GCE) workloads across all regions and zones, including both standard and custom machine types. To activate real-time anomaly detection, you need to enable the feature at the organization level.

Cloud Logging audit logs​

Google Cloud real-time anomaly detection uses estimated on-demand costs based on usage derived from Cloud Logging audit logs.

  • The anomaly detection system creates a log sink named realtime-pipeline-log-sink-prod-<ORG-ID> at the organization level.

  • The log sink captures GCE activities of all the projects in the organization, using the following filter: resource.type="gce_instance" AND logName:"cloudaudit.googleapis.com%2Factivity"

  • The log sink publishes audit logs to a Pub/Sub topic named doit-realtime-pipeline-pubsub-topic in DoiT's Google Cloud organization. Audit logs are then processed for real-time anomaly detection.

Required permissions​

To activate real-time anomaly detection for Google Cloud:

  • Your DoiT account must have the Manage Settings permission.

  • You must have been granted the Organization Role Administrator (roles/iam.organizationRoleAdmin) IAM role in your organization.

Enable real-time anomalies for GCE​

You can enable real-time anomaly detection for GCE when connecting an organization or editing an existing connection (see Connect Google Cloud resources).

Enable when connecting an organization​

  1. Sign in to the DoiT console, select Integrate from the top navigation bar, and then select Google Cloud.

  2. From the Connect drop-down, select Organization.

  3. Select the Real-time Anomalies – GCE checkbox. You can expand the feature to view its required permissions. (See Security and data access policy: Feature permissions for details.)

    Add real-time anomaly to GCP organization

  4. Select Generate gcloud commands.

  5. Follow the instructions displayed in the side panel to configure your service account.

  6. If the connection has been set up successfully, the status of the Real-time Anomalies – GCE feature will show Healthy.

Enable on an existing connection​

To add real-time anomalies to a connected organization:

  1. Locate the service account of interest on the Google Cloud access & features page.

  2. Select the kebab menu (⋮) next to the organization connection, and then select Edit.

  3. Select the Real-time Anomalies – GCE checkbox to add the feature.

  4. Select Generate gcloud commands.

  5. Follow the instructions displayed in the side panel to update your custom role.

  6. Select Done to enable the feature.

Real-time anomaly detection costs​

The real-time anomaly detection feature collects your GCE activities using Google Cloud Audit logs. It doesn't incur additional costs.

Disable real-time anomalies for GCE​

To safely disable the real-time anomalies feature for GCE, we recommend disabling the log sink created by the DoiT anomaly detection system, using the following command (make sure to replace the <ORG-ID> with the organization resource ID used to enable the feature):


gcloud logging sinks update realtime-pipeline-log-sink-prod-<ORG-ID> --disabled --organization=<ORG-ID>

Disabled sinks do not route logs to the sink destination. To check the status of your log sink, go to the Log Router page in the Google Cloud console and select your organization.

If you need to reactivate the real-time anomalies feature, run the gcloud logging sinks update command with the --no-disabled flag. It will enable the disabled log sink and resume sending logs to the DoiT Pub/Sub topic.


gcloud logging sinks update realtime-pipeline-log-sink-prod-<ORG-ID> --no-disabled --organization=<ORG-ID>

Last updated on