Google Cloud real-time anomaly detection

DoiT supports real-time anomaly detection for Google Compute Engine (GCE) workloads across all regions and zones, including both standard and custom machine types. To activate real-time anomaly detection, you need to enable the feature at the organization level.

Cloud Logging audit logs

Google Cloud real-time anomaly detection uses estimated on-demand costs based on usage derived from Cloud Logging audit logs.

• The anomaly detection system creates a log sink named realtime-pipeline-log-sink-prod-<Organization-ID> at the organization level.

• The log sink captures GCE activities of all the projects in the organization, using the following filter: resource.type="gce_instance" AND logName:"cloudaudit.googleapis.com%2Factivity"

• The log sink publishes audit logs to a pubsub topic named doit-realtime-pipeline-pubsub-topic in DoiT's Google Cloud organization. Audit logs are then processed for real-time anomaly detection.

Required permissions

To activate real-time anomaly detection for Google Cloud:

• Your DoiT account must have the Manage Settings permission.

• You must have been granted the Organization Role Administrator (roles/iam.organizationRoleAdmin) IAM role in your organization.

Enable real-time anomalies for GCE

You can enable real-time anomaly detection for GCE when connecting an organization or editing an existing connection (see Connect Google Cloud resources).

Enable when connecting an organization

1. Sign in to the DoiT console, select Integrate from the top navigation bar, and then select Google Cloud.

2. From the Connect drop-down, select Organization.

3. Select the Real-time Anomalies – GCE checkbox. You can expand the feature to view its required permissions. (See Security and data access policy: Feature permissions for details.)

   

4. Select Generate gcloud commands.

5. Follow the instructions displayed in the side panel to configure your service account.

6. If the connection has been set up successfully, the status of the Real-time Anomalies – GCE feature will show Healthy.

Enable on an existing connection

To add real-time anomalies to a connected organization:

1. Locate the service account of interest on the Google Cloud access & features page.

2. Select the kebab menu (⋮) next to the organization connection, and then select Edit.

3. Select the Real-time Anomalies – GCE checkbox to add the feature.

4. Select Generate gcloud commands.

5. Follow the instructions displayed in the side panel to update your custom role.

6. Select Done to enable the feature.

Real-time anomaly detection costs

The real-time anomaly detection feature leverages Google Cloud Audit logs to collect your GCE activities. It doesn't incur additional costs.

• Creating the log sink and routing of logs to DoiT's pubsub topic does not incur a charge. See Cloud Logging: Quotas and limits.

• Audit logs rely on Admin Activity audit logs, which are stored in the _Required bucket free of charge. See Cloud Logging: Storage pricing.

• API requests to compute.instances.get and compute.machineTypes.get have no direct costs.