Data Flow

This document outlines how the Attribute platform securely collects, processes, and analyzes your data to provide granular cloud cost insights. Our architecture is built on a foundation of robust security and transparency, ensuring your data is handled with the utmost care at every stage.

Data Flow Stages

The Attribute data flow can be broken down into three main stages:

• Data Collection within customers' environment.
• Data Processing in the Attribute backend.
• Data Presentation through the Attribute Dashboard.

Data Collection

To provide detailed cost allocations, Attribute utilizes three primary data sources:

• Consumption Data: Collected via the proprietary Attribute Sensor deployed within customers' environment.
• Finance Data: Pulled from your cloud billing reports.
• Telemetry Data: Sourced from cloud APIs and other telemetry sources for resource utilization.

The core of our data collection within your environment is the Attribute Sensor. To assure the security and stability of the customers' environment, the Attribute Sensor was designed and developed with a modern security-first approach:

• Technology: The sensor is built on eBPF (extended Berkeley Packet Filter) technology, which safely inspects network packets from within the kernel of your systems. It operates in a read-only, sandboxed environment, ensuring minimal operational risk.
• Deployment: You can deploy the sensor on your compute entities, such as Kubernetes clusters or standalone virtual machines. Deployment is highly configurable, allowing you to choose precisely which workloads to monitor.
• Function: By inspecting network packets, the sensor aggregates usage metrics. For example, it can identify a specific service running in a Kubernetes cluster and measure the resources consumed by a particular query. The sensor only analyzes network traffic for specific protocols explicitly agreed upon with the customer (whitelist).
• Anonymization and Security: Critically, any sensitive or private information is stripped from the data within customers' environment before it is transmitted. This ensures no sensitive data ever leaves your premises. The aggregated, anonymized data is then securely transmitted to the Attribute backend.
• Transmission: Data is sent using the encrypted OpenTelemetry protocol, which offers full transparency into every piece of data collected. This secure transmission is authenticated using JWT and protected by TLS encryption. In addition, the use of OpenTelemetry ensures full transparency into collected data and allows customers to implement additional restrictions for enhanced security by implementing an optional OpenTelemetry endpoint.

Data Processing

Once customers' anonymized usage data and billing information arrive at the Attribute backend, the processing stage begins. This entire process runs on our secure Google Cloud Platform (GCP) infrastructure.

• Data Consolidation: Data from the sensor and your cloud billing reports are consolidated and parsed to accurately determine true costs.
• Core Processing: We use a robust set of GCP services for processing:
  • Data Processing: Complex ETL (Extract, Transform, Load) jobs are handled by SparkApplication and Big Data solutions.
  • Compute: Containerized applications are managed using Kubernetes Engine.
  • Event Streaming: A Pub/Sub service is used for event streaming.
• Data Security: To ensure the security of your data within our backend, we implement multiple measures, including:
  • Data encryption both at rest and in transit.
  • Separated datasets for each customer within our data stores.
  • A strict least-privilege access methodology.
• Scalability and Reliability: All backend components are designed to be redundant and auto-scaling to ensure high availability, reliability, and an optimal user experience.

Data Presentation

The final stage is the delivery of calculated data and cloud cost insights through the Attribute Dashboard.

• Secure Infrastructure: The Dashboard application layer is also built on a secure and scalable GCP infrastructure.
• Data Repositories: ClickHouse, BigQuery, and Atlas MongoDB® serve as the data backbone for the dashboard, maintaining the same high standards of security, data segmentation, and encryption as our backend.
• User Access: Access to the dashboard is managed through secure Single Sign-On (SSO) authentication, ensuring strict data governance.
• High Availability: The dashboard components are designed for redundancy and auto-scaling to ensure they are always available and reliable.

Appendix A — Secure Handling of Sensitive Data

This appendix explains the strict security measures built into the Attribute platform to address concerns around data privacy and transmission. No sensitive data from your environment is ever sent to us.

The platform is designed to process data locally, within your environment, and only transmits minimal, non-sensitive metadata necessary for cost analysis. This architecture ensures that proprietary or confidential information never leaves your system. We take a security-first, zero-trust approach, anchored in the principle of data minimization.

1. Protocol Parsing and Event-Based Transmission

The Attribute Sensor does not transmit raw network packets. Instead, it serves as an intelligent data processor within your environment.

• Selective Protocol Analysis: The sensor inspects network traffic for specific protocols that have been explicitly agreed upon and whitelisted. Any traffic not matching these protocols is ignored.
• Event Generation, Not Packet Forwarding: For the whitelisted protocols, the Attribute sensor parsers extract only the specific metadata required for cost allocation. This metadata is used to construct structured "events." It is these purpose-built, minimal events, not the original network packets, that are transmitted. This process ensures that superfluous or sensitive data from the network traffic is discarded at the source. For reference, Section 3 includes an excerpt of the code used for building a sample event.

2. Proactive Exclusion of Sensitive Information

The sensor is explicitly designed to detect and exclude sensitive data during analysis. This exclusion occurs entirely within the customer's environment, ensuring that such data is never transmitted to Attribute's backend.

• Secrets and Credentials: The Attribute sensor does not parse or transmit secrets, API keys, passwords, or authentication tokens.
• Cookies: All cookie data within network traffic is ignored.
• Personally Identifiable Information (PII): Any PII that may reside within queries is redacted using common libraries.

3. Example Code for Building an Event

r.metricBuilder.BuildCompoundMetric(func(metric metricbuilder.CompoundMetric) {
    metric = setMetricResourceAttributesFromEnvInfo(metric, envInfo.Region, envInfo.Host, metricDescriptor.Name)

    for fields, values := range data.Values {
        wl := fields.Workload

        scope := metric.NewScope("workload", "")
        scope = setScopeAttributeFromWorkload(scope, wl)
        scope = setScopeAttributeFromEnvInfo(scope, envInfo.Name, envInfo.Type)

        scopeFillAttributesFromMetricFields(&fields, scope)

        gauge := scope.NewGauge("metric")

        measurement := gauge.NewMeasurement()
        measurement.SetTimestamp(eventTime)
        measurement.SetAttribute("attributes", map[string]interface{}{
            "host":                  fields.F0,
            "method":                fields.F1,
            "path":                  fields.F2,
            "request.count":         values.Count,
            "request.total.time.ns": values.V0,
        })
    }
})

---

Trademarks: Google Cloud is a trademark of Google LLC. MongoDB is a registered trademark of MongoDB, Inc. Kubernetes and OpenTelemetry are trademarks of The Linux Foundation. Apache Spark is a trademark of the Apache Software Foundation. ClickHouse is a registered trademark of ClickHouse, Inc. All other trademarks are the property of their respective owners.