Skip to main content

SELECT Databricks Cost Observability & Optimization Service Description

Overview of the Platform​

SELECT is a software-as-a-service (SaaS) application that provides cost observability, usage analytics, optimization insights, and automated compute configuration adjustments for customer Databricks environments. SELECT enables organisations to monitor, analyse, govern, and optimise Databricks compute usage and associated spend through automated analysis of Databricks account metadata and query workload behaviour.

SELECT is delivered as a vendor-hosted web application accessed via browser interface and supported integrations. Customers provision Databricks metadata access credentials, and initial onboarding includes account connection and configuration of usage groups and monitors. Most customers are fully operational in under fifteen minutes.

The following sections describe the major features and capabilities available in SELECT.

Core Analytics and Reporting​

Cost and Usage Observability​

  • Single pane of glass for Databricks spend – SELECT aggregates and visualises Databricks usage and credit consumption, providing a single source of truth for credits spent. The platform attributes compute spend across Databricks resources including compute, workloads, users, queries, and metadata-derived assets. It highlights the annualised costs across Databricks resources—tables, tasks, LLM models and more.

  • Workload-level and query-level cost analysis – SELECT uses query parsing and fingerprinting to deliver workload-level and query-level cost analysis. Users can explore historical spend trends, drill into individual query costs, and identify the specific workloads driving consumption. Query-level insights accelerate performance optimization and help educate business users on query best practices.

  • Forecasting – SELECT includes usage and spend forecasting across supported resource dimensions. Forecasts are accessible in dashboards and budgets, enabling proactive decision-making and planning.

Usage Groups (Cost Allocation and Governance)​

  • Flexible cost attribution – Usage Groups provide a flexible way of creating cost categories within SELECT. They allow customers to allocate Databricks costs to teams, projects, or departments using a variety of Databricks attributes such as users, roles, compute instances, databases, and workload classifications. Usage Groups support showback and chargeback reporting, enabling organisations to decentralise cost management and promote a shared sense of cost ownership.

  • Budgets and forecasting – Customers can assign budgets to each Usage Group and forecast costs on a monthly, quarterly, or annual basis. Teams are automatically alerted when they are forecasted to exceed their budget. Budget threshold alerts and projected overrun notifications are designed to ensure proactive spend governance.

  • Configuration as code – Usage Groups can be configured and updated through the web interface or automatically via JSON or YAML definitions. Version history is maintained for Usage Group changes, allowing customers to review and restore prior configurations at any time. Usage Groups integrate with Insights, Monitors, and the broader analytics experience throughout SELECT.

Insight Generation and Recommendations​

Insights​

  • Intelligent recommendations engine – SELECT’s Insights engine continuously analyses Databricks accounts for optimisation opportunities. It scans across Databricks resources and queries for common anti-patterns and inefficiencies, and surfaces detailed optimisation recommendations to end users.

  • Actionable insights dashboard – The Insights page provides an overview of insight categories with the ability to drill into each category. Users can quickly see total potential savings along with the estimated effort for each opportunity. Insights can be filtered by resource, potential savings, or level of effort to help teams prioritise their work. Query-level insights include performance and cost diagnostics derived from Databricks metadata and query profiles, helping users accelerate query workloads, reduce compute costs, and adopt query best practices.

Automated Optimisation​

Automated Savings​

  • Continuous compute optimisation – SELECT’s Automated Savings feature continuously monitors Databricks workloads and compute utilisation and automatically adjusts compute instance configurations—including worker scaling behaviour and related parameters—to improve utilisation efficiency. The feature is designed to reduce compute spend with no engineering effort required from the customer.

  • Granular control and transparency – Customers can see the potential savings for each compute instance and those with an Admin or Editor role may easily toggle the Automated Savings feature on or off for complete control. SELECT provides full transparency into the exact actions taken and the savings generated, so customers can validate results at any time. All optimization actions are limited to configuration parameters and do not modify customer data, schemas, or query logic.

Alerts and Anomaly Detection​

Monitors​

Spend anomaly alerts – SELECT provides configurable monitors for spend anomalies and usage spikes. Monitors support multiple detection methods:

  • Threshold Monitors allow customers to set explicit spend thresholds for workspaces, compute instances, or other Databricks resources and receive alerts when those thresholds are crossed.

  • Anomaly Monitors automatically determine whether a metric is anomalously high or low, accounting for seasonal patterns. Sensitivity can be adjusted to balance between rapid detection and minimising false positives.

  • Custom SQL Monitors allow customers to define their own SQL-based alerting logic for scenarios where out-of-the-box monitors do not apply. Monitors can run on daily, weekly, or monthly schedules.

  • Recurring digests – Users can subscribe to digests at recurring intervals (daily, weekly, or monthly) summarising spend at the account level or by Usage Group, warehouse, or specific workload. Proactive notifications help maintain cost awareness across the organisation.

Integrations​

  • Data ecosystem integrations – SELECT integrates with popular data tools that sit on top of Databricks and drive significant compute consumption. Supported integrations include tools such as dbt, Looker, Sigma, and other transformation, BI, and ingestion platforms where metadata access is available. These integrations provide enhanced cost attribution and lineage context, allowing customers to understand the total cost of assets originating from these systems and surface optimisation opportunities specific to each tool.

  • Alerting integrations – SELECT supports alert delivery to Email, Slack, Microsoft Teams, PagerDuty, Opsgenie, and custom webhook endpoints, ensuring seamless integration with existing incident management and communication workflows.

Artificial Intelligence​

AI-Powered Capabilities​

SELECT leverages AI to deliver meaningful value while maintaining strict data security and privacy controls. AI capabilities within SELECT include summarisation of key Databricks query data that previously required manual exploration and analysis, generation of understandable and concise descriptions of complex queries, and identification of inefficiencies with proposed remedial actions across customer Databricks accounts.

Data Privacy and Security​

SELECT maintains a rigorous approach to data handling when using AI features:

  • Strict data lifecycle – All data provided to SELECT is processed only for authorised purposes and is securely handled afterward. SELECT does not use customer data to train AI models.

  • Sanitised data transmission – The only data transmitted to AI subprocessors is sanitised query text and performance metrics. No sensitive data is shared. SELECT is working to establish zero-data-retention (ZDR) policies with its AI subprocessors.

  • Compliant vendors – Third-party AI partners are carefully vetted to ensure they meet stringent compliance standards, including GDPR, CCPA, SOC 2, ISO 27001, and HIPAA. Vendors are continuously monitored for ongoing compliance.

Transparency and Explainability​

SELECT prioritises transparent and explainable AI. Every output produced by AI systems is logged in a comprehensive audit trail. Key decision-making logic is hand-written, with AI used only for use cases such as SQL generation where high confidence exists in the initial recommendation. AI features operate on a human-in-the-loop basis: outputs are provided as recommendations for users to apply their own judgement, and no automated actions are taken by AI features without user direction.

Data Security and Compliance​

Security and Access Model​

SELECT operates using read-only access to designated Databricks metadata views and system tables. SELECT does not access, read, copy, or process customer table data contents. Only Databricks metadata related to usage, configuration, and query execution is accessed. A customer-provisioned, limited-privilege Databricks service account is required for operation.

SELECT retains customer usage metadata while they are an active customer, and automatically deletes it upon offboarding.

Compliance and Certifications​

SELECT meets industry-standard security requirements, including SOC 2 Type II certification, demonstrating a commitment to managing data securely and protecting the privacy and interests of customer organisations. The certification covers critical areas including security, availability, processing integrity, confidentiality, and privacy. SELECT conducts routine internal and external audits to verify that practices align with the latest compliance standards.

Architecture and Data Protection​

SELECT employs a defence-in-depth security architecture:

  • End-to-end encryption – Data is encrypted at rest and in transit using AES-256 encryption.

  • Secure APIs – All data exchanges occur through authenticated APIs, ensuring only authorised systems and personnel can access data.

  • Zero-trust policies – Systems operate on a zero-trust basis; every access request is verified, authenticated, and logged.

  • Multi-factor authentication (MFA) – MFA is required for all access points to prevent unauthorised logins.

  • Data siloing – Each client’s data is processed in isolated, virtualised environments, eliminating cross-contamination risks.

  • Real-time monitoring – Systems are monitored continuously, using advanced analytics to detect and respond to potential threats.

  • Disaster recovery – Redundant backups and a robust disaster recovery plan ensure data safety and accessibility.

Access Controls​

SELECT provides role-based access controls (RBAC) that allow customers to grant users access to specific Databricks accounts or the entire Databricks organisation. Customers can expose a subset of Databricks costs from selected resources to each team or department, ensuring that sensitive billing information is shared only with authorised personnel.

Saved Views and Personalisation​

Many screens in SELECT feature filterable and configurable tables and charts. Instead of repeating configuration and filters on each visit, users can save views for easy reuse. Saved views can be favourited, which pins them to the sidebar for quick access. This capability supports weekly cost-review rituals and helps teams maintain consistent visibility into their Databricks spend.

Exclusions​

SELECT does not:

  • Modify customer data tables or records.

  • Replace Databricks native billing systems.

  • Guarantee specific savings outcomes.

  • Provide financial accounting or invoice reconciliation services.

  • Control third-party tool behaviour.

  • Override Databricks platform limits or billing rules.

Conclusion​

SELECT provides a comprehensive, Databricks-focused cost observability and optimisation platform that combines deep usage analytics, intelligent recommendations, fully automated warehouse savings, flexible cost allocation, and proactive anomaly detection. By leveraging Cost and Usage Observability, Usage Groups, Insights, Automated Savings, Monitors, data ecosystem Integrations, AI-powered capabilities, and granular access controls, organisations can achieve deep cost visibility, enforce financial accountability, detect anomalies quickly, optimise resource usage, and democratise spend management across the enterprise. SELECT’s security posture, read-only access model, and rapid onboarding make it suitable for organisations seeking to embed cost management best practices into their Databricks operations.

Last updated on