Table of Contents
Google recently introduced significant changes to its existing BigQuery pricing models, affecting both compute and storage. They announced the end of sale for flat-rate and flex slots for all BigQuery customers not currently in a contract. Google announced an increase to the price of on-demand analysis by 25% across all regions, starting on July 5, 2023.
Main Components of BigQuery Pricing
Understanding the pricing structure of BigQuery is crucial to effectively manage expenses. There are two big components to BigQuery pricing:
- Compute (analysis) pricing is the cost to process queries, including SQL queries, user-defined functions, scripts, and certain data manipulation language (DML) and data definition language (DDL) statements
- Storage pricing is the cost to store data that you load into BigQuery. Storage options are logical (the default) or physical. If data storage is converted from logical to physical, customers cannot go back to logical storage.
Selecting the appropriate edition and accurately forecasting data processing needs is essential to cloud data budget planning and maximizing the value derived from Google Cloud BigQuery.
Introducing Unravel 4.8.1 for BigQuery
Unravel 4.8.1 for BigQuery includes AI-driven FinOps and performance optimization features and enhancements, empowering Google Cloud BigQuery customers to see and better manage their cloud data costs. Unravel helps users understand specific cost drivers, allocation insights, and performance and cost optimization of SQL queries. New Unravel features are align with the FinOps phases:
Inform
- Compute and storage costs
- Unit costs and trends for projects, users, and jobs
Optimize
- Reservation insights
- SQL insights
- Data and storage insights
- Scheduling insights
Operate
- OpenSearch-based alerts on job duration and slot-ms
- Alert customization: ability to create custom alerts
Improving visibility, optimizing data performance, and automating spending guardrails can help organizations overcome resource limitations to get more out of their existing data environments.
Visibility into BigQuery compute and storage spend
Getting insights into your cloud data spending starts with understanding your cloud bill. With Unravel, BigQuery users can see their overall spend as well as spending trends for their selected time window, such as the past 30 days.
The cost dashboard shows details and trends, including compute, storage, and services by pricing tier, project, job, and user.
Unravel provides cost analysis, including the average cost of both compute and storage per project, job, and user over time. Compute spending can be further split between on-demand and reserved capacity pricing.
Armed with this detail, BigQuery customers can better understand both infrastructure and pricing tier usage as well as efficiencies by query, user, department, and project. This granular visibility enables accurate, precise cost allocation, trend visualization, and forecasting.
This dashboard provides BigQuery project chargeback details and trends, including a breakdown by compute and storage tier.
Unravel’s AI-driven cloud cost optimization for BigQuery delivers insights based on Unravel’s deep observability of the job, user, and code level to supply AI-driven cost optimization recommendations for slots and SQL queries, including slot provisioning, query duration, autoscaling efficiencies, and more.
With Unravel, BigQuery users can speed cloud transformation initiatives by having real-time cost visibility, predictive spend forecasting, and performance insights for their workloads.
AI-driven cloud cost optimization for BigQuery
At the core of Unravel Data’s data observability and FinOps platform is the AI-powered Insights Engine. It is purpose-built for data platforms—including BigQuery—to understand all the unique aspects and capabilities of each modern data stack and the underlying infrastructure to optimize efficiency and performance.
Unravel’s AI-powered Insights Engine continuously ingests and interprets millions of metadata inputs to provide real-time insights into application and system performance, along with recommendations to improve performance and efficiency for faster results and greater positive business impact for your existing cloud data spend.
Unravel provides insights and recommendations to optimize BigQuery reservations.
Using Unravel’s cost and performance optimization intelligence based on its deep observability at the job, user, and code level, users get recommendations such as:
- Reservation sizing that achieves optimal cost efficiency and performance
- SQL insights and anti-patterns to avoid
- Scheduling insights for recurring jobs
- Quota insights with respect to workload patterns
- and more
With Unravel, BigQuery customers can speed cloud transformation initiatives by having predictive cost and performance insights of existing workloads prior to moving them to the cloud.
Visualization dashboards and unit costs
Visualizing unit costs not only simplifies cost management but also enhances decision-making processes within your organization. With clear insights into spending patterns and resource utilization, you can make informed choices regarding optimization strategies or budget allocation.
With Unravel, BigQuery customers can customize dashboards and alerts with easy-to-use widgets to enable at-a-glance and drill down dashboards on:
- Spend
- Performance
- Unit economics
Unravel displays user count and cost trends by compute pricing tier.
From a unit economics perspective, BigQuery customers can build dashboards to show unit costs in terms of average cost per user, per project, and per job.
Take advantage of visualization dashboards in Unravel for BigQuery to effortlessly gain valuable insights into unit costs.
Additional features included in this release
Unravel 4.8.1 includes additional features, such as showback/chargeback reports, SQL insights and anti-patterns. You can compare two jobs side-by-side, enabling you to point out any metrics that are different between two runs, even if the queries are different.
With this release, you also get:
- Top-K projects, users, and jobs
- Showback by compute and storage types, services, pricing plans, etc.
- Chargeback by projects and users
- Out-of-the-box and custom alerts and dashboards
- Project/Job views of insights and details
- Side-by-side job comparisons
- Data KPIs, metrics, and insights such as size and number of tables and partitions, access by jobs, hot/warm/cold tables
Use case scenarios
Unravel for BigQuery provides a single source of truth to improve collaboration across functional teams and accelerates workflows for common use cases. Below are just a few examples of how Unravel helps BigQuery users for specific situations:
Role | Scenario | Unravel benefits |
FinOps Practitioner | Understand what we pay for BigQuery down to the user/app level in real time, accurately forecast future spend with confidence | Granular visibility at the project, job, and user level enables FinOps practitioners to perform cost allocation, estimate annual cloud data application costs, cost drivers, break-even, and ROI analysis |
FinOps Practitioner / Engineering / Operations | Identify the most impactful recommendations to optimize overall cost and performance | AI-powered performance and cost optimization recommendations enable FinOps and data teams to rapidly upskill team members, implement cost efficiency SLAs, and optimize BigQuery pricing tier usage to maximize the company’s cloud data ROI |
Engineering Lead / Product Owner | Identify the most impactful recommendations to optimize the cost and performance of a project | AI-driven insights and recommendations enable product and data teams to improve slot utilization, boost SQL query performance, leverage table partitioning and column clustering to achieve cost efficiency SLAs and launch more data jobs within the same project budget |
Engineering / Operations | Live monitoring with alerts | Live monitoring with alerts speed MTTR and prevent outages before they happen |
Data Engineer | Debugging a job and comparing jobs | Automatic troubleshooting guides data teams directly to the pinpoint the source of job failures down to the line of code or SQL query along with AI recommendations to fix it and prevent future issues |
Data Engineer | Identify expensive, inefficient, or failed jobs | Proactively improve cost efficiency, performance, and reliability before deploying jobs into production. Compare two jobs side-by-side to find any metrics that are different between the two runs, even if the queries are different. |
Get Started with Unravel for BigQuery
Learn more about Unravel for BigQuery by reviewing the docs and creating your own free account.