Explore All in This Collection
By Jason English, Principal Analyst, Intellyx
Part 3 in the Demystifying Data Observability Series, by Intellyx for Unravel Data
IT and data executives find themselves in a quandary about deciding how to wrangle an exponentially increasing volume of data to support their business requirements – without breaking an increasingly finite IT budget.
Like an overeager diner at a buffet who’s already loaded their plate with the cheap carbs of potatoes and noodles before they reach the protein-packed entrees, they need to survey all of the data options on the menu before formulating their plans for this trip.
In our previous chapters of this series, we discussed why DataOps needs its own kind of observability, and then how DataOps is a natural evolution of DevOps practices. Now there’s a whole new set of options in the data observability menu to help DataOps teams track the intersection of value and cost.
From ROI to FinOps
Executives can never seem to get their fill of ROI insights from IT projects, so they can measure bottom-line results or increase top-line revenue associated with each budget line item. After all, predictions about ROI can shape the perception of a company for its investors and customers.
Unfortunately, ROI metrics are often discussed at the start of a major technology product or services contract – and then forgotten as soon as the next initiative gets underway.
The discipline of FinOps burst onto the scene over the last few years, as a strategy to address the see-saw problem of balancing the CFO’s budget constraints with the CIO’s technology delivery requirements to best meet the current and future needs of customers and employees.
FinOps focuses on improving technology spending decisions of an enterprise using measurements that go beyond ROI, to assess the value of business outcomes generated through technology investments.
- Based on customer demand or volatility in our consumption patterns, should we buy capacity on-demand or reserve more cloud capacity?
- Which FinOps tools should we buy, and what functionality should we build ourselves, to deliver this important new capability?
- Which cloud cost models are preferred for capital expenditures (capex) projects and operational expenditures (opex)?
- What is the potential risk and cost of known and unknown usage spikes, and how much should we reasonably invest in analysts and tools for preventative purposes?
As a discipline, FinOps has come a long way, building communities of interest among expert practitioners, product, business, and finance teams as well as solution providers through its own FinOps Foundation and instructional courses on the topic.
FinOps + DataOps = DataFinOps?
Real-time analytics and AI-based operational intelligence are enabling revolutionary business capabilities, enterprise-wide awareness, and innovative machine learning-driven services. All of this is possible thanks to a smorgasbord of cloud data storage and processing, cloud data lakes, cloud data warehouse, and cloud lakehouse options.
Unfortunately, the rich streams of data required for such sophisticated functionality bring along the unwanted side effect of elastically expanding budgetary waistbands, due to ungoverned cloud storage and compute consumption costs. Nearly a third of all data science projects go more than 40% over budget on cloud data, according to a recent survey–a huge delta between cost expectations and reality.
How can better observability into data costs help the organization wring more value from data assets without cutting into results, or risking cost surprises?
As it turns out, data has its own unique costs, benefits, and value considerations. Combining the disciplines of FinOps and DataOps – which I’ll dub DataFinOps just for convenience here – can yield a unique new set of efficiencies and benefits for the enterprise’s data estate.
Some of unique considerations of DataFinOps:
- Which groups within our company are the top spenders on cloud data analytics, and is anything anomalous about their spending patterns versus the expected budgets?
- What is the value of improving data performance or decreasing the latency of our data estate by region or geography, in order to improve local accuracy, reduce customer and employee attrition and improve retention?
- If we are moving to a multi-cloud, hybrid approach, what is an appropriate and realistic mix of reserved instances and spot resources for processing data of different service level agreements (SLAs)?
- Where are we paying excessive ingress / egress fees within our data estate? Would it be more cost effective to process data near the data or move our data elsewhere?
- How much labor do our teams spend building and maintaining data pipelines, and what is that time worth?
- Are cloud instances being intelligently right-sized and auto-scaled to meet demand?
Systems-oriented observability platforms such as DataDog and Dynatrace can measure system or service level telemetry, which is useful for a DevOps team looking at application-level cloud capacity and cost/performance ratios. Unfortunately these tools do not dig into enough detail to answer data analytics-specific FinOps questions.
Taming a market of data options
Leading American grocery chain Kroger launched its 84.51° customer experience and data analytics startup to provide predictive data insights and precision marketing for its parent company and other retailers, across multiple cloud data warehouses such as Snowflake and Databricks, using data storage in multiple clouds such as Azure and GCP.
Using the Unravel platform for data observability, they were able to get a grip on data costs and value across multiple data platforms and clouds without having to train up more experts on the gritty details of data job optimization within each system.
“The end result is giving us tremendous visibility into what is happening within our platforms. Unravel gave recommendations to us that told us what was good and bad. It simply cut to the chase and told us what we really needed to know about the users and sessions that were problematic. It not only identified them, but then made recommendations that we could test and implement.”
– Jeff Lambert, Vice President Engineering, 84.51°
It’s still early days for this transformation, but a data cost reduction of up to 50% would go a long way toward extracting value from deep customer analytics, as transaction data volumes continue to increase by 2x or 3x a year as more sources come online.
The Intellyx Take
It would be nice if CFOs could just tell CIOs and CDOs to simply stop consuming and storing so much data, and have that reduce their data spend. But just like in real life, crash diets will never produce long-term results, if the ‘all-you-can-eat’ data consumption pattern isn’t changed.
The hybrid IT underpinnings of advanced data-driven applications evolves almost every day. To achieve sustainable improvements in cost/benefit returns on data, analysts and data scientists would have to become experts on the inner workings of each public cloud and data warehousing vendor.
DataFinOps practices should encourage data team accountability for value improvements, but more importantly, it should give them the data observability, AI-driven recommendations, and governance controls necessary to both contain costs, and stay ahead of the organization’s growing business demand for data across hybrid IT data resources and clouds.
©2023 Intellyx LLC. Intellyx is editorially responsible for the content of this document. At the time of writing, Unravel is an Intellyx customer. Image source: crayion.ai