Explore All in This Collection
By Jason Bloomberg, President, Intellyx
Part 2 of the Demystifying Data Observability Series for Unravel Data
In part one of this series, fellow Intellyx analyst Jason English explained the differences between DevOps and DataOps, drilling down into the importance of DataOps observability.
The question he left open for this article: how did we get here? How did DevOps evolve to what it is today, and what parallels or differences can we find in the growth of DataOps?
DevOps Precursors
The traditional, pre-cloud approach to building custom software in large organizations separated the application development (‘dev’) teams from the IT operations (‘ops’) personnel responsible for running software in the corporate production environment.
In between these two teams, organizations would implement a plethora of processes and gates to ensure the quality of the code and that it would work properly in production before handing it to the ops folks to deploy and manage.
Such ‘throw it over the wall’ processes were slow and laborious, leading to deployment cycles many months long. The importance of having software that worked properly, so the reasoning went, was sufficient reason for such onerous delays.
Then came the Web. And the cloud. And enterprise digital transformation initiatives. All of these macro-trends forced enterprises to rethink their plodding software lifecycles.
Not only were they too slow to deliver increasingly important software capabilities, but business requirements would evolve far too quickly for the deployed software to keep up.
Such pressures led to the rise of agile software methodologies, cloud native computing, and DevOps.
Finding the Essence of DevOps
The original vision of DevOps was to pull together the dev and ops teams to foster greater collaboration, in hopes that software deployment cadences would accelerate while maintaining or improving the quality of the resulting software.
Over time, this ‘kumbaya’ vision of seamless collaboration itself evolved. Today, we can distill the essence of modern DevOps into these five elements:
- A cultural and organizational shift away from the ‘throw it over the wall’ mentality to greater collaboration across the software lifecycle
- A well-integrated, comprehensive automation suite that supports CI/CD activities, along with specialists who manage and configure such technologies, i.e., DevOps engineers
- A proactive, shift-left mentality that seeks to represent production behavior declaratively early in the lifecycle for better quality control and rapid deployment
- Full-lifecycle observability that shifts problem resolution to the left via better prediction of problematic behavior and preemptive mitigation of issues in production
- Lean practices to deliver value and improve efficiency throughout the software development lifecycle
Furthermore, DevOps doesn’t live in a vacuum. Rather, it is consistent with and supports other modern software best practices, including infrastructure-as-code, GitOps, and the ‘cattle not pets’ approach to supporting the production environment via metadata representations that drive deployment.
The Evolution of DataOps
Before information technology (IT), organizations had management of information systems (MIS). And before MIS, at the dawn of corporate computing, enterprises implemented data processing (DP).
The mainframes at the heart of enterprise technology as far back as the 1960s were all about processing data – crunching numbers in batch jobs that yielded arcane business results, typically dot-matrix printed on green and white striped paper.
Today, IT covers a vast landscape of technology infrastructure, applications, and hybrid on-premises and cloud environments – but data processing remains at the heart of what IT is all about.
Early in the evolution of DP, it became clear that the technologies necessary for processing transactions were different from the technology the organization required to provide business intelligence to line-of-business (LoB) professionals.
Enterprises required parallel investments in online transaction processing (OLTP) and online analytical processing (OLAP), respectively. OLAP proved the tougher nut to crack, because enterprises generated voluminous quantities of transactional data, while LoB executives required complex insights that would vary over time – thus taxing the ability of the data infrastructure to respond to the business need for information.
To address this need, data warehouses exploded onto the scene, separating the work of OLAP into two phases: transforming and loading data into the warehouses and supporting business intelligence via queries of the data in them.
Operating these early OLAP systems was relatively straightforward, centering on administering the data warehouses. In contrast, today’s data estate – the sum total of all the data infrastructure in a modern enterprise – is far more varied than in the early data warehousing days.
Motivations for DataOps
Operating this data estate has also become increasingly complex, as the practice of DataOps rises in today’s organizations.
Complexity, however, is only one motivation for DataOps. There are more reasons why today’s data estate requires it:
- Increased mission-criticality of data, as digital transformations rework organizations into digital enterprises
- Increased importance of real-time data, a capability that data warehouses never delivered
- Greater diversity of data-centric use cases beyond basic business intelligence
- Increased need for dynamic applications of data, as different LoBs need an ever-growing variety of data-centric solutions
- Growing need for operational cost predictability, optimization, and governance
Driving these motivations is the rise of AI, as it drives the shift from code-based to data-based software behavior. In other words, AI is more than just another data-centric use case. It repositions data as the central driver of software functionality for the enterprise.
The Intellyx Take
For all these reasons, DataOps can no longer follow the simplistic data warehouse administration pattern of the past. Today’s data estate is dynamic, diverse, and increasingly important, requiring organizations to take a full-lifecycle approach to collecting, transforming, storing, querying, managing, and consuming data.
As a result, DataOps requires the application of core DevOps practices along the data lifecycle. DataOps requires the cross-lifecycle collaboration, full-lifecycle automation and observability, and the shift-left mentality that DevOps brings to the table – only now applied to the enterprise data estate.
Thinking of DataOps as ‘DevOps for data’ may be too simplistic an explanation of the role DataOps should play. Instead, it might be more accurate to say that as data increasingly becomes the driver of software behavior, DataOps becomes the new DevOps.
Next up in part 3 of this series: DataFinOps: More on the menu than data cost governance
Copyright © Intellyx LLC. Unravel Data is an Intellyx customer. Intellyx retains final editorial control of this article.