Table of Contents
Over the past several years, there has been an explosion of different terms related to the world of IT operations. Not long ago, it was standard practice to separate business functions from IT operations. But those days are a distant memory now, and for good reason.
The Ops landscape has expanded beyond the generic “IT” to include DevOps, DataOps, AIOPs, MLOps, and more. Each of these Ops areas are cross-functional throughout an organization, and each provides a unique benefit. And each of the Ops areas emerges from the same general mechanism: Applying agile principles, originally created to guide software development, to the overlap of different flavors of software development, related technologies (data-driven applications, AI, and ML), and operations.
As with DevOps, the goal of DataOps, AIOps, and MLOps is to accelerate processes and improve the quality of data, analytics insights, and AI models respectively. In practice, we see DataOps as a superset of AIOps and MLOps with the latter two Ops overlapping each other.
Why is this? DataOps describes the flow of data, and the processing that takes place against that data, through one or more data pipelines. In this context, every data-driven app needs DataOps; those which are primarily driven by machine learning models also need MLOps, and those with additional AI capabilities need AIOps. (Confusingly, ML is sometimes considered to be separate from AI, and sometimes simply an important part of the technologies that are part of AI as a whole.)
The goal of this article is to help you understand these new terms and provide insight into where they came from, the similarities, differences, and who in an organization uses them. To start, we’ll look at DataOps.
What is DataOps
DataOps is the use of agile development practices to create, deliver, and optimize data products, quickly and cost-effectively. DataOps is practiced by modern data teams, including data engineers, architects, analysts, scientists and operations.
The data products which power today’s companies range from advanced analytics, data pipelines, and machine learning models to embedded AI solutions. Using a DataOps methodology allows companies to move faster, more surely, and with greater cost-effectiveness in extracting value out of data.
A company can adopt a DataOps approach independently, without buying any specialty software or services. However, just as the term DevOps became strongly associated with the use of containers, as commercial software from Docker and as open source software from Kubernetes, the term DataOps is increasingly associated with data pipelines and applications.
DataOps at scale with Unravel
The origin of DataOps
DataOps is a successor methodology to DevOps, which is an approach to software development that optimizes for responsive, continuous deployment of software applications and software updates. DataOps applies this approach across the entire lifecycle of data applications and to even help productize data science.
As a term, DataOps has been in gradually increasing use for several years now. Gartner began to include it in their Hype Cycle for Data Management in 2018, and the term is now “moving up the Hype Cycle” as it becomes more widespread. The first industry conference devoted to DataOps, DataOps Unleashed, was launched in March 2021.
While DataOps is sometimes described as a set of best practices for continuous analytics, it is actually more holistic. DataOps includes identifying the data sources needed to solve a business problem, the processing needed to make that data ready for analytics, the analytics step(s) – which may be seen as including AI and ML – and the delivery of results to the people or apps that will use them. DataOps also includes making all this work fast enough to be useful, whether that means processing the weekly report in an hour or so, or approving a credit card transaction in a fraction of a second.
Who uses DataOps
DataOps is directly for data operations teams, data engineers, and software developers building data-powered applications and the software-defined infrastructure that supports them. The benefits of DataOps approaches are felt by the teams themselves; the IT team, which can now do more with less; the data team’s internal “customers,” who request and use analytics results; the organization as a whole; and the company’s customers. Ultimately, society benefits, as things simply work better, faster, and less expensively. (Compare booking a plane ticket online to going to a travel agent, or calling airlines yourself, out of the phone book, as a simple example.)
What is AIOps
AIOps stands for Artificial Intelligence for IT operations. It is a paradigm shift that allows machines to solve IT issues without the need of human assistance or interaction. AIOPs uses machine learning and analytics to analyze big data obtained via different tools, which allows for issues to be spotted automatically and dealt with in real time. (Confusingly, AIOps is also sometimes used to describe the operationalization of AI project, but we will stick with the definition used by Gartner and others, as described here.)
As part of DataOps, AIOps supports continuous integration and deployment for the core tech functions of machine learning and big data. AIOps helps automate operations across hybrid environments. AIOps includes the use of machine learning to detect patterns and reduce noise.
See Unravel AI and automation in action
The Origin of AIOps
AIOps was originally defined in 2017 by Gartner as a means to describe the growing interest and investment in applying a broad spectrum of AI capabilities to enterprise IT operations management challenges.
Gartner defines AIOps as platforms that utilize big data, machine learning, and other advanced analytics technologies to directly and indirectly enhance IT operations (such as monitoring, automation and service desk) functions with proactive, personal, and dynamic insight.
Put another way, AIOps refers to improving the way IT organizations manage data and information in their environments using artificial intelligence.
Who uses AIOps
From enterprises with large, complex environments, to cloud-native small and medium enterprises (SMEs), AIOps is being used globally by organizations of all sizes in a variety of industries. AIOps is most often bought in as products or services from specialist companies; few large organizations are using their own in-house AI expertise to solve IT operations problems.
Companies with extensive IT environments, spanning multiple technology types, are adopting AIOps, especially when they face issues of scale and complexity. AIOps can make a significant contribution when those challenges are layered on top of a business model that is dependent on IT. If the business needs to be agile and to quickly respond to market forces, IT will rely upon AIOps to help IT be just as agile in supporting the goals of the business.
AIOps is not just for massive enterprises, though. SMEs that need to develop and release software continuously are embracing AIOps as it allows them to continually improve their digital services, while preventing malfunctions and outages.
The ultimate goal of AIOps is to enhance IT Operations. AIOps delivers intelligent filtering of signals out of the noise in IT systems. AIOps intelligently observes IT operations data in order to identify root causes and recommend solutions quickly. In some cases, these recommendations can even be implemented without human interaction.
What is MLOps
MLOps, or Machine Learning Operations, helps simplify the management, logistics, and deployment of machine learning models between operations teams and machine learning researchers.
MLOps is like DataOps – the fusion of a discipline (machine learning in one case, data science in the other) and the operationalization of projects from that discipline. MLOps and DataOps are different from AIOps, which is the use of AI to improve AI operations.
MLOps takes the DevOps methodology of continuous integration and continuous deployment and applies it to machine learning. As in traditional development, there is code that needs to be written and deployed, as well as bug testing to be done, and changes in user requirements to be accommodated. Specific to the topic of machine learning, models are being trained with data, and new data is introduced to retrain the models again and again.
The Origin of MLOps
According to Forbes, the origins of MLOps date back to 2015, from a paper entitled “Hidden Technical Debt in Machine Learning Systems.” The paper offered the position that machine learning offered an incredibly powerful toolkit for building useful complex prediction systems quickly, but that it was dangerous to think of these quick wins as coming for free.
Who Uses MLOps
Data scientists tend to focus on the development of models that deliver valuable insights to your organization more quickly. Ops people tend to focus on running those models in production. MLOps unifies the two approaches into a single, flexible practice, focused on the delivery of business value through the use of machine learning models and relevant input data.
Because MLOps follows a similar pattern to DevOps, MLOps creates a seamless integration between your development cycle and your overall operations process that transforms how your organization handles the use of big data as input to machine learning models. MLOps helps drive insights that you can count on and put into practice.
Streamlining MLOps is critical to organizations that are developing Machine Learning models, as well as to end users who use applications that rely on these models.
According to research from Finances Online, machine learning applications and platforms account for 57% (or $42 Billion) in AI funding worldwide. Organizations that are making significant investments want to ensure they are deriving significant value.
As an example of the impact of MLOps, 97% of all mobile users use AI-powered voice assistants that depend on machine learning models, and thus benefit from MLOps. These voice assistants are the result of a subset of ML known as deep learning. This deep learning technology is built around machine learning and is at the core of platforms such as Apple’s Siri, Amazon Echo, and Google Assistant.
The goal of MLOps is to bridge the gap between data scientists and operations teams to deliver insights from machine learning models that can be put into use immediately.
Conclusion
Here at Unravel Data, we deliver a DataOps platform that uses AI-powered recommendations – AIOps – to help proactively identify and resolve operations problems. This platform complements the adoption of DataOps practices in an organization, with the end results including improved application performance, fewer operational hassles, lower costs, and the ability for IT to take on new initiatives that further benefit organizations.
Our platform does not explicitly support MLOps, though MLOps always occurs in a DataOps context. That is, machine learning models run on data – usually, on big data – and their outputs can also serve as inputs to additional processes, before business results are achieved.
As DataOps, AIOps, and MLOps proliferate – as working practices, and in the form of platforms and software tools that support agile XOPs approaches – complex stacks will be simplified and made to run much faster, with fewer problems, and at less cost. And overburdened IT organizations will truly be able to do more with less, leading to new and improved products and services that perhaps can’t all be imagined today.
If you want to know more about DataOps, check out the recorded sessions from the first DataOps industry conference, DataOps Unleashed. To learn more about the Unravel Data DataOps platform, you can create a free account or contact us.